He had been sitting in third place overall, 3:32 behind race leader Chris Froome (Sky) and just 22 seconds behind Movistar's Nairo Quintana. On the first climb of the day, however, it became clear that something was wrong as van Garderen slipped behind on the category 3 Col des Lèques.
The American was seen back at the medical car as he trailed some two minutes behind the maillot jaune group.
A slowing in the main peloton and a descent from the Col de Touts Aures allowed van Garderen to rejoin the group of race favourites, but an attack by Alberto Contador (Tinkoff-Saxo) sent him out the back once more.
The category 2 Col de la Colle proved to be the end of the road for van Garderen, and he signalled to his team car to stop.
It is the first time in his career that he abandoned a Grand Tour.
The third place spot now goes to Alejandro Valverde (Movistar).