Goss had rocketed into the yellow leader's jersey on stage three after avoiding several fallen riders and outsprinting Heinrich Haussler (Garmin-Cervelo) and Denis Galimzyanov (Katusha).
Stage four was a different story however, with Goss forced to the back of the peloton with cramps.
"Terrible stomach today unfortunately," the 24-year-old said on his Twitter account post-stage. "Cramps and unable to eat for the last 80k. Not good when u have 3000m of climbing to get through."
Goss finished today's 191km stage back in 117th position, 13 seconds behind winner, French champion Thomas Voeckler (Europcar), pushing the Australian back to fourth overall, 21 seconds down on leader Thomas De Gendt (Vacansoleil-DCM).
Goss had targeted the opening three stages of the Paris-Nice for a stage win and the possibility of the yellow leader's jersey. With the race now headed for the hills towards Nice, the Australian's focus will shift again.
"In the mountainous stages it'll be a question of working for the team's GC guys and getting good form for the Classics," Goss said prior to the event.
Tony Martin and Tejay Van Garderen are next-best for HTC-Highroad, 37 seconds behind De Gendt, with the pair expected to play their cards in Friday's time trial.