It was the 2011 Tour Down Under and Vermote, a naive 21-year-old, wanted to impress on his Quick-Step debut by getting in the breakaway. Cavendish, however - a rider with 15 Tour de France stage wins already under his belt - had other ideas.
"I wanted to go in the breakaway but already a few riders had gone, and I tried to go across, but HTC were blocking the road, because they wanted to control the race. I tried to pass but then Cav goes to the right, you know… he wasn't letting me through. He was on the front line and he was blocking me so I couldn't pass.
"But after the stage he came up to me and was like 'ah sorry, sorry.' At the race he apologised already and then in the hotel at dinner he came again to excuse himself. I was impressed, because I was a neo-professional and he came twice to excuse himself."
Little did Cavendish know that this fresh-faced Belgian would become one of his most admired teammates, and also a close friend. Cavendish joined Quick-Step in 2013, with Vermote becoming part of his leadout train. After two years apart a reunion is imminent; Vermote has left Quick-Step Floors to join Cavendish at Dimension Data for 2018.
The 2013 Giro was where the relationship first blossomed, as Cavendish recounts in his 2013 autobiography, At Speed.
"When we arrived in Naples for the Giro d'Italia start in the first week of May, Julien Vermote was a 23-year-old in his third pro season that the team didn't seem to know what to do with. He would leave three weeks later having discovered – and shown me – that he could look after a sprinter in a stage race like few other riders in the peloton, and hence suddenly having found his identity as a rider."
Vermote agrees, and he looks back with fondness on the whole experience and the atmosphere in the team over those three weeks. Cavendish arrived in Brescia at the end of the Giro with five stage victories in the bag, but one stands out for Vermote.
"It was a close call, because it was a short stage and there was a break of five guys. We were pulling all day, rain and full gas all day, and it finished with this circuit and the break was still in front on the last lap," he says of stage 12, Cavendish's third win of the race and the 100th of his career.
"I was so fucked up. A lot of guys had gone already, so there were only two of us to pull, but I was struggling to stay in second position. Cav had to let me past him again to come through and pull again, and he was shouting 'keep going, keep going'. My legs were so tired. But yeah, he won it. It was his 100th victory.
"I crashed in the last kilometre – a car hit me when I was back in the cars – so it took me a while to get to the finish, and he was still waiting there like two minutes after the finish, and he said to me, 'What you did today was world class, without you I wouldn't have won the stage.'"
As Cavendish said, Vermote found his identity as a rider and, four years on he earned worldwide acclaim for his work at the Tour de France, where Marcel Kittel won five stages.
Partly owing to the fact stages were shown live from start to finish, Vermote occupied a huge amount of screen time for the hundreds of kilometres he accrued on the front of the peloton, controlling breakaways. He was held up as a paragon of selflessness and one of the heroes of the race.
It was during the Tour that concrete offers from other teams started to come in, including from Cavendish and Dimension Data.
"Some other teams were interested but then I texted Cav one night saying Dimension Data is interested. He knew it already but hadn't wanted to text me saying 'come over' – he didn't want to influence me," says Vermote.
"For me, too, it had to be my own choice, and it's not only about Cav – it's for my career – but I heard Cav and it was like 'click.' We always had a good relationship, and immediately the click was there and I knew it already."
Vermote follows Cavendish from Quick-Step Floors to Dimension Data but this is not simply a case of him leaving one sprinter to work for another – it's time to start thinking about himself.
"I already worked a lot for other guys but I have also my ambitions," he says. "I still want to discover what I can do."
At 28 years of age, Vermote faced the decision: stay at Quick-Step Floors and continue to slog his guts out in the Classics and Grand Tours, or take the plunge outside his comfort zone and try to add some results to his own palmares.
"The thing was with Quick-Step Floors, I ended up doing all the big races in service of the guys, but then you don't do the other races anymore, the smaller races, and then you're always riding in service of the guys," he explains.
"I have to point out all that Quick-Step Floors gave to me, but in the end I gave them a lot back."
Vermote grew up in Flanders and has always dreamed of the Classics. In a team like Quick-Step Floors, one of the strongest teams in the world and one that lives and breathes the Classics, opportunities of his own were non-existent, but at Dimension Data "there will be more possibilities, more freedom".
During the spring he can share leadership with Edvald Boasson Hagen, and he'll have the chance to take a more aggressive and liberal approach to other races too - something he tasted last year with a stage win and a spell in the leader's jersey at the Tour of Britain. As he says, he'll be able to race again, and not just ride.
"I know the Classics, I know the roads, so I'm quite convinced I can make the finales. But then you need some experience, and also some time. I just want to ride the finales again, and not have have to control the race anymore. In the end with Quick-Step Floors I was just controlling and it's a different kind of racing. That's normal, I don't blame anyone, if you're that strong you'd be stupid not to take the race into your hands.
"I'm just looking forward to finding the joy of racing again. Now we don't have to control, I can race again. I can think differently, then see in the finale what can happen. Like free racing – doing your thing and seeing what happens. To win a race you also need to dare to lose."
Of course, we will still see Vermote on the front of peloton, arms folded over his handlebars, emptying his legs in pursuit of the break, in the name of others. At the Tour de France, and indeed elsewhere, Cavendish will be in tow, but there'll be a significant difference.
"I won't have to pull first anymore. Ok, at the Tour anything can happen, but I don't want to pull anymore like that at the Tour of Dubai or races like that. That kills you. It's the goal now to use me in the finale. I like to work for the team, but there's a difference between working from the start and working later. When you're just pulling all day every day, you get slower, you lose your explosivity. You kill yourself for the next days, if you want go in the breakaway or do anything else."
Doing something else was what drove Vermote to leave his Quick-Step Floors comfort zone. The text from Cavendish might have been a clincher in settling on Dimension Data, but he could also look to others, riders whose careers have been rejuvenated at the South African team.
"You see a lot of guys there, they develop their skills already. Like [Serge] Pauwels, he came from us and he did some nice things in the Tour. Stevo Cummings found his way in this team. For a lot of guys it's a good move in their careers. That's why I was really interested to go," says Vermote.
"At the end of the day, I don't want regrets after my career, or to wonder what was possible. You don't know what's possible but the only way to find out is to try, and have no regrets."