At the start of stage 5 in Lorient on Wednesday morning, the Team Sky rider approached a fan who was holding a banner reading 'Sky – go home', and snatched it from him.
"The rider came over quickly, grabbed my placard and threw it down," said Didier Bregardes, according to reports from the Press Association and Velonews, explaining that he was protesting against the British team's handling of Chris Froome's salbutamol case.
According to the reports, Rowe was quickly asked about the incident but denied it. "No, I don't know what you mean," he said. "It wasn't me."
Later that afternoon, after the finish of the stage in Quimper, Rowe emerged from the Sky bus to admit that he had in fact taken the banner.
"There was a guy outside the bus, and he had a sign saying 'Sky - go home'. When I came out of the bus to sign on, I kind of rode past him and he was shaking it around," Rowe explained.
"He was actually stood next to two kids wearing Sky t-shirts. He was kind of laughing, I was laughing, and I grabbed it."
Despite initially denying it, Rowe was keen to play down any suggestion that tempers had flared.
"I think it was kind of tongue in cheek. Afterwards, I went over and gave the kids two bottles. It was all a bit of light-hearted fun," he said.
"He seemed like quite a light-hearted guy, when I started riding over towards him he started laughing. I think it's was quite light-hearted, and it was no big deal."
The reception for Team Sky from the French public has been one of the big story lines of the start of the 2018 Tour de France. Chris Froome was cleared in his salbutamol case five days before the start of the race, but the boos and whistles that greeted him at the teams presentation and have persisted – albeit at a lower level – show that the French public is still far from convinced.
The team have called on parent company BskyB for security advice and their bus is patrolled each morning and afternoon by a burly French bodyguard.
"The main concern didn't come from the team; we know there might be the odd person on the side of the road who's not the biggest fan. That's sport, right? Man City play Chelsea and half the stadium hates you," said Rowe.
"The cycling fans, in general, are not like that. Even if a cycling fan isn't the biggest fan of Chris Froome and Team Sky, I don't think we have to be in a position where we have to be scared or worried, because that's not the cycling culture, and it's never really boiled over to that extent in the past so I don't see why it should now."
Asked if he'd be grabbing any more banners, Rowe closed the matter on a light-hearted note.
"I think that's my one banner done for the Tour now."