Groenewegen burst around Gaviria, however, to take his first stage win of this year's race and second of his career at the Tour having won on the Champs-Elysees in 2017.
Other riders chatted as they admired the rolling countryside.
Greg Van Avermaet of Belgium, wearing the yellow jersey, took advantage of an intermediate bonus sprint to add two seconds to his overall lead - ahead of BMC teammate Tejay van Garderen of the United States.
Thursday's sixth stage is expected to shake up the peloton as it culminates in the double ascension of the feared Mur de Bretagne climb, a 2km long and 6.9 per cent gradient affair.
"I knew I wouldn't make it back so it was about limiting the time loss", Dumoulin said. "I have the legs to win now, so we celebrate tonight and maybe the same tomorrow".
Greg Van Avermaet retains the yellow jersey and remains first in the general classification for the fourth consecutive stage.
Groenewegen told reporters after the stage: "I hope there's more".
France's Pierre Latour, of Ag2r La Mondiale, finished in second place, followed by Spaniard Alejandro Valverde, of Movistar, in a stage that started in Brest and ended in Mur-De-Bretagne Guerledan.
Trump hits the links at his Scottish golf resort
Trump's history with Scotland traces to his mother, who grew up on the Scottish Isle of Lewis in a village called Tong. A Greenpeace demonstrator paraglides flies over a Trump-owned golf course in Scotland during the president's visit.
"I was pretty disappointing in the first few stages but now I've got the power back in my legs", said the 25-year-old from Amsterdam who dislikes comparisons to Briton Mark Cavendish.
Bardet was already trailing Froome, and after losing 28 seconds is now 1:45 off the leader's pace after he broke his back wheel and had to change bikes with Tony Gallopin.
Peter Sagan was also in the final mix, coming eighth and picking up more points in the green jersey race and extending his lead on Colombian sprinter Fernando Gaviria.
"It was a very fast ascent and I just held on really", said the Belgian, who finished alongside team leader Richie Porte at three seconds off the pace to pull on the jersey for a third day.
Groenewegen grew up in a cycling environment as his grandfather was a bike manufacturer and his boyhood coach was Jean-Paul van Poppel, victor of nine sprint finishes at the Tour de France.
"I'd like to thank Greg (Van Avermaet) because he did me a favour when he went for it, I don't know if he did it on goal but he dropped a lot of people", said Sagan, who has tightened his grip on the points classification's green jersey.
The upcoming seventh stage runs from Fougeres to Chartres.