The German, a four-time time trial world champion, feels the Tokyo course is too hilly to offer a realistic chance of success.
Instead, he will make a fifth world title, at Aigle-Martigny in September 2020, the main objective of his season.
"My plan is to skip the Olympics in order to prepare 100 per cent for the World Championships, which suit me better," Martin said on German radio station MDR on Thursday.
The Olympic time trial course in Tokyo covers two laps of a 22.1km circuit that starts and finishes on the Fuji Speedway motor racing circuit. It's an undulating circuit with a hefty climb at the mid-point and the total elevation gain across the two laps will be 846 metres.
"My manager, Jörg Werner, was there, and he looked at the course and he described it as extremely difficult," Martin said.
"It will hurt to miss it, because of course, the Olympic Games are always something special."
Martin won the silver medal in the time trial at the 2012 Olympics in London but placed a lowly 12th in Rio four years later. By the time of the 2024 Olympics in Paris, he'll be 39.
Martin won the world title three times in a row between 2011 and 2013, before adding a fourth title in 2016. A fifth world title in Switzerland would take him past Fabian Cancellara as the outright record holder. The Worlds take place in the Alps but the 46km course between Aigle and Martigny sticks largely to the valley.
Martin placed 9th at this year's Worlds in Yorkshire, though he was hampered by a heavy crash at the Vuelta a España, where he controlled the peloton for long periods as his teammate Primoz Roglic rode to overall victory.