Starting from the pole position, the Italian fell back early as a number of riders appeared to struggle with a rain-soaked track. But he steadily reeled in his compatriot Rossi, taking the lead back with six laps to go and quickly opening up a gap as Rossi was slowed by tyre issues.
Dovizioso, 30, finished 3.115secs ahead of Rossi for his maiden win of the season and first since 2009.
Rossi, meanwhile, clinched second place in the 2016 world championship over his Yamaha teammate Jorge Lorenzo, who finished third Sunday at the Sepang International Circuit.
Dovizioso wins Malaysia pole, Rossi second
"Well, it was in 2009, a long time," a clearly relieved Dovizioso said of the extended wait for a win. "It's really nice to come to this moment. In these conditions I struggled a lot in the race and had some problems, but I really wanted it."
The race began after a short delay due to one of the Sepang circuit's signature tropical downpours that doused the track.
Riders have complained this week that the newly repaved track was taking too long dry in some spots, forcing them to stay on their toes as they pass from dry to wet patches.
Much of the race was a gripping duel between Rossi and Dovizioso's Ducati teammate Andrea Iannone, who was returning to the track for the first time since he suffered a vertebrae fracture in a crash in early September.
Marquez on pole for Australian MotoGP
After Rossi snatched the lead early, it repeatedly changed hands between him and Iannone over the first 10 laps as they aggressively dived under each other with inches to spare on Sepang's tight curves.
But when Iannone crashed out on lap 12, Dovizioso pounced, quickly overtaking Rossi, stretching a sizable lead and holding on for the win.
Rossi said he began suffering problems as the track gradually dried over the course of the race.
"When the water (became) less, I had a problem with the front tyre and had to slow down," said the 37-year-old nine-time world champion. "It's a shame because we would have liked to win but its good to get second place in the championship."
Honda rider Marc Marquez had already clinched the world championship two weeks earlier in Japan.
COMMENTS
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ