In a high-scoring thriller helped by short boundaries at the packed Green Park stadium, India made a spirited chase of the Proteas' challenging 303-5 before being stopped just short on 298-7.
Sharma nearly upstaged South African captain AB de Villiers' 104 off 73 balls before the tourists' bowlers fought back in front of 40,000 home fans.
Leg-spinner Imran Tahir removed Sharma and Suresh Raina in the space of five deliveries in the 47th over, before 20-year-old fast bowler Kagiso Rabada nailed his side's victory in the final over.
With India needing 11 runs off six balls, Rabada conceded just five runs and claimed two wickets, including the important one of skipper Mahendra Singh Dhoni for 31, to dash India's hopes.
The victory put South Africa ahead in the five-match series, which follows their 2-0 success in the preceding Twenty20 games.
Sharma put on 149 for the second wicket with Mumbai team-mate Ajinkya Rahane (60) after opener Shikhar Dhawan was leg-before to Morne Morkel for 23 in the eighth over.
Sharma, the only batsman in the world to have hit two double-centuries in one-day cricket, stroked 13 boundaries and six sixes before he offered a tame return catch to Tahir.
Earlier, the Proteas had put on a breathtaking batting display after de Villiers won the toss and elected to bat.
De Villiers reached his century off the last ball of the innings, hitting Umesh Yadav for a six. He finished with six sixes and five boundaries.
Farhaan Behardien was unbeaten on a 19-ball 35 as the Proteas plundered 86 runs in the final six overs -- which proved to be the key to success in the end.
"It was an amazing game of cricket," said a visibly tired and relieved de Villiers, who was named man of the match. "It is close games like these that bring the fans in.
"The momentum turned around several times, but I thought Tahir turned the game on its head. Rabada, of course, proved to be a handful with his pace. He's got a bright future."
India were dealt a blow when frontline spinner Ravichandran Ashwin, playing his 100th one-dayer, suffered a side strain while bowling his fourth over and left the field.
Ashwin returned to bowl only one more over before going off again. Fellow off-spinner Harbhajan Singh has been drafted into the squad as a precaution.
Dhoni said he was disappointed by the narrow loss which he blamed on his bowlers leaking runs in the final overs.
"The outcome was not great for us," he said. "There were plenty that went wrong, but we showed character to get close. Frankly, we should have won this game.
"We missed the six overs from Ash (Ashwin) which was crucial because he is our best spinner. But the bowlers should have done a bit better.
"The partnership between Rohit and Rahane was superb. This was a fantastic innings from Rohit. It wasn't an easy wicket to bat on but he made it look so easy."
The second one-dayer will be played in Indore on Wednesday.
COMMENTS (3)
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ