Sanchez, a 46-year-old economist with no government experience, will take the oath of office at 11:00 am (0900 GMT) before King Felipe VI at the Zarzuela Palace near Madrid.
The Socialist leader still must name his cabinet and it is only when their names are published in an official government journal in the coming days that he will fully assume his functions.
Bloody legacy of the ETA haunts Spain's Basque left wing separatists
After leading the Socialists to two crushing general election defeats in 2015 and 2016, Sanchez was forced to resign by the party apparatus.
But he was reelected as party head by members in 2017, and managed to oust Rajoy, a 63-year-old veteran politician who had been in power since 2011.
Sanchez filed a no-confidence motion against Rajoy on May 25, a day after a court found former officials of his conservative Popular Party (PP) guilty of running a slush fund.
Other opposition parties lined up against Rajoy, who was abandoned by his allies, the centrist Ciudadanos and the Basque Nationalist Party (PNV).
The PNV had just helped Rajoy's government approve its 2018 budget in exchange for receiving 500 million ($584 million euros) in investments for the northern Basque region which it governs.
An absolute majority of 180 lawmakers voted for the motion to loud applause and shouts of "Yes we can", even if Sanchez faces a tough road ahead as he will govern with an even smaller minority than Rajoy.
"It's been an honour - there is none bigger - to have been Spain's prime minister," Rajoy told parliament.
In his first comments after winning the no-confidence motion, Sanchez, a former basketball player, vowed to tackle "all the challenges which the country faces with humility".
But he will struggle to govern as his Socialists have just 84 seats in the 350-seat parliament, and his minority government is supported by a hodgepodge of disparate parties, including far-left Podemos, Catalan separatists and Basque nationalists, with diverging interests.
All of his allies in the no-confidence motion stressed their vote against Rajoy was not a blank cheque for Sanchez.
"Our 'Yes' to Sanchez is a 'No' to Rajoy," is how Joan Tarda, speaker of Catalan pro-independence party Esquerra Republicana de Catalunya (ERC), put it in parliament.
Sanchez will only be able to implement policy initiatives "that allow him to obtain an easy majority" in parliament, said Fernando Vallespin, political scientist at the Autonomous University of Madrid.
PP lawmaker Rafael Hernando said Sanchez would be entering the prime minister's office "through the back door" after failing to win 2015 and 2016 general elections.
Spanish PM poised to fall in no-confidence vote
Sanchez has already tied his hands by promising to respect Rajoy's 2018 budget, which includes generous concessions to the northern Basque region.
He has also said he wants to "build bridges" with Catalonia's new separatist government, headed by Quim Torra, which will take office on Saturday at the same time that Sanchez takes his oath of office.
The parties that supported Sanchez will make demands he will not meet, predicted Pedro Fernandez, a 68-year-old pensioner, outside of parliament.
"When he does not do what they want, they will remind him that they brought him to power. And in five or six months we will either have fresh elections or they will oust him," he added.
COMMENTS
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ