The rebels had seized the hilltop complex a day earlier after fierce fighting with supporters of Hadi, who has taken refuge in Saudi Arabia.
"The Huthi militia and their allies withdrew before dawn from the Al-Maashiq palace," said the official in Aden, who did not want to be named.
Under pressure from pro-Hadi fighters, the rebel forces retreated to the nearby central district of Khor Maksar.
A security source and the official Saudi news agency SPA also reported the rebels had quit the palace.
The Saudi-led coalition launched an air war, called Operation Decisive Storm, on March 26 to try to stop an advance by Shiite Huthi rebels and allied military units loyal to former president Ali Abdullah Saleh.
Hadi fled to Aden from Sanaa after the rebels seized power in the capital in February, and the southern palace came under fire from warplanes twice during his stay.
He went into hiding last week as the Huthis bore down on what was his last stronghold, later surfacing in the Saudi capital as Riyadh launched air strikes on the rebels.
UN aid chief Valerie Amos said Thursday that 519 people had been killed and nearly 1,700 injured in two weeks of fighting in Yemen, adding she was "extremely concerned" for the safety of trapped civilians.
COMMENTS
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ