A security official said 34 people were also wounded in the clashes "that lasted from morning until late in the evening," and that eight were in serious condition.
Witnesses said police fired live rounds tear gas to disperse hundreds of protesters while troops deployed on rooftops of buildings around the mission in Sanaa.
Earlier in the day police used water cannon and fired warning shots to expel protesters who breached the perimeter wall.
One demonstrator was shot dead outside the compound as police battled to prevent any new incursion, with three others killed in successive clashes.
The protests came as US and Libyan officials said they were probing a mob attack on the consulate in Benghazi that killed the ambassador and three other US officials on Tuesday, amid growing speculation it was the work of jihadist militants rather than just demonstrators.
As Libya announced it made arrests, US Attorney General Eric Holder told an Arab forum in Qatar, "the FBI has opened an investigation" into the deaths and the attack on the consulate.
Yemeni President Abdrabuh Mansur Hadi apologised to his US counterpart Barack Obama and the American people for the acts of a "mob" and ordered a probe.
"Those who are behind (the attack) are a mob that are not aware of the far-reaching plots of Zionist forces, especially those who made a film insulting the Prophet (PBUH)," said Hadi.
Some protesters said they saw three vehicles being torched by some of the demonstrators after they gained access to the compound through an unguarded security gate.
After being evicted from the complex on their first assault, protesters retreated about 100 metres (yards) from the gate, gathering near a checkpoint where they chanted anti-Jewish slogans.
Chanting "O, messenger of Allah... O, Mohammed," the protesters then launched a second bid to access the compound, prompting police to fire on the crowd, witnesses said, and clashes continued until late in the evening.
European Union foreign policy chief Catherine Ashton condemned the violence and urged Yemeni authorities to boost security at EU diplomatic missions in Sanaa.
"We have urged the Yemeni authorities to reinforce security of EU missions in Sanaa and to take the necessary measures to protect diplomats," Ashton added.
Violence also rocked the Egyptian capital Cairo, where police fired tear gas to disperse protests outside the embassy by stone- and bottle-throwing demonstrators.
A total of 224 people were injured, the Egyptian health ministry said.
On Tuesday night, protesters stormed the Cairo embassy compound, tearing down the Stars and Stripes and replacing it with a black Islamic flag.
COMMENTS
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ