The disturbances came with tensions running high after Israeli Defence Minister Moshe Yaalon last week outlawed two Muslim groups that confront Jewish visitors to the compound, which is holy to both faiths.
Palestinian witnesses said police entered the mosque, Islam's third-holiest site. Police only said they closed the doors to the mosque to lock in rioters throwing stones, fireworks and other objects.
Authorities have used the same tactic in the past in a bid to restore calm and which has seen them briefly enter.
According to police, the rioters had barricaded themselves in the mosque overnight with the aim of disrupting visits by Jews to the site ahead of the start of New Year celebrations on Sunday evening.
"Masked protesters who were inside the mosque threw stones and fireworks at police," a police statement said.
"Suspect pipes that could be filled with homemade explosives were also found at the entry to the mosque."
A Muslim witness accused police of entering the mosque and causing damage, saying prayer mats were partially burned.
Clashes later continued outside the mosque complex, with police firing tear gas and stun grenades.
The defence minister's move last week banned the Murabitat and Murabitun, which he said was necessary to "defend the security of the state, the well-being of the public and public order".
His office said the groups were "a main factor in creating the tension and violence" at the site, venerated by Jews as the Temple Mount, and in Jerusalem at large.
Non-Muslims are allowed to visit the compound, but Jews are forbidden from praying or displaying national symbols for fear of triggering tensions with Muslim worshippers.
Israel seized east Jerusalem, where Al-Aqsa is located, in the Six Day War of 1967 and later annexed it in a move never recognised by the international community.
COMMENTS
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ