Two women -- aged 80 and 70 -- and two men -- aged 83 and 77 -- were killed after the yellow bus careered out of control in the downtown North Point district.
Photographs in local media showed victims sprawled on the ground while others received medical treatment at the scene.
The vehicle rolled down the street after it was parked, terrifying passers-by as the driver ran towards it and tried in vain to stop it, according to the police.
"We believe he might have forgotten to pull the handbrake or hadn't properly pulled it," chief inspector of police Sun Lun-yum told reporters.
The 62-year-old driver became trapped under the bus and was dragged along for about 20 metres before falling onto the road.
Two school buses crash head-on in Germany, 40 injured: police
Police said the bus travelled for a total of 100 metres, hitting two taxis before finally crashing into a building in a narrow lane lined with shops.
In addition to the driver, who suffered injuries to his head, neck and back, a further nine pedestrians aged between 22 and 89 were taken to hospital for treatment. Several remain in a serious condition.
Hong Kong prides itself on having one of the world's best public transport systems but deadly bus accidents are not unknown.
Five people were killed and 32 injured when a coach carrying Cathay Pacific staff to Hong Kong's airport collided with a taxi last month.
A speeding double-decker overturned in northern Hong Kong in February, killing 19 people and leaving more than 60 injured. The bus driver was arrested for dangerous driving.
And in 2003 a double-decker bus collided with a truck and plummeted off a bridge, killing 21 people and injuring 20.
COMMENTS
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ