Japan looked to have been given a crucial advantage when Greek captain Konstantinos Katsouranis was dismissed on 38 minutes for a second bookable offence.
But although Japan dominated possession thereafter, they were unable to find a way through a stubborn Greek rearguard action and now have it all to do if they are to emerge from Group C.
Japan's draw means Colombia are guaranteed a top two finish in the group, but Zaccheroni's men will now have to beat the in-form South Americans to have any chance of joining them in the last 16.
"We're not happy at this stage of the tournament," said Zaccheroni. "We pushed hard to win the game. We do have one match to play, but right now I'm not happy at all.
"We had a lot of possession, had a lot of chances and we didn't capitalise on them.
"I don't know what Colombia will decide and how they line up for us. But we have to play differently from the first game and also from our game tonight.
"We should have won the game."
Rodriguez inspires Colombia win
Another superb performance by playmaker James Rodriguez who scored one goal and created the other inspired Colombia to a 2-1 win over Ivory Coast in their World Cup Group C match on Thursday.
Rodriguez, scoring his second of the tournament, and substitute Juan Fernando Quintero, with his first international goal, struck in the second half while Gervinho replied with a superb individual goal, his second of the finals, for the African side.
"It was a tough match like all of those at the World Cup," said Rodriguez, who has more than made up for the loss of leading striker and fellow Monaco teammate Radamel Falcao through injury.
"It is terrific to have taken three points off our major rival who are extremely physical and know how to play good football."
Meanwhile Colombia's coach Jose Pekerman said, "It gives me great pleasure that we have come through a tough encounter and we have six points from six. It is super that Colombia can compete at this level with players competing in their first World Cup finals."
COMMENTS
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ