Swiss co-investigating judge Laurent Kasper-Ansermet said that his authority to investigate possible third and fourth cases at the tribunal had been constantly thwarted by his Cambodian counterpart.
"The situation is completely blocked," Kasper-Ansermet told AFP.
In October, German judge Siegfried Blunk quit the court, blaming government interference in two potential new cases, in which five mid-level Khmer Rouge members face a string of allegations including mass killings and forced labour during the regime's 1975-1979 reign of terror.
The United Nations named Kasper-Ansermet, the reserve judge, as Blunk's replacement.
But Cambodia refused to recognise the appointment, prompting an unprecedented row and forcing the Swiss to work without the support of his Cambodian counterpart You Bunleng.
In a statement released by the court, Kasper-Ansermet said that "You Bunleng's active opposition to investigations into cases 003 and 004 has led to a dysfunctional situation".
The judge will stay in his role until May 4. It is unclear whether the UN has a replacement ready, but observers said any future judge would likely face similar difficulties.
"I'm surprised by the resignation but I'm not surprised at what has led to it," said tribunal monitor Clair Duffy from the Open Society Justice Initiative.
"The UN and donors have to address the heart of the problem: that the Cambodian government is trying to control who this court investigates and prosecutes."
The court, set up to find justice for the deaths of up to two million people under the hardline communist Khmer Rouge regime, has for years been dogged by claims of political meddling.
The Cambodian government, which includes many former Khmer Rouge members, strongly opposes pursuing more suspects of the regime beyond the current second trial.
The tribunal has so far completed just one trial, sentencing former prison chief Kaing Guek Eav, better known as Duch, to life in jail on appeal last month for overseeing the deaths of some 15,000 people at a torture prison.
Duch told the court on Monday that he would help to discover "the truth" in an ongoing second trial against the regime's three most senior surviving leaders who are accused of war crimes, genocide and crimes against humanity.
"I am ready to cooperate with the court," said the 69-year-old, clad in the country's standard blue prison uniform, when asked if he understood he was there to provide "information concerning the truth" about the brutal regime.
The court's potential third and fourth cases were widely expected to be dismissed in the face of government objections, though observers had hoped Kasper-Ansermet would breathe new life into the moribund cases.
You Bunleng, who came under fire last year for closing the probe into the third case without quizzing the suspects or visiting the crime scenes, has repeatedly stated that Kasper-Ansermet's actions are not legally valid because he is not officially accredited.
"I haven't recognised his rights from the beginning. Everything has to be conducted in accordance with the law," You Bunleng told AFP, reacting to the news of Kasper-Ansermet's resignation.
Cambodia said earlier this year it objected to the Swiss because he had used the microblogging site Twitter to draw attention to the controversy surrounding the cases.
Observers have however voiced suspicions that the government may have used the spat as an excuse to keep stalling on the investigations.
COMMENTS
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ