The grave of Reinhard Heydrich was "dug up in the night between Wednesday and Thursday" and an investigation has been opened on charges of disturbing a burial site, a police spokesperson said.
German media said it appeared nothing was removed.
Heydrich was the powerful head of Hitler's Reich Security Office, which included the Gestapo.
Less well-known than other Nazi leaders, he was nevertheless highly influential and was marked out for his cruelty even within the Third Reich elite.
Adolf Hitler admiringly used to refer to him as "the man with the iron heart", according to the biography Heydrich: The Face of Evil by Mario Dederichs.
Heydrich hosted the Wannsee Conference on January 20, 1942 when leading Nazis discussed the extermination of the Jews in German-occupied Europe.
France, Portugal, Germany drawn together at Euro 2020 as England get Croatia
During the Nazi occupation of what is now the Czech capital, he became known as "the Butcher of Prague".
His car was attacked with an anti-tank mine in the city on May 27, 1942 by Czechoslovak agents trained by Britain's secret Special Operations Executive.
Heydrich died of his injuries a few days later.
His body was brought back to Berlin and buried in the city's Invalidenfriedhof, a military cemetery.
During the Cold War, the cemetery became a no-man's land along the Berlin Wall and his grave — along with the ones of other top Nazis —was dismantled.
But Heydrich's remains were never disinterred and the location of the grave was an open secret.
In 2000, a group of anti-fascists said they had opened up the grave of Nazi storm trooper Horst Wessel in Berlin, taken his skull and thrown it into the Spree River, according to the Tagesspiegel newspaper.
Police at the time said no remains were stolen.
COMMENTS
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ