A French national, a Kuwaiti and 10 Pakistani detainees were sent back to their home countries last month from the Parwan prison, the defence official told AFP on condition of anonymity.
The move left 38 non-Afghan detainees at the prison.
The Defence Department notified Congress of the transfer 10 days beforehand, the official added.
President Barack Obama's administration faces legal difficulties at the Parwan center similar to those posed by the US military prison at Guantanamo Bay, Cuba.
In both cases, US authorities suspect detainees have terror links but often lack sufficient evidence to try them in court and are reluctant to release them.
The Obama administration is trying to shut down the Guantanamo and Parwan prisons as it seeks to draw a line under the extraordinary legal measures adopted in the wake of the September 11 attacks.
The move at Parwan came amid controversy over a prisoner swap involving the release of a US soldier captured by Afghan insurgents and five Taliban militants held at Guantanamo.
Bowe Bergdahl was freed on May 31 but some lawmakers have slammed the White House for agreeing to the transfer of the five Taliban figures.
With the United States and Nato allies withdrawing most of their combat troops from Afghanistan this year, Washington handed over control of Parwan prison to the Kabul government under an agreement.
But US authorities still help oversee the cases of foreign, non-Afghan detainees.
The conditions for the 12 detainees transferred last month remain unclear.
COMMENTS
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ