Police General Ali Shah Paktiwal told the BBC the grave was unearthed in a former military base dating back to the Soviet occupation of Afghanistan in the 1980s.
The grisly find was made after an old man who recently returned to Afghanistan led police to the site where victims were apparently walled up in rooms and left to die.
“An old man told us about the grave. He told us he worked as a driver when there was a Russian military base here,” Gen Paktiwal told the BBC World Service.
“They used to bring people here. They put them in these rooms, they shut the door and then they put bricks and stones and covered the door with earth.”
Several hundred bodies have been discovered in the 15 rooms unearthed so far but it was not known how many were buried there in total, the report said.
Gen Paktiwal, who is overseeing the criminal investigation into the mass grave, said many of the victims had their arms tied and had been blindfolded or gagged.
Photographs on the BBC news website showed policemen digging skulls and other bones out of the underground chambers in the prison.
The find is a grim reminder of the nearly three decades of conflict and human rights abuses Afghans have endured.
More than 1.5 million Afghans were killed and thousands tortured or raped during the resistance to the Soviet invasion and ensuing civil war.
It was the second mass grave reported to have been found near Kabul. Last year Nato troops discovered another one near the notorious Pul-e-Charkhi prison in the east of the city where many opponents of the Soviet invaders were tortured and killed.
In April this year another mass grave, containing the remains of more than 400 Afghans killed during the communist era, was discovered in the remote north-eastern province of Badakhshan.