Four security officers were killed and five others were wounded when a suicide bomber detonated his vest outside one of Islam's holiest sites.
The Monday evening attack took place as thousands of worshippers were about to hold sunset prayers in the mosque where the Prophet Muhammad is buried in Medina.
The mosque is visited by millions of Muslims from around the world annually performing pilgrimage in Mecca.
The interior ministry said the attacker set off the bomb after security officers raised suspicions about him.
The attack took place in a car park outside the sprawling mosque complex.
A suicide bomber struck at one of Islam's holiest sites in the Saudi Arabian city of Medina just as the call to sunset prayers was ending and hours after suicide bombers struck in two other cities.
Sabq news site and other local media said the explosion outside the Prophet Muhammad's mosque was caused by a suicide bomber near a security building.
The bomber reportedly sat briefly with security guards as they were breaking their dawn-to-dusk fast before detonating his vest.
Several cars caught fire and thick plumes of black smoke were seen rising from the site of the explosion as thousands of worshippers crowded the streets around the mosque.
There was no indication of the number of casualties and no group has claimed responsibility for any of the attacks.
The sprawling mosque where the Prophet Muhammad is buried is visited by millions of Muslims from around the world each year during pilgrimages to Mecca.
The area was packed with pilgrims for prayer during the final days of the Muslim holy month of Ramadan, which ends in the kingdom on Tuesday.
Altayeb Osama, a 25-year old Sudanese visitor to Medina and resident of Abu Dhabi, said he heard two large booms about a minute apart as he was heading toward the mosque for sunset prayers.
He said police and fire engines were on the scene within seconds.
"It was very shocking that such a thing happens in such a holy place for Muslims, the second holiest place in the world. That's not an act that represents Islam," Mr Osama said.
"People never imagined that this could happen here."
Qari Ziyaad Patel, 36, from Johannesburg, South Africa, was at the mosque when he heard a blast just as people were breaking their fast with dates.
Many at first thought it was the sound of traditional, celebratory cannon fire, but then he felt the ground shake.
"The vibrations were very strong," he said.
"It sounded like a building imploded."
Saudi Arabia's state-run news channel al-Ekhbariya aired live video of thousands of worshippers praying inside the mosque hours after the explosion.
It also showed footage of Saudi King Salman's son and the Governor of Medina, Prince Faisal bin Salman, visiting security officers wounded in the blast and the site of the explosion.
Also on Monday evening, at least one suicide bomber and a car bomb exploded near a Shiite mosque in eastern Saudi Arabia several hours after another suicide bomber carried out an attack near the US Consulate in the western city of Jiddah.
Saudi Arabia has been a target of Islamic State attacks that have killed dozens of people.
In June, the Interior Ministry reported 26 terror attacks in the kingdom in the last two years.
The possibility of coordinated attacks across different cities in Saudi Arabia on the same day underscores the threat the kingdom faces from extremists who view the Western-allied Saudi monarchy as heretics and enemies of Islam.
Saudi Arabia is part of the US-led coalition fighting the Islamic State group in Iraq and Syria.
The attack in the eastern region of Qatif did not appear to cause any injuries, said resident Mohammed al-Nimr.
His brother, prominent Saudi Shiite cleric Nimr al-Nimr, was executed in January after a court found him guilty of sedition and inciting violence for his role in anti-government protests.
Qatif is heavily populated by Shiites, who are a minority in the Sunni-ruled kingdom.
Mr al-Nimr said that near the body of a suicide bomber was a car bomb that also went off around the same time.
He said the bomber detonated his suicide vest when most residents were at home breaking the daily Ramadan fast.
Several state-linked media reported that two suicide bombers died in the attack, which was aimed at a Shiite mosque.
Earlier on Monday, the Interior Ministry said a suicide bomber had detonated his explosives when security guards approached him near the US Consulate in Jiddah.
The attacker died and the two security men were wounded.
The US Embassy in Saudi Arabia confirmed there were no casualties among consular staff.