He was 85 and finally succumbed to multiple organ failure after being in a critical condition for more than three weeks, his doctors said.
Thousands of Hindu followers massed at the Institute of Higher Medical Sciences hospital where Sai Baba had been treated in his hometown of Puttaparthi, in the southern state of Andhra Pradesh.
Police used barriers to hold back crowds of mourners and appealed for public calm after the death was announced.
“Sai Baba is no more with us physically. He breathed his last at 7.40am and died due to cardio-respiratory failure,” a statement from the hospital said. “His body will be kept for public worship for two days on Monday and Tuesday.”
Devotees had congregated in the town over recent weeks to hold prayers asking for a miracle to allow Sai Baba to recover.
He was credited by millions of people across the globe with having supernatural powers, including an ability to conjure objects out of thin air, remember past lives and cure terminal diseases.
Recognisable for his wild curly hair and saffron robes, he was followed by politicians, judges and generals in India as well as cricketing legend Sachin Tendulkar, Hollywood star Goldie Hawn and legions of the poor.
Indian Prime Minister Manmohan Singh said the nation would deeply mourn the death of Sai Baba, who “was an inspiration to people of all faiths”.
Tendulkar had earlier said on his Twitter page that he was praying for Sai Baba’s health and asked others to join him.
The guru claimed to be the reincarnation of a former holy man, Sai Baba of Shirdi, who died in 1918.
At meetings his showman antics, in which he would miraculously produce gold coins or watches on stage, brought him both fame and notoriety.
He also faced allegations of sexual abuse of young male followers, which his opponents said were ignored by Indian authorities due to his power and popularity.