Rumours were rife in the country that Mr Magufuli had coronavirus, but the cause of death was given as a heart complication. He was 61.
The country's Vice President Samia Suluhu Hassan confirmed his death in a statement delivered on live television.
READ MORE: The countries making dubious COVID-19 claims
Mr Magufuli had not been seen in public for two weeks, and died in a hospital in the city of Dar Es Salaam.
Police in the east African nation have been arresting people in recent days for spreading reports the president was seriously ill.
"There are a lot more who have been spreading false statements," Police Commissioner for Kinondoni Ramadhani Kingai said at a press conference on Saturday.
"We will arrest them all."
Mr Magufuli frequently mocked the threat of coronavirus, declaring last year the disease had been eradicated by three days of national prayer.
He also cast aspersions on the country's coronavirus testing program, claiming that he sent samples of papaya and goat to the facility and they came back positive.
He also discouraged the wearing of masks.
Tanzania stopped reporting coronavirus figures in May last year.
LIVE UPDATES: Europe threatens to block vaccine supplies
Before being elected to parliament and then the presidency, Mr Magufuli was a high school teacher and an industrial chemist.
Popular in Tanzania for his anti-corruption efforts, he was also accused of autocratic tendencies, persecuting critics and members of the opposition.
Ms Hassan has now become Tanzania's first female president.
Located between Kenya and Mozambique, Tanzania has a population of around 58 million.
Source: 9News https://www.9news.com.au/world/tanzania-president-dies-john-pombe-magufuli--coronavirus-sceptic-covid19-africa-news/14305b74-eb31-4460-9640-d2092efa6bc4