The song emerged last week and has been widely shared on social media.
One line in it asks:
“Is there still freedom of expression in the country?” and refers to a “doctor” who can’t tolerate criticism.
“Media owners, let me tell you: ‘Be careful. Watch it. If you think you have that kind of freedom – not to that extent,’” Mr Magufuli said.
Police confirmed the arrest to local media, saying the rapper would be questioned “for releasing a song with words that malign the government”.
He was reportedly held at a hotel in Morogoro, about 190km (120 miles) from Tanzania’s commercial capital Dar es Salaam, to where he was being transferred.
Local media reports said Nay wa Mitego’s lawyer was due to file a request for bail.
Other lines in the song said:
“Who are you now? Don’t you want to listen to advice? Don’t you want criticism?”
The lyric is addressed to someone he calls “a doctor specialising in lancing boils”.
The phrase “lancing boils” has been used repeatedly by Mr Magufuli to refer to people he deems obstructive.