A Brazilian news weekly reported, Temer surprised Brazilian politics watchers this week with the revelation that he has decamped from the Alvorada Palace.
He moved with his former beauty queen wife and their seven-year-old son down the road to the smaller vice presidential residence.
The modernist Alvorada, which means Dawn and was designed by Brazilian architect Oscar Niemeyer, would be a dream home for many.
It has a huge pool, football field, chapel, medical centre and vast lawn.
But Temer, 76, and his 33-year-old wife Marcela, find the cavernous, glass-fronted building spooky.
“I felt something strange there. I wasn’t able to sleep right from the first night.
“The energy wasn’t good,” Temer was quoted as saying by Veja.
“Marcela felt the same thing. Only (their son) Michelzinho, who went running from one end to the other, liked it.”
“We even started to wonder: could there be ghosts?” he reportedly quipped to Veja.
According to a report in Globo newspaper, Marcela Temer brought in a priest to attempt to drive out any evil spirits, but to no avail.
The Temers then moved to the still luxurious but smaller Jaburu Palace nearby.
That automatically put Temer in the top job and in the Alvorada. No one filled his vacant vice presidential post, however, meaning he can now take his pick of palaces.
The house moving comes in the middle of a severe political crisis for Brazil, with many of Temer’s allies face potential corruption probes.
The president himself is battling a case in the electoral court where he is accused of having benefited from illegal donations when he and Rousseff ran together in 2014.