Habre seized power in Chad in 1982, fleeing to Senegal in 1990 after he was overthrown.
The former leader — who is now in his late 70s — was jailed in Senegal in 2016 over abuses committed during years of iron-fisted rule in Chad.
It would be a slap in the face to his victims for him to receive special treatment.
Some 40,000 people are estimated to have been killed under Habre’s leadership of the semi-desert country.
But on April 6, a judge granted Habre a 60-day release from prison after his lawyer argued that his age left him particularly vulnerable to coronavirus.
Habre’s supporters are ramping up efforts to have him freed, which has fuelled fear among his victims that the authorities might be gearing up to release him permanently.
However, Babacar Dione of Senegal’s justice ministry told AFP that Habre would return to prison on Sunday.
The former dictator had spent his 60 days of freedom under house arrest at his home in Ouakam in Senegal’s seaside capital Dakar.
Senegalese authorities have recorded 4,328 cases of coronavirus to date, with 49 fatalities.
A lawyer who represented Habre’s victims said in a statement on Sunday that “no one wants to see Habre himself unprotected,” despite atrocities committed under his rule.
“But let’s remember that Habre was convicted of mass murder,” said Reed Brody, the human rights lawyer.
“It would be a slap in the face to his victims for him to receive special treatment”.