Troops also stormed a TV station and claimed they had targeted criminals around Mugabe.
South African President Jacob Zuma said Mugabe is “fine” but confined to his home.
The country’s Major General SB Moyo said they want to deal with people who “were committing crimes that are causing social and economic suffering in the country”.
He said: “As soon as we have accomplished our mission, we expect that the situation will return to normalcy.
“We are against that act of injustice and we intend to protect every one of you against that.”
It added: “Our wish is that you enjoy your rights and freedoms as Zimbabwe returns to prosperity.
“Provocation shall be met with appropriate response.”
Moyo also claimed the situation in Zimbabwe was not a “military takeover of government”.
The military broadcast ordered all members of the army to regroup as “all leave is cancelled”.
Moyo added: “We call upon all the war veterans to play a positive role in ensuring peace, stability and unity in the country.
“To members of the Zimbabwe defence forces, all leave is cancelled and you are all to return to your barracks with immediate effect.
“To the other security forces we urge you to co-operate for the good of the country.”
It was seen as a power play to make way for his wife Grace to succeed him and establish a dynasty after Mugabe’s 37-year reign.
A statement from a Twitter account claiming to belong to the ruling ZANU PDF party claims Mugabe, Grace and their family are under house arrest.
It said ousted vice-president Emmerson Mnangagwa has returned as leader.
“Last night the family was detained and are safe, both for the constitution and the sanity of the nation this was necessary.
“Neither Zimbabwe nor ZANU are owned by Mugabe and his wife. Today begins a fresh new era and comrade Mnangagwa will help us achieve a better Zimbabwe.
“There was no coup, only a bloodless transition which saw corrupt and crooked persons being arrested and an elderly man who had been taken advantage of by his wife being detained.”
The website of the ruling party now appears to be down.
A military source told Reuters: “They are now in charge of all armoury, all gates and roads leading in or out of the camp. “Arcturus Road (which leads to the camp) is closed and all Support Unit details with guns have been disarmed.”
Zimbabwean finance minister Patrick Chinamasa is among members of the government detained by the army.
Three explosions were heard in Harare just before the military announcement, and gunshots were also heard near the home of Mugabe.
The UK government on Wednesday advised its citizens currently in the capital Harare to remain indoors until political situation becomes clearer.
“Due to the uncertain political situation in Harare, including reports of unusual military activity, we recommend British nationals currently in Harare to remain safely at home or in their accommodation until the situation becomes clearer,” the Foreign & Commonwealth Office said in a statement.