Russia’s commander in Ukraine, Gen Sergei Surovikin, said it was no longer possible to keep supplying the city.
The withdrawal means Russian forces will pull out entirely from the western bank of the River Dnipro.
It is a significant blow as Russia faces a Ukrainian counter-offensive.
For the Kremlin it is also an embarrassment, says BBC Russia Editor Steve Rosenberg.
The military’s top brass were seen making the decision on Russian state TV, with Gen Surovikin reporting on the situation on the ground in Kherson.
President Vladimir Putin did not take part in the staged event. The architect of Russia’s failing war in Ukraine appeared to have left the announcement to his generals.
It was Mr Putin who announced Russia’s annexation of Kherson, and three other occupied regions, at the end of September.
“In these circumstances, the most sensible option is to organise the defence along a barrier line along the river Dnipro,” Gen Surovikin told the meeting.
The retreat was announced shortly after Russian media said the deputy leader of Kherson, Kirill Stremousov, had been killed in a car crash.
Seen as one of the main cheerleaders of the occupation of Kherson, he had warned only six days ago that it was “most likely” that Russian forces would have to cross to the eastern bank.
Although the Ukrainian advance had slowed in recent weeks, Russia’s supply lines across the Dnipro had become increasingly difficult after the few bridges across were destroyed by Ukrainian missiles.
Before the withdrawal, Russia moved thousands of civilians out of the city by boat, in what Ukraine condemned as a forced deportation.
Gen Surovikin – previously a notorious commander of Russia’s recent operations in Syria – was appointed to take over the running of Russia’s invasion a month ago.