Sedgwick withdrew across the Rappahannock at Banks's Ford during the pre-dawn hours of May 5. When he learned that Sedgwick had retreated back over the river, Hooker felt he was out of options to save the campaign. He called a council of war and asked his corps commanders to vote about whether to stay and fight or to withdraw. Although a majority voted to fight, Hooker had had enough, and on the night of May 5–6, he withdrew back across the river at U.S. Ford.^[23]

It was a difficult operation. Hooker and the artillery crossed first, followed by the infantry beginning at 6 a.m. on May 6. Meade's V Corps served as the rear guard. Rains caused the river to rise and threatened to break the pontoon bridges.^[23]

Couch was in command on the south bank after Hooker departed, but he was left with explicit orders not to continue the battle, which he had been tempted to do. The surprise withdrawal frustrated Lee's plan for one final attack against Chancellorsville. He had issued orders for his artillery to bombard the Union line in preparation for another assault, but by the time they were ready Hooker and his men were gone.^[23]