At 7 a.m. on May 3, Early was confronted with four Union divisions: Brig. Gen. John Gibbon of the II Corps had crossed the Rappahannock north of town, and three divisions of Sedgwick's VI Corps—Maj. Gen. John Newton and Brig. Gens. Albion P. Howe and William T. H. Brooks—were arrayed in line from the front of the town to Deep Run. Most of Early's combat strength was deployed to the south of town, where Federal troops had achieved their most significant successes during the December battle. Marye's Heights was defended by Barksdale's Mississippi brigade and Early ordered the Louisiana brigade of Brig. Gen. Harry T. Hays from the far right to Barksdale's left.^[18]

By midmorning, two Union attacks against the infamous stone wall on Marye's Heights were repulsed with numerous casualties. A Union party under flag of truce was allowed to approach ostensibly to collect the wounded, but while close to the stone wall, they were able to observe how sparsely the Confederate line was manned. A third Union attack was successful in overrunning the Confederate position. Early was able to organize an effective fighting retreat.^[19]

John Sedgwick's road to Chancellorsville was open, but he wasted time in gathering his troops and forming a marching column. His men, led by Brooks's division, followed by Newton and Howe, were delayed for several hours by successive actions against the Alabama brigade of Brig. Gen. Cadmus M. Wilcox. His final delaying line was a ridge at Salem church, where he was joined by three brigades from McLaws's division and one from Anderson's, bringing the total Confederate strength to about 10,000 men.^[19]

Confederate casualties totaled 700 men and four cannons. Early withdrew with his division two miles to the south, while Wilcox withdrew westward, slowing Sedgwick's advance. When he learned of the Confederate defeat, Lee started moving two divisions east to stop Sedgwick.