The Union position was untenable, and the men could see the XI Corps retreating from the northern battle, pursued by masses of Confederates. Doubleday ordered a withdrawal east to Cemetery Hill.^[44] On the southern flank, the North Carolina brigade of Brig. Gen. James H. Lane contributed little to the assault; he was kept busy by a clash with Union cavalry on the Hagerstown Road. Brig. Gen. Edward L. Thomas's Georgia Brigade was in reserve well to the rear, not summoned by Pender or Hill to assist or exploit the breakthrough.^[45]

Union troops retreated in different states of order. The brigades on Seminary Ridge were said to move deliberately and slowly, keeping in control, although Col. Wainwright's artillery was not informed of the order to retreat and found themselves alone. When Wainwright realized his situation, he ordered his gun crews to withdraw at a walk, not wishing to panic the infantry and start a rout. As pressure eventually increased, Wainwright ordered his 17 remaining guns to gallop down Chambersburg Street, three abreast.^[46] A.P. Hill failed to commit any of his reserves to the pursuit of the Seminary defenders, a great missed opportunity.^[47]