Historian Andrew Roberts notes that "It is a curious fact about the Battle of Waterloo that no one is absolutely certain when it actually began". Wellington recorded in his dispatches that at "about ten o'clock [Napoleon] commenced a furious attack upon our post at Hougoumont". Other sources state that the attack began around 11:30.The house and its immediate environs were defended by four light companies of Guards, and the wood and park by Hanoverian Jäger and the 1/2nd Nassau.

The initial attack by Bauduin's brigade emptied the wood and park, but was driven back by heavy British artillery fire, and cost Bauduin his life. As the British guns were distracted by a duel with French artillery, a second attack by Soye's brigade and what had been Bauduin's succeeded in reaching the north gate of the house. Sous-Lieutenant Legros, a French officer, broke the gate open with an axe, and some French troops managed to enter the courtyard. The Coldstream Guards and the Scots Guards arrived to support the defence. There was a fierce melee, and the British managed to close the gate on the French troops streaming in. The Frenchmen trapped in the courtyard were all killed. Only a young drummer boy was spared.

Fighting continued around Hougoumont all afternoon. Its surroundings were heavily invested by French light infantry, and coordinated attacks were made against the troops behind Hougoumont. Wellington's army defended the house and the hollow way running north from it. In the afternoon, Napoleon personally ordered the house to be shelled to set it on fire,resulting in the destruction of all but the chapel. Du Plat's brigade of the King's German Legion was brought forward to defend the hollow way, which they had to do without senior officers. Eventually they were relieved by the 71st Highlanders, a British infantry regiment. Adam's brigade was further reinforced by Hugh Halkett's 3rd Hanoverian Brigade, and successfully repulsed further infantry and cavalry attacks sent by Reille. Hougoumont held out until the end of the battle.