Casimir III of Poland urged Louis to intervene in his war with the Lithuanians who had occupied Brest, Volodymyr-Volynskyi, and other important towns in Halych and Lodomeria in the previous years. The two monarchs agreed that Halych and Lodomeria would be integrated into the Kingdom of Hungary after Casimir's death. Louis led his army to Cracow in June 1351. Because Casimir fell ill, Louis became the sole commander of the united Polish and Hungarian army.

He invaded the lands of the Lithuanian prince, Kęstutis, in July. Kęstutis seemingly accepted Louis's suzerainty on 15 August and agreed to be baptised, along with his brothers, in Buda. However, Kęstutis did nothing to fulfill his promises after Polish and Hungarian troops were withdrawn. In an attempt to capture Kęstutis, Louis returned, but he could not defeat the Lithuanians, who even killed one of his allies, Boleslaus III of Płock, in battle. Louis returned to Buda before 13 September

Casimir III laid siege to Belz and Louis joined his uncle in March 1352. During the siege, which ended without the surrender of the fort, Louis was heavily injured in his head. Algirdas, Grand Duke of Lithuania, hired Tatar mercenaries who stormed into Podolia, Louis returned to Hungary because he feared a Tatar invasion of Transylvania. Pope Clement proclaimed a crusade against the Lithuanians and the Tatars in May, authorizing Louis to collect a tithe from Church revenues during the next four years.