Don't if the idler arm is the cause but definitely needs fixing. Most it requires the idler arm, the shouldered washer, and the fender washer to replaced. The fender washer completely out, the shoulder wears one sided, and idler arm where the shoulder washer rides wear thin. All this leads to a sloppy idler arm assembly. Most times here it leads to the belt being thrown.
Also don't forget to check the blades for balance and for any bent pulleys like a spindle pulley that is not spinning true. (ie not wobbling).
Also there is two different belt used depending the year of the mower. LT1050 manufactured before 2008 used a B [5/8] section x 131 belt and 2008 and later used an A [1/2] section x 131.25 belt. If yo are trying to use the A sec on the older mower it will be too loose as it will ride too deep in the pulleys.