On some lawn tractors I have had to replace the idler arm bracket because the deck wears a spot down on it. You'll need to remove the spring and take the pulley off to get down to the bracket itself to inspect it for wear. Usually you have to put a wrench under the pulley while you loosen the pulley nut with another wrench or socket. Good luck and keep us posted.
The idler arm get to notches worn in it where the blade brakes rub constantly. if you take the arm off and inspect it you will probably see the 2 notches. if you have a good metal file and some time, file the areas with notches into smooth rounded edges so the brake arms can slide easily across them. reassemble and your problem should be fixed.