With that much wear I would be removing the housing & either replacing it or building it up and remachining it.
The bearing is obviously cactus & has been spinning in the housing for a long time.
Bearing are not bearings, they come in about 5 levels of fit to compensate for compressive loads when they are pressed into a housing or to allow a bit of extra running clearances.
The mower companies never list the class of bearing .
In your cae I would be considering fitting +1 ( slightly sloppier ) bearing
But why, when the ball bearing in question still spins smoothly would the outer race find it easier to spin in the pulley? Usually, for a pressed-in bearing to spin in the housing, the inner race and balls have to seize up. This bearing doesn't even feel gritty.
BTW - the two pulley bearings are dirt-common 6203-2RS, available for cheap at any bearing supply house or on line.
FWIW, I found a length of thinwall tubing slightly larger than the bearing OD. Cut a length the size of the height of the bearing, cut a slit in the wall and removed a small section to allow the spacer to press in the pulley ID. Using a Dremel as a tool post grinder, I'm sizing the sleeve ID for a press fit on the new bearing. A bit of red Loctite will be insurance.
Bit of trivia, I didn't have the 6203s in my misc bearing drawer, so I ordered a couple on line. An hour later, I was changing a headlight bulb in one of our Saabs. In one of the Saab parts box were two new SKF bearings. Now, if I could just remember for which part of which of the many Saabs those 6203-2RS bearings were purchased as spares.
jack vines