The part you show is called the pivot arm and the part you are trying to remove is called the bearing hoc. There should be a retaining ring which hold the two together. The shiny part is the bearing. Section 4 of this manual should help you out.
https://www.toro.com/getpub/22446. I doubt that the pivot arm is the cause of you problem. More likely the bearing is bad, for which I would replace the bearing and bearing hoc. Please post the model and serial numbers for your unit and I will get you the part numbers if you need them.