If this is true, you are jamming the feeler gauge in way too tightly.The valve can also move when pushing the feeler gauge in.
The drag should be like pulling a piece of paper out of a small book. That's the drag you are looking for when gapping valves. Just a SLIGHT drag is all you need.
Easy way is one valve fully open, SET the other valve lash for that cylinder. Rotate engine around and do the same for the other valve. Set gap with valve on the heel of the cam. NOT the lifting lobe side which is called the toe.