If that crap made it to the float bowl (broken up finer) it can slowly plug the jets due to being drawn in. Once the engine dies, it drops off (but is still in suspension) and the cycle repeats. When you have not run it for a while, it falls to the bottom of the bowl, and takes longer to get stirred up again.
Clean it end to end - tank, lines, and carb . . . Oh, and old disintegrating fuel line can do this as well