I'd be shocked if it was permitted to fill the swale with gravel. That would look bad.
Once grass starts growing, it will be fine. Just have to avoid the bottom when wet, or push mow hat part only.
The ditch (swale) would be "Right of Way" for underground water / gas etc & not actually owned by homeowner.
Can't make changes like that without written permission. But homeowner is to care for it.