As others said, pea gravel is not the best option. Ours also migrated, got flung elsewhere as horses took off running. Since we had put the pea gravel along the fence over drain tile, it got flung onto ME, and it hurts when it hits! I had to get rubber mats to cover the pea gravel along the fence, keep it from being flung or filling the horse hooves to go other places.
Our clay dirt eats fill, disappears in a couple years without the fabric layer to hold fill apart from the clay.
What we use now is crushed concrete which is cheaper, more permanent than gravel or pea stone. It makes excellent “pads” horses like standing on, easily cleaned off, drains well. The chunks kind of knit together with use, get very solid for standing or driving over. Chunks can be any size you want, big or small. Ours vary from a couple inches down to dust in the same load. You might ignore the field mud, make some big, solid pads closer to your gates that horses will use when things are wet, muddy elsewhere. Mine like standing on the crushed cement driveway thru their barnyard to the field gates. Keeps them up out of the mud. I use the tractor bucket to clean off manure with a light scraping. Not enough daytime to be out hand picking it clean with multiple horses!