Hmmmmm… it might depend on what you are calling pea gravel and what it is mixed with.
It would also depend on the depth of it. For example, I use pea gravel for all the walkways on my farm. It packs down to a perfectly even surface that even tender footed barefoot horses are comfortable on. Stays dry and clean and is hard to tear up. I have a client who has much deeper pea gravel in her very large loafing shed and surrounding area… deep enough for their feet to dig into a bit. Her horses have excellent feet and they did not before she moved there. Of course there could be many factors there, but I think staying clean/dry is probably a player.
I can see if you mixed pea gravel (which here is very small angular pieces of grey stone, not round at all) and rubber and sand, it would probably make excellent footing as it would not ball up at all, would maintain its flatness/shape of the arena and probably would help control dust. My guess is not only would you want to drag it, but you might want to roll it on occassion, too. Something to ride in everyday, no. But all-weather consistent footing? Probably would work. In fact, now that I think about it… when I build my very small dressage arena (to be used only in foul weather) maybe I will try it and report back!