Both the crystalline structure and the particle size affect the toxicity of silicon dioxide (sand). Large particulates are trapped in the upper respiratory tract (if you blow your nose when it’s been dusty, you can see how efficient a capture mechanism your nasopharyngeal area has. All that snot you blow out has done a good job of capturing the bigger particulates - and a really cool mechanism, the mucociliary elevator, moves that dirty snot up to your nose and mouth, where you either swallow it or snort it out. That, obviously, is not the technical explanation for how it works).
Very small particulates, those less than about 5 microns in size, do a very good job of getting past that defense mechanism. They can penetrate deep into your alveoli, where the gas exchange occurs in your lungs. Once there, they can scar the lungs (silicosis) - and cause lung cancer.
Worksafe BC has an extremely good video on what happens when you breathe respirable crystalline silica [RCS].
When the sand in your arena (whether it contains silica or not) breaks down, it will form respirable particulates. If it was pure glass, the primary particulate would be amorphous silica. That has not been associated with lung scarring (fibrosis) or cancer. I wouldn’t consider it harmless, though. It’s much less researched than RCS, and there is, overall, probably much less exposure to the respirable form than you find with RCS.
And one more comment: as NoSuchPerson stated, there are different types of sand. Most often, when we say sand, we’re talking about quartz. Gypsum, for instance, isn’t quartz or silica sand - but it can have enough quartz in it as a contaminant to create overexposure to RCS.