I wear a tight-ish vest. Practically every fitted vest has a zipper on the pocket, or a tight, high chest pocket to stash your phone safely. (My favorites are the long-waisted Kerrits vests in summer, or the men’s charcoal wool vests with the four little pockets sewn on front in winter.) I do carry it religiously, and let me tell you what a PLEASURE a giant smartphone is when compared to a huge knocker radio on your hip! Living way out in the wilds of MT, that giant stupid radio was the only way to ensure help in an emergency for years, but MAN did that thing hurt you when you hit the ground (or trotted for an extended period, and my saddle is permanently scarred from it bouncing along…). I carry that smartphone in the space along my ribs below my boobs, and I’ve never had trouble with it yet. I suppose this wouldn’t work if you were gifted with a hefty rack though 
-Phone holder that touch tapes around my upper calf. I bought it on Amazon and use it for running as well. It’s big enough for my phone and ID without being irritating to wear. Actually, I sometimes forget I am wearing it at all. I can reach my cell easily without messing in my pockets and I feel like I am unlikely to damage it if I take a tumble. The holder also stops the irritating pocket jingle/bounce if you are traveling at speed.
When I wear my Kerrits flex tights, my iPhone fits nicely in the pocket on the right hip. When I wear other breeches, I usually stick it in the top of my half chap. There is just enough room there to hold it securely so it doesn’t slide down or fall out. I can hear the music or distance recording apps from there.
If it’s drizzling out I put it in a waterproof arm band holder, but that’s not what I prefer bc those get hot and itchy in the summer as you sweat.
