I would certainly keep a very young colt with the group to learn basic manners but once he gets to about 8-9 months you’d want to shift him out from the mares and fillies. To be left in a group of colts or geldings should be fine, but you need to keep an eye out for his development and move him out on his own should problems start up in the group.
Pasture breeding is not a good idea imo, particularly with a young inexperianced colt. You could well end up with an injured boy. An older, more experianced stallion knows better when to make advances and when to leave the girls alone! He will also have a better understanding of herd manners, behaviours etc, that a youngin doesn’t yet understand.
Breeding should be kept to one place and proceedure so that the colt learns when he is there he can do the studly thing but anywhere else he must be a gentleman. When a colt or stallion is schooled properly, you should be able to take him out in company or to shows without him having a performance about the mares.
You shouldn’t have problems gelding a colt that has served, I know of many young stallions that have spent a couple of seasons in the breeding shed and then gone on to be awesome ridden geldings. Again, it comes back to proper schooling.