Well sure, if you have enough property you can shoot at any time. Obviously the limiting factor is safety and distance. If you suspect that they are shooting onto your property, then you need to report them.
Bird shot doesn’t travel that far - so depending on the size of their property (and yours) it would be hard to “accidentally” shoot into your pasture. How big is their property? I have a clay pigeon thrower in my 8 acre field - no way I can even remotely reach the other side of that with a shotgun. But if they don’t have that much property, they may know that they are shooting deliberately onto your property.
Seems like some well-placed cameras are going to be your best bet with regard to the neighbors and the sheriff.