Your CPT book, Appendix A-Modifiers, would be one place to start. The definition of the modifier would tell you some of the answer. Appendix E has info on modifier 51 exempt, etc.
Payers have guidelines also, example:
https://www.uhcprovider.com/content...mm-reimbursement/COMM-Add-On-Codes-Policy.pdf
Most Encoder type programs have information on which modifiers are applicable to codes.
There could be some payers that want certain modifiers appended even when it doesn't make sense or is not required by CMS or others. You'll probably hear multiple different answers depending. Some say "never append" but like I said, some payers want them some don't. In some cases they have to be appended to add-on codes.
You wouldn't use 51 on add-on codes because they aren't subject to multiple procedure rules. (See CPT introduction + add-on codes).