Yes, E&M should be billed by a primary care physician who is treating medical problems that just happen to be occurring during the postop period of a surgery. No modifier is needed as your physician is not the surgeon nor part of the surgeon's group.
If the situation is (this does sometimes happen mostly in rural areas where doctors are not plentiful) that the surgeon does the surgery, and then transfers the care to another physician, then the surgeon should be billing the surgery with -54 and the physician doing postop care bills the same surgery code with -55. It does not seem to be the case here, but I want to cover all bases.
I have some concern about the wording "seeing this patient for a follow up after surgery, fatigue and refill." What type of follow up after surgery? Did the patient not follow up with the surgeon? Why did this patient need follow up after surgery by your physician? I'm thinking it's likely just poorly worded, leading to the possible confusion. The patient is likely actually following up on hypertension or diabetes or arthritis, etc.
Here is the full CMS global surgery booklet referenced above by Dorinda.
https://www.cms.gov/outreach-and-ed...oducts/downloads/globallsurgery-icn907166.pdf
It specifies, among other things "The global surgical package, also called global surgery, includes all the necessary services normally furnished by a surgeon before, during, and after a procedure. Medicare payment for a surgical procedure includes the pre-operative, intra-operative, and post-operative services routinely performed by the surgeon or by members of the same group with the same specialty. Physicians in the same group practice who are in the same specialty must bill and be paid as though they were a single physician."
Since your physician is not the surgeon/surgeon's group, the global surgical package does not apply (unless transfer of care, see page 8).