No. For starters, no E/M is "always" anything.
E/M coding (until 2021) is based on history, exam & medical decision making OR time if counseling/coordination of care is > 50% of total time.
A follow up visit could be anywhere from 99211-99215. If you wanted to say it's often 99213, that would be accurate.
The items you describe, worsening vs improving, ordering tests, medication change, etc all fall into the level of medical decision making, which is only 1 of the components to determine E/M level.