It's possible that it could 'throw a red flag', but if you don't have documentation to support the claims, then I don't see that you have much choice in the matter. Once the problem has been identified and you are aware that an overpayment has been made to your clinic, you're under a legal obligation to get those claims voided and payments refunded to the payers and the clinic could face penalties if this is delayed.
If the volumes are high, you might want to speak with a payer rep or someone in the claims department who can help coordinate the voiding and refunding/recouping of the claims as a special project to make it easier, rather than trying to do this claim-by-claim. For government payers, you'll want to make this a top priority as you're under a time limit that starts on the date that the overpayments were identified - save your commercial payers for last. Your compliance department, if you have one, should be involved in this and be giving you direction. Your management may also want to take some legal action against the provider who left without completing their work if the financial impact is significant.
I'm not sure what else to suggest here - it's sounds like you have a mess to clean up and there's not really a good way to get around that. My recommendation, once this is cleaned up, would be to put a process in place to prevent this from happening again. Services that haven't been documented and signed yet should never be billed. If there is a red flag triggered, claims for undocumented services is exactly what they'd be looking for and your clinic needs to make sure that is stopped.