We want him gone, but we don't want to go to court...?

Hate to say it but you can't get him gone without opening a can of worms especially under those accusations. Good luck.