There's no easy way around that one unfortunately. You need to tell him straight up how you feel. Not too harsh with him, but make him understand. Yes, it will be awkward for a little while. But not forever. But it's the only way.