I had this happen to me. You just need to let him go and let him cry or help him through his anger or get some help for him.
He might just have a clinging issue. If he never wants to leave it's just the things that make him angry when you're around him which causes him to go to you. I recommend you have some alone time.