Asorry your having such a hard time. I'm afraid I can't really suggest anything other than you need to sit down and talk properly with no kids there. For him to say you know where the door is is hurtful and doesn't show much love. As hard as it is you need to find out why he is behaving like this and expecting you to stay. I'm not suggesting packing your bags , most issues can be worked through. Has he got worse since you Lo arrived?
I hope you have woken and do feel a bit better this morning x x x