Supersquish and TriTread are right, you can take your leave as a normal employed person any time you like. Is there an HR department at your company? If so, it's definitely worth having a quite chat about this with them - they'll probably have a quiet chat with your boss (attitude adjustment!!). Otherwise you could ask them to be present in a meeting with your boss while you tell him why you feel that he's being unfair. That ought to stop him in his tracks and prevent the problem escalating.
If there isn't an HR department, you'll want to ask him for a meeting where you tell him that you think he's being unfair. Take notes, email him the notes after the meeting.
HTH, Jules