Breaking the rules once can (but not always is) a mistake...breaking them a second time is an 'on purpose'. He is being disrespectful to you...and you are allowing it to happen. Neither of you are ready for this lifestyle and should take a break until your relationship is much stronger and in a better place. I know you love him, but I remember an old story about a man having to break up with tequila. See, he loved tequila with all of his heart, but the tequila didn't love him back and treated him poorly. Not saying that this is what you need to do, just saying sometimes you loving someone isn't enough. Until he loves you and RESPECTS you enough to put you ahead of everything else, you both need to stop.
When my SO (or anyone for that matter) leaves the room for any reason when we were playing, we all stop until she returns. We always play together meaning everyone has to be there. If she were to leave without saying something, everything would immediately stop and I would want to know what was going on. That's the way this has to work. He needs to be told this (point him to this board). You should come first always. He seems to know that you are having issues, but he doesn't want to talk about them is not good. You need to have trust and communication on an epic scale. Not wanting to talk is a lack of trust and/or sign of poor communication.
Finally, while some couples have been successful with playing with friends, we strongly advise against it since there is already a connection of some sort between friends, but also there is the risk of everyone in your group of friends finding out if things do go south. Bottom line, you need to stop until you both can work on the relationship and he is better at understanding what you want and need. Not for one talk or a week or two, but until the relationship is in a much stronger and better place. Good luck and let us know how things progress.