I fully agree. To answer the devil's advocate, however, the fact is that mistakes like these will inevitably happen. The important part is how you deal with them. Here is the same conversation, albeit a bit truncated, said two different ways with opposite outcomes.
Her: Honey, you know tonight when you went down on that girl? Did you realize that you hadn't asked me if you could do that, and that broke one of our rules because you didn't ask me?
Him: I hadn't realized that until now. I was so in the moment, and she nearly pushed my head down into her crotch. It was a bit of a surprise, and also so quick that I wasn't able to think quite that fast. I'm sorry that I didn't ask you. How did you feel about it in the moment?
Her: Well, to be honest, it was quite hot to watch, but it left me a bit perturbed afterwards because you didn't follow our rule.
Him: Okay, I get that, and will have it in mind a bit so that I won't get caught off guard quite so easily. Did you want to keep the rule intact, about asking first or is that something that you're past now? You said it was kind of hot, so I wanted to check in on how you feel about it.
Her: No, I think we'll keep the rule intact for now, only because I like to stay informed of what's going on.
Him: Cool. Lets have some hot sex while fantasizing about that girl forcing me to go down on her, where it's safe and you can enjoy the sight without any broken rules.
Her: I like that idea, let's go fuck!
------------
V2:
Her: Why did you break our rules??!!! You know that it hurts me so bad when you don't ask if you can go down on the other girl.
Him: Why are you so jealous? Can't I have any fun at all? It makes me feel like I have a collar around my neck when you say stuff like that! I barely had time to think before she shoved my face in her pussy!
Her: I don't care if you didn't have time to think. You broke a rule and now I can't trust you anymore.
Him: You're being unreasonable, I didn't like that rule anyway. Sometimes you just have to go with the flow, baby. You know you liked it, I could see it in your eyes. You're just ashamed that you liked it.
Her: How can you say that? I'm mad at you and you don't care! We're never swinging again, and you're not getting anything from me for a month, not even a handy!
-------------
As you can see, conversation 1 is the cool head, forgive, re-negotiate, empathetic, constructive version. V2 is the accusatory, unforgiving, refusing to let go, and most importantly, destructive kind of conversation. As far as I'm concerned, this constructive, forgiving dynamic is more important than the playtime itself. It's the only way that my wife and I personally could survive the specific kinds of stresses that swinging puts on our relationship. Version 2 only happens when a person is being selfish, and actively clutches to and feeds the jealousy and hurt. Quit hugging the thorn bush, and you'll quit getting hurt.
There's a catch though. BOTH partners have to agree to act constructively, it simply doesn't work otherwise. The above conversations are not gender-specific either. All people have to play by the same rules.
We're glad you had both good and bad experiences in your update. Robin Williams quoted a director he worked with one time, "When you make a mistake, go with it because it's a Buddhist gift." I always try to look at mistakes as opportunities to learn. That's kind of the basis of my comment about being confident that your partner isn't intentionally trying to hurt you. Also, bad experiences teach us the warning signs and red flags about which couples to stay away from. It really does train the spidey-sense.
Happy hunting EBC!