Getting back to the original topic, swinging is fun, but relationships involve complicated emotions, especially for women. The problem is that you're mixing the fun of alternative sex with the emotions of a relationship. From reading your post, it appears your wife is having a hard time distinguishing between her relationship with you and sex with others.
You talk about her "boyfriends" and your "girlfriends", and mention that you had a few "girlfriends" at a time when your wife is clearly having trouble with the whole extramarital sex thing. There's the problem. Women fall in love with boyfriends, especially when they feel their relationship is being threatened by a girlfriend ... and that's causing a conflict in her own head. And no matter how much you want to insist that your "Mrs" is different from the rest of the human race, this emotion is hard-wired into women's brains ... including hers.
So what do you do? Swinging is for sex - period. You need to establish a feeling in your relationship that the others are sex toys, not "boyfriends" and "girlfriends". These objects are dildos and pussies, nothing more. As soon as you introduce the possibility that they are people with personalities, feelings, and emotions that you care about, you WILL cause a problem. Hopefully you haven't used terms of affection about your "girlfriends" or comparisons about how her pussy is "tight", or she give the "best blow jobs", etc. And don't tell me you treat your "girlfriends" as impersonal sex toys because you already said it always ends in "drama". That tells me they think they are in a "relationship".
Also - swinging is best done WITH your wife, not off in private with someone else while she's at home feeling so unsexy that she hasn't had sex in 9 months. And don't encourage her to go off in private with a boyfriend. Women have an instinctive need to form a pair-bond. And when you encourage a relationship with a "boyfriend", so that you feel good about having a "girlfriend", they instinctively feel that you're trying to weaken that all important pair-bond.
And one last thing - stop calling your wife "The Mrs". That's an antiquated term popular when women were considered property. Call her by her name or, if you wish to remain anonymous, use a fake name. But stop depersonalizing her while you form friendships with "girlfriends".