Hmm... there is a lot to chew on here. The previous poster is right in that every couple should do what makes them BOTH feel comfortable. I cannot possibly know what your relationship is like, but I cannot help but wonder why your wife is more comfortable "swinging" without you. There could be a problem lurking there, especially given that she said something hurtful to you when you tried to get her to hold back in the name of fairness.
I wonder why you think swinging alone would help you in your relationships with other women. As a swinging wife, I like a man much better when I see that his relationship with his wife is solid and loving. I feel much more comfortable developing a friendship with a playmate who is in a strong marriage that I can see. And not least, like many swinging wives, men swinging without partners are more or less off-limits to me, because Mr. Fuse and I swing together. Those are two big strikes against you in your situation. Can you describe what the advantages are?
And about whether you are being "fair"... well... as I mentioned above, I personally think that is not quite the right question, because "counting" can get really technical. Mr. Fuse and I have had many a discussion about this subject, especially when we first started swinging. Women get many more chances to play than men, in general, and Mr. Fuse didn't want me to leave him behind. It would have been very easy for me to do that, if he hadn't been careful about it. And at times I resented not being able to do things I wanted to do. But for us, it is better to stay together and make swinging a joint activity except for the occasional time.
In my opinion, the answer is: If you feel like she is taking advantage of the situation or trying to do too much while you are not getting to do enough, then you are right for asking her to put the brakes on. From the discomfort you expressed, it sounds like you are past that threshold.