Going to go a different direction here than some of the other posters but trying to be 'fair' in swinging is one of the fastest ways to kill the experience. Every experience we have, the wife get's to fuck 3 people (the wife, the other husband and me) I get the whole 'jipped' feeling you mention but this is about an experience the two of you are having together and if you want fairness, become bi and suck the other guys cock and vice versa and all settled.
If your like me and many of the other men who are totally grossed out at the thought, then you really don't want fairness.
Be happy in your wife's enjoyment because it's not about tit-for-tat sex. We run into so many couples where it's like, they feel they have to match everything the other couple is doing, I feel like we are having sex in front of a fucking mirror...like a kid that keeps repeating everything you do...you just want to say quit-it!! If you can go 3 times in the time it takes me to go once, as long as your partner is into it then I say go for it! Why are we all so wrapped up on fairness and not on the enjoyment?
You are a really nice guy. I can tell by your posts you are pretty sincere and your wife is awesome too in her desire to want to make sure you are just as happy as she is but I really believe this is mostly in your head. It's easy to feel that the other woman is into your wife more than you but I say this...most swingers won't take one for the team and if they are playing with you then the wife digs you! Don't let your mind screw with your head and the last thing I would say is why the fuck are you and the other husband just sitting around watching? Get on a boobie and participate!!!!