My wife felt about the same way you seem to when we were in the lifestyle years ago. She never wanted to see me even flirting with the other wives, much less having sex.
Now I loved watching her with the other men and never had any negative feelings. She always seemed (and she's not that good an actor so I think it was real) to really enjoy herself during but always felt badly the day after.
We even tried separate room swapping but she still had difficulty handling the afters.
So, we decided that this was just one activity that one of us loved that the other could't handle.
In more recent years she did decide to give it, or at least a mitigated version of swinging, another try and we found a couple that we both really liked the wife. During the flirting process my wife didn't have the same bad feelings about my flirting and even exchanging provacotive emails and photos with the lady and she seemed to enjoy doing the same with the husband. But, when we met in person my wife didn't like the husband at all. I don't blame her because I didn't either. If he'd been as nice and considerate as his wife I think my wife and I would have found what we'd been searching for. Too bad because the wife was exactly what my wife was confortable with and I found her sexy as can be.
OK, now to where I'm going..., I think that it might help if you like, trust, and befriend the female your husband will be with beforehand. And, always have yourself occupied with a man you find hot while your husband is with that lady.