Concur with Rackir about the sense of "fair". Drop the issue from a fairness perspective, because swinging isn't about being fair. What works for you is what works for you.
I recall a poll here a while back, well participated in, that showed the biggest fantasy of male participants of this board wasn't being with two women, it was being in an MFM. I think that surprises most people. Me personally, I thoroughly enjoy watching my wife have sex with another man, and participating in that. She absolutely loves having sex and giving head at the same time, which is of course impossible without another man involved. As Rackir said, how much she enjoys it is a seriously major turn on for me, and I'm very happy to oblige her and make that fantasy a reality.
Some men, like some women, can be insecure. There can be any number of reasons why your partner might feel threatened by another man having sex with you. It would be good to ensure your partner that this is sexual, and not emotional, that you are utterly devoted to him, love him, will not leave him over this, etc. You might also want to make it clear to him that you want him directly involved in the MFM, not as a spectator (if you do). My wife doesn't mind me watching, but she prefers me being involved. After all, she can have one-on-one sex with me. She doesn't need the other guy for that.
I agree with Rackir; make your partner aware, through various means, how much the fantasy turns you on.