I've played "together," as a foursome, with hubby, and we've played separately (under the same roof, in different locations, and totally solo). I'm fine with them all. I am most relaxed, however, when I'm with only one partner in a room. Why? It has nothing to do with not wanting hubby around. It has everything to do with being able to focus on who I'm with and not worry about or, in the alternative, be distracted by hubby and/or his partner. That's me. I don't swing to be an exhibitionist or voyeur, even though those roles are sometimes fun. I do it to be able to enjoy the sexual experience with another. Maybe your wife feels the same. By the same token, if my hubby were uncomfortable with the thought, I wouldn't do it.
Different comfort levels can be managed. Just consider carefully why you feel the way you do. Pushing through insecurities can be a good thing.