What you're experiencing is common. You're at the crossroads and have to decide if you're going to accept it or not. Your problem is that you seem to give a shit how the other wife feels about you. Don't be a pussy and make it clear that if you're going to let the dude have the honor of fucking your wife, you're going to fuck his, it's a package deal. If his wife finds you hideous and says no, just walk away. The biggest newbie mistake is to let swinging get personal or developing "feelings" for playmates. Swinging is NSA sex on your terms, nothing more, nothing less.