Susan here-- Well, you replaced sexual intimacy with the thrill of sexual acts. The latter is more thrilling in the short term, yet if that's all you get, it's like candy, too much of a good thing and you've forgotten the importance of dining on good food. Take a break from swinging and reconnect with your sensual selves. Touching, sensualness with a buildup to a great sexual encounter and it's just you two.
It has effected you because you let it effect you. It snuck up on you like an addictive drug. Sex can be VERY addictive. Good luck reconnecting with what is truly important.