I'm with padoc on this. I'm seeing red flags popping up all over the place on this one. We're missing some details here, but it sure as heck sounds like your wife is cheating on you. Even if you're in the lifestyle, you can still cheat. She is. Further, your friend J is saying he won't tell his wife and the other guys won't tell theirs. Guess what? That means they're cheating on their wives too. If they're willing to lie to their own wives, who won't they lie to? This guy isn't your friend, and neither are the other guys. They're cheating on their wives, your wife is cheating on you, and has kept you in the dark.
You don't want to shut down all facets? Well, you've got two choices here. Either you allow yourself to continue to be lied to and cheated on, and 'enjoy' this somehow, or you confront your wife and maybe, just maybe get your marriage back on an even keel where you are open and honest with each other. Swinging REQUIRES great, honest communication. You don't have that. Nevermind that this group of 8 couples is seriously dysfunctional and not to be trusted. If ever you do get back to swinging, it sure as hell better not be with this group. They will just backstab you again.
This isn't just a mess. It's far, far worse than that. Good luck.