Where there's smoke, there's fire, and if you feel uncomfortable with his actions with this woman, there's a dang good reason. I think you're being played like a DVD, and your husband is running rough-shod over your feeling and boundaries. You really need to sit him down and tell him exactly how serious this is. And, you need to stop seeing this couple, or any other couple for that matter, until you can agree how you'll procede with swinging. Do not deny your gut--it typically is correct.
Now, this is good advice:
Trust me, they're not the only couple in Pensacola....not by a long shot!
Pepper