The point may be that the lifestyle helped the OP even though they didn't get into it. How? By being the instigator of communication. Their relationship was a classic one of not communicating and getting along by agreeing to ignore some deep and hurtful issues. Something about the possibility of swinging inspired communication and both learned that both had been hurtful. As with most relationship issues, both play a part in the negativity.
Communication is what makes possible real relationships as well as playing and sharing sexually together.