Ah, this is the other thought I had that I forgot and expands on my #3. Many people are in it for themselves. Even the relationship/marriage. Many people marry someone for what they represent to them, not who they really are. Is she the the hottest and make his friends envious? Does he have money and can buy her the biggest house and car and make her friends envious? They really don't care about their spouse beyond how they make them look to others. It's selfish. So is cheating.
However swinging is one of the most unselfish things you can ever do for your spouse. Swinging is about you because it is about them. It's getting joy from their joy. You've had a great time because your spouse has had a great time.
Cheating is about none of that. It's all about the cheater's happiness without any regard whatsoever for their spouse's happiness.