In terms of the last couple, they were arrogant jerks and you are well rid of them. Friends (of any persuasion) should not browbeat or manipulate you. They had a perfect right to ask for what they wanted, not right at all to blow up when they didn't get it.
As for your method of swinging...didn't you get the handbook with the three correct methods clearly described and illustrated with line drawings? Seriously, don't let anyone tell you that you aren't doing it 'correctly'.
As for the level of drama or number of marriages breaking up, remember that many marriages end in divorce without swinging, so it might not be the swinging. Or it might be that people are getting into swinging as a last ditch effort to save a marriage (which is about as effective as having a baby for that reason).
And finally, your actual question! For me (and I think I speak for Mr. Ivory too) is that swinging actually isn't a lifestyle. It's a hobby. That is, it's a thing we do for fun, to enhance our lives, but not a central defining characteristic. And as with our other hobbies, we have 'hobby friends' very very few of whom make it over to real friends. My husband, for example, has been seeing the same set of people every other weekend for years and yet would never think of inviting them over to dinner.
This isn't a bad thing. It's a realization that a) getting 4 people to like each other and spend time together is hard b) we all have very very limited time and so we are protective of how we spend it and with whom c) people you get along with for a short period of time or with whom you share one interest may not be so likable in general.
I think the problem we all run into is that this seems so much like dating we think there's something wrong if we don't go the next step and establish a relationship. Not true!