That is far too common an occurrence and it just tars the whole group. For good single guys, yes, it sucks to be lumped in with the worst of the group, but that's how it is. Just accept that going into it you are down at the lowest common denominator but then show them through your words and actions that in fact you are just the opposite. Do that and you will have a lot of success and make some great friends. If you can't be bothered to do that or don't think you should have to, then you just reinforced the stereotype and made it that much harder on yourself and every other single guy.
Talk to just about any swinger couple that interacts with single guys and they can tell you more bad single guy stories than they can good single guy stories. You can't say the same about other couples. Sure, there are plenty you didn't hit it off with or whatever, and couples do sometimes show the same bad behaviors. Odds are a lot less with couples though, and that's just a fact. Couples get into swinging for fun, and having to police the single male ranks is not fun. So, many choose to just avoid it.