Nothing, because there is no difference. Most single guys like to fuck. If they're horny enough, they'll fuck anything. Fat women. Old women. Other guys wives. Other guys AND their wives. Guys dressed up like other guys wives. Single males will fuck anything and anybody, and those who hang around swingers clubs and websites aren't "swingers," they're opportunists. Nothing really, but if a single guy were to say "Hey, would any of you like to fuck my girlfriend? I'll just stand here and watch" he might have a little more credibility as a "swinger." NOW we're getting to the heart of the matter, the "litmus test" that defines whether you're a swinger or not. I think you answered your own question
"Single male swingers" do NOT have "monogamous relationships" with women who are not in the lifestyle.
This lifestyle is obviously a very important part of your life. Why would you want to be in a "monogamous relationship" with somebody who doesn't share it with you? Is it like Julie said, that you "respect your woman too much" to bring her around people like us?
Personally, and this is just my own, PERSONAL opinion, I think ALL swingers start out as couples. I believe it's the only way you can truly know all the emotions and issues that are involved in this. Sometimes, when a person becomes single either through death or divorce, they can continue in the lifestyle as a "single swinger" if they choose. We know 2 males and 1 female who did just that for a while, but now they've all dropped out until they find new partners.
I don't think you can go from being a "single male" to a "single swinging male" without passing "Married swinging couple"