Part of the reason swingers can be human sex toys for each other without attachments is that we are already attached.
I already have love, I am not looking for love, I am happy with who I am with. It would take a VERY remarkable woman to somehow make me think 'if only her instead of Mrs. Chicup'. Personally I've never come close to meeting one and I only list it as a possibility because the world is big.
Singles are different, singles don't have someone, singles by default are often looking for someone even if they don't THINK they are looking for someone. Your drama potential is always higher with singles, and this is part of the reason.
Always though remember, they are people with their own wants and desires, you need to treat your partners with respect, and at the same time be wary of their motivations. Really basic 'human' stuff.