Strip away the details, and the question is about values--yours and theirs. As EdisonCarter points out, the swinger value system is orthogonal to that of the Hypocrites and the Puritans.
The Hypocrites are fine with cheating as long as no one gets caught or, if they do, there is a juicy divorce that can be gossiped about.
The Puritans are at best lukewarm on sexual activity even within marriage.
The idea that couples can and do enjoy sex with others purely for pleasure, and moreover do so in a way that does not devalue either partner is often difficult for them to comprehend. (It was difficult for most to comprehend before we started down this path, wasn't it?)
Telling them that your values differ from theirs "lets the genie out of the bottle". Once out, he doesn't go back inside. And as many posters point out, friendships can quickly be lost.
Truly vanilla folks do not want to know that they know lifestylers.