This is a 2 cent version--I'd consider seeking some familial counseling--maybe with someone new.
I feel for you... everyone takes things differently. I know my parents swung with at least one other couple, and my mom had a lot of extra-marital sex... I never walked in on them, but I could have. They didn't particularly try to hide it.
My friend's parents swung downstairs after all the kids went upstairs to bed. I was spending the night, and we snuck down and watched them, and crept back up to bed without getting caught. (We didn't actually see them fucking, but they were kissing and fondling with the neighbor couple.
Another friend I had had parents who were outta-the-closet swingers; my friend and his two sisters were well aware of what was happening.
Our parents, at the time, were all WWII vets, had seen a lot, done a lot, and fucked a lot while overseas. They were made of a little bit harder stuff than our generation. While they tried to be discreet--for the kid's sakes, they really didn't give a flying frick to whether or not we knew their sex lives or whether or not we approved of it.
We were the kids; they provided for us; loved us the best they could, and, as my mom used to say, hoped we stayed out of jail til we were 21.
To be honest, I don't know that these parental swinging or non-marital events had that much drastic affect on any of us. I believe we all just kind of took it as normal because that's what we were used to. And our parents expected us to act normally and civilized.
As far as you guys not being someone he could respect??? That sounds a lot more like him being a 16 year old than anything else. Explain to him you could give him a million different things about yourself that he couldn't respect.
Have you ever told him why you swing? How it helps you enjoy each other more--maybe the variety of menu from day-to-day analogy. (Maybe, or maybe not, play the immature/inexperienced card--you don't have enough life experience. What if someday you woke up and realized we were right??)
All of this is to say, I do not like the fact that your son threatens you.
It sounds like he's deeply unhappy. Explain that he, himself, has done, and will do things that make you unhappy and/or mad, but you still love him regardless.
Also, point out that you, too, are human. You've done things in the past that made him mad or unhappy, and he might as well face it, he can expect that in the future. But that doesn't mean you love him any less. That sometimes you just have to bite the bullet, and move on instead of making an issue about it.
Explain that your marriage is fine and that he needs to accept that. End of the story.
Then, maybe ignore him a little--as in giving him a little space to work out what the reality is. As my mom would have said, "Let him put that in his pipe and smoke it."
If he continues to threaten you, just raise one eyebrow and say, "Some cultures would regard you to be a man, now, son. So you just go on and do what you gotta do." If he starts telling people, so be it. (Scandal lasts from 2 to 6 months; after that, though the fact remains the same, everyone just kind of gets bored gossiping over it and just moves on.) Bottom line, if your son did tell someone, you really could weather it. Not fun, but possible. (Not to minimize it, but if you watch the growth of SLS in your area, swinging really is coming out of the closet in an exponential way.)
Bottom line, let him know you love him and never let him see you flinch. And again, it sounds far more like 16 year old behavior than "caught-my-parent's-swinging" behavior.
Also, in case I totally missed the boat, and did nothing but lay on a crock on your painful situation, (which I am quite capable of doing) you might check out the familial counseling route).
Best of luck and peace