This is a perennial concern. Here is the situation.
1. What happens among consenting adults is a lot less scandalous than the usual reported behaviors: cheating, sex for money, etc.
2. Taboo becomes mainstream over time. For example, going to a nude beach or resort used to raise eyebrows. Not so much any more. People are increasingly open about various aspects of sexuality.
3. A bit of discretion goes a long way.
So there are really two issues.
1. You might run into someone you know at a swing venue. Or someone who knows someone you know. They are there at a Swing Club or event for the same reason. There's no percentage in 'outing' you. Just have a plan--quietly acknowledge their presence (so they know that you have spotted them), exchange a simple "good evening" and move on. Pretend they are not there.
2. You find a couple you like, you end up seeing each other and you are at a restaurant or some other public place and someone you know (co-worker, neighbor, pastor) comes up to you and says" hi". Just introduce your new friends. If asked how you met, give the standard response "Oh, John and Mary are friends of friends."
Bottom line, be respectful, be discreet. People pay you little mind. It's especially true if you're a little older and fairly conservative in dress and manner. Candidly, your biggest risk is outing yourselves. People will ask why you're so bouncy and happy on Monday when they have had an ordinary weekend. Just smile and respond "quality time with my spouse!".
Most swingers are not openly swingers for the simple reason that potential playmates might be concerned about being outed and steer clear. Once and again, discretion matters.