You've learned an important lesson - not all the fish in the sea make for a good fish fry.
When you aren't feeling a connection, it's best to say so up front, be honest with yourself, with everyone. In this case, you should have said, "no, not tonight," and abandoned play.
As far as just playing with the girl and not the husband, turn it around. How would your husband feel if the other couple wanted to play with you, but they didn't want anything to do with your husband? I hope you'd agree that nothing should happen in that scenario.
Yeah, these people might make good friends, but it's likely they'll never make good swapping partners. Throw 'em back, go fishing again.