My BS meter is still pegging 9.9 on this one.
Reality; his jealous feelings are making you feel rotten. What's more important? Her feeling good or you feeling good?
Bottom line; you're not attracted to either person in this couple. They aren't the one couple you are going to be one couple couple with, period. Therefore, regardless of your husband's behavior, this is over. It doesn't matter what he says, does, thinks, feels, or behaves. It's over. Done. Kaput.
Look, you don't NEED this couple. I just did a search on SLS using 32514 as the zip code. I restricted the search to age 21 to 39, active within the last two weeks on SLS, and within 20 miles of 32514. The search came back with 323 matches. I'm very sure there's another couple out there that is willing to be a one couple couple, willing to go at whatever the slowest pace is, and not have the problems of emotional attachments beyond friendship getting in the way.
Everybody here is telling you the same thing. DROP this couple. NOW. Get more assertive with your husband and insist he cut it off cold turkey, and that the two of you need to step back and make darn certain what the rules are and they will NEVER be broken again. He should count his lucky stars he has a wife who is willing and happy to swing. If he thinks the only possible person he could ever have swinging sex with is this other woman, he's deluding himself, being dishonest with himself and probably with you. Well, he's already been dishonest with you anyways.
For what it's worth, my wife and I have a rule of no cuddling when not engaged in play with a playmate. It's intimate, reserved for us. The closest I ever got to actually cuddling another playmate was holding her, and lightly playing with her while both of us were intentionally watching our spouses enjoy each other. That lasted all of a few minutes before we had to return to our own activities I get so turned on by watching my wife with someone else... But anyways, cuddling is right out. Your husband cuddling with the other woman when he didn't expect you to come back in so soon was flat, utterly, completely wrong. He was cheating on you, and trying to cover it up by saying it was a joke was a blatant lie. That he's acting jealous to make this other woman feel good strikes me similarly as a lie.
I STRONGLY agree with IAPR here. Fool you once, shame on him. Fool you twice, shame on you. You know what the score is. You know what the deal is. Don't play the fool. Put an end to this immediately. If your husband can't cut it off with someone you are not comfortable with, then he has no business swinging at all. That's another rule my wife and I have; if either of us says it has to stop with X person/couple, it stops with X person/couple. Period. No trying to convince, no trying to pressure, no trying to get one more opportunity to have sex with X person/couple, nothing. The most we might do is sit down and talk about why it needs to stop, to help understand what is happening to prevent it from happening in the future with new people. No means no. It's not negotiable.
If this affair is allowed to continue it will only get worse. Your radar is strongly telling you there are emotions developing here, and many people here agree with that. Once he starts having sex with her (and worse when it's on a regular basis) this is going to become far, far worse.
Mind, we've only been talking about you and your husband. The other woman certainly knows what is going on and is being utterly disrespectful of you and your marriage. That's a huge red flag in and of itself. If she can't respect you and your marriage, she has no business fucking your husband. She's intentionally screwing with your marriage. For my wife and I, if another couple or single is not willing to respect our marriage, there's no way in heck they get to play with either of us. Our marriage comes first, period. We're in swinging because we have an amazing marriage. We're not in swinging to screw that up. We are in no way fearful of anyone screwing our marriage up, but we are vigilant to keep away people who do not respect our marriage.
STOP this affair. Now. I can't over emphasize this. The house is burning in front of your eyes.