I also agree that it is not that common for this to happen, but if you look for the same type of arrangement (that is, sharing a lot of vanilla/family time with a couple, only playing with one couple) then the risk is higher. I would branch out and find a few nice couples and space out the times that you see them. I would wait quite awhile before bringing family into your relationship with swing partners until you have a clearer idea of their intentions.
If the problem couple ask why you are no longer interested, you can just say you're moving in a different direction after your break or just tell them that her behavior made you uncomfortable. Maybe she would be able to respect specific boundaries in the future if you do want to stay friends with them. Your husband probably needs to be very clear about what type of communication is ok and not ok.