I see that you are a new member here at Swingersboard so I want to say WELCOME to the board.
I feel sorry for you that you have found yourself in a situation of this kind. How do you feel deep down about your relationship? If you value the relationship and want to recover it, a professional marriage counselor you should both see. If relationship give you more bad feelings than good feelings, I recommend that you find a good divorce attorney. In either case, you should both agree to stay away from swing activity.
You said, "but if done properly, it can be seen as constructive criticism"
So you say! There are people who, no matter how you put it, take it as an insult. So the safest, easiest thing to do is say, "No, thank you," and walk away clean.
Sounds like it hasn't totally disappeared though. In another 7-10 years the kids will be doing their own things and not even want to be vacationing with the parents. It may just be on hold...