Kitty, good to see you're thinking about it.
As you pointed out, Xavier is a ticking time bomb, because he knows a secret that Bob doesn't, that he's been swinging with you. And, as you pointed out, he may spill the beans either unintentionally or intentionally. And it's all complicated by the fact that you're in a semi-polygamous lifestyle with both of them. What to do?
Well, you can end the relationship with Xavier. That would hurt, I know. But there are advantages. I think Xavier is hurting your relationship with Bob, this path would stop that, give your primary relationship with Bob a chance to heal. Plus (unless they know each other,) your indiscretions with Xavier would never be known to Bob.
Or, you can keep going with both of them under the new guidelines (Swinging with Bob, sex with Xavier *telling Bob each time*) and hope that it hangs together. With this one, you'll be subjected to some tremendous pressure from Xavier to do things you may not want.
You're in a pickle. Hope it works out for you.