Cutting off contact with the offending party is the first real step and you appear to be taking that one.
You should shelve even thinking about swinging or sex with others until you and your husband build that trust factor back up.
You are running around behind his back checking on him, he is lying to you about things. You two have lost the most important pars of a relationship, trust, communication and honestly.
You have no business even thinking about swinging or having sex with others at this point until you rebuild your own relationship.
Time to shut down the "Swinging couple" and rebuild the family couple. Maybe once you rebuild you might want to think about swinging together, or not.
Swinging will NEVER fix a messed up relationship but it will sure has hell brake one up that has cracks in it already.