Well, we are very new to the idea of swinging, and we are in no way experts here. However, we do have an opinion about this.
We have had several discussions about bringing other people into our sex life and, while it has not progressed beyond conversation and fantasy, one thing our research told us is that we should have rules in place before starting anything with someone else. First among these rules for us is that we talk about everything, absolutely everything, both before and after. If both of us agree that something is going to happen, then it is not cheating for us.
It is a respect issue and an integrity issue. Integrity means honoring your word, doing what you say you will do. Sometimes, what you say you will do is what is expected by others (say, your wife for example) even though you may not have said so verbally. In this case, your word is that you would be honest with your wife. According to your original post, you have not done this.
Now, I will not pass judgement here. I ask you to look for yourself. You asked "Does that make me just a cheating asshole?" Only you can answer this, and you already have. Looking for validation here is inauthentic and a lie.
Now, that said, cleaning up your integrity may seem daunting to you. Yes, it means coming clean about what you have been doing and who you are. I invite you to consider that you have no idea what may happen in this conversation with your wife. It could go any number of ways, including several you have not thought about.
My question to you is, what kind of marriage do you want? You get to choose. Honest and respectful? Full of lies and deception? Something else? Once you choose that, then the actions to take will be obvious to you.
Again, just an opinion from someone who is no expert.