Think of it this way...
If you dated a woman who jacked you around, didn't show up for a date, and when she did she brought along another guy and you never got a moment alone with her to even talk, would you be kinda rethinkin dating women?
I don't think so
If you and your wife had fun, take the fun. Don't over analyze. You probably were had. So what?
For the future, decide if you want to accept a situation like this. If you don't, a polite "thanks but no thanks" is all it takes. Let him know that the two of you play with couples only, no exceptions. That's how it works for my wife and I. Couples only, and same room. Else, it's a non-starter and we won't play. The important part here is YOU get to decide, not the person you're meeting up with.
To my wife and I, lying is ugly and a complete turn off. We don't care if you don't tell us everything, or even 1%. But, what you do tell us should be the truth.