I'm not saying you're wrong, but I believe it's more nuanced than that. For me, and many other men, it's not about competition - it's about having fun. For us, once we decided what we wanted, it wasn't 'taboo' anymore.
We often played with couples, but there were many times when we made a choice to play with a single male, either late at night after a swinging experience at a club, or just for a night's pleasure. My wife (like, I believe, a lot of women,) had greater stamina than me, she could easily have multiple men in one night. So we sought out men for the simple pleasure it brought to her. And the fact that she was being pleasured brought me pleasure.
I'm not saying there wasn't a part of me that enjoyed the sights, sounds and smells of her getting laid, but there wasn't a 'cuckold' portion to it for us. I had no desire to be humiliated, nor did she want to humiliate me. It was simply for the fun of it.
So, if you believe that I'm a guy who simply 'won't admit it', well, that's on you.