Wow, are you sure you're not me? I had exactly this same issue come up last time my wife went out to a club. We ended up on seperate beds, me with a woman I had no attraction to and her having a great time. Afterwards, I realized that a big part of my problem was that I am, in general, more interested in being "engaged" with what my wife is doing than in doing anything myself. The times we have hooked with up another couple, we've been in the same bed and I've felt "connected" with her even while I was with someone else. It was US swinging together. Being across the room, glancing over at her while also playing with a woman I had no attraction to made me feel seperated and alone. I figured that out while I was talking to my wife about out last unpleasant experience and we resolved to make sure we're always doing things together because being seperate doesn't work for me.
I suggest that you talk out your feelings with your wife and somewhere in that conversation you will find the truth. I think you're close to it when you say that you have a communication issue. Swinging is all about communication. You can't have a truely good swinging experience without good communication. You need to work out your communcation and talk to your wife before you go further.