One thing we have learned in swinging is that there is never a rush. You don't need to run out and meet every couple you can. Take you time to choose what you want and don't be afraid to say no. Make your choices when you are sober or close to it. If you found the couple to not be attractive to you, then signal your SO early about that so the evening can come to a close without playing.
And yes, you are drinking too much and that is contributing to your bad experiences. You have realized this, but I (guy half) learned that more than 2 drinks in short amount of time impacts my performance (I'm 50). So i spread them out through the night and I am also the driver on the way home so I need to be ready for that. If you need some alcohol to loosen up, that's fine..just pace yourselves