Nights like that are rough, feeling left out, whether intentional or unintentional, is never fun.
Was this group new to you but not to them? Maybe the others not only know each other better, but also have sort of a routine going? If that's the case, just be patient and give it time and let things evolve and soon you will feel that same level of comfort they do.
The other idea is to play with each other right in among everyone else. No rule that you HAVE to swap to have sex, right? Having sex with each other in front of others is still very exciting, and in fact, can be awesome since you get the swinging vibe and excitement factor but don't have the "am I doing this right for him/her?" anxiety that goes with being with someone new.