In some ways you are correct but the problem that occurs is a lot of single guys want to get into swinging because they think it is going to be "easy" uncomplicated sex...it isn't. In swinging you still have to work for it and it still gets complicated.
Consider this - in order to have even the slightest success in the swinging world you are going to have to be pleasing to the female eye (much more so than in the vanilla world), you are going to have to be charming and flirtatious, you are going to have to have very solid social skills and very importantly you are going to have to be able to "read" people and know their motivations and be able to read between the lines. You are also going to have to very skilled at communicating your own boundaries and limits as well as your objectives and motivations.
Ok stay with me here - if you are able to develop and hone all of those traits and skills enough to be even a slightly mediocre single in the swinger world, you will be a lady-killer in the vanilla world.
So I guess the bottom line of what I am trying to say is that the time and energy that you spend trying to score a piece in the swinger world will be a lot more effective and you will probably have more enjoyment from it in the vanilla world.
The issue here isn't that vanilla chicks are all fucked up and that swinger chicks are going to be easier and less complicated. The issue is you need to work on your "game" a little more.
Whether you go the swinger route or the vanilla route you will still need to look your best, have solid social skills, and be able to communicate effectively and be able to read people well enough to know if they are pulling your chain or have a completely different set of motivations and objectives that you do.
Once you have those skills down there is A LOT more options in the vanilla world and deep down your core objectives are to have a meaningful relationship with your own special someone and not just be a stunt cock for someone else's.