You all are communicating and talking, and that's good. No sense in rushing things, that is when bad things happen. If you have talked through everything you can think to talk about ahead of time, then when a curve ball comes your way, which it surely will sooner or later, then much easier to deal with.
All I have to go on is your post, so my apologies if I'm coming out of left field, but the thing that catches my eye a little is it seems like there is a little more focus on trying to make a scenario happen that he wants, and less on what you want. You sound pretty sure being with another woman isn't for you right now, so I would drop that idea right away. The single male route interests you a little more, but again, there is apprehension already from both of you based on the experience of your friends. That leaves another couple. That is the way we swing so admittedly I am biased here, but I think that would probably be the best way to go for you both. The only caveat there is the more people you get involved, the harder it gets to find the chemistry all the way around though. But, good things come to those willing to invest the time and energy to make them happen
Your concerns about liking them more than your spouse, they are better in bed, etc. are all very common when considering whether to swing or not. I look at that like this - we've been married 20+ years now, and other than the horny teenager stage when we first starting going out, our relationship hasn't been focused on or survived because of sex. A lot of people are willing to have sex with you, but how many will give you a shoulder to cry on when life isn't going well, be there for you when you are sick, etc. Those are the ties that bind, and none of those have anything to do with swinging.
This is one of my favorite pieces of advice, so I'll use it here too. Get on the internet, research on-premise swingers clubs in your part of the country, then make a weekend getaway together to check one out with no expectations other than having a good time together. Even if you go but never even emerge from the shadowy table in the corner, you will have learned a lot about swingers and swinging, and then you can decide from there what your next step will be.
Good luck, welcome to the Swingers Board, and please register so you will be able to reply to this thread. We'd love to have you as a regular member