Hello, this is Petra. Welcome and congratulations on taking your first steps into what we all here find to be a wonderfully new part of life. You have identified a number of issues for you and your husband, and they seem to be tangled together. Try to separate them, consider them individually, resolve them one at a time. Not everything has to be done, or even taken on, at once.
First, as you both seem to recognize, this does not have to be a symmetric situation where both of you do the same thing at the same time. Perhaps you both can start with a another woman in your life and it may (or may not) become a polyamorous situation. As your husband becomes more comfortable with the entire idea, you may develop a relationship with another man on you own. Your husband seems to be open to the idea of you having sexual freedom, he just doesn't want to be involved, at least at this point. Communication and honesty do not necessarily mean that every (or even any) detail is wanted. Some people want to hear about and/or watch their spouses sexual advntures and take great pleasure in it. Others for various reasons, do not.
You also call what you are looking for "polyamory" instead of "swinging." That is the type of relationship we are in, and it has taken some twists and turns to get there. I prefer it because it is for me the right mix of sexual variety and personal relationship, but it takes longer to achieve than you two going to a swinging club on Saturday night, finding someone that is physically attractive, and having sex with them. Two ways poly relationships can develop: there are friends you have feelings for that become sex partners as well, or you look for someone, like dating. For hubby and me it was a combination of both - I had a bf since before we were married, while hubby "dated" women (that I recruited) that became his sex partners for varying lengths of time until we found the right one. She is now both his gf and mine (something I hadn't done before). So we are now a closed circle of four.
Good luck working through this, poly is definitely worth it both for the sex and the relationships. Being unselfish with our spouse and opening up to other relationships satisfies two basic human desires: to be in a stable relationship with a home to go to, and to find sexual variety and excitement. Go slowly and patiently with your husband. Lead him by example, letting him have the freedom to enjoy other sexual relationships and he will come around. He seems to already have made a start recognizing that because he loves you, you deserve the same sexual freedom and pleasure that you are planning for him. It may just take a while and be something he prefers to keep a mystery, in a box. That is what happened with hubby and me; for several years after we married I kept my bf while hubby was monogamous, then I opened up on my own to the fact that sharing a spouse is a genuine pleasure in itself.