Don't misunderstand. A lack of jealousy does not mean you should swing. There is much more to it. The fact that you don't want him with another woman, may not have anything to do with jealousy. Unfortunately, there are many things that in our head don't bother us but when presented with them in reality we make very different choices than we would have thought we would. For many, swinging is one of those things. Some times it should be left as a fantasy.
You have referred several times to how your situation of him being with another woman made you feel "less special". To me this says that your issue isn't jealousy but rather simply a self-esteem issue. You need to be the center of attention to "feel special". When you are with other guys or in a MFM situation or just with your hubby then you are the center of that universe. When he was with someone else, he turned his back to you for a brief time and took away some of the attention you thought should have gone to you and gave it to someone else.