Hey Violet75 - So, I can relate to this pretty well. The only difference was that I wasn't envious of her and it never bothered me. When we took this step it was really because we wanted that "lust" that you can ONLY get when being with someone new.
In your case, it sounds like he feels slighted. Men before him got to do something with you that he hasn't...
Swinging is the greatest thing and if you two do it for the right reasons then it opens up incredible possibilities but I worry when I hear things like envy, well, envy and jealousy go hand in hand.
The number one thing is communication. This is a great place to read about other peoples experiences both good and bad. Use it to it's fullest. I hope you two are successful and it all works out but just make sure that you really know why you are doing it.
Welcome to the boards and good luck!