You sound a lot like my husband and I. He always had these juicy fantasies about seeing me with another guy and I thought they were just that until I was propositioned by an old classmate and my husband said "you should go for it!"
We talked about it for about a year after that. I knew that he thought he would like this, but what if I went through with it then my husband felt horrible? I couldn't un-fuck the guy! Once it's done it's done, so I think that if you get to the point where you are going to go for it, you have to agree that if there are regrets afterwards that you are going to have to let them go (easier said than done). No holding grudges or blaming each other for following through with something that you both agree to ahead of time. Then you can either decide to proceed or call it quits and say "well, that happened".
We have just started dating a couple, but my husband's main interest is seeing me (or hearing me) with another guy. He belongs to an online forum for the "hotwife" lifestyle which is a little different than regular "swinging".
If you'd like to get his perspective, I'm sure he'd be happy to get on this site and chat with you about it.
One thing we did that helped ease us into things is chat with guys online. I've actually met some really nice men through a chat room I frequent and it turned on my husband (and me) for him to watch/listen/read me having erotic chats with these guys. It's a "safe" way to dally with another guy and gave us some confidence that we were going to like pursuing this lifestyle.
Go slow...keep communicating openly and honestly.