It is normal since, for me anyway, it has usually been the result of sexual chemistry.
I'm now 61 and still remember vividly a few women...well 3 in particular, 2 ONS and one 3 week relationship where the skin contact was electrifying.
Now I have been in the LS on and off since the late 70s..never with a SO that was very significant, so whenever my GF of the moment would after a swap be glowing because the male of the other couple had been amazing, it would not hurt my ego but definitely pinch it.
The 4 women I had long relationships with(including my soulmate ex-wife) were against swinging and I knew a situation like the one you're in would have been very difficult for me to deal with. With these 4 women I was all-in emotionally, mentally and physically. In that mode it is impossible for me to separate sex from love and I can't imagine a random evening after a day of work watching TV with my SO and hear her tell me that the last few days she has been longing for that swap partner that I know she likes a lot.....even if she tells me that I'm still #1, I would wonder if she is not just trying to spare my feelings and that would have made sex with her not impossible but a huge challenge.
I've known when in relationships that I was not the biggest cock, best looking guy or best body my GFs had been with, but because of my passion, intensity and very high sex drive I needed to feel like the one she would think of when getting horny.
But when you start swapping the possibility of a very special connection for your SO with another partner exists and even if I know I'm number 1 in real life as the life partner, the one she chooses to share the highs and lows of life 24/7 with the bills, mortgages, maybe kids etc...knowing I'm not number 1 anymore in bed, in my ''all-in relationships'' would have been devastating.
Now I suppose your husband is different, is not threatened by your feelings for that other guy, reads your posts and knows about this...and if I'm wrong let me know.
By the way, if I had a dollar for every time I've read a swinger saying that a certain partner was not better but different then their SO, I would be living on a beach somewhere....to me it's a PC way swingers have found to spare their SO's feelings.
I've been sexually hyper-active for 44 years...yes all the women were different in bed, but can I make a top ten list? you bet I can and the funniest thing is that with these 10 all-time best sexual partners I would rather have root canal treatment than be their bf!
So now the question is are your feelings evolving? becoming stronger? do they have an impact on your sex life at home or on your couple? Do you see yourself spending a weekend or maybe spending more time than that with that man?
You should look into this and be very very careful, feelings have a life and a mind of their own. Affective reactions defy reason and logic.
Blaise Pascal, a French mathematician, physicist, inventor, writer and philosopher said:
'The heart has its reasons, which reason knows nothing of'