You are guessing and assuming. Ask him. Send him an email that says "Are you OK?, I haven't heard from you. I miss seeing you". Remember only 10% of things are as they seem, the other 90% is a mere guess without the facts.
What you are describing is referred to by some as hotwifing or the hotwife lifestyle. Some here get their panties in a twist if you try to include it in the definition of "swinger". I have looked elsewhere and found forums for this specific topic.
"Darling, Let's talk about Scott. He's really good in bed and I'm loving it!... but I want to make sure you and I are alright. How do you feel about my hall pass, so far?"
Alura