I totally agree with PB&J on this one.
My guy and I work opposite shifts with opposite off days. For us to organize much is a challenge. so, we do have a girl who we play with individually and together. She is fully aware of our commitment to one another and knows she is just a FWB for us. We are both in contact with her. We don't have new play partners unless we have all met together. It keeps the communication and boundaries we have set in check, and enhances what we have.
It sounds like he wants his cake and to eat it too. If you are not comfortable with any of this, put the brakes on it fast!
Swinging may or may not be for you, and you won't know until you've found the right guy to try this with, he doesn't seem like the right one. Honestly, this isn't really swinging to me.
If you haven't met his play partners, chances are they don't know you exist.
When the Mr or I seek a play partner, we are both involved.
I see things he may miss and vice versa.
Example, one girl said she was looking for FWB. She was bi and wanted a couple with a bi girl. Well, turns out she was looking for a guy only who could be her sugar daddy. There were signs I saw that she was only interested in him or a man in general that he did not. He even thought I was jealous. When I went straight to the point with her the truth surfaced.
She did not become a playmate for he, I or us.
Communication is key and so is honesty and it sees like he is falling short with these two things.
If you have concerns, he should listen. My Guy always listens, even when I've not said something.
I think the relationship is too new to be swinging to be honest. He went from one to another and there hasn't been enough time to build a solid foundation.