I think part of this is his lack of tact, and another part of this is your body issues.
You shouldn't take one for the team. Right. But, what's the motivation behind your will to ban another couple? That you dislike the guy? Or that your husband likes a gal that doesn't look the way you do? (with your body issues, I am pretty sure you'd read this a "a gal prettier than you"... but, that's just you and your body issues).
I guess you two have a lot to talk... but given your body issues, some topics become difficult to bring up.
So, I'll talk of myself, a seldom fat guy with my own body issues. I know my wife likes me... and that she liked my body better years ago, when I was slim. She likes slim guys, and I am fat. Let's deal with this. She's still next to me, awakes next to me, no matter which adonis had sex with her the evening before. She keeps choosing me, over and over and over. Not because of my body (obviously) but because of everything else of what I am.
And this is the way it have to be.
Now, if I were overly sensitive about my body issues, it's very likely that se won't tell me. She would do her best to halp me feel better about myself, to avoid hurting me... and she would avoid commenting "she liked me better when I was slim". And again, this is the way it have to be.
As for my wife, she may pick an adonis... and also pick guys like myself. And this makes a difference. And I wouldn't ban a couple just because he's the adonis I am not... and that makes a difference.
Honesty is overestimated. You're asking him to be honest, and with your every day attitudes, you're also aking him to be very carefull about your body issues, to avoid hurting you. You also want to swing, and want to indulge his desires... and yes, it's very likely he'll look in someone else for features he knows you don't (and probabily won't) have, for a little while, just for fun, before returning to your arms.
So, you're giving him a thin thread where to walk... and he doesn't seem to be too proficient in walking on thin threads.
He may relinkish some of his wishes for a playmate to protect you. Or you could valuate yourself fow what you are besides your body, and allow him to openly admit what's binding him to you... besides your body, and EVEN with your body.
Anyway, I am not sure if you two are swinging for the right reasons today. If he does it looking for features you don't have... and if you do it looking for a reasurance about his tastes. Should he were picking women more like yourself... wouldn't this help you "indulge yourself"?
You have to work on your body issues, better say, in your self esteem beyond your body issues... and it would be nice if he helps you out a little bit.