Please see above short answer:
Short answer: He loves swinging and having sex with whoever he wants more than he cares for you. Sorry, but from the info given, that is the one conclusion that can be drawn.
Even you have come to the same conclusion:
All the additional information only clouds the issue. Lets see, he is still having sex with other women, but he's only looking for women that he has zero attachment to??? Including married women??? Bottom line: It's the same question. Is he treating you with the respect and love you deserve or is he TELLING you that you need to do things and let him do things you don't agree with or enjoy?
Is it possible to love and want someone who just isn't ever going to be able to give you what you need? Yes, but that doesn't mean he will ever give you what you need or deserve. Wishing that someone was different will never make them so any more than wishing for a million dollars. Does it hurt...does it suck...yes it does, but better to hurt now than waste years only to come to the same conclusion only then having to wish you had that time back. You deserve better, go out and find it.