You "have to understand", you are talking about a subset of prostitutes, not all of them. Personally, I don't really "have to understand" anything about this particular subset; I've worked professionally with this particular subset once upon a time and know that it's just that - a subset. Not necessarily the majority. Some DO have a crappy existence rife with abuse and drug use just like some do who don't engage in prostitution.
Actually, I'm not inclined to admit, much less "have to admit", anything without hard and fast numbers and certainly not based simply on issues of morality or personal viewpoint about cheaters (yet another subset of the group that uses prostitutes).
I'm not inclined to follow the morality path as an argument for or against legalization of prostitution. We, as swingers or practitioners of other forms of alternative lifestyles, could have a similar argument used against us. A strict definition of infidelity/cheating could be used against our group; how could we condone a practice that thrives on cheating spouses? I guess I'm saying I personally am not inclined to let moralistic or paternalistic viewpoints override legal or practical viewpoints as going the moral route tends to dictate the rules, norms and laws for all individuals.
In the end, I'm with George Carlin on prostitution. "Why should it be illegal to sell something that's perfectly legal to give away?"