Making judgments is perfectly normal and acceptable in my mind depending on what you are judging, for what purpose and how you act toward people.
I think it is perfectly fine to decide who you want to have sex with based on looks, personality, manners and chemistry.
I think it is fine to decide not to associate with someone based on moral conflicts, illegal or unethical behavior or their bigoted attitudes.
I do not think it is acceptable to treat people unfairly or infringe on their rights based on their race, gender, religion etc. Unfortunately, too may people assume that if you don't care for them, it is based on one of those characteristics rather than the fact they are crude, rude or just flat out jerks. I guess it is easier to explain away rejection by attributing it to the worst part of the other persons nature, rather than realizing it may be something that makes them unappealing or even they are simple not what the person is looking for.
As for swinging, I have no right to anyone's body nor they to mine. So if one of us decides that we are just not interested, for what ever reason, then that is just how it goes, regardless how petty or "judgmental" the rejected party thinks it may be. Why waste time being upset about it.