There are two parts to this, really.
1) You don't want to fuck someone you're not attracted to - totally your decision and your right, every time, with or without reason, and even if it means no fun for your partner as a result.
2) If your partner is attracted to someone you don't like and you treat them with contempt because of this, that can't be right. Unless we're talking about some extreme cases of poor hygiene or heavy drug use or something of that kind, I don't think it's kind of you to pass a judgement on your partner's sexual preferences. Again, he might not get to play with this person anyway but surely he can at least fantasize and talk about it openly, without being judged?..
Again, if this happens all the time, then the question arises how do you screen the couples to start with? Do you (the female) need to be more involved in that part?