Based on your description of what you're looking for (somewhat, physically fit, for example), it doesn't sound like you're too picky. But, since I haven't seen the people you've turned down, I can't really judge.
However, I can say that we were too picky when we first got into the lifestyle. We were both nearly 40 when we started swapping, but we looked like we're in our early 30s; maybe 35 max. We met when we were 31 and 32, so we'd never had sex with anyone who looked 40-50. I (Adam) wasn't too thrilled about being with some who looked 45 when the oldest looking woman I'd ever slept with looked about 35 -- and there's a big difference between the two (I say this as I close in on 43).
Our first couple was in their early 20s, but our second was older. She was 44 and he was like 48. They kind of took charge and we just went with it. We had a great time and definitely lightened up after that. Since then, I've learned that being too picky just means I lose out on some great sex. In fact, some of the best sex I've had has been with girls who were older or just average looking at best.
Don't get me wrong, a hottie is always nice, and I have to be attracted to a woman to have sex with her, but what I find attractive is different today than two years ago, and I'm better off for it. In fact, I would have missed out on my favorite partner (other than Eve) if my standards had remained where they'd started. And I'm extremely happy that I didn't miss out on her.
Of course, as everyone has already stated: having sex with someone you don't find attractive is not an option. Don't do it! But, at the same time, I would say that if you're only open to models, I think you're going to miss out on some fun times.