I've been doing a lot of thinking and soul-searching after starting this thread and reading all the replies, and it's truly changed my perspective. I had let other people stuff my sense of a sexual self into a box and they watched while I locked it away in the basement. Now I have complete freedom to let it out, not only for my own development into a whole person but also to have fun in this lifestyle.
I realized that my confidence problem wasn't that women weren't paying attention to me, it was that I wasn't paying attention to myself. When I shut off my sexual identity, I also shut off both my sense that anyone could be attracted to me and also my identity as an attractive person. I had become all but literally blind and deaf to their interest. No wonder when my wife said she saw women looking at me I couldn't see it. And when someone would complement my appearance, it fell on deaf ears.
Now I'm thinking back on the past 15 years, remembering the comments that people have made about me. And you know what? That whole time, there was only one, ONE, person who has ever insulted my looks. And that's stacked up against comments like "X and I think you look like Gavin Rossdale", "Isn't he cute, Z?", and "You and your wife look like members of the Cullens!" (from Twilight ), among many others. When girls wouldn't go out with me in high school, it was because I was a snobby, morally uptight, neurotic mess. They liked my looks (though I dress a lot better now), but not what was underneath them. But high school me is long gone. I have come so far from where I used to be.
And you know what else? Yesterday, when I looked in the mirror, I actually liked what I saw.