For me, you are hitting the key part when you talk about a natural pace. The only difficulty I have experienced has been when things progressed faster than the natural pace. In one case, the way that events had transpired almost by chance, the spouses had spent more time together before myself and my playmate had really had much of a chance to get acquainted. We had to some degree, since we both knew we were good to go, but less than usual. Once they got the go-ahead that all were good, the other two went into high gear pretty fast but my trusty gear shift wasn't ready for overdrive yet, which was a first for me. It is extremely frustrating, and the harder you try, the worse it gets. Everything turned out well though, and the pacing was key - even though it was a same room experience and we were all on one bed, once me and her stopped trying to play catch-up and took the time to settle into our own pace (at her wise suggestion, obviously this wasn't the first time she had dealt with the problem) with some pillow talk and more foreplay even though the others were well beyond that phase right beside us, then in short order everything started working just fine and dandy.
Based on that experience, maybe the problem is you are trying to match your pace too much to your wife's? Have you tried separate rooms, or if y'all aren't comfortable with that, at least something with more separation to let your and your playmates pace develop on it's own working off of cues from just each other and not others in the room?
As sort of a related question, when you do get hard, do you ejaculate? That is where I have difficulty, and again it's pace. I aim to please, and so I certainly don't want to come too early, and when you are with someone new you have little to no idea of what sort of pace they want, for how long, and when and how they want it to end. Throw in the decreased sensitivity with condoms, and the anxiety that I would much rather error on the side of not coming at all versus too early, and that is one I struggle with. When or even if she tells you she is ready, then it's hard to just do a 180 on something you have been doing your damndest to not even think about up until that point. Me and the Mrs. have 20+ years of learning our rhythm and pace, and when you get out of that, then your body just doesn't respond in typical ways always.