I have to ask if instead of falling asleep your wife had then had sex only with you with the SM watching or getting something to drink or whatever would it have been a good experience.
I've been in more than a few FMF and some of them as part of "the couple" and rarely are two alike, even with the same people involved. The best advice I can give is go with the flow and enjoy.
Maybe the best advice would be to use communication. On the board here, we hear everybody say that a couple has to communicate, communicate, communicate. The same holds true for ALL involved. Non verbal communications are often not noticed, misread, misunderstood. Verbal communications are best.
Once, the husband of a couple I swung with pretty regular told me after a play session he thought I had spent too long performing oral sex on his wife at one point that he had wanted to do it too. I told him I was sorry, I was kind of involved and since she was performing oral sex on him at the same time I thought everything was OK. My view was pretty limited so I couldn't see any non verbal cues. I told him next time, tell me. After that he had no problem saying what he wanted to do next so to speak, we all did and it worked great.
Was the SM an experienced swinger? I try to "check on" the husband fairly frequently if he's just watching to see if he wants to join in or take over. Sometimes I'll ask for him to join in. If you can't talk to the people you're having sex with or in front of, I'm thinking you shouldn't be having sex with them.
Not every SM you swing with is going to be very experienced and not many are going to have swung as a couple and every situation is different.
It sounds like more of a lack of communication or a misunderstanding than anything else.
I know I was talking to a male half of a couple one time I had played with and I told him he was lucky as his wife was so interested in pleasing him. I talked about how she frequently made sure that he was having fun and a couple of times whispered to me what he liked to see and we should do that etc. There was a moment of silence then "I really needed to hear that right now". Evidently he had a little bit of similar feelings to what the OP did but hadn't realized his wife had been doing what she thought was a turn on for him, what she was doing was trying to please him. It all got sorted out in the end because of verbal communication.
Talk, talk, talk, and every now and then talk dirty.