Yes you certainly have every right to be upset, with this "friend" as well as your wife. Even if you had sex with his wife previously he was undoubtedly using both of you under the guise of having a couple/couple relationship to get your wife into bed while shunting you aside. And I speak from experience from this having gone through something very similar. I do believe this happens more than some are willing to admit.
How you deal with what your wife did is obviously up to you and your relationship. Just because she was sloshed doesn't let her off the hook. Especially with you not there she needed to have exercised greater self-control, knowing that alcohol and swinging are a bad combination. Some will say that liquor is the ultimate truth serum, uncovering someones innermost desires. I would say that rings true in this case.
What I don't get is if the situation were reversed and this thread were started by your wife. If it were you going to the party alone and ending up in bed with the other woman people would be all over you. It simply has to go both ways; the rewards and responsibilities of swinging, as well as owning up to the transgressions.