Don't worry, it will burn itself out. There hasn't been much discussion of it lately here, but new relationship energy (NRE) is common. It flames fast and hot, then dies quickly.
You're not wrong, he's wrong. The approach I suggest is to tell him that it's ok for him to have feelings for the other woman, it's natural and you respect him for being more than a sexual animal with her. BUT, the rules of the game are that he needs to share his feeling for her with you and show you everything. It's part of what makes swinging and your marriage special. And from what you imagine in your mind that they are saying, tell him that you think that it is hot. Then when he does share, be gentle and don't criticize him. Kiss him, tell him that you understand and love him, because in doing what he is doing, he doesn't love you any less; by understanding and letting him continue, he will love you more. Also, be extraordinarily nice to her.