It is time to stop and take a break.
You two need to sit down, talk and find something that works for both of you. This is not to be just about one of you. You are a couple, sharing life together and it has to work for both or it won't work for either in the long run.
It takes hard decisions at times but someone has to make them. Sit her down, tell her what you told us and tell her the swinging such as it is with you two is off until you can come up with a plan that works for both of you.
Swinging is not for everyone and what you have is not working so why do it?