Please consider this and accept it without judgement.
This pursuit; "swinging", "the lifestyle", etc, what ever you wish to call it is predicating on communication.
Communication within the couples involved and communication between the couples involved.
Open, honest, frequent communication creates trust which increases the chances of having a good experience.
You have stated that "anytime we try and talk through this problem, it turns into an argument".
Getting past this will be key to moving forward for the two of you, clearly.
If you can't communicate as a couple, your chances are poor of communicating effectively with another couple
as you will not be moving forward with one voice and aligned in all respects.