Years prior to my wife and I considering swinging, we often had debates/discussion about my having been briefly involved in a polyamorous triad in the year before I met her. She had rather strong opinions that it was wrong, and that the married couple had broken their vows, and I was wrong for aiding them in doing that. My response to that was that no cheating happened, everybody knew and approved of what was going on, and no promises were broken...even marital vows...that they (the couple) had not previously agreed were no longer valid.
Life isn't set in stone. A decision you make when your 25 doesn't mean you have to be chained to that decision for the rest of your life (in this case a vow of sexual monogamy) if you both agree to change that decision.
Also, agreeing to have sex with others does not have to mean you are somehow despoiling your marriage. For many swinging couples, it's rather the opposite. It can add so much to a marriage that a "vanilla" marriage will never be able to experience.
Marriage is an institution. But, as with any institution you are not being forced to partake of an external definition of what that institution is. If you enter a monastery, taking vows of celibacy, and later decide to leave the monastery...have you broken the institution? No, you've decided the definition of that institution given to you is one that does not work for you anymore. You are not chained to that institution. There are many ways in which you can serve a purpose that while initially might be embodied in one institution, you find at a later juncture does not have all the answers for you. Marriage is such an institution.
More; marriage is not a monolithic definition that applies in absolute rigidity across all cultures, all times, all people. The very definition of marriage is highly subjective, viewing through those lenses.
What is YOUR definition of marriage? That's the most important question. Not someone else's idea of the definition, or some church, group, culture, etc. Figure out your own definition. It is your life, and only your life. Nobody else should get to dictate to you in what manner you choose to express, enjoy, and revel in your marriage.
For my wife and I, our definition is our unending commitment of love, of souls, of family, of life to each other. Nobody can hold a candle to that. Whether someone is enjoying a lunch time conversation with my wife or is having sex with her without my being there (but of course in the know), they can not violate our marriage, and what it means to us.