In the posts above, there isn't much about the actual negative effects on the person who is writing. It's all about perceptions of society and others who cheat, etc.
Here's personal experience: The only negative has been the health issue. Almost as soon as we started swinging, my wife started to have yeast infections after almost every new partner. When we laid off for a year because we were out of the country on business, suddenly no more yeast infections. The connectons between swinging and yeast were pretty clear! The yeast infections were a minor annoyance which was not even close to making either of us want to give swinging.
Then, some son of a bitch gave my wife herpes and HPV. She had an outbreak one week which was pretty unpleasant. I had one very small lesion a week later which was hardly even noticeable. That was 8 years ago. Very fortunately, neither of us has had any outbreak or other health problem since. The HPV in her is not the bad kind and it cleared spontaneously. On me, the doc used some kind of lotion that made all of the warts go away.
It has not affected our own sex life at all, except for giving up swinging.
Only recently, have I started swinging again evey couple of months with a longtime friend who is in an open marriage and who is willing to take the chance. Having not had any signs of outbreak in over 8 years, the risk of contamination is very low according to my Mayo clinic urologist and several other sources.
Another of our swinging friends, with whom we had only soft swing experiences and who NEVER, EVER allowed penetration by any swinging partner, got herpes from oral contact (not us). She had two really bad outbreaks in the first year but no more for several years. He never had any, inspite of the fact that they were screwing each other several times a week.
A very minor negative was the discomfort that my wife had while trying to keep knowledge of our swinging activities from our children until they were old enough to understand it. (They probably knew anyway.) Our swinging had no bad effect on the kids. They are now grown, responsible, married adults. One has been swinging on occasion for years, and the other has talked quite a bit about starting soon. Unlike us, both had many partners before marriage, so swinging is not quite the same thrill for them as it was for us (who had not had other partners before marriage).
Let's now have more personal experiences in this thread and less generalizations and speculation.