First, I will say that religion is not for me or for the mrs, as both of us were brought up in a somewhat hypocracy of religion (christianity to be specific, and primarily baptist). Does that mean that all christianity is filled with hypocracy or that all baptist is even? I don't know, but what we've seen is filled with it. Then with so many people that live forever that are scum bags in my opinion, and so many people that are good that go through much bad stuff, I find it hard to beleive that a loving god exists. If you believe it, fine I'm not going to bash you over the head.
However, I will point out something to you. Religion or religous books (aka the bible, the koran, etc.) were translated by man in we'll say "oldern days". It was probably translated several times by man, resulting in a version that due to the motivations of the translators and varying opinions it probably isn't in it's original version and there are probably some changes that were motivated by personal gain of individuals. After all if I was to be the leader, and I could say pay me a percentage of your earnings as "a gift to god" or "tithe", I might do it to. So I would get this great salary or income from you and lots of others to run my organization or government. Or if I wanted you to live my way and my way only; and I were translating a religous book, I might be tempted make a rule that said you had to live my way. I'm not like that though and I obviously don't have a religion that I dictate to people.
How does this relate to guilt? I believe religion is a man made set of rules, and at a minimum it was greatly modified depending on the powers of time when particular translations were made of what ever religous book in question. On this note: In order to feel guilty over breaking a rule, we have to whole heartedly agree with the rule being broken. I'm pretty sure some where I heard that King James was homosexual (yes of the King James Old English Version of the Christian Bible). I don't think many in the Christian Religion today would openly agree that that was ok.
As this relates to swingers, the rules may be different for different couples, as many of us have different boundaries between us and our significant others.
So, I would not feel guilty over whether I'm in line with some religous beleifs.