I'm playing devils' advocate here.
When thought about logically, the OP's situation seems harmless.
But...then there's the real world, where the incest taboo is very much a factor. This kind of thing, if anyone finds out, is a brush that will paint all swingers. It would be like if one swinger were caught abusing a child. Then all swingers would be cast as child abusers (as if we weren't already!).
I admit, it makes me squeamish. Okay, you're not going to have babies. But let's say you wanted to fuck your sister -- would that be okay too? Logically, it would be just as okay as being with your cousins.
I wish you the best of luck. When looked at unemotionally, you're not doing anything wrong. But c'mon, we all know that's not what matters in many cases. I just hope no one finds out because it would be bad in so many more ways than just being outed as a swinger. And the "ick" would get all over those of us who keep it out of the family, out of the workplace, away from vanilla friends, etc.