Seems like semantics for sure. A lifestyle includes a variety of aspects: the way a person lives, their social relationships, consumption, entertainment, attire, etc. A lifestyle is a mixture of habits and ways of doing things, and reflects an person's attitudes, values, and worldview. EVERYONE has a lifestyle, and for swingers sex-with-others is a part of it. (A big part for some, less-so for others: either way, it's an aspect of their "lifestyle".)
A hobby is something a person does for entertainment, because they are interested and enjoy it, rather than for financial reward. Not sure how one can have any hobby that cannot be seen as part of their lifestyle. Seems what people really mean when they say "for me it's a hobby, not a lifestyle" (or vice versa) boils down to a matter of emphasis, a statement of their priorities and values.