Yes, I also feel the age of swingers is tilted towards later in life.
Many couples get married in their twenties and early thirties, believe that monogamy is the way to live, I don't think the interest in other people goes away, it's just subdued. Then as the couple reaches their late forties and fifties, the children are growing into high school, they begin to look at being empty nesters. They see their sex life as ordinary, if not just plain boring. This is the point when some couples start fantasizing and consider non-monogamy. And some take the next step, enter the LifeStyle.
Certainly not all swinging couples follow this pattern, it just seems typical.