As far as terms go, when your wife does it and you don't 'swap', it's hotwifing. If you have a girl while she has a guy, then it's swinging. And if you both agree on it, and it's fun for both of you, I don't see why not. You're keeping safe, she's not letting her (and your) identity out, go for it when it's safe.
As far as the guy goes, the ex-boyfriend, you make it sound like she hasn't seen him in years, that you don't run in the same social circles. And his wife knows and is allowing it. Again, I can't see anything problematic here.
Have a good time!