It seems to me you may be at a cusp in the LifeStyle. Assuming your husband would like to play with her, he is being careful about your feelings. And he is to be commended for that.
Now, assuming you really don't mind if he plays with her without you participating, you may choose to encourage him. This will open up your relationship to soloing, which is simply another flavor. If you have that talk, and he decides to have fun without you, then when/if you decide you'd like to play without him present, he should be willing to allow you this boon.
On the other hand, if this is a step either he or you don't wish to take, then you'll still be having fun in foursomes or the other flavors you're already participating in, right?