Again, does it matter? It could be anything that makes me feel fulfilled. It could be as simple as being touched more, to having a completely different sexual partner. If I feel as if there is something missing in my life, that I need, want, desire, crave, (call it what you will), then why is it so wrong to find someone to fulfill that need?
But fulfilling that "need" may not be worth the possible costs.
Take a "conventional" marriage. Do you think it's right for a spouse to cheat just because they have this desire that's going unfulfilled? Most of the time I do not believe it's ok to cheat. But I've recently learned that maybe, just maybe it's at least justified sometimes.
On the one hand, life can be tough sometimes and we need to just deal with it. But like you've recognized, we have every right to do what makes us happy.
I'm not saying it's wrong to fulfill our desires, but we can't always have our cake and eat it to.