Eight years ago, when Whitney Goodman was a newly qualified therapist counselling cancer patients, it struck her that positive thinking was being “very heavily pushed”, both in her profession and the broader culture, as the way to deal with things. She wasn’t convinced that platitudes like “Look on the bright side!” and “Everything happens for a reason!” held the answers for anyone trying to navigate life’s messiness. Between herself, her friends and her patients, “All of us were thinking, ‘Being positive is the only way to live,’ but really it was making us feel disconnected and, ultimately, worse.”

This stayed with her and, in 2019, she started an Instagram account, @sitwithwhit, as a tonic to the saccharine inspirational quotes dominating social media feeds. Her posts included: “Sometimes things are hard because they’re just hard and not because you’re incompetent…” and “It’s OK to complain about something you’re grateful for.” It took off: the “radically honest” Miami-based psychotherapist now has more than 500,000 followers.

Goodman’s new book, Toxic Positivity, expands on this thinking, critiquing a culture – particularly prevalent in the US and the west more broadly – that has programmed us to believe that optimism is always best. She traces its roots in the US to 19th-century religion, but it has been especially ascendant since the 1970s, when scientists identified happiness as the ultimate life goal and started rigorously researching how to achieve it. More recently, the wellness movement – religion for an agnostic generation – has seen fitness instructors and yogis preach about gratitude in between burpees and downward dogs. We all practise it in some way. When comforting a friend, we turn into dogged silver-lining hunters. And we lock our own difficult thoughts inside tiny boxes in a corner of our brains because they’re uncomfortable to deal with and we believe that being relentlessly upbeat is the only way forward. Being positive, says Goodman, has become “a goal and an obligation”.

Toxic Positivity is among a refreshing new wave of books attempting to redress the balance by espousing the power of “negative” emotions. Their authors are hardly a band of grouches advocating for us to be miserable. But they’re convinced that leaning into – rather than suppressing – feelings, including regret, sadness and fear brings great benefit. The road to the good life, you see, is paved with tears and furrowed brows as well as smiles and laughter. “I think a lot of people who focus on happiness, and the all-importance of positive emotions, are getting human psychology wrong,” says Paul Bloom, a psychology professor at Yale and the author of The Sweet Spot, which explores why some people seek out painful experiences, like running ultra marathons and watching horror movies. “In a life well lived, you should have far fewer negative than positive emotions, but you shouldn’t have zero negative emotions,” adds Daniel Pink, the author of The Power of Regret. “Banishing them is a bad strategy.”

The timing of these new works – which also include Helen Russell’s podcast (following her book of the same name) How To Be Sad – is no coincidence. In light of the pandemic and now the conflict in Ukraine, it seems trite to suggest a positive outlook is all we need. Strong negative emotions – fear, anxiety and sadness – are a natural response to what’s happening around the world right now and we shouldn’t have to deny them.

These authors want you to know that “negative” emotions are, in fact, helpful. Russell talks about sadness being a “problem-solving” emotion. Research from the University of New South Wales shows that it can improve our attention to detail, increase perseverance, promote generosity and make us more grateful for what we’ve got. “It’s the emotion that helps us connect to others,” she adds. “We’re nicer, better people in some ways when we are sad.”

It’s tougher making an argument for regret, which might be the world’s most maligned emotion, but Pink is game. From a young age we are instructed to never waste energy on regrets. The phrase “No regrets” is inked into arms and on to bumper plates and T-shirts. Seemingly every famous person has a quip about living without regrets (I would know: as someone who tends to linger on thoughts of what might have been, I’ve read them all). Pink says we’re getting it all wrong. “A ‘No regrets’ tattoo is like having a tattoo that says ‘No learning’,” says Pink, who was also a speechwriter for Al Gore, speaking from Dallas, Texas. He became interested in this topic because he couldn’t shake his own regrets about the fact that, while a university student, he wasn’t kind to fellow pupils excluded at social events. “If it has bothered me for a month, a year, or in this case 20 years, that’s telling me: ‘Hey, you might not realise it, but you care about kindness,’” he says. “Regrets clarify what matters to us and teach us how to do better. That’s the power of this emotion – if we treat it right.”

READ RELATED: ASK CAROLINE: My teenage daughter won't talk to me

The problem? We’re not taught how to effectively process these difficult emotions. A good starting point is to familiarise ourselves with these feelings by acknowledging them and sitting with them for a beat. That takes practice, says Goodman. “It can include learning how your emotions feel in your body, and what to call them. When we’re able to put a name to a feeling, it makes it less scary. And when something is known, we can figure out what we want to do with it.”

Telling others about it lightens the weight. Complaining is perfectly natural, says Goodman. And articulating it helps us pinpoint what it is that’s bothering us, because language converts this “menacing cloud” into “something concrete”, says Pink. That disclosure could be to a friend, therapist or total stranger. In his Regret Survey, 18,000 people anonymously shared their biggest regrets, while Russell suggests a “buddy” system, in which you make a reciprocal agreement with someone to talk about your worries without interruption. (A note, if you are comforting a friend: listen and ask questions rather than immediately reaching for pick-me-ups.)

Your next step will likely depend on the nature – and severity – of the emotion. To help us sit with sadness, Russell advocates being in nature. Cultural pursuits can help, too. “It sounds a little ‘woo’, but there are lots of studies about the effectiveness of reading therapy and looking at a piece of art – and how music can change our moods,” she says. “Sad music can act as a companion when we’re feeling sad, rather than making us feel lower. I do think it’s liberating when you finally kind of surrender to it all.”

Pink, whose approach is a little more structured, differentiates between regrets of action (wrongs you’ve committed) and inaction (opportunities not seized). For both, you must comfort yourself with the knowledge that everyone has regrets – and recognise that that single thing doesn’t define you. “Don’t look at a mistake as St Peter at the gate passing final judgment on your worth,” he says, but as “a teacher trying to instruct you.” He recommends stepping outside yourself and considering what you would recommend a friend do in a similar situation, whether that’s making amends for past acts, grasping a new opportunity, or ensuring you don’t make a similar misstep in the future.

Crucially, processing negative emotions “should all feel somewhat productive in the end”, says Goodman. Meaning: instead of ending up in a funk of wallowing, with your feelings replaying on a loop, “the wheels are turning, you’re making connections, you’re figuring things out,” she says. That doesn’t mean you need to come out of it feeling happy, or with a neat fix. “Sometimes you just get to a place where you say, ‘That was really hard, and now it’s over or now I’m not dealing with that any more’,” says Goodman. “And if it comes up for me again, I’ll deal with it.”

Leaning into negative thoughts should ultimately leave you with a sense of fulfilment. While we might instinctively think that filling our days solely with joy and excitement is the dream, “if we want to live a meaningful and purposeful life, a lot of pain is going to be part of it”, says Bloom. “What I really want is for people to be able to enjoy the full range of the human experience,” adds Goodman. Armed with the knowledge that you can do it in a methodical way, don’t be afraid to let the darkness in.

Toxic Positivity by Whitney Goodman (Hachette) is available from guardianbookshop.com for £13.04