Prince Andrew is slowly starting to lose the support of the royal family amid his sexual abuse scandal.

According to Woman’s Day, Queen Elizabeth has expressed her desire to strip off Prince Andrew’s royal title. Months ago, the Duke of York announced that he would be stepping back from his royal duties. However, he is still using his official royal title even though he’s no longer a senior working royal.

“Andrew keeps denying any wrongdoing, but there’s also no denying the walls are closing in around him. There’s nothing his mommy can do to protect him now,” the source said.

But the insider said that Sarah Ferguson continues to stay by her ex-husband’s side. And the Duchess of York previously slammed Jeffrey Epstein for sexually abusing underage girls.

“The only person left on his side, it seems, is his ex-wife [Sarah Ferguson], who he is leaning on heavily. She’s doing her best to boost the York family’s reputation by dripping into all manner of charity endeavors, but she’s getting slated for that too. But it will be nothing compared to the backlash against her if she gets dragged into Andrew’s investigation,” the source said.

Last year, the mom of two spoke about Prince Andrew and Epstein’s friendship during her interview with Vogue Arabia. Princess Beatrice and Princess Eugenie’s mom said that it’s been incredibly difficult for her, Prince Andrew, and their entire family.

Ferguson also said that when she talks about the Duke of York, she talks about family because the past couple of months have been hard on them.

“To see such a wonderful man go through such enormous pain. He is the best man I know. It’s just incredible what he has done for Britain, and it’s all nonsense [her voice raises as she alludes to the Epstein scandal], so I talk about familyhood, and I’m very strong about it,” she said.

Ferguson also posted a photo of Prince Andrew on her Instagram account and she wrote a lengthy caption praising her ex-husband.

The Yorks have an incredibly unique relationship because even though they’ve been divorced for over two decades, they are still a family unit.

In fact, there are reports suggesting that Ferguson continues to live with Prince Andrew and their two children before they moved out of the property.

And even though they are divorced, Prince Andrew and Ferguson continue to make public appearances together.