Simon Cowell baby drama gets more scandalous: Adultery alleged Simon Cowell's baby-on-the-way news got a bit more tawdry Thursday, with words like "adultery," "surprise" and "gold digger" starting to swirl in the mix. Cowell, 53, is reportedly expecting a child with New York socialite Lauren Silverman, 36, the estranged wife of the "X Factor" host's friend Andrew Silverman, 37. The question being raised now is, was the married couple estranged before or after the the unmarried couple made the baby? It is an unbelievable story of betrayal. It is a sad story and a tragic story," Andrew's brother, Alexander, told the New York Post. PHOTOS: Celebrity splits of 2013 In divorce papers filed two weeks ago in New York, a no-fault state when it comes to ending a marriage, Andrew Silverman chose to go the fault route instead. He alleged adultery, and along with his future ex-wife listed Cowell as a co-respondent, the Post reported. What does that mean? Well, if the matter went to trial, Cowell could be called to testify in court about his relationship with Lauren and his financial situation (he's estimated to be worth $350 million), and there might be DNA tests involved to officially determine the father of the child, according to TMZ. The Silvermans have a 7-year-old son together - and Dad is seeking sole custody. But despite the recent nature of the divorce filing, "Lauren and Andrew have been unhappy in their marriage for some time, and their divorce has been in the works for a while," a source told Us Weekly. As their marriage deteriorated, she and Simon became close. The three have known each other for years, and have been repeatedly photographed together on Cowell's yacht. The former "American Idol" meanie was keeping mum when accosted by paparazzi as he left an "X Factor" taping Wednesday. The pregnancy has taken [Cowell] by surprise. He assumed she was using birth control," a source close to the entertainment mogul told the New York Daily News. Simon thought this was a casual relationship - friends with benefits. The pregnancy was not planned. Sources told the Daily News that Cowell only found out about the baby around July 10. "She is a total gold digger," one friend of the Silvermans told the Post, with another adding, "The pregnancy was by design." See Beyonce's adorable boot-inspired family portrait