Nicole Scherzinger and Paula Abdul not returning to The X Factor USA At this point, the only ones staying are Cowell and record producer L.A. Reid. Despite ratings of 13 million viewers, The X Factor USA has failed to match the success of American Idol, which Simon left to concentrate on developing the X Factor stateside. Earlier this month, the music mogul admitted he became "a bit too cocky" after a successful year in 2010, but that in 2011 his "ego was put in check."