Joaquin Phoenix opens up about growing up in the Children of God cult Joaquin Phoenix attends the screening of "Inherent Vice" during AFI FEST 2014 presented by Audi at the Egyptian Theatre on Nov. 8, 2014, in Hollywood, California. Jason Kempin/Getty Images Though the typically reclusive Joaquin Phoenix rarely talks about his unconventional upbringing, the Oscar-winning actor gets candid with Playboy about his experience being born into the Children of God cult, as well as his infamous 2009 "Late Show with David Letterman" appearance. Phoenix's parents joined the controversial religious group Children of God in the early 1970s and traveled throughout South America along with him and his siblings -- including the late River Phoenix, who died of a drug overdose in 1993 -- but left when they became disenchanted with the group. "When people bring up Children of God, there's always something vaguely accusatory about it," the "Her" star, now 40, tells the magazine. It's guilt by association. I think it was really innocent on my parents' part. They really believed, but I don't think most people see it that way. I've always thought that was strange and unfair. "I think my parents thought they'd found a community that shared their ideals," he further explains. Cults rarely advertise themselves as such. It's usually someone saying, 'We're like-minded people. This is a community,' but I think the moment my parents realized there was something more to it, they got out. Phoenix also comments on his 2010 faux documentary "I'm Still Here," in which he pretended to reinvent himself as a rapper. This being the notoriously intense Phoenix, he took it to the next level by making one of the most bizarre appearances ever on "Late Show with David Letterman," which lots of people didn't take too kindly to. "David Letterman was not in on the joke," he reveals. But look, David Letterman is one of the smartest guys on television. There's no way that guy doesn't know what's going on in some way. That's what I'll say about it. Despite the controversial reception to the Casey Affleck-directed film, he calls the entire experience "liberating." "Unlike when you're acting and everyone is there to support you and you can do take after take, when I did those live music shows and the movie, the safety net wasn't there," he says. I can see how people felt like they'd been duped. I think I would have had a similar reaction. I totally understand people getting defensive and scared because they don't want to be taken advantage of. I think now everyone knows it was never our intention to attack people. We were clearly attacking ourselves. As for his trouble dealing with fame, Phoenix compares it to being an attractive woman. "I think I know what it's like to be an attractive woman," he says. I think that's basically what the experience [of fame] is, right? But that sycophantic energy is uncomfortable to be around. Nobody wants to experience that. Though as he gets older, Phoenix admits he feels like he knows less and less. "All I know is that I've been fortunate, and my good fortune continues," he reflects. Other than that, the older I get, the more I know that I don't f**king know anything at all. .... I'm trying to get better at being open to the mystery of it all. Playboy's December issue hits newsstands this Friday. In 2001, ET caught up with the endearingly off-beat actor at the Academy Awards, when he admitted he can't tie a tie! Check it out in the video here.