Saints' Sean Payton returns to work at Senior Bowl (AP) - New Orleans Saints coach Sean Payton likes to jot down to-do lists and notes on yellow pads. Payton has had time to fill a nice stack of those this season. The Saints coach returned to work Wednesday following a nearly yearlong suspension for his role in the team's bounty program with some catching up to do. The newly reinstated Payton watched the Senior Bowl practices and planned to meet later in the day with the staff to evaluate a defense that struggled during a 7-9 season and opened with four straight losses. Payton said there are plenty of challenges, from fixing the defense to replacing departed coaches and evaluating staff and players. Then, of course, there's getting ready for the draft and studying available free agents. Payton said during a 45-minute meeting with reporters that he and NFL Commissioner Roger Goodell agreed not to discuss specifics about whether there was an organized bounty system or of their 4-1/2-hour meeting on Monday. We're at a point where it's time for closure. It's time for us as a team, it's time for us as a league, to take this next step forward," Payton said. Goodell lifted Payton's suspension Tuesday, nearly two weeks earlier than expected. Payton characterized his relationship with the commissioner as "very good" and said the discussion also ranged to topics like player safety, youth football and the NFL's current crop of quarterbacks. He also said he feels that New Orleans fans will extend Goodell a "gracious" welcome during Super Bowl week despite any residual anger over the bounty saga. The coach watched the Senior Bowl's North squad practice from the stands, chatting with assistant head coach Joe Vitt and general manager Mickey Loomis. The NFL suspended Payton, Vitt, Loomis and four current or former Saints players, including linebacker Jonathan Vilma, after an investigation found the club had a performance pool offering cash rewards for key plays, including big hits. The player suspensions eventually were overturned. Payton said he wasn't involved in the appeals process. "I followed the process but I wasn't really privy to how those appeals went, how the meetings went, the specifics, and I'm still not," he said. Payton said his situation was different because as head coach, "it all falls under your umbrella." He said he wouldn't allow himself to get "disappointed or upset, and I was just going to be patient to keep waiting and making sure I did everything the right way." Loomis was suspended for eight games, Vitt for six and former defensive coordinator Gregg Williams remains suspended indefinitely Payton said he has not spoken to Williams and "I have no interest in talking to Gregg." He said he missed football "quite a bit" but that the hardest part was missing the daily interaction with colleagues like Loomis and Vitt and players like quarterback Drew Brees and defensive lineman Will Smith. Brees said Wednesday from the Pro Bowl that he is excited to have Payton return. He said he texted briefly with the coach and they plan to talk more during Super Bowl week. "That's just overtime," Brees said. It's not all going to happen in one sit-down. Saints guard Jahri Evans said he looked forward to catching up with Payton just to see how he's been doing. He called Payton "a special guy" for his play calling and coaching ability. Evans expects the Saints will benefit from Payton's observations from afar during this season. "I think he had a different view of the game and he's going to use that to his" advantage, Evans said. One advantage of the forced hiatus for Payton was getting to coach his son's youth league team for two practices and a game each week. "We got to the Super Bowl but we lost that game," Payton said. I needed those players on that team probably more than they needed me. He has had quite a bit of success with big boy football, too. Payton, who recently signed a five-year contract extension through 2017, has steered the franchise to its most successful period, including three NFC South division titles and four postseason appearances. Two of his teams advanced to the NFC Championship and the 2009 squad won Super Bowl XLIV. He said the challenges of rebuilding after a losing season will be painful and that his comeback won't guarantee an immediate return to winning. Payton said the Saints' struggles defensively and running the ball early in the season, along with problems with special teams at times, made it tough to win - with or without the bounty scandal. Payton, citing advice from Bill Parcells, noted that his presence won't mean an automatic return to big success. It was the Saints first losing season since 2007 "Coach Parcells said to me that this mind-set that you're back and all of a sudden you're back to winning 11, 12, 13 games doesn't exist, because you very well could win five the next year," he said. He's right. We've got a lot of things we've got to correct to get to where we want to go. That's just the truth. He said he hadn't started talking to Loomis regarding any possible changes in the coaching staff . The rest of the staff conducted its evaluation of the offense on Tuesday. Offensive line coach Aaron Kromer has left to become the Bears offensive coordinator. Defensively, the Saints allowed 7,042 yards, the most ever in a season, under first-year coordinator Steve Spagnuolo but also improved enough to later record the club's first shutout in 17 years. "This meeting we're going to have is going to be painful," Payton said. We've got a lot of tough meetings ahead. That's what 7-9 is. Those are our biggest challenges. Associated Press Writer Oskar Garcia in Honolulu contributed to this report.