“I just couldn’t talk to people all day long. I’m not very good at talking,” the America’s Got Talent judge and executive producer revealed

Simon Cowell. Photo: Amy Sussman/WireImage

Simon Cowell does not believe he’s talk show host material.

In a recent interview with E! News, the America’s Got Talent judge revealed that he has no plans to host his own daytime talk show — and even backed out of a previous offer to do so.

“I got to the point where they built the set and I started to get anxious and then really stressed out, and I just walked out the meeting,” Cowell, 63, told the outlet after being asked if he would consider hosting a daytime talk show like his American Idol discoveries Kelly Clarkson and Jennifer Hudson.

He continued, “I just said I literally couldn’t do this. I just couldn’t talk to people all day long. I’m not very good at talking.”

While he may not be the right person for the gig, Cowell continues to be great at identifying talents and is resuming his role judging alongside Howie Mandel and Heidi Klum on AGT: All-Stars, a new extension of the beloved series on NBC, which premiered on Monday.

America’s Got Talent: All-Stars. Trae Patton/NBC

Speaking previously to PEOPLE of the show, Cowell said, “It was terrific and it was really fascinating to see people who’ve competed on the American show and done really well, competing with people who won the show in other countries.”

“The dynamic was fascinating … and what really was great and cool about the show was that so many people decided to come back and compete,” he added.

Much like the original series, AGT: All-Stars — which is hosted by AGT emcee Terry Crews — sees dozens of variety acts compete for the winning title. However, this show is different in that it features 60 of Got Talent franchise’s past winners, finalists, fan-favorite acts and viral sensations as they compete to be the best of the All-Stars.

“We were reunited with so many great people, and then met so many great people, and there were a lot of surprises on the show as well,” Cowell told PEOPLE in November. “I really, really enjoyed it.”

In addition to AGT, Cowell served as a judge on American Idol from 2002 until 2010. During his time on the singing competition series, he helped launch Clarkson, 40, and Hudson, 41, among other notable names, into stardom.

In September 2019, Cowell visited the The Kelly Clarkson Show beside his fellow original Idol judges Paula Abdul and Randy Jackson and reminisced about their time on the competition show, as well as if they’d ever sit together on the judge’s panel again.

“It does make me think,” Cowell said. “If I had one wish it’d be to make another show with us again. That would be my number one wish.”

Cowell also stopped by Hudson’s self-titled talk show in September to talk about her time on Idol, which shockingly ended early after her performance of Barry Manilow’s 1976 single “Weekend in New England.”

Cowell believed poor song choice may have led to Hudson’s demise on the show, telling her, “That night, I will never forget… I remember thinking, ‘This is not a great song.’ Wasn’t your fault. And then, of course, what happened, happened.”

Elsewhere in the interview, Cowell credited Hudson’s personality for making Idol’s early seasons such a success.

“It was the combination of talent, determination and real personality, and even though we had that kind of banter, you and I, it was always that … I always knew how determined you were and you were funny and you took it with grace,” he said. “You kind of got it, I always thought that about it. You got it.”