(CNN) A small group of House Democrats are publicly saying it's time for House Minority Leader Nancy Pelosi to step aside as the face of the party ahead of the 2018 midterms following Tuesday night's disappointing loss in a special election for a Georgia House seat.

"We need leadership change," New York Democratic Rep. Kathleen Rice told CNN outside the House chamber. "It's time for Nancy Pelosi to go, and the entire leadership team."

Rice attended the closed-door House Democratic caucus meeting on Wednesday morning but said she did not raise the issue in the session, and no other members brought up the idea of a leadership change. "I think that people were in shock" after hearing a report from Pelosi and the head of the House campaign arm that Democrats lost, but were doing better in contests in other GOP districts.

Massachusetts Democratic Rep. Seth Moulton echoed Rice's complaints in an interview with CNN's Jake Tapper. "This is certainly something that we have to discuss because it's clear that, I think, across the board in the Democratic Party we need new leadership. It's time for a new generation of leadership in the party."

Asked about her colleagues urging her to step down, Pelosi waved off the question and said she would address it at her weekly press conference on Thursday. But pressed if she had any plans to go anywhere, she replied "no."

Pelosi didn't directly respond to Rice's call, but stressed that now was the time for Democrats to come together.

"I respect the comments of some in our caucus, but right now we must be unified in order to defeat Trumpcare," the minority leader said in a written statement to CNN.

In a written statement the California Democrat didn't directly respond to the criticism, but stressed that now was the time for Democrats to come together.

A Rice spokesman clarified to CNN that the congresswoman was referring to the top three leadership positions: minority leader, Democratic whip, and assistant Democratic leader. Pelosi, Steny Hoyer of Maryland and Jim Clyburn of South Carolina were elected in November to those positions for the current session of Congress.

The moderate New York Democrat said she had talked to a number of House Democrats since last night's defeat and some expressed concerns about keeping the same leadership team. She said she is not interested in running for Pelosi's post or other leadership posts, but said there are discussions among some Democrats about next steps. She didn't call for Pelosi's immediate ouster, but said that she hopes Pelosi will ultimately decide not to run for another term as the top Democratic leader in the next Congress and several others emerge as alternatives to lead the caucus.

"There are about a handful of people who are now seriously considering it," Rice said.

Rice has been a vocal critic of Pelosi's since the 2016 election and was supportive in the fall when Ohio Democratic Rep. Tim Ryan challenged her for the top leadership position. Ryan lost that internal election, receiving only 63 votes from the 194 members.

Tuesday's stinging loss in Georgia, where Democrats spent tens of millions of dollars has many in the party frustrated, with most griping privately that their current strategy isn't working.

"Look we need to win, everything else is bulls***," Democratic Rep. Sean Patrick Maloney of New York told reporters leaving the meeting.

"This is not about me," Rice explained. "This is about being able to take the Democratic party in a direction that is actually going to help us win seats and get back into the majority. We need a vision -- where are we going? And we need a message -- how are we going to get there? We don't have either one of those. We just don't have either one of those in the present leadership."

Ryan told reporters Wednesday "there's a level of depression" among members and said bluntly, "Our brand is toxic."

Several House Democrats believe the GOP playbook of linking Democratic candidates to Pelosi is hurting.

"They tried it once and it worked. It's like the gift that keeps on giving," Rice said.

Pelosi spokesman Drew Hammill brushed aside that argument, saying about the GOP tactic, "the politics of personal destruction has been their hallmark. "When you are effective, you are a target. It goes with the job."

Rice said she hoped that Pelosi would ultimately decide on her own to announce she won't run for re-election following the 2018 midterms to remove that issue and demonstrate there are other Democrats running for the post.

Pelosi told members at the Wednesday morning meeting about Georgia, "unfortunately this is a loss for us, but it's not good news for them," referring to Republicans.

She and Rep. Ben Ray Lujan of New Mexico, chair of the House Democrats' campaign arm, stressed that in all of the special elections that took place in solidly red districts Democrats had significantly narrowed the gap. They pointed out that there are 70 more House seats that are more competitive than the one in the Atlanta suburbs that the GOP retained on Tuesday. Lujan distributed a memo to members that declared "the House is in play" and ticked through polling and recent trends.