OTTAWA -- The federal Conservatives return to the House of Commons today after a tumultuous two weeks that included losing one of their best-known MPs to a sexual-misconduct scandal.

But for leader Andrew Scheer and company, it's not Tony Clement's online activities, the potential spillover of harassment allegations against provincial Tory politicians, or even the growing popularity of former Tory Maxime Bernier's new political party that are keeping them up at night.

The Tory nightmare now is the fall of the NDP.

"That's where I'm actually worried, frankly -- from a strategic perspective, that's what I'm watching," said Alexandre Meterissian, a conservative strategist with Montreal firm Hatley Strategy Advisors. "The collapse of the NDP is a big problem for the party."

Meterissian in particular is concerned about a Mainstreet Research Poll published Nov. 14 that had the Liberals with a comfortable lead and the NDP down at 10 per cent, barely ahead of the Green Party. That was an unusually bad poll for the New Democrats but Meterissian said there is a downward trend for the party and leader Jagmeet Singh, who is struggling with caucus discontent and communications problems.

An NDP official was livid at the suggestion.

"If the Conservatives want to say it's the NDP's fault their message is failing to resonate with more Canadians, that's up to them, but it's a pretty feeble argument," said James Smith, the NDP's senior communications advisor.

"While the Conservatives are blaming others for their own problems, the NDP is focused on pushing this government to deliver solutions Canadians urgently need in their everyday lives like reducing the cost of prescription drugs and getting serious on the housing crisis."

The Liberals ability to win a majority in the last election however, came in large part because of the collapse of the NDP vote, most of which went to them.

Meterissian said the Conservatives need to figure out how to attract NDP voters in Quebec and northern Ontario, because otherwise those votes will all go to the Liberals.

Scheer said in an interview that going back into the House of Commons after a break week for Remembrance Day, his party is going to focus on being united and not dwelling on the difficulties of the last few weeks or even months.

Scheer said sexual harassment is not a partisan issue, all the parties have had to contend with it, and all the parties have to figure out how best to investigate and address it. Clement is the fourth MP since the last election to be forced out of a caucus because of allegations of improper behaviour. Two have been Liberals and one was a New Democrat.

"We've had our issue, we've dealt with it and now we're focusing on going back to holding this government to account and putting forward innovative policy options," Scheer said.

He acknowledged losing an MP as experienced and well-known as Clement is a bit of a blow. Clement has been a cabinet minister in the Ontario and federal governments, a leadership candidate and the party's justice critic.

"Obviously Tony held significant roles in our caucus," Scheer said. "That said, we have a great team with a lot of depth."

Clement -- who admitted to sending explicit photos to women online -- is just one of at least five conservative politicians and staff members at the federal level and in Ontario and Manitoba who have quit or were forced out in the last few weeks after allegations of sexual misconduct.

The publication last week of former Ontario Tory leader and former federal MP Patrick Brown's book, in which he threw deep shade at many of his former staff and fellow Tories, added to the disarray.

At best, the stories were a distraction for Scheer, whose major policy announcement on how the Conservatives would address gang and gun problems went largely under the radar.

The Tories almost had a big win against the Liberals when the government changed rules and sent child-killer Terri-Lynne McClintic back to prison from a healing lodge. But then came the news Nov. 13 that during the Conservatives' own time in office, 10 people convicted of killing children had been in such lodges for at least some of their sentences.

The Conservatives are also about to receive a report on how the party handled allegations in 2015 that then-MP Rick Dykstra had been accused of sexual assault. The party, including leader Stephen Harper, allowed Dykstra to stay on as a candidate and the report is to examine the propriety of that decision. (Dykstra has never been charged in the matter and denies the allegation.)

Meterissian said most of these stories are gossip that will not live long in the minds of voters.

The Conservatives are also, however, facing a challenge from the right in Scheer's one-time leadership rival Maxime Bernier. Bernier quit the Conservative Party of Canada to launch his own. Bernier's People's Party of Canada is growing in popularity, claiming more than 30,000 members signed up and more than $300,000 raised in just its first few weeks.

Bernier's party is polling between three and five per cent on most surveys at the moment. But a lot of Conservatives worry that even if the party can't win a lot of seats, it can steal enough votes from the Conservatives to help the Liberals win.

Meterissian isn't convinced of that, believing Bernier is really attracting people who otherwise would not vote.