Prime Minister Stephen Harper has countered claims made by Sen. Mike Duffy, saying he never threatened to expel the embattled senator from the upper chamber if he failed to repay ineligible expenses.

Harper did say, however, that he informed the entire Conservative caucus, including Duffy, that any improper expense claims should be repaid.

“You’re darn right I told him to repay his expenses,” Harper said in the House Wednesday.

"I told our entire caucus and staff that my view was these expense claims were inappropriate and they should be repaid.”

Harper maintained that he never said the spending scandal was about perception among the Conservative Party’s “base,” as alleged by Duffy.

"What I said to our caucus is ‘you cannot claim an expense you did not incur.’ That is not right, that is not proper and that will not be tolerated in this party."

He said when inappropriate expense claims are made, he expects "corrective action" to be taken.

"And if it is not taken, the person who does not take corrective action could not expect to continue to sit as a member of the Conservative party."

Harper was asked again of his knowledge of a $90,000 cheque made to Duffy by his former chief of staff Nigel Wright to cover the senator's ineligible expenses. He repeated that Wright acted alone and said if he knew of the payment that he “would have never approved such a scheme.”

Speaking to the Senate on Tuesday, Duffy said he was under immense pressure to accept the funds from Wright -- and when news of the $90,000 payment appeared in media reports, he was ordered to resign from the Conservative caucus immediately.

Duffy said he had followed the Senate rules when it came to his expenses, and insisted he had the documentation to back it up.

On Wednesday, NDP Leader Thomas Mulcair said Duffy's speech showed Harper is "directly" implicated in the ongoing Senate expense scandal.

"This is a profound scandal that directly implicates Stephen Harper," Mulcair told reporters in Ottawa. "This is no longer a question of Nigel Wright, or Mike Duffy: this is about Stephen Harper."

Mulcair said Harper has dodged questions about his knowledge of the $90,000 cheque. For months, Harper has maintained that Wright acted alone in writing the cheque to Duffy.

However, CTV News has learned that at least 13 Conservative insiders were aware of the $90,000 payment.

"No one else in his office knew, those were his exact words," Mulcair said. "We now know that he's got a baker's dozen around him who were actually implicated in this thing, and there's one thing they have in common: they all work for Stephen Harper."

Mulcair later told CTV’s Power Play that everyone familiar with Harper knows he’s “a control freak.

“He knows everything that goes on around him. So do you really think it’s plausible that every single person in his office knew this except him?”

Harper disputed claims that other upper-level Conservative insiders knew about the payment.

"There wasn't 13 people. Wright made the decision himself," he said. "He took responsibility and he is being held accountable."

Mulcair said the Senate scandal is a “serious” issue for the prime minister.

“What we’ve had is a constantly evolving version of facts, all of which can’t be true at the same time because they contradict each other,” he told Power Play.

Liberal Leader Justin Trudeau said Wednesday that it's time for Harper to show "leadership" in the matter.

"The Prime Minister has consistently avoided answering questions in a fulsome and open way. (He) has consistently avoided taking any responsibility for the mess that the Conservative Party and the Government of Canada of created around this issue," Trudeau told reporters.

He's calling for all the parties involved in the spending scandal to testify under oath.

"That's unfortunately, regrettably, now the only thing that I believe is going to restore confidence to Canadians across the country who are extremely disappointed on the lack of leadership from this Prime Minister."