Prime Minister Stephen Harper’s chief of staff secretly intervened to help Conservative Sen. Mike Duffy pay back tens of thousands of dollars in improperly claimed expenses while an external audit was still underway, CTV News has learned.

Two months before the audit was released, Harper’s top advisor Nigel Wright had a PMO lawyer work on a letter of understanding with Duffy’s legal counsel.

Sources told CTV's Ottawa Bureau Chief Robert Fife that the deal involved Duffy reimbursing taxpayers in return for financial help and a promise from the government to go easy on him.

At one point, Duffy expected the PMO to cover all of the money he’d improperly claimed – more than $90,000.

In a Feb. 20 email, Duffy said Wright worked out a “scenario” where all of his claimed living expenses would be covered, including “cash for the repayment.”

Two days later, Duffy publicly vowed to reimburse the taxpayers, saying he "may have been mistaken" when he filled out Senate housing allowance forms, claiming a cottage in Prince Edward Island as his primary residence.

In March, Duffy repaid $90,172.

“If there is some kind of agreement that somehow Senator Duffy is going to be compensated, then I think Canadians would be appalled,” Liberal MP Ralph Goodale said.

“It doesn’t pass the smell test and the Prime Minister’s Office needs to come clean.”

NDP MP Mathieu Ravignat called the matter “troubling.”

“This is supposed to be an independent forensic audit. There shouldn’t be interference,” he said.

The PMO and Duffy released identical statements Tuesday, saying: “Mr. Duffy had paid back the expenses in question – and no taxpayer resources were used.”

A senior official refused to comment when asked if anyone helped Duffy with the repayment, including a loan from the Royal Bank.

The Conservative Party told Fife late Tuesday that party funds were not used.

Records show that Duffy is carrying a $360,000 mortgage on his Ottawa-area home. His P.E.I. residence is mortgage-free.

Duffy, Liberal Sen. Mac Harb and Independent Sen. Patrick Brazeau were all audited over concerns about their housing allowance claims.

Senators can claim an annual housing allowance of up to $22,000 if their primary residence is more than 100 kilometres outside of Ottawa.

Harb has claimed as his primary residence a bungalow in Westmeath, Ont., but neighbours told CTV earlier this year that no one seems to live in the house year-round. He was ordered to repay $51,482, including interest, for the period from April 1, 2011 to March 31, 2013.

Brazeau claimed his father’s address in Maniwaki, Que., as his primary residence, despite the fact that it is believed he lived in a home in Gatineau, a short drive from Parliament Hill. He must pay back $48,744, including interest.

Both Harb and Brazeau have said that they will fight the Senate’s repayment orders.

Meanwhile, Duffy refused to co-operate with the auditors. In one email he wrote: “I stayed silent on the orders of the PMO.”

After the Deloitte audit was released, the Harper government praised Duffy for showing "leadership" and voluntarily paying back the money.

But Liberals accused the Tories of whitewashing Duffy’s expense reports. They also questioned why the audit of Duffy’s expense claims did not mention that the Senate rules were clear about primary residence claims. That was mentioned in Brazeau and Harb’s audits.

Both Liberals and the NDP also accused the Conservative chairman of the internal economy committee, Sen. David Tkachuk, of giving Duffy a heads up about improperly claimed per diems. Duffy reimbursed the Senate for that as well.

After the Deloitte audit was tabled in the Senate, government leader Marjory LeBreton said the matter would not be referred to the RCMP.

But CTV News later reported that the Mounties are poised to open a criminal investigation into the senators' expenses.

With a report from CTV’s Ottawa Bureau Chief Robert Fife