Anyone who wants to use public office for their own benefit should find a new line of work, Prime Minister Stephen Harper said Tuesday -- for the first time addressing a deepening scandal related to Sen. Mike Duffy's repayment of improperly filed expenses.

In a rare move, Harper invited reporters and cameras to witness him address the Conservative caucus before he flies to South America later in the day for trade meetings. He took no questions from reporters and wasn't in the daily House of Commons question period Tuesday afternoon.

"I don't think any of you will be very surprised to hear I'm not happy, I'm very upset, about the conduct we've witnessed, the conduct of some parliamentarians and the conduct of my own office," Harper said as he opened his speech.

Last week, CTV's Ottawa Bureau Chief Robert Fife reported that Harper's chief of staff, Nigel Wright, had written a personal cheque to Duffy for over $90,000, allowing him to repay the money he owed for improperly filed housing expenses.

Ethics Commissioner Mary Dawson confirmed late Tuesday that her office has launched an investigation into Wright’s cheque to Duffy under the Conflict of Interest Act.

“The Commissioner and her Office cannot comment on the examination, as all examinations are conducted in private,” Dawson’s office said in a statement.

The Canadian Press reported Tuesday afternoon that Liberals in the Senate are trying to trigger special parliamentary hearings in an effort to force Wright and other Conservatives to testify.

Liberal Senate leader James Cowan is expected to argue that the PMO violated the privileges of parliamentarians. If Senate speaker Noel Kinsella agrees, he could send the issue to a special committee.

Wright resigned Sunday , while Duffy quit the Conservative caucus on Thursday. Another Conservative-appointed senator facing an expenses audit, Pamela Wallin, recused herself from the Conservative caucus on Friday.

On Monday, sources told Fife that Harper’s former special counsel and legal adviser Benjamin Perrin helped draft a letter of understanding that called for Duffy to publicly declare that he would repay the money. In return, sources said, Wright would give a personal cheque to Duffy to cover the $90,000. Sources also said the agreement stipulated that a Senate investigation into expense claims would go easy on Duffy.

On Tuesday, Perrin said in a statement he was “not consulted on, and did not participate in, Nigel Wright’s decision to write a personal cheque to reimburse Senator Duffy’s expenses.”

“I have never communicated with the Prime Minister on this matter,” he added.

The opposition hammered the government about the $90,000 cheque and the agreement during question period Tuesday.

“Duffy says he agreed to ‘stay silent’ on the orders of the PMO. In exchange, the Prime Minister’s Office agreed to cover the cost of the senator’s fraudulent expenses,” NDP Leader Thomas Mulcair said. “Why were taxpayer-funded lawyers used to negotiate this secret backroom deal between the prime minister’s chief of staff and Senator Duffy? Was taxpayers’ money used to bankroll Senate-gate, yes or no?”

Foreign Affairs Minister John Baird responded, saying: “I reject much of the premise of his question. The government’s been very clear that the prime minister was not aware of this payment until media reports surfaced last week.”

In response to a separate question, Baird also denied that taxpayers’ money was used to pay Duffy’s expenses.

Liberal Leader Justin Trudeau accused the government of having “lost its moral compass.”

Trudeau asked: “Apparently when Conservatives break the rules, they get their debts secretly paid off by their friends in high places. It boggles the mind. Nobody over there even thinks anybody did anything wrong except get caught. When will they release this secret document, allow for a full investigation, and while they’re at it, how about apologizing to Canadians?”

Baird denied the existence of a legal deal.

“With respect to a legal agreement that the member opposite refers to, our understanding is there is no such agreement. This issue has been referred to two independent authorities, which will look into the matter and we look forward to them reporting back to Parliament and to Canadians.”

On Tuesday, Harper touted his own government's efforts to introduce accountability and transparency since they were first elected in 2006, saying "Canada now has one the most transparent and accountable systems of governance in the whole world and this is something Canadians are rightly proud of."

However, he said it is something Canadians can never take for granted.

Without getting into specifics about the recent controversy, Harper said Sen. Marjory LeBreton, leader of the government in the Senate, has been tasked with closing loopholes in the way senators claim expenses. Those not willing to play by the rules should get out, Harper said.

"Anyone who wants to use public office for their own benefit should make other plans or better yet leave this room," Harper said to rousing applause.

He also said that he "didn't get into politics to defend the Senate" and said the Conservatives have pushed to reform the Red Chamber but have been blocked by the opposition parties.

The NDP on Monday asked the RCMP to look into Wright's actions as well.

The New Democrats have called on Harper to give Canadians a public explanation of exactly what he knew and when.

Charlie Angus, the NDP ethics critic, said Harper missed an opportunity Tuesday to clear the fog surrounding the repayment of Duffy's expenses, calling it an "epic failure."

"When you write a secret cheque to a politician and there appears to have been perhaps an agreement to go easy on him in the Senate, that breaches all the ethical rules, that breaches the rules of any politician and it also may breach the Criminal Code," Angus told CTV News Channel.

"And this happened out of the Prime Minister's Office and the prime minister completely dodged that, he refused to take responsibility for it, he refused to talk to the media, he's refused to talk to Canadians."

On Tuesday, Minister James Moore told reporters that any Parliamentarian who doesn’t respect taxpayers’ money “should leave.”

Harper departed Tuesday afternoon for Peru, where he'll take part in bilateral trade discussions before heading to Colombia for talks on the Pacific Alliance trade group. Chile, Colombia, Mexico and Peru are permanent members of the trade group, while Canada has observer status.

However, Canada already has free-trade agreements with all four countries and critics have questioned the timing of the out-of-country trip with so much going on in Ottawa.