OTTAWA -- Prime Minister Stephen Harper says the three words from the Mike Duffy scandal that have hung over him - "good to go" - were not his.

Those three words are likely to be repeated in the coming days as Duffy's trial returns to an Ottawa courtroom, with Harper's former chief of staff Nigel Wright a key witness scheduled to testify this week.

It was Wright who wrote those words in an email to fellow staffers in the Prime Minister's Office on Feb. 22, 2013, when the discussion in Harper's office was about using Conservative party funds to cover Duffy's questionable housing claims. At the time, it was thought that Duffy's expense tab was only $32,000.

The email chain, outlined in RCMP court documents, show Wright wanted to speak with Harper "before everything is considered final."

Then the notorious words that the opposition has used to repeatedly try to link Harper to the Duffy affair: "We are good to go from the PM."

"The words you're quoting are not my words. They're somebody else's," Harper said when asked about the Duffy trial at a campaign event in Ottawa on Sunday.

That plan, outlined in court documents, fell through when Wright and the Conservatives' chief fundraiser, Sen. Irving Gerstein, found out Duffy owed more than $90,000. That's when Wright went rogue, Harper maintained.

"I've said repeatedly, and I think the facts are clear, I did not know about that Mr. Wright had made a payment to Mr. Duffy, and as soon as I learned that, I made that public and Mr. Wright has been clear about that," Harper said.

He also added that the RCMP in the same court document, said there was nothing to suggest the prime minister knew of the $90,000 payment Wright gave Duffy. That sparked applause from the partisan supporters in the room.

That payment was part of a plan to have Duffy repay the expenses he charged for his long-time Ottawa-area home, and in return keep Duffy out of trouble with a Senate-ordered audit of his housing expenses.

Some Tory supporters have questioned the prime minister and the details of the Duffy scandal in letters to Harper's office. The opposition have also tried to turn Harper's ballot box question of leadership against the incumbent prime minister, saying that the Duffy affair - among others in the Conservative ranks - are symptomatic of a government that has become tired, disconnected and doesn't see itself as being accountable for its actions.

"I don't for a moment pretend, no leader should ever believe that everything is going to go well," Harper said.

"But when bad things happen, what a leader does is he takes immediate action to make sure people are held accountable. That's exactly what I did in this case and that is exactly what is happening."

Duffy now faces 31 criminal charges stemming from his Senate expenses, including that he allegedly billed taxpayers for travel to campaign for the Tories in the 2011 election. The RCMP didn't charge Wright for his role in the affair, saying the evidence didn't support laying charges.

Duffy's trial wasn't originally supposed to coincide with the federal election campaign, but that changed when it took longer than expected to go through witnesses and Harper kickstarted the official campaign period one month earlier than anticipated.

Harper, however, seemed unfazed by questions about Duffy.

"This is the purpose of the (legal) process: Those who are responsible will be held accountable and I'll let the court do its work," he said.