MONTREAL -- A former prime minister and an ex-Quebec premier say they don't think the Senate expenses scandal is a good enough reason to get rid of the upper chamber.

Senate abolition and reform was just one of the topics that was broached by Brian Mulroney and Jean Charest on Friday during a panel discussion on federalism.

It was part of a two-day conference held to mark the 10th anniversary of the death of Claude Ryan, a strong federalist and former Quebec Liberal leader.

Charest, who was premier until his provincial Liberals were beaten by the Parti Quebecois in September 2012, made it clear that he was not impressed with all the talk about abolishing the Senate over alleged expense account misspending.

"Is there a business where, if there's a matter involving an expense account, where it would say, 'we're going to close the business?' " Charest wondered out loud. "That doesn't happen."

He conceded the Senate scandal was important, "but it's about an expense account . . . and we say we're going to scrap, we're going to abolish, the Canadian Senate."

Charest added that even if popular opinion suggested the majority wanted the Senate scrapped, he would not just bow to public opinion.

"I know my responsibility as premier of Quebec is to say that our fundamental interest is not to sell off the interests of Quebec because, in the news, a senator is cheating on an expense account," he told the audience.

Charest added that the Senate would never be abolished without the approval of the Quebec legislature.

Charest's comments came as a report on polling for the Conservative government suggested Canadians are fed up with Senate shenanigans.

Mulroney, who stepped down as Conservative prime minister in 1993, told the federalist crowd that he had appointed 59 senators when he was prime minister and knew something about it.

He pointed to a formula that he had proposed to the provincial premiers during discussions on the so-called Meech Lake constitutional accord in 1987.

Mulroney's proposal was that senators should be named from a list "of the best citizens" provided by a provincial government -- whether it was Liberal, Conservative or New Democrat.

He also agreed with Charest's statement that the Senate could not be abolished without the approval of the Quebec national assembly.

The former Conservative prime minister also applauded Liberal Leader Justin Trudeau's attempt at reform of the upper house when he removed senators from his Liberal caucus.

"The idea is good," Mulroney said. "It's not bad."

Mulroney's lack of partisanship continued at one point into personal style.

Mulroney, a long-time Conservative, joked about the red tie he was wearing and he couldn't help but point out that Charest, a former Liberal premier, was wearing a blue tie.

Also on Friday, a report on public-opinion polling ordered by the Harper government suggests the Senate scandal has frustrated Canadians. The newly released report indicated Canadians are fed up with "rich politicians" and their "lush lifestyles," and wondering how deep the flagrant spending abuse runs.

Those views dominate the polling and survey groups conducted by private pollster Leger last August for the Privy Council Office, the prime minister's own department.

The survey specifically asked about the economy, pipelines and telecom regulation -- but it was the Senate scandal that seemed to get people's blood boiling.

The $112,000 report included a telephone survey of 3,000 Canadians, and 12 focus groups in six cities, including a French-language group in Quebec City.

Leger was hired to ask Canadians specifically about the Senate scandal, which then involved allegations of spending abuse by Conservative senators Mike Duffy, Pam Wallin and Patrick Brazeau, all of whom were subsequently voted out of the current Senate session, losing their paycheques.

The report, dated Oct. 30, said the allegations regarding misspending were perceived as a sign that more accountability was needed for all use of public money.

The Harper government took a tough line on the Senate scandal in the fall session of Parliament, as senators were pressured to suspend Duffy, Wallin and Brazeau from their posts, despite concerns about due process.

The government kept up the heat in this week's budget, which promised legislation that would ensure senators do not accrue pension benefits while suspended. The new law is not expected to apply retroactively.

With files from Dean Beeby in Ottawa