Immigration Minister Jason Kenney denied Sunday any political involvement in the caregiver controversy surrounding Liberal MP Ruby Dhalla.

On CTV's Question Period, Kenney scoffed at the suggestion, made by Dhalla's lawyer, that there may have been some sort of political conspiracy to bring down the Brampton, Ont., MP.

"I don't know what conspiracy this would be," he said. "Between the Conservative party, the NDP, three Filipino nannies and the Toronto Star? I think it undermines credibility when people start talking about conspiracies."

This past week, three foreign workers came forward to allege they all worked illegally at Dhalla's family home, and were mistreated and underpaid. None of the allegations have been proven in court.

Dhalla's lawyer, Howard Levitt, said Friday the allegations may have been part of an orchestrated attempt to ruin his client's political career.

"There is no truth to them," he said. "The only question is, who's really behind them and who orchestrated or assisted or enabled these former employees of her brother to suddenly come forward one year after the last of them worked providing care for her mother?"

The Toronto Star was the first to report on the controversy. Two of the nannies alleged they had their passports confiscated during their time working for Dhalla's family in early 2008. They also alleged they were forced to work 12 to 16 hour work days performing many non-nanny jobs such as polishing shoes, shoveling snow and cleaning family-owned chiropractic clinics.

They claim they were paid $250 a week, but no overtime.

A third woman also came forward, alleging she too was overworked during her brief time with the Dhalla family in 2008.

Dhalla denied those claims on Friday and said that the "false and unsubstantiated" allegations have been troubling and hurtful to her family.

"Anyone that has ever entered our home has always been treated with love, with care, with compassion and respect," she said.

On Sunday, Kenney said he didn't want to comment specifically on the Dhalla case, but said, as immigration minister his "broader concern is ensuring that we protect the rights of caregivers."

He added that "sometimes people fall into vulnerable situations."

Kenney also said he only became aware of the case after the Toronto Star released its report. He also said that, to the best of his knowledge, he has never met the caregivers. But he said it may be possible they attended some of the roundtable discussions he's held for caregivers in the Greater Toronto Area.

However, he did add that his department has indicated to the nannies' legal counsel that "if they're acting as whistle blowers that will not be held against them, that if they're providing information about offences we're not going to penalize them."

With files from The Canadian Press