Conservative MP Larry Miller is apologizing and retracting some of his comments after he told a radio station that Muslim women should "stay the hell where you came from" if they wish to keep their faces covered during the citizenship ceremony.

Miller made the comments Monday to a call-in show on CFOS Radio, which is in his Ontario riding of Bruce-Grey-Owen Sound.

The host, Bill Murdoch, asked about the recent debate over whether women should be required to uncover they face when they swear the oath of citizenship.

A caller named Joseph said he thinks "that's wrong," and Miller said he agreed.

"You know, like frankly, if you, if you're not willing to show your face in a ceremony, that you're joining the best country in the world, then frankly …" Miller said, before the caller interrupted to say, "send ya back."

"Yeah," Miller said. "Frankly, if you don't like that or don't want to do that, stay the hell where you came from, is the way."

Miller went on to say that "I think most Canadians feel the same."

"I'm so sick and tired of, of people wanting to come here because they know it's a good country and then they want to change things before they even really officially become a Canadian," he said.

After the audio of his comments made the rounds on social media Tuesday morning, Miller issued a statement to say that some of his comments were "inappropriate."

"I stand by my view that anyone being sworn in as a new citizen of our country must uncover their face," the statement read. "However, I apologize and retract my comments that went beyond this."

The Prime Minister's office issued a brief statement to acknowledge Miller's apology for "inappropriate comments that went beyond our clear position."

The statement went on: "We believe most Canadians, including new Canadians, would find it offensive that someone would cover their face at the very moment they want to join the Canadian family."

Amira Elghawaby, human rights co-ordinator with the National Council of Canadian Muslims, told CTV Power Play that Miller’s comments were anti-immigrant and anti-Canadian.

“(Canadian) values include freedom of choice, freedom of religion, freedom of expression,” she said. “These women do show their face for security purposes, so there’s really no harm for anyone at all.”

Elghawaby said she worries such comments from political leaders could expose Muslims to harassment and discrimination

“I’m not really sure why the leaders keep bringing this up, keep fanning the flames with this kind of rhetoric,” she added. “It’s really disturbing.”

Elghawaby does not wear a niqab, which covers the lower portion of the face and, but does wear a hijab, which covers only the hair.

“It’s not something I, for example, believe is intrinsic in my practice,” she says. “But it’s just not my place or anyone’s place -- or the state’s place -- to dictate what women should or should not be wearing.”

Quebec Liberal MP Emmanuel Dubourg, who was born in Haiti, issued a statement Tuesday calling Miller's comments "incredibly offensive.

"As an immigrant, I was shocked and appalled to hear these divisive and disgraceful comments from another MP in Mr. Harper's Conservative Party," Dubourg said. He added that "Canada needs new leadership that recognizes we are strong not in spite of our differences, but because of them."

Later Monday, Miller told the radio station that a man who self-identified as Muslim showed up at his constituency office wearing camouflage and carrying a flag on his arm, objecting to the MP's comments from earlier in the day.

Miller said he can tolerate someone "expressing their displeasure" about his views, but "he scared the ladies in my office, and I'm not going to tolerate any of that from anybody."

Office staff called Owen Sound police over concerns about the man's attire.

In a statement, police said they identified and spoke to the the 44-year-old man, who "apologized for causing such alarm that police would have to be contacted."

The man admitted that he was "angry and emotional" about Miller's comments, but only wanted to speak with him.

"There was no criminal offence committed and as a result no charges," the statement concluded.

This isn’t the first time Miller has made controversial remarks.

On Feb. 7, 2012, during a speech in the House of Commons on the Conservatives’ bill to end the long gun registry, Miller explained “How we got here” by referring to comments apparently made by Allan Rock in which the former justice minister had stated that the only people in Canada who should have firearms are police and the military.

Miller went on to comment: “Sound familiar? 1939? Adolf Hitler. Mr. Speaker, this statement is the reason that we are here today.”

Miller later rose in the House to apologize for making a reference to Hitler.

“While the reference is to the gun registry, and what this evil guy did to perpetrate his crimes are very clear, it was inappropriate to use his name in the House and I apologize to anybody it may have offended.”