BERLIN -- German Chancellor Angela Merkel won a fourth term Sunday, but now faces the tricky prospect of forming a coalition with two disparate new partners after voters weakened her conservatives and a nationalist, anti-migrant party surged into parliament.

Merkel's centre-left challenger, Martin Schulz, conceded his Social Democrats had suffered a "crushing election defeat," with projections showing the party's worst performance in post-World War II Germany.

He vowed to take his party, the junior partner in Merkel's outgoing "grand coalition" of Germany's traditionally dominant parties, into opposition.

"We have a mandate to form a new government, and no government can be formed against us," Merkel told cheering supporters. She added that it wasn't a "matter of course" to finish first after 12 years in power, and that the past four years were "extremely challenging."

Stressing that "we live in stormy times" internationally, she declared: "I have the intention of achieving a stable government in Germany."

The biggest winner was the 4-year-old Alternative for Germany, or AfD. It finished third after a campaign that centred on shrill criticism of Merkel and her decision in 2015 to allow large numbers of migrants into Germany, but also harnessed wider discontent with established politicians.

One of AfD's leaders, Alice Weidel, said it will provide "constructive opposition." But co-leader Alexander Gauland struck a harsher tone, vowing that "we will take our country back" and promising to "chase" Merkel.

Final results released shortly before 4 a.m. (0200 GMT) on Monday showed Merkel's Christian Democratic Union and their Bavaria-only allies, the Christian Social Union, winning 33 per cent of the vote -- down from 41.5 per cent four years ago. It was one of their weakest post-war showings.

Schulz's Social Democrats were trailing far behind, with 20.5 per cent support, down from 25.7 per cent in 2013 and undercutting their previous post-war low of 23 per cent eight years ago.

AfD won 12.6 per cent of the vote. It was followed by the election's other big winner -- the pro-business Free Democratic Party, which returned to parliament after a four-year break with 10.7 per cent.

The Free Democrats were Merkel's coalition partners in her second-term government from 2009-2013, but lost all their seats four years ago.

"In a country that is big on schadenfreude, our comeback is an encouraging message -- after failure, a new beginning is possible," party leader Christian Lindner told supporters.

The Left Party took 9.2 per cent of the vote, coming slightly ahead of the traditionally left-leaning Greens who won 8.9 per cent, completing a parliament that now has six caucuses rather than the previous four.

All mainstream parties have ruled out working with AfD and Merkel's conservatives won't form a coalition with the Left Party.

That means two politically plausible governments are mathematically feasible: continuing the "grand coalition" or a combination of Merkel's Union bloc, the Free Democrats and Greens.

That alliance is known as a "Jamaica" coalition because the parties' colours match those of the Caribbean nation's flag. It has been tried, with mixed results, in state governments but never in a national government.

The Social Democrats were adamant Sunday night that they wouldn't continue to serve under Merkel.

"It is completely clear that the role the voters have given us is as the opposition," Schulz said.

Referring to AfD's third-place finish, he said "there cannot be a far-right party leading the opposition in Germany."

Cobbling together a "Jamaica" coalition is likely to be time-consuming. The Free Democrats and Greens are traditional rivals. Four years ago, Merkel's conservatives and the Greens held exploratory talks on a two-party coalition but they came to nothing.

The underwhelming result also looks set to re-ignite pressure within Merkel's bloc for a tougher conservative image. Bavarian Governor Horst Seehofer, who feuded with Merkel over the migrant influx before putting aside their differences this year, said the outcome showed that the conservatives need to close "an open flank to the right."

Reiner Haseloff, the conservative governor of eastern Saxony-Anhalt state, said it would be wrong to ignore AfD's strong result.

"We need an answer -- there must be no democratic alternative to our right," he added. "As long as it is there, we haven't completely done our homework."

AfD is the first party to the right of the conservatives to enter parliament in 60 years.

Merkel pledged a "thorough analysis, because we want to win back AfD voters by solving problems, by taking account of their concerns and fears, and above all with good policies."

Outside AfD's election party in Berlin after the results were announced, at least 500 protesters shouted "all Berlin hates the AfD," "Nazi pigs," and other slogans, while several protesters threw bottles as police kept them away from the building.

Similar protests broke out in Cologne, Hamburg and Frankfurt but police said they were mostly peaceful.

Major Jewish groups expressed dismay at the AfD's showing, with the World Jewish Congress calling the party "a disgraceful reactionary movement which recalls the worst of Germany's past."

Mainstream parties' leaders vowed a robust response to AfD's entry into parliament. Greens co-leader Katrin Goering-Eckardt told supporters: "there will again be Nazis sitting in parliament."

"We will not let one single attack on German democracy stand," she said, to applause.

AfD leaders dismissed such talk. Asked by The Associated Press what signal the vote sends internationally, chairman Joerg Meuthen said: "That there is conservative politics in Germany again. And that there are patriots in the German parliament again."

"I want to emphasize that there is absolutely no risk of extreme right politics in the German parliament," he said.

Fellow European right-wing populists hailed AfD's performance. The Netherlands' Geert Wilders wrote on Twitter: "The message is clear. We are no Islamic nations."

"Bravo to our allies from AfD for this historic score!" tweeted Marine Le Pen, the runner-up in France's presidential election. "It's a new symbol of the awakening of the peoples of Europe."

------

Associated Press journalists Frank Jordans and Kirsten Grieshaber contributed to this story.