Analysts said the record 61% turnout reflected a widespread fear among liberal Serbs that a win for Mr Nikolic would stall reforms and complicate Serbia’s path to the EU.
“People realised it’s a very important election as depending on who wins, Serbia will decide its future orientation, to the EU or Russia,” said Djordje Vukovic of the CESID think-tank.
Analysts say the vote will determine the country’s future ties with the West as it grapples with the expected loss of its breakaway Kosovo province.
Both candidates will need to court third-party voters for the February 3 round. According to first exit polls, Mr Nikolic got 39.5% of the vote, versus 35.3% for Mr Tadic.
To win the run-off, the two candidates will try to attract third-party votes with promises of higher living standards, jobs and the defence of Kosovo, which is heading for independence, with western backing, within months.
Mr Nikolic, whose Radical Party supported the policies of late autocrat Slobodan Milosevic during the 1990s, puts his trust in Russia and favours a neutral stance between East and West.
He has toned down his rhetoric to appeal to moderates as well as the one-third of Serbs who live just above the poverty line.
He has rejected accusations of isolationism and war-mongering.
“If the EU wants to open up its doors and no longer impose obstacles, we will be glad to join the EU,” Mr Nikolic said on Sunday, vowing “no more humiliations”.
The main humiliation Serbs see coming from the West regards Kosovo, a UN protectorate since 1999 when NATO expelled Serb troops accused of atrocities against ethnic Albanians there during a guerrilla war.
Indications by Washington and most EU member states that they will recognise Kosovo as independent within months have irked Serbs who feel the country has paid enough for its role in the wars of the 1990s, and could cause a nationalist backlash.
Mr Tadic also opposes independence for Kosovo, regarded by Serbs as their historic heartland, but favours signing a first-level agreement with the EU even if the bloc takes over Kosovo’s supervision as a prelude to recognising the territory.
The incumbent, who defeated Mr Nikolic in a run-off in 2004 after his rival won the first round, has warned that a Nikolic victory would drive the country back to the Milosevic days.