The In camp were ahead of Leave by 52% to 48%, according to an Ipsos Mori survey for the Evening Standard.
To add further uncertainty to the outcome, 12% of those polled said they could switch sides as they headed for the polling booth.
With turnout key to the result, torrential rain storms in Remain stronghold London meant the unseasonally bad weather could deter voters casting their ballots.
Ukip leader Nigel Farage insisted his side could pull off an historic victory if “soft” In voters stayed home.
“Actually I do think we are in with a very strong chance, I do genuinely. But it’s all about turnout and those soft Remainers staying at home. I do think that the people who have decided to vote Leave have a real conviction and passion,” said Mr Farage.
However, early indications showed long queues snaking outside voting centres as the nation braved the storms to decide Britain’s future role in Europe.
But some polling stations were forced to close, and many others were flooded due to the heavy rain sweeping the capital.
Kingston upon Thames Council in south west London moved two polling stations after they became inundated with water.
Prime minister David Cameron ignored questions about the weather, saying only “Good morning” to the gathered media from across the world, as he and wife Samantha cast their votes at Methodist Hall in Westminster.
Labour leader Jeremy Corbyn arrived in a positive mood to cast his vote in his Islington constituency in north London.
Asked if he was feeing confident, Mr Corbyn smiled and said: “Extremely, it’s a very good day.”
Pro-Leave justice secretary Michael Gove said he was feeling “quite excited” as he accompanied his wife, Sarah Vine, to a polling station in North Kensington.
The referendum has seen one of the longest, and most personally bitter, campaigns in recent British political memory.
Both sides of the campaign have been locked in fierce fighting for months, and things came to a frenetic close on Wednesday as senior politicians criss-crossed across the country to try to sway undecided voters.
The prime minister and his Remain colleagues from across the political spectrum have warned of the potentially severe economic consequences of a Brexit vote amid fears of financial market turmoil and another recession.
But Leave campaigners, led by Tory heavyweight Boris Johnson, have urged voters to “take back control” of the country.
Mr Farage would not elaborate on his reasons for missing Wednesday night’s final Channel 4 debate of the campaign, saying only that it was “for family reasons”.
He came in for particularly stern criticism after unveiling a Brexit poster showing a queue of hundreds of immigrants arriving in Europe with the slogan “breaking point”.
Meanwhile, dozens of celebrities have intervened during the course of the campaign to make their feelings known.
Footballer David Beckham, James Bond actor Daniel Craig, and Harry Potter author JK Rowling were just three of the high-profile names to back the Remain campaign, while Leave won support from the likes of comedian John Cleese, former cricketer Ian Botham and former England football player Sol Campbell.