Slovenia's prime minister is resigning after the country's top court annulled last year's referendum on a key railway project and ordered a new vote.