Unionists must present options to maintain Scotland's relationship with the European Union or risk the break up of the UK, a think tank has warned.