European Union nationals could "tip the balance" towards Scottish independence if a second referendum was held before Brexit, academics have said.