A ferry carrying more than 100 people started drifting off the Scottish coast after it suffered engine trouble on its way to Northern Ireland, the coastguard said.