Royal Bank of Scotland is considering merging its troubled Ulster Bank business with Irish rivals, according to a report.