Royal Bank of Scotland pursued last-minute settlement talks with a group of investors to avoid a potentially embarrassing trial over allegations the lender misled them about a 2008 capital increase.