Royal Bank of Scotland Group, Britain’s biggest taxpayer-owned lender, is being investigated by the Federal Reserve and Justice Department for potentially violating sanctions against Iran.