ECB narrowly avoids payment disaster after tech issue meltdown
Turmoil at the ECB on Thursday was linked to a piece of malfunctioning hardware.
The European Central Bank's (ECB) payments crash last week meant salaries and welfare funds were delayed for thousands of people. It could have been much worse.
If the same disruption had struck, or persisted into, the following day — the end of the month, and payday for many public-sector workers, pensioners and welfare recipients — the mayhem could have hit millions of people and businesses, and strained the banking system.
"If it had lasted until Friday, there would have been big risk-management questions for banks," said Alistair Milne, a professor of financial economics at Britain's Loughborough Business School.
"Bank risk managers would have to decide: Are we willing to credit the customer account on the trust that the money will eventually turn up?"
When the system went down on Thursday, the chaos that descended in the 10 hours it took to identify and fix the problem blocked the welfare payments of more than 15,000 mostly elderly and poor Greeks, a large number of salaries and pensions in Austria, plus several financial trades.
At the heart of the escalating turmoil was a piece of malfunctioning hardware, but it took hours for the ECB's technicians to spot the problem after an initial, erroneous diagnosis of database issues, according to the ECB messages and officials at the ECB and three other eurozone central banks.
This forced central bank staff across many eurozone countries to work through the night to fix the defective equipment and clear a backlog of transactions in time for pay day, according to officials who, like the other sources, requested anonymity to discuss this sensitive matter.
An ECB official told Reuters the affected hardware, which he declined to identify, did have multiple back-ups and the bank was analysing why they did not kick in.
The ECB had recently completed an overhaul of its payment system and crisis management before this incident, the official added, as recommended in a report by consultancy Deloitte following a string of outages in 2020.
The Target payment system, which handles trillion of euros of daily transactions, is so vital to financial and economic stability in the 20-nation eurozone that the locations of its four servers are a closely guarded secret.
The situation started going south shortly after 8am Frankfurt time on Thursday, when the ECB's system for settling financial trades, Target 2 Securities (T2S), crashed. The Target 2 (T2) network, which handles large payments between central banks and commercial lenders, followed suit two hours later, according to ECB updates to market participants.
Both T2 and T2S are run by the ECB with the central banks of Germany, Italy, France and Spain.
In phone calls that morning between crisis managers at the ECB and the other central banks, the blame was put on errors in the systems' shared database, which would make a transfer to a back-up version — or "failover" — impossible without copying over the same issue.
This required staff at the central banks to laboriously parse through transactions to identify the mistake while the system was offline.
At 11.30am, the crisis managers activated the Target systems' emergency channel for critical payments, such as those involving foreign currencies or margin calls.
This process, which requires participants to submit every transaction manually, is ill-suited for the hundreds of thousands of payments that go through T2 and T2S every day.
A few dozen payments went through in that way but all others had to be placed in a queue, waiting for the problem to eventually be fixed.
It was not until the afternoon that the ECB established there was actually what it later described as a "defective hardware component" at one of the system's four locations.
Staff then began moving all transactions to the back-up, or failover, shortly before 4pm. The move was completed at 5.15pm and settlement resumed at 6pm, lasting through the evening until a delayed midnight deadline, the ECB messages showed.
The officials at the ECB and three other central banks said work continued until the early morning to clear most of the transactions that had been queued up over the previous day.
Some transactions had been cleared too late for the banks to process in time on Friday, leading to the delays of thousands of people in Greece and Austria receiving salaries, pensions and welfare payments.
Brokers were also left fuming over delayed trades, according to three market sources who declined to give details about the affected transactions.
In a message after the event, the ECB described the breakdown as a "major incident" that had "adverse consequences for market participants as well as for their clients" and said it had started a "thorough analysis" of the episode.
CONNECT WITH US TODAY
Be the first to know the latest news and updates
