WASHINGTON, Nov 14 (Reuters) – A massive, nearly five-hour shutdown of the US aviation system that has disrupted more than 20,000 flights since Friday may have been the fault of a single router installed by government contractor L3Harris, government officials said Saturday.
The incident is the latest painful example of the aging US air traffic control telecom system and the struggles to replace systems while keeping the old communications working.
The FAA’s aging telecom systems in recent years have failed after errant fiber cuts, accidental deletion of files and a strong chemical smell caused by a melted component. The system will fail again until it is properly upgraded, officials said.
“This was an FAA system that didn’t meet the moment,” Transportation Secretary Sean Duffy said Saturday.
Duffy said on Friday L3Harris was conducting routine maintenance to switch telecommunication lines from old copper wire to fiber at at least one remote location. Officials told Reuters there may have been problems with routers installed in both Arkansas and Texas.
“We’re working to determine whether we had incompetence, meaning was it user error or installer error from L3Harris, or did they install a faulty router? We haven’t determined what the fault was,” Duffy said.
L3Harris said it took the issue seriously and was “supporting the FAA in its investigation to understand the conditions that led to the Nov. 13 ground stop of our nation’s air travel.”
L3 noted it has completed more than 70% of the planned airspace communications conversion from legacy copper wires to a fiber-optic system.
The FAA spends $1 billion annually to maintain its old copper wire system and is racing to replace it with more modern fiber-based communications.
“We need L3Harris or whoever provided that router to us to do a better job,” Duffy said, noting that “virtually every single institution in America has gone from copper to fiber, from analog to digital.”
The issue began overloading communications at 17 of the FAA’s 22 en route centers across the country, which prompted the FAA to issue a nationwide ground stop because it could not identify a root cause.
Duffy said the new routers started flooding other routers with over 1,000 connection attempts every four to five seconds, which sparked communications issues with planes and radar at FAA facilities.
Bedford said the FAA had to take the action when it lost “the ability for those routers to communicate data.” The FAA is still trying to understand the root cause of what went wrong.
The FAA said communications did not go down but led to instability in data at the centers. It did not lead to centers going dark. The routers being replaced are about 25 years old, Duffy said, and were supposed to be replaced in 2006.
NOT THE FIRST TIME
Small mishaps leading to catastrophic consequences are not new to the US air traffic control network.A 2023 FAA nationwideground stop that lasted 90 minutes and disrupted 11,000 flights was caused by a government contractor unintentionally deleting files.
That led to the first nationwide ground stop since the hijacked airplane attacks on September 11, 2001.
Friday’s outage hit as a surge in travelers began weekend travel. Noting the poor timing, Duffy said they would reevaluate when work is done at smaller facilities to prevent future disruptions.
The FAA got $12.5 billion to upgrade its badly aging air traffic control and telecom system from Congress and Duffy has asked for $50 billion more for air traffic control and airports.
The FAA only last week said it would replace by the end of 2028 outdated floppy disks used for weather and other critical information at more than 400 air traffic control facilities.
The FAA relies on a telecoms system it is racing to replace as well as decades-old Compaq computers and replacement parts bought on eBay.
In September, thousands of flights across the densely populated Northeast were delayed or canceled due to the dual failures of a circuit at a Philadelphia air traffic facility and the accidental severing of a fiber-optic cable in New Jersey by a construction crew.
The FAA has also been working for years to end a long-ridiculed, decades-old practice of air traffic controllers using paper flight strips to keep track of aircraft. But adopting the change, which started in 2022, at 49 major airports will take until at least 2028.
(Reporting by David Shepardson; editing by Chris Sanders and Franklin Paul)
