Oct 24, 2023·edited Oct 25, 2023Liked by N.S. Lyons
For several years I was a principal engineer at Amazon and I was a periodic member of the "availability team" which did post-mortem analysis of every event that caused the web site to go down. Amazon is/was the most operationally competent company I have ever worked for but the scale at which they operated required a level of complexity that no one could really get their arms around. No one fully understood the inter-relationships and dependencies between all the micro-services. The system required intense, continuous automated monitoring to achieve our goals in terms of uptime/availability. (At one point, fully 20% of the entire system's compute capacity was being consumed merely to process faux requests that checked on whether the system was up.) Everything you say about the challenges of complex systems is true. But I will offer a couple of other observations.
First, when faced with complex systems, human beings have a strong propensity to develop superstitious explanations for the visible phenomena they observe. (The last talk I gave to the engineering organization before leaving Amazon was called "Superstitious Architectures: How to Avoid Them".) Prying understanding from complex systems is hard and people are often happy to settle for superstition. (I offer as evidence a lot of what the lay press is saying about AI right now.) What this means in practice is that complexity causes human actors to operate in a state of greater general ignorance. Not a happy circumstance when the sh*t hits the fan.
Second, people are being driven to despair of their own human agency as their dependence on technology they can never understand grows. This is not entirely accidental I suspect and is probably even a little sinister. This has the effect of increasing dependency within the general population and reducing the sum total of initiative in the population at large.
Your advice to focus on simplifying our own lives is wise and even prescient I suspect.
Not Amazon, but a couple of tech orgs as well. And wow, well put. Was not familiar with the thought of "superstitious architectures." Nicely sums it up. Hell, I notice it in my relationship with cooking. Until getting into it a couple of decades ago the combo of heat/cold, spices and time was all magic to me. Only now do I see it without superstition, and it took years.
The lucky accident of making computing a career means I walk around having (or at least thinking I have), a basic grasp of how practically everything now a days works. The look of bewilderment, frustration, even fear, on the faces of my fellows about things modern has always hit me. "Superstitious architecture" in so much of what's around for many means it's like living in a world of evil spirits and malevolent gods.
The failure cascade of complex systems is not limited to war. Consider the collapse of Just in Time inventory systems during the COVID lockdowns. One bottleneck proved to be able to shutdown whole industries. A major disaster loomed when the Governor of Pennsylvania shut down all of the rest stops and restaurants along the PA interstates. This effectively cut the major supply line to NYC. President Trump and the President of the Teamsters "explained" things and got the order rescinded.
Going back to the defense of Israel, they had apparently forgotten the failure of the Bar Lev Line in 1973. Technology aside crust defenses are inherently fragile. See also Maginot Line and Great Wall of China. If you depend solely on a crust defense, once the enemy breaks the crust, they have a free run in the vacuum behind the crust. What is needed is defense in depth to contain the attack while larger forces mobilize to drive it back. Especially in an age of terrorism, an armed citizenry is indispensable. Citizens are the targets of terrorism so by definition they are the first to fight. Behind them are police, regional military (National Guard in an American context) and then the regular military.
I was shocked to read how few firearms there are in Israel. Something like less than 3% of the population is armed. I'm unclear whether this is by choice (of the people) or by design (hard application process), but it seems insane in such a dangerous part of the world.
The widespread private ownership of firearms is essentially crowdsourcing your defense. Thus, if someone does "pierce the crust" they're not getting far.
Agreed that even without an overreliance on tech, hard and thin defenses are fragile. The Roman's eventually had to give up their limes (correct spelling) for a defense in depth strategy. This Hamas attack would have been a military disaster if it had been 15,000 men instead of 1500, and Hizboulah coordinated their rocket attacks with Hamas. Instead, Israel suffered a hit-and-run raid. The real threat for Israel is not the Palastinians, it's a multi-front war. It did in Napoleon and Hitler, and it could take out Netanyahu and Biden as well. Israel cannot fight a united Arab world, and the US cannot fight a combined Russia, China, and Iran.
Wonderful read. Insightful. Old wisdom, "the more moving parts, the more points of failure," applies to everything in life. Making anything more complex adds risk. Sometimes that complexity is really worth it; things are better with it than without it. But to not know, or deny, or ignore that old wisdom is foolish.
I laugh to myself, because so often reading your essays validates something I think already. And that's dangerous. Seductive. That feeling of "Yes! Exactly!" is just so good.
Oct 24, 2023·edited Oct 24, 2023Liked by N.S. Lyons
This feels like a timely critique of technology for technology's sake. It feels like humans are being encouraged to be less and less human, while being more and more entwined in technology that is unnecessary and not more useful than tried and true solutions from the past.
Hopefully more people will take a step back and push back against the wave of fragile tech that weakens us.
The "wave of fragile technology" may derive more from our class and caste systems than some inanimate tendency. It is to the personal business interest of PMC types to concoct complex, hard-to-control, hard to fix technologies, since it tends to increase the employment and sales opportunities of the technologists and those who manage them and sell their services.
True. A lot of people, men mostly, are fairly mechanically minded. My childhood 30+ plus years ago was filled with dads tinkering with cars and assorted engines, some other old codger offering advice.
The young fathers don't know where to start and the old boys' knowledge is no longer fit for the modern day
Home Ec (short for "Economy") and wood/metal/drafting shops have long since been removed from US public schools for budget reasons. There is a "Maker" (DIY or do-it-yourself) movement in all things once considered trades try to pass on their knowledge, which is terrific, but it does not compare to the DIY can-do repair spirit I remember as well, from 30+, 40+ years ago.
Humans are EXPLICITLY being encouraged to be less human and the word human itself is being hijacked to mean post-human. There are multiple dovetailing ideologies aspiring to end the anthropocene.
The towers being staffed exclusively by young women seems distinctly under-analyzed here. If we're serious about "people, ideas, machines - in that order" then the people aspect must have harsh questions be raised here.
Beyond the questionable wisdom of having an army with female soldiers to begin with, how is 100% of towers having female soldiers even demographically possible without extreme levels of wokeness and anti-male bias in staffing decisions? According to Wikipedia, in 2014 women made up only 4% of IDF combat roles with all the rest in support positions? Is it possible the IDF classified border posts as non-combat roles and used them as a form of demographic ballast to score virtue points? If so, why is nobody talking about that?
According to reports, women were selected because they tested as superior for the attentiveness required in such positions. I can't put my hands on the reference right now, but apparently women were chosen because they scored in the top 4% for attentiveness. I did find these that suggest there is something to this concept:
Strangely, the article left me in mostly in agreement with the solutions, though the analysis is poor. There are many assertions made about the fence as a system—this is nonlinear, emergent behavior, etc.—without citing any real sources about how it worked. The use of failure cascade here does not match my understanding either. A failure cascade is the proverbial house of cards collapsing as a result of a single point of failure. What Hamas executed was a systematic dismantling of a complex system, predicated on digital communication and surveillance. Anyone that's had to work a radio in wartime will tell you both how important communications are and how unreliable they are. The worst part was the example of EVs as being a less complex system. In fact, EVs are far more simple devices (see how much unions worry they'll lose jobs from having fewer parts to install). They do not have the benefit of a hundred years of refinement and innovation that internal combustion engines enjoy in both their design and production. So then, which is preferable? Simplicity or mature, proven technological systems?
Where this analysis succeeds is rooting the solutions in people. Boyd's principles are excellent and a great inclusion that illuminate the strategy for successfully deploying technology. I am curious if the IDF ever attempted a penetration test of their own fence, and if they came up with Hamas/Iran's plan. If not, why not? Similarly, they could have employed the "chaos monkey" approach, where you simply remove a component of your system and see how resilient it is in the face of one or multiple failures. Simply removing wireless communications would have highlighted how fragile this system was. Frankly, I would not be surprised to learn there was no stress-testing of such a system usually because they're politically unviable. If you invested political capital in making the fence, you're unlikely to want evidence from experts it doesn't work. The root failure mode, though, is a leadership failure and maybe a lack of imagination, neither of which can be solved with technology. My goal here is not to argue the fence is a good idea (evidence shows it's pretty terrible), but simply how future systems (including men with guns on a wall) should be evaluated and tested to gauge their merit.
There was no "border failure," nor was there a "technology" failure. This whole event was known about ahead of time and deliberately allowed to happen for strategic purposes. Nothing could be more obvious, and many Israelis are now speaking out.
Tell that to the Israelis, including those in the IDF, many of whom are saying the exact same thing. You don't think Netanyahu wants an excuse to erase Gaza from the map? He has made no secret of it. He boosted and funded Hamas for years for that very purpose (as evidenced by at least a dozen Israeli newspaper articles about that very thing in the last two weeks). I wonder if you know the history of the many leaders in this world who have done just that, allowing their own people to be killed to create a pretext for larger objectives. That is not a conspiracy theory; it is history. The conflict between Israel and the Palestinians is not a "political dispute," and I think you know that and used the term dishonestly. Let's perform a thought experiment: Aliens from outer space appear and attack the US, butchering thousands and driving millions west of the Mississippi. After some resistance, the aliens offer to split the US down the middle, taking the entire eastern half in a "peace deal." Fighters in the US say hell no and continue to resist. But the aliens claim they lived here 100,000 years ago, and so they have a right to the land over people who have just lived here a measly several hundred years, so because the US fighters resisted, they round up the Americans and put them in a concentration camp. Would you say that's a "political dispute"? Or would you say that's an existential crisis brought on by questions central to what it means to have self-determination, what it means to have property rights, what it means to consider yourself superior to other beings, etc. And because those questions are so big, and so consequential, the leader of the invaders tries to think of some way to solve his "American resistance" problem for good.
You are suffering from a lack of knowledge of both history and current information.
sorry there is nothing more pointless than discussing anything with a conspiracy theorist. it is somehow even worse than dealing w religious fundamentalists.
see u at Building 7 with the Sandy Hook crisis actors!
Yes, it's pointless because you have absolutely zero knowledge of history or current events, so you can't argue. You just throw the term "conspiracy theorist" around indiscriminately and hope someone considers that an argument. You probably also think the COVID response was scientific and the vax was safe and effective. You know that dissenting from that was also a conspiracy theory, right? And it's a conspiracy theory that mainstream media is a stenographer for government narratives, right? Except now you apparently think that the current government narrative is A-OK, because anything else is a conspiracy theory, right? The term "conspiracy theory" was invented by the CIA after the assassination of Kennedy, which you are probably convinced was carried out by Lee Harvey Oswald. Believing forensic analysis of what happened that day is just being a conspiracy theorist, right?
The IDF have knowledge of Israeli defense systems we don't have. So your claim is they don't know what they are talking about when they say this was obviously not an intelligence failure? Israel has the most sophisticated surveillance and multiple overlapping systems of protection. All of that was ignored or turned off. The women operating the border surveillance IMMEDIATELY reported to the government that the border had been breached. They were ignored. IDF members say the system is set up to respond to any such breach within five minutes. So...you just choose to believe they are wrong about a system they have worked within every day for years?
There's a workplace politics aspect to explore here as well. Either the people responsible for the wall were too incompetent or too insecure to attempt red-team exercises against their designs.
If it's truly the case that disabling the cell networks effectively disables the guard towers and the gun bots, then I can't believe they ever made an attempt to defeat their own system. Redundancy here is cheap and critical. Hundred dollar handheld radios could have kept comms online between guard towers.
Static defense is usually vulnerable to dynamic, innovative, offense. Monash's synchronized offensive battle plans probably ended the First World War a year earlier than it would have otherwise. Synchronizing the offensive battlefield tactics of the new armaments, tanks, airplanes, artillery, in an engineered offensive with limited objectives decisively ended the stalemate. France's Maginot Line was pathetically ineffective in 1940 against the German Blitzkrieg.
It was the effectiveness of Hamas' cheap drones that foreshadow the future of war. There is nothing we can do to stop "everybody" from building lethal, autonomous, killing machines that operate without human control. Israel can build a wall of drones to surround Gaza that simply execute human beings who venture across the line. No paraglider lands alive. No human beings cross back alive. The killing machines will be as ubiquitous in battle as smartphones are in a queue. Why would we give advantage to the enemy and restrain our technology? Combatant? Non-combatant? That's a history lesson.
I'd like someone to refute my prophesy. Please, please, please, tell me I am wrong.
William, I don’t think you are wrong, but it’s not a binary choice--either tech or people. It requires both. And no machine will ever be able to “out think” a human as multi-factor, multi-scenarios unfold in real time. Recall John Boyd’s mantra, “People, ideas, machines. In that order.”
AI can’t even accomplish autonomous driving on American suburbia streets during peacetime.
Drones don't think, they are programmed. They do what they are told. We are getting pretty good at programming. "Can't even..." in regards to autonomous motor vehicles twenty years ago was, "can't even imagine" The aviation training and ferry fatalities during World War II accounted for almost four percent of the total US military fatalities. I have a three-volume set that catalogs each of the seven thousand plus accidents.
What sort of an arms race are we looking at? I'd be very interested in John Boyd's thoughts on the future of drones - - - programmed to kill autonomously.
Drones don’t think is precisely my point. AI is simply a model, and all models say what the programmer tells it to say.
A great source on Substack for realistic commentary on AI is Erik Larson’s Colligo. Autonomous driving was “imagined” in the 1940s when AI was first conceived/invented. It’s taken 80 years to do the “cool” things like play chess and write essays because it took that long for the power, speed and memory of computers to bring it to fruition.
According to my reading the Maginot Line worked exactly as it was supposed to; the failures bringing about the fall of France lay elsewhere which the Maginot Line was helpless to correct since they were outside its domain. Sometimes static defenses do exactly what they are supposed to do, the Kursk Salient for example.
I was just reading that it was one of the last components of the French military to give up, and that occurred only because many of its personnel were withdrawn from it to fight elsewhere. Even under these conditions much of the Line held out until the French government itself gave up. I can't speak to a "false sense of security" because I wasn't there, but as far as I know the French did not feel secure in the least about Germany in the 1930s given its political evolution during that period. I think the Maginot Line has been given a bum rap.
Bum raps are like, "if only." Six weeks, that's how long it took the Germans to defeat France. My point is offensive maneuver always has a dynamic potential that static defense is denied. A defensive wall of autonomous drones that kill on sight has a different stasis - but it's still a known quantity. Offensive innovation is the locus of surprise.
Perhaps the way to overcome a wall of drones that kill as efficiently as computers play chess is to go through a wall of drones with a massed drone swarm. The response? The remaining defensive drone wall goes kill indiscriminately within the breached enclosure? The possibilities for maneuver are endless. A counteroffensive begins -- all on autopilot.
I don't know much about drone warfare, but I do know about the Maginot Line. When I was in school, and for many years, we were repeatedly told that the Maginot Line "didn't work." But it did work; it did exactly what it was supposed to do. Of course it wasn't a universal panacea; it was supposed to force the Germans to go elsewhere, where presumably the French commanders thought they could be stopped. That they weren't is a different issue. The durability of the fiction about the Maginot Line is quite remarkable and I wonder what purpose it serves -- maybe the sacredness of such holy texts as "The best defense is a good offense." Sometimes it is, sometimes it isn't.
Part of this problem seems to be credentialism. Insofar as I was employed between 1965 and 2010, I was dealing with computer systems, I noticed a gradual change in competence levels, which I attributed to the ability of certain people to get certain jobs based on passing tests, possibly inflated resumés, connections, and political considerations. I used to think it was the influence of Academia, but as I look around these days I wonder if the problem isn't more widespread than the Education Industry.
Complex systems require highly competent people to design/manage them. However, half of any population is below average and the supply of really competent people is even more limited. So you have to design a system that can be operated by average or even below average people. Meanwhile, complexity spreads into non-vital functions, spreading the supply of really competent people even thinner.
When Zionism was in its infancy, surveyors were sent to look over the proposed new land and report back their findings about the suitability of the territory for settlement and agriculture. They reported back that almost all the arable land had Arab farmers living on it. This information was ignored.
This policy, of ignoring the Palestinian problem, ran into its first violent failure with the 1928 riots and massacre of Jews in Hebron, followed by the Arab revolt in 1936 (crushed by the British as a colonial war) the 1948 war, with the expulsion of 750,000 Arabs and the creation of the refugee problem, then the ’67 war and the creation of the occupied West Bank, and so on and so on.
By the time of the October 7 massacre of Israeli towns along the Gaza border, Israeli’s were living with this prolonged denial of the existence of a Palestinian problem, by protecting itself with a barrier wall separating Israel proper from the West Bank, the Iron Dome and the Gaza security fence. They were able to convince themselves that they were safe from any Palestinian violence because the Palestinians were walled away, out of sight, out of mind. Demands for an actual solution to the problem, Oslo, two state solution, whatever, faded away. Like the early Zionists, they ignored the problem.
It was technology that abetted this denial, but in the end it was the human tendency to avoid the difficult, to look for easy solutions, to wish away the distasteful. The result was October 7th.
The 750,000 Arab refugees can't be looked at without the context of approximately the same number of Jews expelled from Arab countries at the same time, and absorbed by Israel. Without trying to weigh the justice of any of the situations, it was not all that different from similar population exchanges between, for example, Greece and Turkey after WWI, or Germans expelled from what is now Poland after WWII.
Thanks for your thoughtful comment. I would suggest that there is limited utility in looking at similar events in history as if it were the same or comparable, without context, particularly if you remove any ethical value from the equation. Think about the resistance to comparing genocidal events, like the Holocaust, to the destruction of the Kulaks by Stalin, or American Slavery, for instance. It is a fairly hollow argument. Jews in particular bristle at these kinds of comparisons because it was Jews that suffered and died, in the case of the Holocaust. The Holocaust has become an event of monumental historical and symbolic significance to almost all Jewish people, but if you take away what you call “Justice” and just count up the bodies you could make a pretty good argument that the Holocaust was just another dreary act of human indifference and cruelty in history. The expulsion of the Arabs from Palestine in 1948, like the Holocaust for the Jews, is a deeply meaningful and symbolic event and one which has come to almost define a people. Moreover, the displacement and suffering of the 1948 event has festered and expanded since that time, and remains the cause of ongoing suffering and death to this day. If we remove what you call justice or the ethical weight of an historical event, the humanity of it, we reduce our own humanity in the process. With respect, I would suggest that you do this to reduce the moral weight of the forced expulsion of 750,000 Palestinian people. What is most significant about that event, historically, is that the Palestinians were not allowed to return to their villages after hostilities ceased, as is generally the case in most refugee situations. Those villages were taken by the new Jewish state to house the hundreds of thousands of displaced Jews from Europe, many of them Holocaust survivors, at a time where even the United States refused them, because widespread anti-semitism in the US made it politically impossible. Wouldn’t you say it was difficult to see events with this significant moral complexity in the purely mathematical terms of population exchange?
What I was suggesting was that you were ignoring an important aspect of the context that a roughly equal number of Jews were driven out of the various Arab countries at the same time. Israel has a response that it took in Jews expelled from Arab countries roughly proportional to Palestinians expelled from Israel. I don't think the moral weight what happened to the Palestinians can be divorced entirely from what happened to the Arabian Jews. The difference is that Israel integrated the Mizrahi and the Arab countries did not do the same to the Palestinians.
Certainly the Greek-Turk situation is a reasonable historical analogy as it was relatively close in time and space, neither side got to return to their historical homes, and yet it yielded a rough equilibrium. Not to say there are not still historical grievances -- the Greeks had a presence in Asia Minor as old or older than the Jews in Israel -- but on balance the rough justice of population exchange tended to dampen rather than exacerbate ethnic tensions. One could ask whether the Arab nations absorbing the Palestinian refugees proportionally to the Jews driven out and promoting assimilation over festering and expansion of historical grievance might not have produced a more stable situation.
I only mentioned putting justice aside because there is always the argument that each historical situation somehow stands on its own. I am not sure there is a neutral principle for settling such historical grievances, which is why that art of getting past them is important.
Thanks for the interesting response. With respect, I think we are talking past each other. You are taking a historical event (the expulsion of Jews from Arab countries) and comparing it without context or an acknowledgment of its complexity to a similar but larger expulsion of another people also without an acknowledgment of its complexity, to make a very simplistic point, a painful example of whataboutism.
So here is some context to keep in mind. The Jews were expelled from those Arab countries as a direct result of the 1948 war and the expulsion of the Palestinian Arabs. 650,000 of these people settled in Israel and the process took from from 1948 to the early 1970s, with one final expulsion from Iran in 1979–80 following the Iranian Revolution. The expulsion of the Palestinians took place over a comparatively short amount of time with the specific intent of using their homes to house European refugees. These are very different events.
An Israeli psychiatrist once told me that the expulsion of the Palestinian Arabs in 1948 was “Israel’s original sin”. He said it is difficult for many Israelis to actually take ownership of this fact. In our own country, the displacement and destruction of the American Indians was treated as an heroic story until the recent past. All nations have painful histories they try to avoid.
The actual history of the Arab expulsions of 1948 was kept out of Israeli textbooks in favor on mythic stories of Arab leaders ordering Palestinians to flee, for almost 50 years, and the historical facts of the military expulsion of the Arabs by Jewish forces was kept hidden in classified army documents until the 80’s, so explosive were these facts considered by the Israeli government. When the archives were opened, Israeli historian Benny Morris shocked the Israel public by publishing these facts.
Is it helpful to suggest that expulsions happen all the time and all kinds of people do them so what is the big deal, what does it matter? I do not believe so. I think these things have moral weight.
It comes as a relief to read a 'right-wing' thinker who does not default to conspiracism.
The instant 'inside job' theories again reminded me that it's psychologically comforting to believe that wired-in human frailty isn't behind intelligence failures. The entirely normal human tendency to err and fall into hubris only underlines our vulnerability. That is a much more frightening forever threat than evil plotters getting together to do something bad every now and then.
Thought-provoking essay, as always. I hadn't yet bothered to learn about the tactics of the Hamas attack (maybe because, as an American, I assume Southern borders are naturally porous---but that's a separate issue), so this was an enlightening summary of what went down on 10/7. I've nothing to add, except to re-emphasize one aspect of the attack which you touched on twice:
"Hamas, however, used small, off-the-shelf drones rigged with mortar rounds and other explosives to attack and disable the communications towers powering the network. These drones were too small and low-flying for radar to detect, so would have had to have been spotted by eye and ear."
The WSJ article you linked to is behind a paywall, but I assume it describes the same phenomenon that observers of the Russo-Ukrainian war have noted over the past year and a half: these small drones are lethal. They constitute a revolution in military affairs. The Russkies use them for reconnaissance / surveillance tasks, as well as generating telemetry data for artillery strikes, naturally, but will also pilot them, silently, over a Ukrainian trench, then drop grenades onto its occupants. This is terrifying for infantry, and unimaginable before this era. Not to mention Russia's medium-size Lancet kamikaze drones, which casually conduct precision strikes against tanks and armored troop carriers. It used to be the case if, from the air, you wanted to take out one of these, it required a burst from an A-10 Warthog's auto-cannon or an expensive air-to-ground missile. No longer.
So here we are, back at a new drawing board. Think what you will about the villain of A Few Good Men, Colonel Jessup, but it's difficult to discount his supremely realist perspective on such matters:
"You can't handle the truth! Son, we live in a world that has walls, and those walls have to be guarded by men with guns. Who's gonna do it? You[, Lieutenant Kaffee]? You, Lieutenant Weinberg?* I have a greater responsibility than you can possibly fathom. You weep for Santiago and you curse the Marines. You have that luxury. You have the luxury of not knowing what I know: Santiago's death, while tragic, probably saved lives. And my existence, while grotesque and incomprehensible to you, saves lives. You don't want the truth because deep down in places you don't talk about at parties, you want me on that wall---you need me on that wall."
The Collapse of Complex Societies by Joseph Tainter makes similar points. He’s an archaeologist but his analysis isn’t limited only to those complex societies that have left only physical evidence but few records.
I don't know if you have seen this - very interesting, fact-based explanation implies the move to one source of software was big part of the problem, rather than having multiple systems and backups.
For several years I was a principal engineer at Amazon and I was a periodic member of the "availability team" which did post-mortem analysis of every event that caused the web site to go down. Amazon is/was the most operationally competent company I have ever worked for but the scale at which they operated required a level of complexity that no one could really get their arms around. No one fully understood the inter-relationships and dependencies between all the micro-services. The system required intense, continuous automated monitoring to achieve our goals in terms of uptime/availability. (At one point, fully 20% of the entire system's compute capacity was being consumed merely to process faux requests that checked on whether the system was up.) Everything you say about the challenges of complex systems is true. But I will offer a couple of other observations.
First, when faced with complex systems, human beings have a strong propensity to develop superstitious explanations for the visible phenomena they observe. (The last talk I gave to the engineering organization before leaving Amazon was called "Superstitious Architectures: How to Avoid Them".) Prying understanding from complex systems is hard and people are often happy to settle for superstition. (I offer as evidence a lot of what the lay press is saying about AI right now.) What this means in practice is that complexity causes human actors to operate in a state of greater general ignorance. Not a happy circumstance when the sh*t hits the fan.
Second, people are being driven to despair of their own human agency as their dependence on technology they can never understand grows. This is not entirely accidental I suspect and is probably even a little sinister. This has the effect of increasing dependency within the general population and reducing the sum total of initiative in the population at large.
Your advice to focus on simplifying our own lives is wise and even prescient I suspect.
Thanks for this essay.
Not Amazon, but a couple of tech orgs as well. And wow, well put. Was not familiar with the thought of "superstitious architectures." Nicely sums it up. Hell, I notice it in my relationship with cooking. Until getting into it a couple of decades ago the combo of heat/cold, spices and time was all magic to me. Only now do I see it without superstition, and it took years.
The lucky accident of making computing a career means I walk around having (or at least thinking I have), a basic grasp of how practically everything now a days works. The look of bewilderment, frustration, even fear, on the faces of my fellows about things modern has always hit me. "Superstitious architecture" in so much of what's around for many means it's like living in a world of evil spirits and malevolent gods.
A quantum leap forward in time and space
The universe learned to expand
Mess and magic, triumphant and tragic
A mechanized world out of hand
Computerized clinic for superior cynics
Who dance to a synthetic band
In their own image their world is fashioned
No wonder they don't understand
Wheel within wheels in a spiral array
A pattern so grand and complex
Time after time we lose sight of the way
Our causes can't see their effects
From Rush: Wheels Within Wheels
Great comment, many thanks.
Excellent essay.
The failure cascade of complex systems is not limited to war. Consider the collapse of Just in Time inventory systems during the COVID lockdowns. One bottleneck proved to be able to shutdown whole industries. A major disaster loomed when the Governor of Pennsylvania shut down all of the rest stops and restaurants along the PA interstates. This effectively cut the major supply line to NYC. President Trump and the President of the Teamsters "explained" things and got the order rescinded.
Going back to the defense of Israel, they had apparently forgotten the failure of the Bar Lev Line in 1973. Technology aside crust defenses are inherently fragile. See also Maginot Line and Great Wall of China. If you depend solely on a crust defense, once the enemy breaks the crust, they have a free run in the vacuum behind the crust. What is needed is defense in depth to contain the attack while larger forces mobilize to drive it back. Especially in an age of terrorism, an armed citizenry is indispensable. Citizens are the targets of terrorism so by definition they are the first to fight. Behind them are police, regional military (National Guard in an American context) and then the regular military.
I was shocked to read how few firearms there are in Israel. Something like less than 3% of the population is armed. I'm unclear whether this is by choice (of the people) or by design (hard application process), but it seems insane in such a dangerous part of the world.
The widespread private ownership of firearms is essentially crowdsourcing your defense. Thus, if someone does "pierce the crust" they're not getting far.
Why do people need automatic weapons? Why do people need so many rounds? This is why
Agreed that even without an overreliance on tech, hard and thin defenses are fragile. The Roman's eventually had to give up their limes (correct spelling) for a defense in depth strategy. This Hamas attack would have been a military disaster if it had been 15,000 men instead of 1500, and Hizboulah coordinated their rocket attacks with Hamas. Instead, Israel suffered a hit-and-run raid. The real threat for Israel is not the Palastinians, it's a multi-front war. It did in Napoleon and Hitler, and it could take out Netanyahu and Biden as well. Israel cannot fight a united Arab world, and the US cannot fight a combined Russia, China, and Iran.
Wonderful read. Insightful. Old wisdom, "the more moving parts, the more points of failure," applies to everything in life. Making anything more complex adds risk. Sometimes that complexity is really worth it; things are better with it than without it. But to not know, or deny, or ignore that old wisdom is foolish.
I laugh to myself, because so often reading your essays validates something I think already. And that's dangerous. Seductive. That feeling of "Yes! Exactly!" is just so good.
Read Barba-Kay's 'A Web of Our Own Making.' You'll feel like that on every other page. I kid you not.
This feels like a timely critique of technology for technology's sake. It feels like humans are being encouraged to be less and less human, while being more and more entwined in technology that is unnecessary and not more useful than tried and true solutions from the past.
Hopefully more people will take a step back and push back against the wave of fragile tech that weakens us.
The "wave of fragile technology" may derive more from our class and caste systems than some inanimate tendency. It is to the personal business interest of PMC types to concoct complex, hard-to-control, hard to fix technologies, since it tends to increase the employment and sales opportunities of the technologists and those who manage them and sell their services.
True. A lot of people, men mostly, are fairly mechanically minded. My childhood 30+ plus years ago was filled with dads tinkering with cars and assorted engines, some other old codger offering advice.
The young fathers don't know where to start and the old boys' knowledge is no longer fit for the modern day
Home Ec (short for "Economy") and wood/metal/drafting shops have long since been removed from US public schools for budget reasons. There is a "Maker" (DIY or do-it-yourself) movement in all things once considered trades try to pass on their knowledge, which is terrific, but it does not compare to the DIY can-do repair spirit I remember as well, from 30+, 40+ years ago.
If we lose computers in the hospital we are completely crippled
If they lose computers in PetSmart, they are completely crippled.
Humans are EXPLICITLY being encouraged to be less human and the word human itself is being hijacked to mean post-human. There are multiple dovetailing ideologies aspiring to end the anthropocene.
The towers being staffed exclusively by young women seems distinctly under-analyzed here. If we're serious about "people, ideas, machines - in that order" then the people aspect must have harsh questions be raised here.
Beyond the questionable wisdom of having an army with female soldiers to begin with, how is 100% of towers having female soldiers even demographically possible without extreme levels of wokeness and anti-male bias in staffing decisions? According to Wikipedia, in 2014 women made up only 4% of IDF combat roles with all the rest in support positions? Is it possible the IDF classified border posts as non-combat roles and used them as a form of demographic ballast to score virtue points? If so, why is nobody talking about that?
According to reports, women were selected because they tested as superior for the attentiveness required in such positions. I can't put my hands on the reference right now, but apparently women were chosen because they scored in the top 4% for attentiveness. I did find these that suggest there is something to this concept:
The "girls who sit there":
https://allisrael.com/the-female-idfsoldiers-manning-the-border-observation-cameras-speak-out-they-sniped-the-cameras
https://www.reuters.com/article/us-israel-egypt-lookouts/israeli-women-soldiers-have-right-stuff-for-border-watch-idUSBRE90A0GB20130111
"18-year-old girls took care of their wounded friends, hid for more than 8 hours from terrorists" ...... wow
Strangely, the article left me in mostly in agreement with the solutions, though the analysis is poor. There are many assertions made about the fence as a system—this is nonlinear, emergent behavior, etc.—without citing any real sources about how it worked. The use of failure cascade here does not match my understanding either. A failure cascade is the proverbial house of cards collapsing as a result of a single point of failure. What Hamas executed was a systematic dismantling of a complex system, predicated on digital communication and surveillance. Anyone that's had to work a radio in wartime will tell you both how important communications are and how unreliable they are. The worst part was the example of EVs as being a less complex system. In fact, EVs are far more simple devices (see how much unions worry they'll lose jobs from having fewer parts to install). They do not have the benefit of a hundred years of refinement and innovation that internal combustion engines enjoy in both their design and production. So then, which is preferable? Simplicity or mature, proven technological systems?
Where this analysis succeeds is rooting the solutions in people. Boyd's principles are excellent and a great inclusion that illuminate the strategy for successfully deploying technology. I am curious if the IDF ever attempted a penetration test of their own fence, and if they came up with Hamas/Iran's plan. If not, why not? Similarly, they could have employed the "chaos monkey" approach, where you simply remove a component of your system and see how resilient it is in the face of one or multiple failures. Simply removing wireless communications would have highlighted how fragile this system was. Frankly, I would not be surprised to learn there was no stress-testing of such a system usually because they're politically unviable. If you invested political capital in making the fence, you're unlikely to want evidence from experts it doesn't work. The root failure mode, though, is a leadership failure and maybe a lack of imagination, neither of which can be solved with technology. My goal here is not to argue the fence is a good idea (evidence shows it's pretty terrible), but simply how future systems (including men with guns on a wall) should be evaluated and tested to gauge their merit.
There was no "border failure," nor was there a "technology" failure. This whole event was known about ahead of time and deliberately allowed to happen for strategic purposes. Nothing could be more obvious, and many Israelis are now speaking out.
yes, nothing is more obvious than that the israeli govt allowed its own citizens to be tortured and massacred...because? to win a political dispute?
conspiracy theorists are like maggots, they love to grow fat dining on fresh corpses.
Tell that to the Israelis, including those in the IDF, many of whom are saying the exact same thing. You don't think Netanyahu wants an excuse to erase Gaza from the map? He has made no secret of it. He boosted and funded Hamas for years for that very purpose (as evidenced by at least a dozen Israeli newspaper articles about that very thing in the last two weeks). I wonder if you know the history of the many leaders in this world who have done just that, allowing their own people to be killed to create a pretext for larger objectives. That is not a conspiracy theory; it is history. The conflict between Israel and the Palestinians is not a "political dispute," and I think you know that and used the term dishonestly. Let's perform a thought experiment: Aliens from outer space appear and attack the US, butchering thousands and driving millions west of the Mississippi. After some resistance, the aliens offer to split the US down the middle, taking the entire eastern half in a "peace deal." Fighters in the US say hell no and continue to resist. But the aliens claim they lived here 100,000 years ago, and so they have a right to the land over people who have just lived here a measly several hundred years, so because the US fighters resisted, they round up the Americans and put them in a concentration camp. Would you say that's a "political dispute"? Or would you say that's an existential crisis brought on by questions central to what it means to have self-determination, what it means to have property rights, what it means to consider yourself superior to other beings, etc. And because those questions are so big, and so consequential, the leader of the invaders tries to think of some way to solve his "American resistance" problem for good.
You are suffering from a lack of knowledge of both history and current information.
sorry there is nothing more pointless than discussing anything with a conspiracy theorist. it is somehow even worse than dealing w religious fundamentalists.
see u at Building 7 with the Sandy Hook crisis actors!
Yes, it's pointless because you have absolutely zero knowledge of history or current events, so you can't argue. You just throw the term "conspiracy theorist" around indiscriminately and hope someone considers that an argument. You probably also think the COVID response was scientific and the vax was safe and effective. You know that dissenting from that was also a conspiracy theory, right? And it's a conspiracy theory that mainstream media is a stenographer for government narratives, right? Except now you apparently think that the current government narrative is A-OK, because anything else is a conspiracy theory, right? The term "conspiracy theory" was invented by the CIA after the assassination of Kennedy, which you are probably convinced was carried out by Lee Harvey Oswald. Believing forensic analysis of what happened that day is just being a conspiracy theorist, right?
hey im just a random guy on the internet, i assume that my not engaging you will not cause you major pain or hardship. have a great day!
Random guy on the internet contributes zero content.
“Absolutely zero knowledge of history” belongs on Reddit.
Yet there are conspiracies, after all.
Lots of Israelis agreeing with you doesn’t make it right.
The IDF have knowledge of Israeli defense systems we don't have. So your claim is they don't know what they are talking about when they say this was obviously not an intelligence failure? Israel has the most sophisticated surveillance and multiple overlapping systems of protection. All of that was ignored or turned off. The women operating the border surveillance IMMEDIATELY reported to the government that the border had been breached. They were ignored. IDF members say the system is set up to respond to any such breach within five minutes. So...you just choose to believe they are wrong about a system they have worked within every day for years?
“So your claim is...” also belongs on Reddit.
There's a workplace politics aspect to explore here as well. Either the people responsible for the wall were too incompetent or too insecure to attempt red-team exercises against their designs.
If it's truly the case that disabling the cell networks effectively disables the guard towers and the gun bots, then I can't believe they ever made an attempt to defeat their own system. Redundancy here is cheap and critical. Hundred dollar handheld radios could have kept comms online between guard towers.
Static defense is usually vulnerable to dynamic, innovative, offense. Monash's synchronized offensive battle plans probably ended the First World War a year earlier than it would have otherwise. Synchronizing the offensive battlefield tactics of the new armaments, tanks, airplanes, artillery, in an engineered offensive with limited objectives decisively ended the stalemate. France's Maginot Line was pathetically ineffective in 1940 against the German Blitzkrieg.
It was the effectiveness of Hamas' cheap drones that foreshadow the future of war. There is nothing we can do to stop "everybody" from building lethal, autonomous, killing machines that operate without human control. Israel can build a wall of drones to surround Gaza that simply execute human beings who venture across the line. No paraglider lands alive. No human beings cross back alive. The killing machines will be as ubiquitous in battle as smartphones are in a queue. Why would we give advantage to the enemy and restrain our technology? Combatant? Non-combatant? That's a history lesson.
I'd like someone to refute my prophesy. Please, please, please, tell me I am wrong.
William, I don’t think you are wrong, but it’s not a binary choice--either tech or people. It requires both. And no machine will ever be able to “out think” a human as multi-factor, multi-scenarios unfold in real time. Recall John Boyd’s mantra, “People, ideas, machines. In that order.”
AI can’t even accomplish autonomous driving on American suburbia streets during peacetime.
Drones don't think, they are programmed. They do what they are told. We are getting pretty good at programming. "Can't even..." in regards to autonomous motor vehicles twenty years ago was, "can't even imagine" The aviation training and ferry fatalities during World War II accounted for almost four percent of the total US military fatalities. I have a three-volume set that catalogs each of the seven thousand plus accidents.
What sort of an arms race are we looking at? I'd be very interested in John Boyd's thoughts on the future of drones - - - programmed to kill autonomously.
Drones don’t think is precisely my point. AI is simply a model, and all models say what the programmer tells it to say.
A great source on Substack for realistic commentary on AI is Erik Larson’s Colligo. Autonomous driving was “imagined” in the 1940s when AI was first conceived/invented. It’s taken 80 years to do the “cool” things like play chess and write essays because it took that long for the power, speed and memory of computers to bring it to fruition.
According to my reading the Maginot Line worked exactly as it was supposed to; the failures bringing about the fall of France lay elsewhere which the Maginot Line was helpless to correct since they were outside its domain. Sometimes static defenses do exactly what they are supposed to do, the Kursk Salient for example.
Maginot Line also created a false sense of security and was easily overcome.
I was just reading that it was one of the last components of the French military to give up, and that occurred only because many of its personnel were withdrawn from it to fight elsewhere. Even under these conditions much of the Line held out until the French government itself gave up. I can't speak to a "false sense of security" because I wasn't there, but as far as I know the French did not feel secure in the least about Germany in the 1930s given its political evolution during that period. I think the Maginot Line has been given a bum rap.
I guess my perspective is it doesn’t matter if parts of the Line were the last to give up; the point is the enemy simply went around it.
Bum raps are like, "if only." Six weeks, that's how long it took the Germans to defeat France. My point is offensive maneuver always has a dynamic potential that static defense is denied. A defensive wall of autonomous drones that kill on sight has a different stasis - but it's still a known quantity. Offensive innovation is the locus of surprise.
Perhaps the way to overcome a wall of drones that kill as efficiently as computers play chess is to go through a wall of drones with a massed drone swarm. The response? The remaining defensive drone wall goes kill indiscriminately within the breached enclosure? The possibilities for maneuver are endless. A counteroffensive begins -- all on autopilot.
I don't know much about drone warfare, but I do know about the Maginot Line. When I was in school, and for many years, we were repeatedly told that the Maginot Line "didn't work." But it did work; it did exactly what it was supposed to do. Of course it wasn't a universal panacea; it was supposed to force the Germans to go elsewhere, where presumably the French commanders thought they could be stopped. That they weren't is a different issue. The durability of the fiction about the Maginot Line is quite remarkable and I wonder what purpose it serves -- maybe the sacredness of such holy texts as "The best defense is a good offense." Sometimes it is, sometimes it isn't.
This is a great complement to Harold Robertson's recent speculations about another source of failure within our complex systems--namely, the purported decline in competence-based hiring and promotion: https://www.palladiummag.com/2023/06/01/complex-systems-wont-survive-the-competence-crisis/
Tech won't save us from ourselves.
Part of this problem seems to be credentialism. Insofar as I was employed between 1965 and 2010, I was dealing with computer systems, I noticed a gradual change in competence levels, which I attributed to the ability of certain people to get certain jobs based on passing tests, possibly inflated resumés, connections, and political considerations. I used to think it was the influence of Academia, but as I look around these days I wonder if the problem isn't more widespread than the Education Industry.
Complex systems require highly competent people to design/manage them. However, half of any population is below average and the supply of really competent people is even more limited. So you have to design a system that can be operated by average or even below average people. Meanwhile, complexity spreads into non-vital functions, spreading the supply of really competent people even thinner.
That’s a fantastic piece, great recommendation
When Zionism was in its infancy, surveyors were sent to look over the proposed new land and report back their findings about the suitability of the territory for settlement and agriculture. They reported back that almost all the arable land had Arab farmers living on it. This information was ignored.
This policy, of ignoring the Palestinian problem, ran into its first violent failure with the 1928 riots and massacre of Jews in Hebron, followed by the Arab revolt in 1936 (crushed by the British as a colonial war) the 1948 war, with the expulsion of 750,000 Arabs and the creation of the refugee problem, then the ’67 war and the creation of the occupied West Bank, and so on and so on.
By the time of the October 7 massacre of Israeli towns along the Gaza border, Israeli’s were living with this prolonged denial of the existence of a Palestinian problem, by protecting itself with a barrier wall separating Israel proper from the West Bank, the Iron Dome and the Gaza security fence. They were able to convince themselves that they were safe from any Palestinian violence because the Palestinians were walled away, out of sight, out of mind. Demands for an actual solution to the problem, Oslo, two state solution, whatever, faded away. Like the early Zionists, they ignored the problem.
It was technology that abetted this denial, but in the end it was the human tendency to avoid the difficult, to look for easy solutions, to wish away the distasteful. The result was October 7th.
The 750,000 Arab refugees can't be looked at without the context of approximately the same number of Jews expelled from Arab countries at the same time, and absorbed by Israel. Without trying to weigh the justice of any of the situations, it was not all that different from similar population exchanges between, for example, Greece and Turkey after WWI, or Germans expelled from what is now Poland after WWII.
Thanks for your thoughtful comment. I would suggest that there is limited utility in looking at similar events in history as if it were the same or comparable, without context, particularly if you remove any ethical value from the equation. Think about the resistance to comparing genocidal events, like the Holocaust, to the destruction of the Kulaks by Stalin, or American Slavery, for instance. It is a fairly hollow argument. Jews in particular bristle at these kinds of comparisons because it was Jews that suffered and died, in the case of the Holocaust. The Holocaust has become an event of monumental historical and symbolic significance to almost all Jewish people, but if you take away what you call “Justice” and just count up the bodies you could make a pretty good argument that the Holocaust was just another dreary act of human indifference and cruelty in history. The expulsion of the Arabs from Palestine in 1948, like the Holocaust for the Jews, is a deeply meaningful and symbolic event and one which has come to almost define a people. Moreover, the displacement and suffering of the 1948 event has festered and expanded since that time, and remains the cause of ongoing suffering and death to this day. If we remove what you call justice or the ethical weight of an historical event, the humanity of it, we reduce our own humanity in the process. With respect, I would suggest that you do this to reduce the moral weight of the forced expulsion of 750,000 Palestinian people. What is most significant about that event, historically, is that the Palestinians were not allowed to return to their villages after hostilities ceased, as is generally the case in most refugee situations. Those villages were taken by the new Jewish state to house the hundreds of thousands of displaced Jews from Europe, many of them Holocaust survivors, at a time where even the United States refused them, because widespread anti-semitism in the US made it politically impossible. Wouldn’t you say it was difficult to see events with this significant moral complexity in the purely mathematical terms of population exchange?
What I was suggesting was that you were ignoring an important aspect of the context that a roughly equal number of Jews were driven out of the various Arab countries at the same time. Israel has a response that it took in Jews expelled from Arab countries roughly proportional to Palestinians expelled from Israel. I don't think the moral weight what happened to the Palestinians can be divorced entirely from what happened to the Arabian Jews. The difference is that Israel integrated the Mizrahi and the Arab countries did not do the same to the Palestinians.
Certainly the Greek-Turk situation is a reasonable historical analogy as it was relatively close in time and space, neither side got to return to their historical homes, and yet it yielded a rough equilibrium. Not to say there are not still historical grievances -- the Greeks had a presence in Asia Minor as old or older than the Jews in Israel -- but on balance the rough justice of population exchange tended to dampen rather than exacerbate ethnic tensions. One could ask whether the Arab nations absorbing the Palestinian refugees proportionally to the Jews driven out and promoting assimilation over festering and expansion of historical grievance might not have produced a more stable situation.
I only mentioned putting justice aside because there is always the argument that each historical situation somehow stands on its own. I am not sure there is a neutral principle for settling such historical grievances, which is why that art of getting past them is important.
Thanks for the interesting response. With respect, I think we are talking past each other. You are taking a historical event (the expulsion of Jews from Arab countries) and comparing it without context or an acknowledgment of its complexity to a similar but larger expulsion of another people also without an acknowledgment of its complexity, to make a very simplistic point, a painful example of whataboutism.
So here is some context to keep in mind. The Jews were expelled from those Arab countries as a direct result of the 1948 war and the expulsion of the Palestinian Arabs. 650,000 of these people settled in Israel and the process took from from 1948 to the early 1970s, with one final expulsion from Iran in 1979–80 following the Iranian Revolution. The expulsion of the Palestinians took place over a comparatively short amount of time with the specific intent of using their homes to house European refugees. These are very different events.
An Israeli psychiatrist once told me that the expulsion of the Palestinian Arabs in 1948 was “Israel’s original sin”. He said it is difficult for many Israelis to actually take ownership of this fact. In our own country, the displacement and destruction of the American Indians was treated as an heroic story until the recent past. All nations have painful histories they try to avoid.
The actual history of the Arab expulsions of 1948 was kept out of Israeli textbooks in favor on mythic stories of Arab leaders ordering Palestinians to flee, for almost 50 years, and the historical facts of the military expulsion of the Arabs by Jewish forces was kept hidden in classified army documents until the 80’s, so explosive were these facts considered by the Israeli government. When the archives were opened, Israeli historian Benny Morris shocked the Israel public by publishing these facts.
Is it helpful to suggest that expulsions happen all the time and all kinds of people do them so what is the big deal, what does it matter? I do not believe so. I think these things have moral weight.
It comes as a relief to read a 'right-wing' thinker who does not default to conspiracism.
The instant 'inside job' theories again reminded me that it's psychologically comforting to believe that wired-in human frailty isn't behind intelligence failures. The entirely normal human tendency to err and fall into hubris only underlines our vulnerability. That is a much more frightening forever threat than evil plotters getting together to do something bad every now and then.
Thought-provoking essay, as always. I hadn't yet bothered to learn about the tactics of the Hamas attack (maybe because, as an American, I assume Southern borders are naturally porous---but that's a separate issue), so this was an enlightening summary of what went down on 10/7. I've nothing to add, except to re-emphasize one aspect of the attack which you touched on twice:
"Hamas, however, used small, off-the-shelf drones rigged with mortar rounds and other explosives to attack and disable the communications towers powering the network. These drones were too small and low-flying for radar to detect, so would have had to have been spotted by eye and ear."
The WSJ article you linked to is behind a paywall, but I assume it describes the same phenomenon that observers of the Russo-Ukrainian war have noted over the past year and a half: these small drones are lethal. They constitute a revolution in military affairs. The Russkies use them for reconnaissance / surveillance tasks, as well as generating telemetry data for artillery strikes, naturally, but will also pilot them, silently, over a Ukrainian trench, then drop grenades onto its occupants. This is terrifying for infantry, and unimaginable before this era. Not to mention Russia's medium-size Lancet kamikaze drones, which casually conduct precision strikes against tanks and armored troop carriers. It used to be the case if, from the air, you wanted to take out one of these, it required a burst from an A-10 Warthog's auto-cannon or an expensive air-to-ground missile. No longer.
So here we are, back at a new drawing board. Think what you will about the villain of A Few Good Men, Colonel Jessup, but it's difficult to discount his supremely realist perspective on such matters:
"You can't handle the truth! Son, we live in a world that has walls, and those walls have to be guarded by men with guns. Who's gonna do it? You[, Lieutenant Kaffee]? You, Lieutenant Weinberg?* I have a greater responsibility than you can possibly fathom. You weep for Santiago and you curse the Marines. You have that luxury. You have the luxury of not knowing what I know: Santiago's death, while tragic, probably saved lives. And my existence, while grotesque and incomprehensible to you, saves lives. You don't want the truth because deep down in places you don't talk about at parties, you want me on that wall---you need me on that wall."
*Apropos
The Collapse of Complex Societies by Joseph Tainter makes similar points. He’s an archaeologist but his analysis isn’t limited only to those complex societies that have left only physical evidence but few records.
You did good by not rushing to come out with a take. I assume you're familiar with https://tinkzorg.wordpress.com/2021/10/20/welcome-to-jurassic-park/?
I don't know if you have seen this - very interesting, fact-based explanation implies the move to one source of software was big part of the problem, rather than having multiple systems and backups.
https://twitter.com/PatrickByrne/status/1712536185065844775?s=20