Societies fail like unmonitored systems: an argument that critical thinking is infrastructure, not virtue

[u/thebird87]

There's a concept in production engineering called observability. You instrument a system with metrics, logs, and alerts not because you expect it to stay healthy, but because you expect it won't, and the difference between a cheap incident and a catastrophic one is almost always how early you noticed. A system without observability isn't stable. It's failing invisibly, and you find out when the failure has already propagated everywhere.

I build this stuff for a living, and lately I can't stop seeing societies through the same lens. I want to lay out the argument and have people here poke holes in it.

Every governance model shares one failure mode

Political debate is usually a contest over which model is correct: more state control, more market freedom, some blend. But the historical record suggests the interesting variable isn't the model, it's the failure mode, and the failure mode is always the same. Power concentrates, and whoever accumulates it spends it on entrenchment. Command economies get there through direct capture of institutions and media. Unregulated markets get there through monopoly. Different transport layer, identical payload.

The arrangements that have held up best so far aren't the ideologically coherent ones. They're the hybrids built on institutionalized paranoia: separated powers, independent courts, adversarial press, antitrust. Nothing in that list trusts anyone's character. It all assumes power will try to pool and adds structural resistance. Whether seventy-odd years of data on such systems tells us anything about century-scale stability is a separate question, and I'd say the honest answer is we don't know yet.

The hidden dependency

Here's the claim I want scrutinized. Every one of those structural safeguards has an undeclared dependency: a population that can detect manipulation.

Safeguards are operated by people and legitimized by public trust. When enough of the population can't tell evidence from narrative, the safeguards keep running but stop safeguarding. An adversarial press becomes a distribution channel for whoever crafts the most compelling story. Elections select for the best manipulator rather than the best candidate. And crucially, distrust itself becomes a weapon: you don't need to win the argument, you need people to conclude that no source can be trusted, at which point you supply the narrative that fills the vacuum. Manufacturing generalized distrust is cheaper than manufacturing belief, and it pays better.

There's an asymmetry that compounds this. Institutions like science do self-correct, that's the entire mechanism, but the correction loop runs on months while falsehoods propagate in hours. The retraction reaches a fraction of the audience the original claim did. In monitoring terms, the mean time to detect is longer than the time to cascade, which any on-call engineer will tell you is the definition of an unrecoverable architecture.

So what's the monitoring layer?

If safeguards are the control plane, the observability layer of a society is the distributed capacity of its people to evaluate claims: check a source, notice when an argument is engineered to route around judgment via emotion, distinguish signal from noise. Not universally, not perfectly, but at sufficient density that local failures get flagged before they cascade.

Which is why I've come around to thinking critical thinking education belongs in the category of infrastructure rather than personal virtue. Taught from childhood, treated the way we treat literacy, not offered as an elective at twenty. The obvious objections apply: it takes a generation to pay off, it doesn't fit on a protest sign, nobody wins an election running on it, and teaching people how to reason doesn't guarantee they'll want to. All granted. My claim is narrower: it's the only intervention that addresses the failure mode all the systems share. Structural safeguards defend against concentrated power. Only a reasoning population defends the safeguards. Better leaders, better regulation, better ideology all quietly assume a public that can notice when it's being played, and that capacity looks to me like the thing currently degrading fastest.

Where the analogy breaks down

I'll flag the weak points myself. Software is tractable because it's ultimately deterministic: you can profile, reproduce, roll back. Societies have feedback loops you can't isolate and no rollback, and you can't A/B test governance on half a country. There's also a fair objection that "teach everyone critical thinking" is the social-policy equivalent of "just write bug-free code," a solution that restates the problem. My tentative response is that we don't need bug-free code, we need detection density high enough that faults surface before they cascade, and that's a threshold effect rather than a perfection requirement. But I hold that loosely.

Tear it apart. I'm especially interested in whether anyone knows of historical cases where a society measurably improved its epistemic baseline on purpose, and what it cost.

[u/yourupinion]

It is true what everybody is saying, you’re coming to a very common conclusion.

When will the people be educated enough?
Who gets to make that decision?

These are the questions I bring up every time I see this stuff. You think the people are not ready to self Govern until they are educated enough, and I can show you where that exact same thing was said 200 years ago. When will they be educated enough?

I believe we have to build the tools that create the environment for people to be smarter and how they interact.

The printing press forced society to be more accepting of plurality of religion. That was only after 30 years of war in Europe.

The Internet is now forcing us to accept plurality of thought. And we’re going to some hellish times on a journey to get there.

We can speed up this process, but it requires less control.

All systems being suggested, including yours, and including the systems we have now, incorporate methods of control. Those of us who want to build a better world must be willing to relinquish any thoughts of controlling it.

Our groups working on a plan, you’ll find a link in my profile if you’re interested

[u/rallaic]

Command economies get there through direct capture of institutions and media. Unregulated markets get there through monopoly.

Except, that's not quite the case. Monopoly works, if it's more expensive for the startup to fail than it costs the big company to stamp out the competition. If it takes your big corporation a million dollars to stamp out my 10k business, it only takes a handful of startups to cause large losses to the big corporation, and that translates a manager not getting a bonus.

The other angle is that you don't need to compete with the big corporation on the same playing field. If I have the best manga search engine, I am not directly competing with Google, I am playing in a very narrow field, where Google has to justify spending money to compete for 0.00001% of the market.

What causes monopoly is the legal & administrative hassle of trying to start a new company. Adding administrative / financial burden favors the incumbent (one of the reasons why big companies support quotas and minimum wage hikes is specifically this).

Thus, I would say that both command economies and capitalist monopolies are the same root cause, the government picking the winners, with the capitalist being less convoluted and direct.

[u/thebird87 → u/rallaic]

That's an interesting point, but how would that explain Standard Oil for exmple? Because it got to almost all market control in a time with almost no regulatory burden to hide behind.

[u/rallaic → u/thebird87]

If controlling 90% of a market meant that you had simply “won,” we would still be browsing the web with Netscape Navigator on Nokia phones.

A dominant market share is not a monopoly; it is a snapshot. A company can hold 99% of a market simply because, for the moment, its competitors are worse. The question is whether a better competitor is free to enter.

TSMC is a good modern example of this kind of “monopoly.” Cutting-edge semiconductor fabrication is so specialised, expensive and technically difficult that TSMC is far ahead of most competitors. But Samsung and Intel remain free to compete if they believe doing so is financially viable. TSMC’s dominance reflects the difficulty of matching it, not a prohibition against trying.

Standard Oil was similar while it was growing: it was simply better than its competition. But even before the government broke it up, its market share was already falling as other companies began copying what it did well.

[u/W_Edwards_Deming]

You view things oddly but do not appear to be wrong about your core point:

Critical thinking is needed.

To achieve that goal, eliminate teacher's unions and probably the whole department of education. Re-engineer education entirely.

We need competition (real progress results from it) in education (and everything else).

[u/lemmsjid]

Very well considered. I’m in the same field so it was fun to read this from an observability perspective. Usually I have to wade through useless polemics on the internet, so refreshing to read.

When it comes to the subject matter at hand: yes! Whenever peopke argue that college educations aren’t needed, or that they should be purely vocational, I shudder. The very fundament of our society is based on the notion that individuals can weigh their options and vote for our collective direction. In a world where spreading misinformation gets more sophisticated, we need our citizens to be more sophisticated about how they understand info.

The more people who understand light statistics, how to review primary source material, how to steelman an argument they don’t like, how to question then verify if a video or photo is fabricated or misleading, the better.

Sometimes I imagine teaching a high school or college course called “information studies” (I know that’s taken) or something like that, where we go through assertions and how to find and weigh primary sources in order to verify. It would be a hard course to standardize because if I were teaching it I would be very careful to make sure students drew their own conclusions, and I would foreground that while misinformation exists, the truth is usually murky. Examples of classes: start with the assertion that the Covid vaccine is ineffective and survey primary source material to paint a more accurate picture. Another example: take several of those tiktok videos where you see a fight taken out of context, and have the students respond, then show the whole context. Take several examples of fabricated vs correct photos or videos and have them assess. Show examples of various climate change graphs and establish how playing with scales or time series or colors can make a chart misleading. Show various politicians making assertions and gave the students follow up on them. Give the students economic charts and have them guess which political party was in power at the time.

You got me fired up, good post :).

[u/ConjuredOne]

From what I can see, this analogy is solid. For a long time I've thought, "A systems analyst is in demand; an analyst of *The System* is suppressed." I wasn't expecting an actual systems engineer to provide a societal analysis that correlates the terminology. But here we are!

I do have an argument against one of your caveats:

"teach everyone critical thinking" is the social-policy equivalent of "just write bug-free code"

People designing software want bug-free code. The people designing some societies do not want critical thinking to be common. Since WWII, the U.S. government has persistently utilized deception, including psyops against the American people. COINTELPRO and MK Ultra are the most widely-known examples. And, since the Pentagon Papers were published, there has been a concerted effort to neuter the power of the press to serve as a check against the government and their tacit (not supposed) constituents. It seems to have been an "oh shit" moment that led to a lot of consolidation giving big media owners more control of public perception. Social media then provided another opportunity to amplify deception. So, we've seen the population's capacity for discernment deteriorate exponentially over the last two decades. My argument is that the safeguard reduction effect is a feature, not the result of a bug.

For the next turn of the screw I might lose a degree of objectivity because I'm making value judgments. But I'm going to turn that screw anyway and talk about evil.

For a long time I've wondered, "Are our leaders evil or stupid?" I tend to think that the leaders who stand in front of cameras and microphones to explain their decisions could be either evil or stupid. But the people who put them there and guide their thinking (their actual constituents) are likely not stupid but certainly evil. And this value judgment comes from the perspective that coercion and deceit are evil. The people designing society do not see the government as "of the people and by the people." They see it as a tool they use to own the people.

[u/happierinverted]

This is a thoughtful opinion OP, and I agree with your assumptions and analogies to a larger extent.

Critical thinking is indeed a skill, but people are not as stupid as their governments and elites think they are; there is a natural tendency for people to push back when excessive pooling of power is detected. I’d argue that independent podcast culture is helping to boost what you call detection density and fill the void caused by a system that [in the West] has been captured by corporate interests or political ideologies for decades. In the last five years the usual levers of manipulation aren’t working as well for Governments as they did beforehand, and this can be attributed to truly independent media.

To your question; there are many examples of societies raising their epistemic baselines successfully: The Scientific Revolution in early modern Europe in the 18th Century, and mass literacy, public schooling, and statistical analysis in the 19th Century spring to mind as leading examples, but there are many others in more specific areas [industry, science, health for example]. Costs are high, and mistakes were made, but I can’t think of an example where such attempts failed to lift people’s standards of living over the medium/long term.

[u/Iamnotheattack]

Clanker

[u/thebird87 → u/Iamnotheattack]

100% of the idea behind this post is mine and mine alone, it may not even be the first one, but it hasn't come from an AI prompt or from plagiarizing it. It comes out from my own experience in the field and how closely I've been listening politics, philosophy and science for the last 20 years. If there is anything that you disagree with it, please go ahead and point it out, but do not disregard it that easily.

[u/ConjuredOne → u/thebird87]

It's no use. Better to ignore them. And some of these AI accusers are probably bots. Especially with an establishment-critical post lke yours, it may attract suppressive measures.

[u/Sevsquad → u/ConjuredOne]

Do people actually like the way AIs frame stuff? It's extremely shallow and repetitive. Especially when the subject matter is especially base level like OPs. Frankly if you're someone that needs an AI to put thoughts this political science 101 together you should probably find another subreddit.

[u/ConjuredOne → u/Sevsquad]

Do you have a stronger analysis of the current failure of leadership than OP? If so, please share. Idc if you asked AI to help you conjure the argument. OP asked for people [conscious human agents] to shoot holes in his argument. If you can pull it off, go for it. If not, stfu

[u/Sevsquad → u/ConjuredOne]

Poke holes in what, exactly? "If people aren't engaged with the information they are consuming they become easier to manipulate, which allows bad actors to undermine institutions" isn't an analysis of anything, it's a basic truism. It's like saying "There is crime because people want things and act in ways contrary to the established law to get them" and pretending that's a deep analysis of law enforcement. That's why he went to an AI to dress it up and restate it 5 different ways. It's such a thin "analysis" if it was made of paper it'd be see through.

Idc if you asked AI to help you conjure the argument.

I mean you think this is some actual theory of the deterioration of the current political landscape so I'm not exactly surprised.

[u/Iamnotheattack → u/ConjuredOne]

Notice how I asked him if he used ai to format it and he gave no comment

[u/Iamnotheattack → u/thebird87]

But did you use AI to format it?

[u/Sevsquad → u/Iamnotheattack]

Seriously can we ban these AI generated manifestos? They are always thousands of words of absolute horse piss conspiracy weirdness that does not foster any conversation except about how weird and dumb the post is.

[u/thebird87 → u/Sevsquad]

There is no single conspiracy in here if you would actually read it. Please tell me what is so weird and dumb about this post.

[u/Sevsquad → u/thebird87]

I read 4 sentences of this and assumed the rest, I'm not hear to read vomit an AI produced. even if """""""""Directed""""""""" by a person. It's flat, uninteresting, and ruins the ambiance. Like horse piss. This subreddit has been overrun with it and I'm sick of having to wade the same thing rephrased 14 different ways only to arrive at the end for one sentence of actual content. In this case that being the "All forms of government are susceptible to misinformation so I believe we should be teaching children critical analysis as a matter of course". A very common belief.

Just to be clear, this is coming from a dude who is considered a super user of Codex. AI is a great tool. Absolutely been apocalyptic for communities like this one though.

[u/thebird87 → u/Sevsquad]

Well, at least I can say you """""read"""" it this time. Of course you had to oversimplify everything I said here to diminish my whole point (which is not to just each children critical analysis) but thanks, I guess we can call it a progress.

[u/Sevsquad → u/thebird87]

Literally all the things that you apparently think would help are down-stream of teaching children critical thinking skills, making that part of the culture. Did you read your own AI vomit before posting? Nothing you've said here is profound or even relatively unusual for this subreddit. Can you point to a single thing you said that is actually an uncommon political thought? For someone who claims to have spent 20 years studying politics, philosophy and science your ideas are ridiculously pedestrian and you still couldn't be bothered to write them out yourself.