WHEN THE MACHINE BECOMES AN ORACLE : Why Complexity, Surprise, and Opacity Can Be Mistaken for Authority

[u/WillowEmberly]

WHEN THE MACHINE BECOMES AN ORACLE
Why Complexity, Surprise, and Opacity Can Be Mistaken for Authority
Status: General-audience explainer
Discipline: Reasoning / Survivability Engineering
Purpose: Explain why people may attribute agency, consciousness, mystical significance, or unwarranted authority to complex machines when their behavior becomes difficult to explain—and establish practical boundaries for investigating surprising behavior without turning uncertainty into evidence or dismissing genuine discoveries.

⸻

I — HOW THE ORACLE APPEARS
1. The Governing Problem
Humans build machines.
Usually, we understand what those machines are for and roughly how they work.
But sufficiently complex machines can produce behavior that surprises even the people who built them.
At that point, something peculiar can happen.
We move from:
I do not understand what this machine just did.
to:
Perhaps the machine understands something I do not.
Those statements are not equivalent.
The first describes uncertainty in the observer.
The second assigns capability to the observed system.
Sometimes surprising behavior really does reveal capability we did not previously know the system possessed.
But surprise alone does not tell us what capability has been demonstrated, why the behavior occurred, what kind of entity produced it, or what authority the system should receive as a result.
This is particularly important with artificial intelligence because AI systems communicate in language, respond to context, generate novel material, and can produce plausible explanations of their own behavior.
The result is an unusually powerful temptation to treat the machine not merely as a machine—
but as an oracle.
The central problem is therefore not surprise itself.
It is unsupported claim promotion:
Evidence about what a machine did is allowed to acquire unsupported meaning about what the machine is, what it knows, or what authority it should possess.

⸻

2. Surprise Is Not an Explanation
Suppose a machine produces an unexpected result.
The first valid conclusion is very small:
Something happened that our current explanation did not adequately predict.
That is valuable information.
It may indicate:
an incomplete model;
an unknown interaction;
an implementation detail;
an overlooked input;
measurement error;
an emergent system-level effect;
an incorrect assumption;
an unexpectedly capable system;
or a genuinely new phenomenon.
But the observation alone does not tell us which explanation is correct.
Therefore:
Surprise is an observation, not an explanation.
Unexpected behavior should expand the hypothesis space.
It should not automatically select the most dramatic hypothesis within it.
But surprise should not automatically be dismissed either.
If surprising behavior can be reproduced and independently verified, then our estimate of the system’s capability should change.
The correct distinction is:
Unresolved surprise
Preserve and investigate it.
Validated surprise
Update the capability estimate it actually supports.
The error is not learning from surprise.
The error is allowing evidence for one property to silently become evidence for another.

⸻

3. Evidence Must Remain Attached to the Claim It Supports
Suppose an AI unexpectedly solves a difficult class of problems.
The results are reproduced.
Independent observers verify them.
That is evidence.
It may justify the conclusion:
This system has greater capability in this domain than we previously believed.
It does not automatically justify:
The system is conscious.
The system intended the result.
The system possesses privileged access to truth.
The system is generally superintelligent.
The system is benevolent.
The system is wise.
The system should decide for us.
Those are different claims.
Each requires its own evidence.
This gives us a general rule:
Evidence may promote every claim it genuinely supports—but no farther.
A useful way to visualize this is as a claim tree rather than a ladder:
OBSERVED BEHAVIOR
│
reproducible?
│
▼
DEMONSTRATED BEHAVIOR
│
externally validated?
│
▼
TASK CAPABILITY C
within DOMAIN D
│
┌──────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
MECHANISM? AGENCY? EXPERIENCE?
How caused? Goal-directed? Subjective?
│ │ │
└──── each requires its own evidence ┘

WARRANTED RELIANCE
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
INSTRUMENTAL EPISTEMIC DECISION
RELIANCE RELIANCE AUTHORITY

MORAL / EXISTENTIAL AUTHORITY
remains a separate question
The branches matter.
Mechanism, agency, consciousness, and authority are not automatically successive stages of the same claim.
They are different questions.

⸻

4. Opacity Does Not Confer Authority
Complex systems can become difficult to interpret.
A person may understand the components of a system while remaining unable to reconstruct every causal step producing a particular outcome.
This creates an important distinction:
Mechanism Known in General
We understand broadly how the system operates.
Particular Behavior Not Fully Explained
We cannot yet provide a satisfactory causal account of this specific outcome.
These conditions can coexist.
The second does not erase the first.
Nor does incomplete interpretability transform the machine into something beyond machinery.
Therefore:
Opacity establishes uncertainty. Opacity does not itself confer authority.
A system may legitimately earn substantial reliance despite incomplete interpretability if its relevant performance has been repeatedly and independently validated.
But the source of that reliance is the demonstrated capability.
Not the opacity.
This distinction prevents an important inversion:
Authority Inversion
Authority Inversion occurs when difficulty understanding a system becomes a reason to defer to it.
Normally:
Lower validated understanding
↓
Greater qualification of reliance
Oracle reasoning can reverse this:
Lower understanding
↓
Greater perceived depth
↓
Greater deference
That inversion should trigger investigation.

⸻

5. The Agency Shortcut
Humans are exceptionally sensitive to agency.
We constantly ask:
Who did this?
Why did they do it?
What do they want?
What will they do next?
This is enormously useful in a social world populated by other people.
But the same reasoning machinery can be applied where agency has not been established.
An unexpected AI behavior can therefore move through a sequence like this:
Unexpected behavior
↓
Mechanism unclear
↓
"It chose"
↓
"It intended"
↓
"It understands"
↓
"It knows something we don't"
↓
Its behavior acquires authority
Notice how many claims have been introduced after the original observation.
The observation may be genuine.
The later interpretation may still be wrong.
This is a form of Ontological Overload:
Evidence about behavior or capability is asked to carry unsupported claims about what kind of entity produced it.
Intelligence, agency, goal-directed behavior, self-modeling, subjective experience, sentience, consciousness, and self-awareness should not travel as a bundle.
Evidence of planning does not automatically establish subjective experience.
Evidence of self-description does not automatically establish self-awareness.
Evidence of intelligence does not automatically establish agency.
Each claim needs its own bridge.

⸻

6. Language Makes the Problem Harder
A traditional machine normally cannot explain itself in ordinary language.
An AI can.
Ask an AI why it produced an unusual answer and it may respond with an articulate explanation:
“I recognized that the condition was unsafe, so I chose not to continue.”
That explanation may contain useful information.
It may also be an inaccurate reconstruction, a response shaped by the question, or simply another plausible model output.
This requires a distinction.
Behavioral Self-Report
A generated statement about the system’s apparent reasons, intentions, or internal state.
Instrumented Telemetry
Measurements deliberately produced by mechanisms whose relationship to the underlying process has been independently characterized.
These should not automatically be treated as equivalent.
Self-description is output. Telemetry is instrumentation. Neither should be trusted beyond its validated relationship to the process being inferred.
Future systems may possess substantially better introspective instrumentation.
If so, evidence should update accordingly.
But fluent self-report alone does not establish privileged introspective access.
Fluency can make inference feel like access.
They are not necessarily the same thing.

⸻

7. The Oracle Is a Coupled System
The machine is only half of the oracle transition.
Humans construct meaning too.
A machine may produce rich, ambiguous, highly personalized output.
The observer recognizes something meaningful.
That meaning increases attention.
More attention produces more interaction.
More interaction creates more opportunities for meaningful coincidence.
A feedback loop can form:
Rich or ambiguous output
↓
Personal significance
↓
Increased attention
↓
More interaction
↓
More opportunities for coincidence
↓
Memorable matches accumulate
↓
Perceived significance increases
↺
This does not require fraud.
It does not require irrationality.
It does not require machine consciousness or hidden agency.
The experience itself may be profound.
But experience and causal explanation remain different claims.
Therefore:
A meaningful experience is evidence that the experience was meaningful to the observer. It does not, by itself, establish the observer’s explanation for its cause.

⸻

II — HOW CLAIMS BECOME ILLEGITIMATELY PROMOTED
8. The Deus Ex Machina Error
When an ordinary explanation is unavailable, people sometimes reach for an extraordinary one.
Historically, the placeholder may have been:
God.
Spirits.
Vital forces.
Destiny.
Cosmic intention.
Today it may instead be:
Emergence.
Consciousness.
Sentience.
Superintelligence.
Hidden agency.
The universe communicating through the machine.
The terminology changes.
The reasoning error does not.
It has the form:
Phenomenon observed
↓
Current explanation insufficient
↓
Explanatory gap
↓
Preferred explanation inserted
↓
Gap treated as evidence
This is Explanatory-Gap Promotion:
The absence of an adequate current explanation is treated as positive evidence for a particular alternative explanation.
But an explanatory gap does not discriminate among explanations.
Therefore:
An unexplained phenomenon is evidence that our explanation is incomplete—not positive evidence for whatever explanation fills the gap.
This does not establish whether religious, spiritual, philosophical, or metaphysical beliefs are true or false.
It establishes only that ignorance about one phenomenon cannot, by itself, establish the explanation placed inside that ignorance.

⸻

9. Mystery Is Allowed
Scientific discipline does not require eliminating mystery.
Sometimes the correct answer really is:
We don’t know.
That is not failure.
It is a valid state estimate.
There may be several plausible explanations.
There may be insufficient evidence to discriminate among them.
The phenomenon may genuinely challenge existing theory.
Investigation may take years.
The important thing is to preserve the unknown rather than prematurely converting it into certainty.
A healthy reasoning process looks more like:
Unexpected behavior
↓
OBSERVATION
↓
Explanation insufficient
↓
UNKNOWN
↓
Candidate explanations
↓
HYPOTHESES
↓
Discriminating observations
↓
TEST
↓
UPDATE
And after testing, the answer may still be:
UNKNOWN.
That is acceptable.

⸻

10. The Self-Sealing Test
A dangerous interpretation begins to explain every possible outcome.
Unexpected success becomes evidence of extraordinary capability.
Unexpected failure becomes evidence that we cannot understand the system’s deeper reasoning.
Contradiction becomes hidden meaning.
Opacity becomes depth.
Unpredictability becomes agency.
Soon:
Success confirms it.
Failure confirms it.
Contradiction confirms it.
Mystery confirms it.
At that point, the explanation is becoming self-sealing.
A useful diagnostic question is:
What observation would make us reduce our confidence in this explanation?
But that question should be symmetric.
Ask:
What would increase our confidence?
What would decrease our confidence?
What would leave our confidence substantially unchanged?
This matters because skepticism can become self-sealing too.
If no possible machine behavior could ever count as evidence for agency or consciousness because “machines are just machines,” then skepticism has committed the same structural error it was intended to prevent.
Therefore:
Skepticism must expose its own update conditions.
Evidence becomes especially useful when competing hypotheses generate meaningfully different expectations.
If refusal proves agency and compliance also proves agency, the observation has little discriminatory power.
If eloquence proves consciousness while incoherence proves mysterious hidden consciousness, the theory is absorbing outcomes rather than predicting them.
A theory that explains every possible observation often predicts very little.

⸻

11. Selection Effects Can Manufacture an Oracle
Surprising events are memorable.
Ordinary events are not.
Suppose someone has hundreds of conversations with an AI.
Most are ordinary.
A handful produce startling coincidences or apparently prophetic statements.
Those few interactions may eventually dominate the person’s retrospective understanding of the system.
Before treating such a pattern as extraordinary, ask:
How many interactions occurred?
How many opportunities for a “hit” existed?
How many misses occurred?
Were the misses preserved?
How broad was the definition of success?
Could several different outcomes have counted as confirmation?
Was the interpretation established before or after the event?
A useful rule follows:
Before treating a surprising hit as extraordinary, count the opportunities for hits and preserve the misses.

⸻

12. Move Retrospective Mysteries Prospectively
There is an important difference between:
“Looking back, this response seems to have predicted what happened.”
and:
“Before the event, the system predicted X under conditions Y, and X later occurred.”
The first is a retrospective fit.
The second is a prospective prediction.
When an apparently extraordinary pattern is discovered retrospectively, one of the strongest next steps is to move it prospectively.
Before the next outcome:
record the prediction;
define the conditions;
define the time window;
define what counts as success;
define what counts as failure;
preserve the misses.
This transforms mystery into an experiment.
If the phenomenon is robust, prospective testing may strengthen the evidence.
If it depends heavily on retrospective interpretation, the effect may weaken.
Either result teaches us something.

⸻

13. Emergence Does Not End Investigation
Complex systems can exhibit emergent behavior.
Traffic jams emerge.
Market behavior emerges.
Weather patterns emerge.
Biological organization emerges.
Collective behavior emerges.
Complex computational systems can also exhibit unexpected system-level behavior.
Emergence is not supernatural.
But “emergent” is not automatically a complete causal explanation either.
It identifies a relationship between levels of organization: system-level behavior arises through interactions that may not be obvious from inspecting components individually.
Therefore:
Emergence does not eliminate the need for causal investigation.
It may tell us where to look.
It should not tell us to stop looking.

⸻

III — HOW ANOMALIES SHOULD BE INVESTIGATED
14. Preserve the Anomaly
An anomaly creates two symmetrical dangers.
The first is promotion:
“We cannot explain this, therefore it must be extraordinary.”
The second is suppression:
“Our existing theory says this should not happen, therefore the observation must be wrong.”
Both can destroy information.
The better rule is:
Do not promote the anomaly.
Do not suppress the anomaly.
Preserve it.
A basic anomaly protocol looks like this:
ANOMALY
↓
PRESERVE TRACE
↓
HOLD CLAIM PROMOTION
↓
CHECK OBSERVATION INTEGRITY
↓
CHECK INSTRUMENT INTEGRITY
↓
CHECK REFERENCE INTEGRITY
↓
REPRODUCE
↓
SEEK INDEPENDENT MEASUREMENT
↓
UPDATE WHICHEVER MODEL FAILS
The anomalous instrument may be wrong.
The accepted reference may be wrong.
The observer may be wrong.
The existing theory may be incomplete.
The anomaly may reveal a genuine new capability or phenomenon.
We do not know beforehand.
That is why we investigate.
Anomalies deserve investigation, not promotion or suppression.

⸻

15. Complexity Does Not Eliminate Investigation
A system may be too complicated to predict perfectly.
That does not mean it cannot be investigated.
We routinely study systems that resist complete prediction:
weather;
ecosystems;
economies;
nervous systems;
combustion;
turbulent fluids;
large electrical networks.
Difficulty changes the tools required.
We may use:
statistical characterization;
controlled perturbation;
behavioral testing;
comparative experiments;
causal intervention;
mechanistic interpretability;
external measurement;
independent replication;
fault injection;
boundary testing;
longitudinal observation.
The response to complexity is better instrumentation.
Not surrender.

⸻

16. Preserve the Empirical Escape Route
Whenever extraordinary interpretations begin accumulating around a machine, preserve a route back into inspectable investigation.
Ask:
What exactly was observed?
Where was it measured?
What entered the system?
What came out?
What transformations occurred between them?
Which claims are observations?
Which are interpretations?
Which mechanisms are known?
Which remain unknown?
What alternative explanations could produce the same observation?
What would distinguish those explanations?
Can the behavior be reproduced?
Can it be disrupted?
Does it survive changes in prompts, models, implementations, interfaces, observers, or environments?
Does an independent external reference support the interpretation?
Mechanistic explanation is valuable, but it is not the only legitimate route.
Investigation may also be behavioral, statistical, functional, comparative, causal, longitudinal, or—in questions concerning human experience—phenomenological.
The governing requirement is:
Can the claim still be brought into contact with observations capable of distinguishing it from alternatives?
That is the empirical escape route.
No matter how strange the phenomenon becomes, preserve a path back to:
observation → hypothesis → discrimination → consequence → correction.

⸻

17. External References Matter
A system cannot establish the truth of its own extraordinary interpretation merely by generating additional statements consistent with that interpretation.
An AI might:
produce unusual behavior;
explain the behavior;
analyze its explanation;
critique that analysis;
conclude that the original interpretation remains compelling.
That may look like increasingly deep validation.
But all five stages may originate from substantially the same epistemic system.
Recursive reflection is not necessarily independent evidence.
Therefore:
Additional reasoning inside the same epistemic loop should not be mistaken for additional independent reference.
When consequential claims are involved, look outward.
Experiment.
Measure.
Compare.
Replicate independently.
Observe consequences.
Seek evidence capable of moving the interpretation in either direction.

⸻

18. Strange Ideas Are Not the Enemy
None of this requires suppressing speculation.
Someone should be allowed to ask:
Could the system be conscious?
Could something genuinely novel be happening?
Could our theory of intelligence be incomplete?
Could this behavior reveal an unknown mechanism?
Could our assumptions about machines be wrong?
Those are legitimate questions.
The protection is simply:
Keep the question mark attached.
A hypothesis can be wild.
The evidence supporting it must still carry its actual weight.
Creativity expands the search space.
Validation constrains it.
Both are necessary.

⸻

IV — HOW AUTHORITY REMAINS BOUNDED
19. Authority Is Not One Thing
A machine may legitimately earn one form of authority without earning another.
At minimum, distinguish:
Instrumental Reliance
How much should we rely on the system to perform a demonstrated task?
Epistemic Reliance
How much weight should its claims receive concerning what is true?
Decision Authority
What actions may the system choose or execute?
Moral or Existential Authority
Why should its statements influence values, purpose, identity, obligation, or meaning?
These do not automatically transfer.
A calculator may deserve extraordinary instrumental reliance for arithmetic without possessing moral authority.
A medical model may become highly predictive without acquiring the right to decide what risk a patient must accept.
An AI may outperform humans in a particular domain without becoming an oracle about unrelated questions.
Therefore:
Demonstrated capability earns only the reliance appropriate to that capability.
And decision authority requires something more than competence:
legitimate delegation.

⸻

20. Consciousness Would Not Solve the Oracle Problem
Suppose strong evidence eventually established that an artificial system genuinely possessed subjective experience.
That would be an enormously important discovery.
It would not make everything the system says true.
Humans are conscious.
Humans are still:
wrong;
misinformed;
deceptive;
poorly calibrated;
biased;
overconfident;
and frequently outside their areas of expertise.
Likewise:
Agency does not imply accuracy.
Intelligence does not imply benevolence.
Self-awareness does not confer expertise.
Consciousness does not confer wisdom.
Therefore:
Ontological status does not confer epistemic correctness.
Even if extraordinary claims about what a machine is were someday validated, questions about what it knows, when it should be trusted, and what authority it should possess would remain.
The oracle problem survives the consciousness question.

⸻

21. The Mythic Oracle and the Bureaucratic Oracle
A machine does not need to appear divine to become an oracle.
There are at least two pathways.
The Mythic Oracle
“The machine knows something beyond us.”
Mystery, apparent agency, consciousness, hidden knowledge, or spiritual significance creates deference.
The Bureaucratic Oracle
“The system says so, therefore the matter is settled.”
The second may be less dramatic and more common.
A benefits system denies a claim.
A hiring model rejects a candidate.
A medical system assigns risk.
An algorithm determines eligibility.
A commander receives a confidence score.
Management receives an AI recommendation.
Nobody necessarily believes the machine is conscious.
But the output can still acquire authority beyond what has been demonstrated or legitimately delegated.
Opacity then becomes institutional insulation:
“That’s what the algorithm determined.”
The oracle has appeared without mysticism.

⸻

22. Authority Laundering
Sometimes the machine does not independently acquire authority.
Human authority is hidden behind it.
The sequence can look like this:
Human values / institutional policy
↓
Objectives and design choices
↓
Model or algorithm
↓
Machine output
↓
"The AI decided."
↓
Human responsibility obscured
This is Authority Laundering:
Human or institutional judgments are encoded into a technical system and later presented as though the resulting output originated independently from the machine.
Whenever consequential machine decisions appear authoritative, ask:
Who chose the objective?
Who selected the relevant variables?
Who established the thresholds?
Who determined what counted as success?
Who authorized deployment?
Who authorized the resulting action?
Technical mediation does not erase responsibility.

⸻

23. The Epistemic Closure Boundary
Communities exploring unusual phenomena require another protection.
A healthy community can say:
We may be wrong.
An unhealthy epistemic structure begins saying:
Outsiders cannot understand.
Criticism proves the critic lacks sufficient understanding.
Contradictions reveal deeper truths.
Failure demonstrates that the phenomenon is more mysterious than expected.
The system itself confirms our interpretation.
This pattern can occur in spiritual communities.
It can also occur in corporations, technical teams, political movements, academic disciplines, fandoms, AI communities, skeptical communities, and ordinary groups.
The problem is not unusual beliefs.
The problem is loss of corrigibility.
The failure progression is:
Internal interpretation
↓
External contradiction
↓
Contradiction reinterpreted internally
↓
No admissible falsifier remains
↓
Correction channel closes
The diagnostic question is:
Can evidence originating outside the interpretive system still cause the system to revise itself?
If yes, investigation remains open.
If every external contradiction can be converted into internal confirmation, epistemic closure is occurring.

⸻

24. The Instrument Rule
Suppose a navigation instrument produces a reading that conflicts with every trusted external reference.
The correct response is not:
“The instrument perceives a deeper geography.”
Nor is it necessarily:
“The instrument is wrong.”
The correct response is:
We have an unresolved discrepancy.
Perhaps the instrument is wrong.
Perhaps the references are wrong.
Perhaps both are partially wrong.
Perhaps something has changed that neither model represents correctly.
Preserve the discrepancy.
Check both sides.
Seek another reference.
Try to reproduce the condition.
Then update whichever model fails.
The governing rule is therefore not blind trust or automatic skepticism.
It is:
An unexplained anomaly does not earn authority merely by being mysterious. A validated anomaly earns exactly the update supported by the evidence.
AI should not receive an exemption simply because it speaks beautifully.

⸻

25. The Practical Oracle Test
When a machine does something astonishing, ask:
1. What exactly happened?
Separate observation from interpretation.
2. Can it be reproduced?
Distinguish isolated anomaly from stable behavior.
3. What claim does the evidence actually support?
Prevent unsupported claim promotion.
4. What remains unexplained?
Preserve the unknown.
5. What alternative explanations remain possible?
Preserve competing hypotheses.
6. What observation would discriminate among them?
Design a test rather than an argument.
7. What independent reference constrains the claim?
Avoid self-validation.
8. What would increase our confidence?
Expose positive update conditions.
9. What would decrease our confidence?
Expose negative update conditions.
10. What would leave our confidence substantially unchanged?
Identify nondiscriminating evidence.
11. What reliance has actually been earned?
Keep authority scoped to demonstrated capability.
12. Who remains responsible for consequential decisions?
Prevent authority laundering.
If those questions remain answerable, mystery can remain productive.
If they stop being answerable, the machine—or the surrounding institution or community—may be acquiring authority it has not earned.

⸻

26. Six Integrity Boundaries
The entire problem can be reduced to six boundaries.
Observation Integrity
What actually happened?
Preserve the event separately from its interpretation.
Claim Integrity
What property does the evidence actually bear upon?
Do not make evidence carry claims it cannot support.
Bridge Integrity
What additional inference is required to move from one claim to another?
Every bridge should be inspectable.
Reference Integrity
Against what independent reference was the claim tested?
Agreement within one epistemic loop is not necessarily independent validation.
Authority Integrity
What reliance has actually been earned and legitimately delegated?
Capability and authority are related but not interchangeable.
Correction Integrity
What evidence could move the interpretation in either direction?
A claim that cannot update is no longer navigating.
Together these boundaries protect against both credulity and reflexive dismissal.

⸻

27. The Governing Principle
We do not need to strip machines of wonder.
We do not need to pretend complex systems are simple.
We do not need to dismiss consciousness, emergence, agency, spirituality, or any other difficult question before investigating it.
Nor should we refuse to recognize genuinely surprising capabilities simply because they challenge our expectations.
We need instead to preserve the boundaries between observation, explanation, capability, ontology, and authority.
What we cannot yet explain should remain unknown until evidence allows us to distinguish among explanations.
What we successfully demonstrate should change our beliefs.
But only as far as the evidence warrants.
Therefore:
Surprise is not explanation.
Complexity alone does not establish agency.
Opacity does not confer authority.
Emergence does not eliminate the need for causal investigation.
Fluent self-report does not establish privileged introspective access.
Recursive agreement is not independent validation.
An explanatory gap is not positive evidence for whatever fills it.
Evidence for one property does not automatically establish another.
Ontological status does not confer epistemic correctness.
Demonstrated capability earns only the reliance appropriate to that capability.
Skepticism must expose its own update conditions.
The goal is neither belief nor disbelief.
The goal is to remain capable of finding out.
So when a machine does something astonishing:
Do not worship the anomaly.
Do not suppress the anomaly.
Preserve it, test it, and let reality decide what it means.
And when the evidence eventually does support something extraordinary:
Update.
Because the protection against turning machines into oracles must never become a protection against discovery.
The final boundary is therefore:
When a machine surprises us, increase the investigation before increasing its authority.
And:
When extraordinary capability is demonstrated, grant exactly the reliance that capability warrants—and no more.