The evening started with an experiment of a narrow technical irritation: the effect of learning rate on gradient descent. The expectation was straightforward. Boyd’s classical convergence results prescribe an optimal step size. Theory was clear, for function:
Hessian eigenvalues are 2 and 200, the optimal constant learning rate for strongly convex quadratics gradient descent is:
Yet empirical results refused to cooperate:
An arbitrary "hand-tuned" learning rate was outperforming theoretically best one. The "wrong" parameter was winning.
The resolution emerged only after extending the simulations. Run long enough, the pattern inverted. The optimal learning rate overtook the naive one and secured the lower asymptotic error:
This trivial coding exercise exposes a structural fact: optimality is conditional on duration.
Gradient Descent and the Robbins–Monro Structure
Gradient descent, in its deeper form, is an instance of the Robbins–Monro stochastic approximation algorithm. The same mathematical skeleton appears across disciplines:
- Exponential smoothing
- Stochastic approximation procedures
- Simultaneous perturbation stochastic approximation (SPSA)
- Online filtering and tracking algorithms
All such schemes adjust an estimate incrementally using noisy data. Each update contains an adaptation parameter, commonly denoted α.
This parameter governs a fundamental trade-off:
- Variance increases proportionally with α.
- Tracking lag (bias in adapting to a moving or non-stationary target) increases inversely with α.
Small α → stable but sluggish. Large α → responsive but noisy.
This is simply the bias–variance dilemma in dynamic form. Any adaptive system must pay one cost or the other.
Relation to Human Condition: Illusion of Superiority
Short horizons reward aggressive learning rates. They adapt quickly and appear superior. Long horizons reward conservative rates. They reduce steady-state variance and eventually dominate.
The conclusion is neither profound nor comforting: "better" depends on how long the game is played. Optimization without a fixed horizon is ill-defined. This whole sitation mirrors human condition succinctly:
Most agents operate by a primitive, non-profound, unreflective and scripted goal: conduct existence in such a way that, when aggregated across time, it constitutes a life worth living.
So how does one do that? Typically the advice follows as a "life strategy" advice:
- Some advocate hedonism, immediacy, and momentary gains. High α.
- Others advocate discipline, asceticism, long-term spirituality. Low α.
The first may win in short horizons. The second may win in long horizons. Neither is inherently superior.
Optimality depends on environmental volatility, personal adaptability and lifespan.
None of these variables are known.
No accessible ground truth graph exists that plots "lifetime reward" against a "strategy". No one has observed the full trajectory of a life under counterfactual policies. No one has measured its variance or its asymptotic error.
Any individual claiming to have solved the optimization problem of life is either deluded or a fraud. The parameters of the system are unknown. The horizon is unknown. The noise model is unknown. Assertions of universal optimality are baseless.
In biological organisms, adaptation speed is not freely tunable. It is constrained by neurobiology, developmental history, personality structure and evolutionary design.
The learning rate is partially hardcoded. It cannot be dialed to zero or one without ceasing to function as an organism.
Worst of all, the most decisive variable - lifespan - is unknown to the individual system executing the policy.
There is no Such Thing as "Being Smart About Life"
Every adaptive strategy pays in bias or variance: overcommitment produces rigidity and lag, hyper-flexibility produces noise and instability.
There is no configuration that escapes this structural constraint.
What is commonly labeled "wisdom" is merely a temporary luck of balance under specific conditions. Change the environment or the horizon and the evaluation shifts.
There is no universal optimum.
Those who present themselves as "sages" who have solved the equation of life claim access to information that they cannot have. The system's graph is hidden.
The sober conclusion is that these proclaimed "life strategies" amount to little more than improvised narratives authored by cognitively bounded primates, unaware of their epistemic limits and ignorant of their actual position within the structure of the universe:
- Hedonism — maximization of pleasure and minimization of pain as the primary evaluative criterion.
- Nihilism — denial of inherent meaning, value, or objective purpose.
- Religiosity — submission to transcendent authority, doctrine, or divine teleology.
- Moralism — strict adherence to ethical codes as the organizing principle of life.
- Collectivism — prioritization of group welfare over individual preference.
- Individualism — elevation of personal autonomy and self-determination above communal claims.
- Stoicism — disciplined emotional restraint and acceptance of uncontrollable events.
- Asceticism — voluntary self-denial to cultivate purity, clarity, or transcendence.
- Materialism — pursuit of wealth, status, and tangible accumulation as measures of success.
- Utilitarianism — optimization of aggregate well-being across agents.
- Existentialism — self-authorship of meaning in an indifferent universe.
- Fatalism — belief that outcomes are predetermined and largely immune to intervention.
- Pragmatism — valuation of beliefs and actions strictly by functional outcomes.
- Altruism — systematic prioritization of others’ interests over self-interest.
- Egoism — explicit centering of one’s own benefit as the guiding rule.
- Transhumanism — commitment to technological self-enhancement and biological transcendence.
- Traditionalism — adherence to inherited norms and historical continuity.
- Revolutionism — orientation toward radical restructuring of existing systems.
- Minimalism — deliberate reduction of desires and possessions to essential elements.
- Ambitionism — relentless pursuit of achievement, power, or recognition.
Contemplate the spectacle of constructing entire ethical architectures, metaphysical systems, and ceremonial disciplines - rather than uttering the only actual honest position available: "I do not know".
Addendum
Examples of different circumstances/universes where either the "adaptive" vs "disciplined" person wins:
- The left graph is the signal graph that shows adaptation through time
- The right graph is the "regret" graph, how much person "pays" for not adapting to the signal precisely/carefully/fast enough.
- The orange plot is the strategy of the "disciplined" person with low parameter α, who says shit like "Consistency beats intensity" or "Delay gratification" or "You need to think about long term investements"
- The blue plot is the strategy of the "adaptive" person with high parameter α, who says shit like "Move fast or get left behind" or "Exploit the opportunity now" or "You should live in the moment"
- The first plot shows the "steady universe" type of scenario, where the disciplined person can precisely "attune" to the form of environment and win.
- The second plot shows the "stochastic" type of scenario, where the universe changes rapidly with no warning, the adaptive person quickly attunes to the changes in environment.
