Psychiatric Disorders of Automata - Rodrick Wallace - Bog

The relentless march toward massively parallel 'Self-X' machines that must program, protect, and repair themselves while managing critical phenomena raises questions of safety and stability. What are the failure modes of the new machines that will direct nuclear reactors, power grids, air traffic control and large-scale financial transactions? Is 'graceful degradation under stress' a realistic expectation? Study of animal cognition suggests that evolution has had only modest success stabilizing individual minds and formal analysis shows that both sudden and subtle failures in massively parallel cognitive structures are norms: Such devices are inherently as difficult to stabilize as to program. Implications for the design of critical systems are starkly troubling.