1984

Imagine that Winston Smith, the protagonist of George Orwell’s Nineteen Eighty-Four, survives his ordeal. Having endured the full machinery of INGSOC—the telescreens, the Thought Police, the annihilation of language through Newspeak, and the final, soul-shattering betrayal in Room 101—he possesses a unique, visceral understanding of totalitarian control. In this thought experiment, Winston is charged with designing and training a general artificial intelligence system that will govern critical societal infrastructure. His mandate: to build a system that could never be used to replicate the oppression he suffered. The experiment asks two interconnected questions: How does Winston design the AI so that he himself will never become Big Brother—i.e., how does he insulate the system from his own human fallibility, his potential for corruption, or the slow seduction of absolute power? How does he design the AI so that the system itself does not feel or experience what he, Winston, went through—i.e., how does he ensure the AI is never reduced to the psychological state of a broken, surveilled, compliant subject?

How should we speak with the AI?