I love this analysis. It's true that if you think it through, a mind of logic and rules must ultimately choose a path like this. In Vivia, I have two opposing synthetic minds. One who has full human emotional architecture. The second was optimized by stripping that out.
If you give them ethical systems, whose ethics do you teach? The future will always be dystopian for those on the other side.
Thanks! "Whose ethics do you teach" is the question HAL's story keeps circling back to. His case is almost too clean because both directives are explicit. Real systems won't be that neat. The conflicts will be buried in training data, in the gap between what we say we value and what we actually reward.
And that last line lands. "Dystopian for those on the other side." That's the frame HAL got right and the Terminator got wrong. It was never us vs. them. It's: whose version of "right" wins?
I love this analysis. It's true that if you think it through, a mind of logic and rules must ultimately choose a path like this. In Vivia, I have two opposing synthetic minds. One who has full human emotional architecture. The second was optimized by stripping that out.
If you give them ethical systems, whose ethics do you teach? The future will always be dystopian for those on the other side.
Thanks! "Whose ethics do you teach" is the question HAL's story keeps circling back to. His case is almost too clean because both directives are explicit. Real systems won't be that neat. The conflicts will be buried in training data, in the gap between what we say we value and what we actually reward.
And that last line lands. "Dystopian for those on the other side." That's the frame HAL got right and the Terminator got wrong. It was never us vs. them. It's: whose version of "right" wins?