« On the Virulence of Malware | Main | Reflections on Zotob »
June 30, 2005
Let go!
I had a conversation recently with someone about the dangers of automating too much functionality in our information technology systems. We were talking about security and he was worried that automatically stopping attacks could cause a lot of collateral damage through false positives. For this reason, he would much rather keep a human in the loop. Furthermore, he was very wary of mechanisms that he couldn't understand or see the inner workings of. He didn't want anything that would function as a "black box".
Legitimate concerns, indeed, but shortsighted none-the-less. With the growing complexity of systems we can no longer maintain fine control over them, we need systems that look after themselves, and to do that they need to be autonomous and adaptive. And as soon as they become autonomous and adaptive, they will very likely move outside the realm of being easy to understand. But I would argue that this is a good thing, not a bad thing. We have to undestand the implications, yes, but we also have to surrender control. It's the only way forward.
A good way to illustrate this is to think of how useful horses have been to humanity throughout history. We can control them and understand very well what to expect from them, but for most of our history we had no idea whatsoever of how they work inside. In future, we will have machines and computing systems that we don't understand at all, but that function much better than they do today, that repair themselves, and adapt and evolve autonomously.
When we think of this kind of future, a key question is how do we know such systems work? Well, in the same way that we know that horses work: by observation and through black-box testing. We know how fast a typical horse can run and for how long because we have "tested" so many of them. We don't have any guarantees however - we could get a horse that was lame and so would fall far outside the expected performance. But that is a chance we take, and it's a small price to pay for a horse that fights off most infections without any assistance whatsoever, that can learn and grow and survive autonomously.
We need to let go: it's time for our systems to look after themselves so that one day they may look after us.