“Technology, in general, and computer science in particular, have been hyped up to such an extreme level that we’ve ignored the importance of not only security but broader notions of ethical computing.”
We like to think that things are going to get better. That, after all, is why we get up in the morning and go to work, in the hope that we might just be making a difference, that we’re working towards something.
That’s certainly true across the technology landscape. And in cybersecurity in particular, the belief that you’re building a more secure world – even if it’s on a small scale – is an energizing and motivating thought.
However, at this year’s USENIX Conference back in August, Harvard Professor James Mickens attempted to put that belief to rest. His talk – titled ‘Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible?’ – was an argument for scepticism in a field that is by nature optimistic (not least when it has a solution to sell).
So, what exactly does Mickens have against keynote speakers? Quite a lot, actually: he jokingly calls them people who have made bad life decisions aand poorrole models.
Although his tongue is firmly in his cheek, he does have a number of serious points. Fundamentally, he suggests developers do not invest time in questioning anything since any degree ofintrospection would “reduce the frequency of git commits”.
Mickens argument is essentially thatsoftware developers are deploying new systems without a robust understanding of those systems.
Why machine learning highlights the problem with computer science today
Mickens stresses that such is the hype and optimism around modern technology and computer science that the field has largely forgotten the value of scepticism. In turn, this can be dangerous for issues such as security and ethics. Take Machine Learning for instance. Machine learning is, Mickens sayss “the oxygen that Silicon Valley is trying to force into our lungs.” It’s everywhere, we seem to need it – but it’s also being forced on us, almost blindly,
Using the example of machine learning he illustrates his point about domain knowledge:
- Computer scientists do not have a deep understanding of the mathematics used in machine learning systems.
- There is no reason or incentive for computer scientists to even invest their time in learning those things.
This lack of knowledge means ethical issues and security issues that may be hidden at a conceptual level – not a technical one – are simply ignored.
Mickens compares machine learning to the standard experiment used in America since 8th grade: the egg drop experiment. This is where students desperately search for a solution to prevent the egg from breaking when dropped from 20 feet in the air. When they finally come up with a technique that is successful, Mickens explains, they don’t really care to understand the logic/math behind it.
This is exactly the same as developers in the context of machine learning. Machine learning is complex, yes, but often, Mickens argues, developers will have no understanding as to why models generate a particular output on being provided with a specific input.
When this inscrutable AI used in models connected with real life mission critical systems (financial markets, healthcare systems, news systems etc) and the internet, security issues arise. Indeed, it begins to raise even more questions than provide answers.
Now that AI is practically used everywhere – even to detect anomalies in cybersecurity, it is somewhat scary that a technology which is so unpredictable can be used to protect our systems.
Examples of poor machine learning design
Some of the examples James presented that caught our attention were:
- Microsoft chatbot Tay- Tay was originally intended to learn language by interacting with humans on Twitter. That sounds all good and very noble – until you realise that given the level of toxic discourse on Twitter, your chatbot will quickly turn into a raving Nazi with zero awareness it is doing so.
- Machine learning used for risk assessment and criminal justice systems have in