
Using the example of machine learning he illustrates his point about domain knowledge:
- Computer scientists do not have a deep understanding of the mathematics used in machine learning systems.
- There is no reason or incentive for computer scientists to even invest their time in learning those things.
This lack of knowledge means ethical issues and security issues that may be hidden at a conceptual level – not a technical one – are simply ignored.
Mickens compares machine learning to the standard experiment used in America since 8th grade: the egg drop experiment. This is where students desperately search for a solution to prevent the egg from breaking when dropped from 20 feet in the air. When they finally come up with a technique that is successful, Mickens explains, they don’t really care to understand the logic/math behind it.
This is exactly the same as developers in the context of machine learning. Machine learning is complex, yes, but often, Mickens argues, developers will have no understanding as to why models generate a particular output on being provided with a specific input.
When this inscrutable AI used in models connected with real life mission critical systems (financial markets, healthcare systems, news systems etc) and the internet, security issues arise. Indeed, it begins to raise even more questions than provide answers.
Now that AI is practically used everywhere – even to detect anomalies in cybersecurity, it is somewhat scary that a technology which is so unpredictable can be used to protect our systems.
Examples of poor machine learning design
Some of the examples James presented that caught our attention were:
- Microsoft chatbot Tay- Tay was originally intended to learn language by interacting with humans on Twitter. That sounds all good and very noble – until you realise that given the level of toxic discourse on Twitter, your chatbot will quickly turn into a raving Nazi with zero awareness it is doing so.
- Machine learning used for risk assessment and criminal justice systems have incorrectly labelled Black defendants as “high risk” – at twice the rate of white defendants.
It’s time for a more holistic approach to cybersecurity
Mickens further adds that we need a more holistic perspective when it comes to security. To do this,, developers should ask themselves not only if a malicious actor can perform illicit actions on a system, but also should a particular action on a system be possible and how can the action achieve societally-beneficial outcomes.
He says developers have 3 major assumptions while deploying a new technology:
#1 Technology is Value-Neutral, and will therefore automatically lead to good outcomes for everyone
#2 New kinds of technology should be deployed as quickly as possible, even if we lack a general idea of how the technology works, or what the societal impact will be
#3 History is generally uninteresting, because the past has nothing to teach us
According to Mickens developers assume way too much. In his assessment, those of us working in the industry take it for granted that technology will always lead to good outcomes for everyone. This optimism goes hand in hand with a need for speed – in turn, this can lead us to miss important risk assessments, security testing, and a broader view on the impact of technology not just on individual users but wider society too.
Most importantly, for Mickens, is that we are failing to learn from mistakes. In particular, he focuses on IoT security. Here, Mickens points out, security experts are failing to learn lessons from traditional network security issues. The Harvard Professor has written extensively on this topic – you can go through his paperon IoT security here.
Perhaps Mickens talk was intentionally provocative, but there are certainly lessons – if 2018 has taught us anything, it’s that a dose of scepticism is healthy where tech is concerned. And maybe it’s time to take a critical eye to the software we build.
If the work we do is to actually matter and make a difference, maybe a little negative is a good thing. What do you thi