Beguiled by Artificial Intelligence

High performing AI systems can lead to unpredictable outcomes unless its results are interpreted

Experienced educators know the importance of using good quality teaching aids to deliver effective lessons to students. If tasked to introduce shapes to small children, the teachers could start off with initially cutting out different coloured shapes such as a triangle, circle or a square. Next, they could visibly hold one shape at a time towards the students and announce its name. So, initially, they could display the blue triangle and loudly say ‘triangle’. Then, they could pick up the red circle and call out ‘circle’, and finally, they could lift the green square and announce ‘square’. This process can be repeated a few times to ensure that the shapes and their associated names are well registered with the students.

However, if the above steps are carefully evaluated, we will observe an underlying problem that results in teaching wrong concepts to the students. As the shapes consist of different colours, and as the teacher holds shape and loudly calls out its name, some students are likely to relate the name of the shape to its respective colour. For example, in the above scheme of things, it is completely reasonable for students to learn that triangle means the colour blue, whereas, circle means red and that the square is referred to the colour green. Although these are incorrect conclusions, nevertheless, in this instance they are logical since the called-out name can be linked to either a shape or a colour. Of course, good instructors also know how to avoid this issue in the first place by simply ensuring that all the shapes have the same colour. This small modification in ensures that the children will correctly associate the name with its appropriate shape and not confuse it with the colour.

In summary, if different colours were indeed used for the shapes, the training could result in students developing two distinctly opposite but consistent types of understanding. The first type of conclusion would associate the name with its correct shape, whereas, the second type of learning could relate the name of the shape to its colour. The key observation is that both outcomes are rational with respect to the data that was presented to the students, but only the first one is correct.

Incidentally, what is true for humans is also true for AI systems which are equally prone to draw wrong conclusions that are albeit consistent with data. To highlight this point, Marvin Minsky, a leading cognitive psychologist who is considered one of the pioneers of AI, narrated an incident pertaining to an earlier version of Artificial Neural Network (ANN) that was being developed to distinguish between friendly and enemy tanks. ANN, as well as its recent instantiation called Deep Learning Systems, consists of several interconnected processing units that transmit signals to each other. The technical architecture of the network broadly mimics the human brain that consists of an extremely large number of linked neuron cells with synaptic connections.

Minsky explained that the network was trained on a set of images duly labelled as being that of a friendly tank or otherwise. To their pleasant surprise, the network was able to learn, and it achieved good performance in terms of recognising one group of tanks versus the other. Paradoxically though, this high accuracy was achieved without it having learned anything about the tanks. Coincidently, what had happened is that the images belonging to the enemy tanks were cloudy, whereas, the images of friendly tanks were clear. Therefore, the network achieved its performance by simply learning to distinguish between cloudy versus clear images. Hence, just as before, the learnings of the artificial neural network were accurate in terms of the input data, but nevertheless also incorrect in terms of its goal to identify the two categories of tanks.

The critical issue here is that modern-day intelligent systems such as ANN are quite powerful in terms of both software and hardware capabilities. If presented with a large set of data with different categories, their learning process consists of uncovering patterns that uniquely identify each of the respective classifications. To achieve this objective, they bring to bear their enormous analytical capacity to discover a plethora of signature patterns that are unique for each category. This overabundance of patterns includes a large proportion that is merely coincidental or insignificant or exists due to outright data errors. The key challenge is that the network has no great means to prefer one such pattern over the other, and therefore, it often converges to a solution that may be accurate with respect to data but also outright incorrect as we saw in the case of the tank example.

Hence, like how it is with humans, the learnability of an AI system is tightly coupled to the quality of data that is fed to it. Alarmingly, this strong dependency has often led to perpetuating hidden biases that exist in the data which is used to train the network. For example, consider a hypothetical case of a finance expert that initially evaluates customer details and their banking transactions to categorize potential defaulters. This labelled dataset could then be used to train a network that could automatically classify risky customers and recommend mitigation steps. However, if the so-called expert secretly discriminated against people from a specific geographical location, this hidden bias is likely to be ‘learnt’ by the network in making predictions regarding defaulting customers.

Although it is important to use good quality data to train AI systems, it may not be enough to completely address the issue of bias. Currently, most of the AI systems operate as black-boxes which often achieve great accuracy but without providing any explanations for its results. Hence, one of the most needed requirements is the ability to interrogate and interpret the predictions made by an AI model to check for logical gaps as well as conformity to ethical standards. An enhanced ability to audit the results will also make people more trusting of AI-based solutions and support large scale deployment.

Although there has been significant progress in developing models that provide better transparency, the artificial neural networks are particularly disadvantaged due to their knowledge representation being distributed across several units. This is currently an active focus of research and unless significant progress is made in the area of interpretability of AI systems, the goal of broader commercial adoption is likely to stay elusive.