A key concern when designing Machine Learning models is to try to avoid bias which might lead to unfair models. When thinking about how things might go wrong it is useful to think about human cognitive biases. We can then consider how similar issues might manifest in Machine Learning systems. This then allows us to take corrective action to avoid problems. So what cognitive biases might we need to consider? There are many cognitive biases, lets consider a few of the most famous.
Confirmation Bias
Confirmation Bias occurs when too much weight is given to information in favour of a hypothesis as compared to information which contradicts it. Problems similar to this could manifest in Machine Learning models if too much attention is paid to measures such as accuracy which look mainly at true positive rates and do not also consider metrics like precision and recall to consider issues such as false positives and false negatives. It can also manifest in situations such as “p-hacking” where we cherry pick which experiments to report so that an effect appears more definite than it actually is.
Survivorship Bias
Survivorship Bias occurs where we only consider the the known instances of a situation and miss an important unseen class of data. This could occur in certain medical contexts. Imagine comparing the average ages of death of the general population to the the average ages of death of Nobel prize winners. It would seem that Noble prize winners are likely to live longer. However the effect may be exaggerated because the importance of work leading to a Nobel Prize is often not immediately evident leading to a delay in award. Since Nobel Prizes are only awarded to living people, being long-lived is an advantage for receiving a Nobel Prize. This effect would need to be accounted for in any comparison. This could be done by matching the Nobel winners with people who were alive and the same age at the time of the Nobel award and then tracing their subsequent life span.
Availability Heuristic
The Availability Heuristic is the tendency to overestimate the likelihood or effects of situations for which information exists as compared to those where it does not. It is very easy to see how this could prove to be a problem in Machine Learning situations. For instance in a classification problem if you are missing certain labels from your training set, then it is impossible to correctly classify those labels with your trained model. More insidiously suppose you miss an important subset (gender, race, age) from the training data for a system designed to interpret emotion. It would not be surprising to expect that the model would perform less well when presented with an image representing a member of the missing subset
Anchoring Bias
Anchoring Bias is the tendency to give early information greater weight than later information. In Machine Learning it could manifest in situations like transfer learning. If the underlying model is not a good fit for the new subject being trained by transfer learning then you could get a poor model from retraining. It could also occur in cases of model retraining particularly if the retraining avoids overfitting by early stopping. This is because the model may not have converged and may still give to much weight to the original data set. A key check to ensure this kind of problem does not occur is to compare the scoring metrics on new test data with that of old test data. Then we can ensure that performance is similar on new data rather than only assessing performance on both sets together.
Normalcy Bias
Normalcy Bias is the tendency to fail to consider issues which have not occurred before. In some ways it is an extension of the Availability Heuristic. In Machine Learning contexts this can manifest as a failure to consider certain unlikely corner cases. These cases might cause the system to behave in unpredictable ways. An example might be a failure to include aeroplanes in a dataset used to train a self-driving car because planes very rarely land on roads. However planes landing on roads does occasionally happen. How would an untrained system react to the eventuality?
Barnum Effect
The Barnum Effect is the tendency of people to interpret general statements in such a way that they make them fit more detailed known facts. This could well be a factor when attempting to assess the quality of Large Language Models (LLM). It was noted very early in attempts to produce conversational AI that people will fill in the blanks. they then give a model more credit than it deserves for its answers. The classic case is the ELIZA chatbot from the 1960s.
Fundamental Attribution Error
The Fundamental Attribution Error is the tendency to incorrectly partition blame and credit between behavioural and environmental actors. While not a direct analogy, this can manifest in Machine Learning problems as a lack of clarity as to whether the training dataset or the structure of the Machine Learning model is responsible for a given bias or failure mode. This kind of confusion can extend to issues such as data leakage and poor choice of or engineering of features. Failures here can result in attempting to simply add more data to improve a fundamentally flawed model. On the other side it could mean abandoning a reasonable model because insufficient data is readily available.
Rhyme as Reason Effect
The Rhyme as Reason Effect is a human tendency to give higher credence to assertions which rhyme, alliterate or have other memorable patterns. This may be due to the Fluency Heuristic. At first glance it would seem that there would be little chance of this posing a problem for machine learning models. However consider Large Language Models. These tend to be trained on larger volumes of human generated text. Since humans suffer from these heuristics, it is likely that the data will encapsulate them. At present it is unclear whether this will manifest solely with the explicit examples. It is possible that LLMs will generalise from specific rhyming truisms to tend to accord all rhyme greater veracity. Indeed given how they are trained it is possible that LLMs like ChatGPT will be at risk of many human biases. There is also the possibility of having their own biases due to their structure and datasets. These will only become apparent with time and research.
What to do about Bias
The first thing to do about bias is to be aware of the different types and think about how they might affect models under development. This is not a complete solution since people tend not to notice their biases, we all have Bias Blind Spots. However conscious effort to consider possibilities for bias, especially if undertaken by a diverse team can often help to mitigate the problems bias can cause. External auditing and model transparency can also help to surface issues with models to allow corrective action.
