It was thought that you could corrupt an entire neural network by letting it sometimes return a ‘No answer’ error. According to the theory, such an error would infect the model like a virus, rendering it and all its real answers useless. So the engineers accepted occasional nonsense in the hope that it was accurate enough everywhere else. And they made use of confidence scores to recognise most of the nonsense: ML models would say how certain they are of the answer. Any score less than 50% (0.5) was looked upon with a distrustful eye.
But the mysterious models were often confident by mistake. One time Sophie passed in the instruction ‘Recommend a movie’, expecting something like ‘The Lion King’ in reply. But no, her model responded with some odd advice: ‘Do something that looks like life’. That wasn’t a movie name. Sophie checked the confidence score, imagining it would be very low, yet was surprised again when she read ’0.93’. It was easy to forgive her model for the language processing failure – that happened daily – but the excessive confidence was irritating, in much the same way as it is in humans. ‘No answer’ would have been better.
However, Sophie didn’t want to corrupt her model, so training it on poor data just to trigger the error was out of the question. Nor could she null the answers with low confidence scores, since that metric could no longer be trusted. Eventually she decided to pair her network with a second network that would assess the answer and determine whether or not it was useful. This technique – perhaps reminiscent of reinforcement learning – let the outer network judge her inner network’s responses. When trained on different datasets, the two could be trusted to work independently of one another. ‘Why don’t you combine the networks to make one better model?’ asked a lab assistant. ‘Ah, the inner one is trained entirely on my very own data, and I would prefer to keep it that way,’ replied Sophie. ‘This way the answer comes from my model, and I benefit from the other unknown dataset just to compare the question and answer.’