A chatbot blunder and responsible AI.
When AI is responsible, it is time for responsible AI.
The Guardian picked up on the recent case in Canada where a chatbot gave the wrong advice to a person seeking to book a bereavement flight. Moffatt v. Air Canada, 2024 BCCRT 149 (CanLII), <https://canlii.ca/t/k2spq>, retrieved on 2024-02-17 Excerpts from the judgement below.
In November 2022, following the death of their grandmother, Jake Moffatt booked a flight with Air Canada. While researching flights, Mr. Moffat used a chatbot on Air Canada’s website. The chatbot suggested Mr. Moffatt could apply for bereavement fares retroactively. Mr. Moffatt later learned from Air Canada employees that Air Canada did not permit retroactive applications.
Air Canada argues it cannot be held liable for information provided by one of its agents, servants, or representatives – including a chatbot. It does not explain why it believes that is the case. In effect, Air Canada suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission. While a chatbot has an interactive component, it is still just a part of Air Canada’s website. It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.
I find Air Canada did not take reasonable care to ensure its chatbot was accurate. While Air Canada argues Mr. Moffatt could find the correct information on another part of its website, it does not explain why the webpage titled “Bereavement travel” was inherently more trustworthy than its chatbot. It also does not explain why customers should have to double-check information found in one part of its website on another part of its website.
It isn’t clear if this specific chatbot is a generative AI chatbot or not, but it is an excellent example of the risks of a poorly configured chatbot, and of hallucinational dangers. The customer was smart enough to have screenprints of the chatbot conversation.
The PR tone deafness is going to outweigh the 800 dollars or so Air Canada thought they could retain but didn’t.
I do wonder if Air Canada actually followed some sort of responsible AI practice before rolling this out. The sentiment in the case gives me a strong vibe they didn’t. This worries me.
Back to HR Tech
As we apply chatbots more assertively in HR tech, it going to be vital to mitigate for this sort of failure. How these systems are trained, tested , deployed and retested is going to determine whether they are trusted by employees and other users. There is a danger of enshittification.
I have been spending sometime with fundamental ML lately. I’ve been learning the basics while dealing with some long buried childhood maths trauma.
Linear regression, bag of words, principle component analysis, random forests, ground truths of interests, confusion matrix and more. I now know enough to be mildly dangerous. Learning about how accuracy, precision and recall interact has been a highlight so far. Put simply, probability means being wrong some of time.
I’d encourage all vendors building AI solutions to work on a clear responsible AI practice now. And those buying AI solutions to demand the same. A good starting point would be to ask your vendors about NIST AI RMF or SO/IEC TR 22989:2020: or IEEE P7009/D7 or the OECD principles. While the standards are new, and others are still emerging, if you are vendor is doing AI on HR data, they should be on top of the latest developments in standards.
One of the tools I’ve been using to learn about AI is dataiku, and I rather like how they communicate their approach to responsible AI. They call it RAFT.
If you are interested in more about chatbot risks, I found this academic paper about prompt injection risks sobering. Yu, J., Wu, Y., Shu, D., Jin, M., & Xing, X. (2023). Assessing Prompt Injection Risks in 200+ Custom GPTs. https://arxiv.org/abs/2311.11538
The EU AI Act is moving along through the approval process, so that will raise the stakes, but even without that, just because you are using AI, it doesn’t give you a free pass on other laws. As is the case above, contract law is contract law. Labour law is labour law and so on.
So what would be a responsible AI practice in HR tech? I’ll pick up on some examples of what HR tech vendors are doing on this in a future post, but if you have any good examples, send them my way.
I’m arranging a session with some leading AI experts for our portfolio companies to discuss responsible AI in the context of HR tech. If there is interest, I’ll open this to a wider audience.
I forgot again to add a song. let’s fix that.
I appreciate your coverage of responsible AI here. As an AI practitioner in HR Tech, this topic sometimes doesn't get enough of my attention. Dataiku's RAFT model is nicely concise but still thorough.
That's so great you're learning some foundations of ML! I get frustrated when people cite the overall accuracy of their systems (we reached 90%!) without discussing precision and recall as well as the distribution of outcomes. Of course if you're trying to predict a rare outcome, you can get to really high overall accuracy just by always predicting the more common outcome. But your recall will be zero and your precision will be undefined.
As for chatbots, the generative ones seem to be the most powerful (they really seem to understand what you're asking, and they provide such fluent responses) but more basic ones that merely classify intent and provide standard responses are probably safer.
I question whether chatbots are generally a good interaction paradigm. For certain cases they are, but it seems many companies are introducing them based on ChatGPT right now just so they can say they've got a GenAI capability in production. I'm writing a newsletter article about that this morning, so we seem to be thinking about the same thing.