Feb 2035: A robot makes a lawyer sit an exam
Linklaters has set exams for its Artificial Intelligence programmes, to assess how much help they can provide to lawyers.
The 'LinksAI English law benchmark' tests are designed to check if AI "can reasonably replicate some tasks carried out by a human lawyer," said the firm.
The exams comprise of 50 questions from 10 different practice areas, at a level which "would require advice from a competent mid-level lawyer specialised in that practice area".
Senior lawyers at the firm marked the AI answers, with points awarded based on substance (if the answer was right), citations (relevant statutes/case law), and clarity.
The firm tested six different AI models this month, and top of the class was OpenAI - awarded the highest score of 6.4 out of 10, which might result in it being labelled as a swot by the other robots, and having ink flicked on its back.
Linklaters noted there was a "significant improvement" in the results when compared with tests it ran in October 2023. The firm said that the AI models are starting to perform at a level where they should be able to assist in legal research, such as providing a first draft or "as a cross-check on an existing answer," and could also be useful "for tasks that involve summarising relatively well-known areas of law”.
However, the firm warned that "expert supervision" would still be required, as AI sometimes gives wrong responses, and there are "dangers" in using it if lawyers "don't already have a good idea of the answer." The firm also reported that the models lack "nuance" and are not so good at applying the law to facts or interpreting clauses.
The warning not to completely rely on the robot's homework, follows cases where lawyers have landed in hot water for using non-existent case law, due to AI hallucinations. While City firms are increasingly making use of AI, the SRA has also told firms not to rely on machine learning.
Last week Hill Dickinson sent a notice to staff, reminding them of the firm's restrictions when using AI, as they are required to get approval before using such platforms. A Hill Dickinson spokesperson told RollOnFriday: "AI can have many benefits for how we work, but we are mindful of the risks it carries and must ensure there is human oversight throughout."
RollOnFriday was an early abandoner of AI after trying to extract sound legal advice from a so-called robot lawyer in 2016, requesting a chatbot write about lawyers, and creating posters of what law firms would like if they were movies.
Comments
61
60
I use AI all the time, mostly Google Gemini. I treat it like a competent trainee in my room. I can ask it questions about an area or to find references but I need to check them. It saves me huge amounts of time.
83
62
Our trainees are told on Day 1 that "you have all been in a world where 80% right is top marks. Here, 80% right is negligence".
64% as the top mark is pretty underwhelming.
I asked our tech crew if they had put our vac scheme or training contract assessment day exercises into the AI programmes and if so how they did. I didn't get a meaningful answer, but my assumption based on usage of the tools is that they would fail those tasks.
61
64
Surely OpenAI would get toner fllicked on its back?
71
57
Anything other than creative writing is where I see the limits of ai.
Communications are over the top.
Research results in made up results.
Not a fan.
57
58
A lot of the success of AI is based on the prompting - if you prompt well you can remove most of the issues, and for every fact in there can quickly link to a citation.
64
55
I imagine AI could replace the SRA already
69
60
"I use AI all the time, mostly Google Gemini. I treat it like a competent trainee in my room."
Obviously bullshit.
There is absolutely no way that you have managed to get Google Gemini to fetch tea, laugh at your woeful jokes, and wear that spectacular pencil-skirt and heels combo for the third time in a week because "the client might be in tomorrow so would you mind dressing smart just in case?" even though you know full well they're using Docusign.
Call me when it can do the real trainee tasks that trainees do.
73
66
Our trainees are told on Day 1 that "you have all been in a world where 80% right is top marks. Here, 80% right is negligence".
You have all been in a world. This is a different world; a new world. Forget the world you have all been in. In the world of this law firm’s world, the same rules do not apply. You have all been in a world where 80% right is top marks. You have all now travelled to another other-worldly world where if you score 80% in an exam, that is negligence. Please just bear in mind that you’ve all been in a world, but you’re now in a separate world which not the same world. This is the only lesson you’ll need to learn, delivered on Day 1.
62
58
We laugh at the Knights model (lampooned as lawyers with Wikipedia level legal knowledge but marketed as legal geniuses) but that’s basically what’s going to happen with AI.
Clients won’t care, indeed an AI programme will look slicker.
62
62
A real world example.
I'm putting together a response to the business about defences in a complaint. I need to set out that the customer needs to complain to FOS within 6 months of the FRL. I quickly ask my favourite AI where the rule is in the FCA handbook and it tells me in 3 seconds and takes me another 30 seconds to check the answer is right. I could have skimmed through the FCA handbook as I have a pretty good idea but that would have taken me more time.
I'm more effective with an AI tab open.