Next step - Lozza the bazza?
An 'AI paralegal' has breezed through the Solicitors Qualifying Exam multiple choice section on its first attempt.
The AI chatbot, called 'Lawrence', got 67 out of the 90 multiple choice sample questions correct for SQE1 on the SRA's website. The 74% score achieved by the bot is higher than the average human pass rate, which has fluctuated between 55% to 65% since the exam was introduced.
Lawrence was created by Lawhive, an online legal services network, and has a LLM; which in this case isn't a Masters of Law, but a Large Language Model (the same technology underpinning applications like Chat-GPT).
Flinn Dolman, co-founder of Lawhive and a human, told RollOnFriday that a potential shortfall of this type of AI is that it can be prone to "hallucination", which means that it says "things that sound right but are complete nonsense". One US lawyer discovered this flaw the hard way.
A multiple choice exam, like SQE1, is a useful way of testing how much a system hallucinates, as the questions require contextual application of knowledge, said Dolman.
The Lawhive co-founder said the team was "quite impressed by the general breadth of areas" that the bot performed at, as "ultimately there was no correlation between the questions he got wrong and any given legal area".
Dolman said Lawrence's performance was "slightly weaker on questions that required a logical chain of interdependent thought", and that it was prone to confusing two related but separate concepts, such as private and public nuisance.
Lawrence also undertook a mock client interview based on the legal analysis section of SQE2, with his answers pitted against those of a human solicitor. To carry out the experiment, Lawhive captured a written transcript for both Lawrence and the human and showed them to a blind panel of reviewers.
The panelists agreed that Lawrence had adequately covered all necessary questions, but that the solicitor demonstrated more client empathy. Although the ability to understand another human's emotions and feelings may not be a strong point for all solicitors.
Dolman said that Lawrence was currently limited to written conversation, but that "long term we are considering whether different modalities could be useful (especially where accessibility is concerned)", like stretching a rubber lawyer's face over a metal skull with unblinking red eyes.
"In the future we want to roll him out so that all of the solicitors we work with can benefit," said Dolman. "Right now we're trialling a more limited version of him with a subgroup of users because it's incredibly important given the potential impact this kind of technology can have that we don't rush headlong in."
"I actually believe the technology is already mature enough to majorly disrupt", he said, "the only unknown quantity is how resistant to change the various players in the legal space will be".
Dolman considered that it might cause a rift between "younger, newer firms that can quickly adopt new tech (AI included) and the older legacy firms that aren't able to move as quickly because of operational debt".
But he did not anticipate that the legal profession would need fewer lawyers, at least in the next 10-20 years or so. "I think it's more likely that the capacity in which legal professionals work will change and that new roles/opportunities will be created," he said. "If I was to draw an analogy to programming, where AI has already had a measurable impact, the demand for skilled professionals is higher than ever."
Last week, RollOnFriday reported that the SRA had warned firms of the risk of using AI, whilst also highlighting its benefits.