Linklaters tests whether AI can replace human lawyers

Reading time: two minutes

Magic circle law firm Linklaters LLP has updated its LinksAI English law benchmark to test two of the latest large language models (LLM) and find out whether “AI models can reasonably replicate some tasks carried out by a human lawyer”.

The law firm first tested four different LLMs in October 2023, such as early versions of GPT, and concluded that AI shouldn’t be used for English law advice without human supervision. It was found that the AI tools often generated “wrong” answers with frequent “fictitious citations”.

The LLMs were up against an exercise comprised of 50 “hard” questions from 10 different practice areas that are said to require knowledge from a mid-level lawyer who specialises in that practice area. The answers were then marked out of 10 by senior lawyers for substance, citations and clarity. The best scoring AI model in the 2023 test was Bard with a score of 4.4 out 10.

In the most recent test this February, Linklaters noted a significant improvement in the AI models’ performance. OpenAI o1 and Gemini 2.0 were the latest LLMs tested with scores of 6.4 out of 10 and 6.0 out of 10, respectively. Despite the higher scores, the second round of testing has led the law firm to the same conclusion – AI models are still “not always right” and “we recommend they should not be used for English law legal advice without expert human supervision”.

The report said: “However, if that expert supervision is available, they are getting to the stage where they could be useful, for example by creating a first draft or as a cross-check. This is particularly the case for tasks that involve summarising relatively well-known areas of law.”

In addition, the report highlighted that none of the models used in the test had been trained to provide legal advice and that other LLMs may perform better.

Linklaters plans to reapply the LinksAI English law benchmark to future versions of this technology and will provide updates to its report.