AI paralegal passes SQE1 but empathy “no replacement” for human solicitor

Reading time: two minutes

Legal technology company Lawhive has put its new AI paralegal, Lawrence, to the test, posing 90 sample questions from the Solicitors Qualifying Exam (SQE) SQE1 assessment. Having answered 67 out of 90 questions correctly, Lawrence’s total mark was 74%.

According to the Solicitors Regulation Authority (SRA), the pass rate for these two sets of 45 sample questions would be around 55% to 60%.

Lawhive created Lawrence to support its nationwide team of solicitors and legal experts. To ensure that the AI lawyer could support solicitors with real legal work, the company made it complete 90 SQE1 sample questions to test its “legal knowledge in line with the SRA’s expectations of junior solicitors”.

According to Lawhive, the questions that Lawrence got incorrect were more complicated, regarding complex chains of logic and wider context. The AI paralegal also found questions that involved two concepts with shared similarities more difficult – for example, public nuisance versus private nuisance.

In addition, as part of Lawhive’s assessment of its AI, the company gave Lawrence and a human solicitor the same client’s will and probate case to compare tone, empathy and legal knowledge. The research found that while Lawrence’s email focused the conversation on attaining the necessary information about the client’s late relative’s will and assets, it was largely transactional and half the length of the human solicitor’s email.

Read more about the implications of AI in the law with this Commercial Question: ‘Use of AI in the legal industry: should we, shouldn’t we?’ by Shoosmiths.

The client provided feedback for both responses, critiquing the AI paralegal’s email for not being as empathetic as the human one. Talking about this comparison, Flinn Dolman, co-founder and AI expert of Lawhive, said the AI response was “no replacement for the empathy shown by one of our human solicitors”.

Dolman added: “At this time, it’s clear that AI will not be replacing the value that human connection provides in a solicitor and client experience – but using AI that’s equipped with theoretical legal knowledge allows our legal experts and solicitors to create efficiencies and seek support with more minor legal tasks as client relationship and outcome takes priority.

“To ensure Lawrence returns value to the company, we intend on testing his response and skill to various situations and supporting our solicitors with legal tasks. We’ll be closely monitoring Lawrence’s progress, and asking for client feedback on their interactions with him as he supports casework.”