A group of enthusiasts from the startup Ansible Health tested the ChatGPT neural network for the ability to pass one of the most difficult US exams – USMLE (medical license exam). It consists of three parts, preparation for it takes 300-400 hours, the list of questions varies from fundamental medical knowledge to the paradoxes of bioethics. But most importantly, ChatGPT has no knowledge of medicine, it only skillfully manipulates textual information to produce relevant answers.
The result surprised all participants – the text AI almost managed to pass this exam, its pass rates ranged from 52% to 75% against the standard of 60%. Important clarification: AI was not specially trained before passing the exam, he still knows nothing about medicine. And this was reflected in his answers – ChatGPT “invented” several non-existent clinically important discoveries and studies to provide a reasoned answer to some questions.
During the exam, AI showed a brilliant ability to build logical chains to justify their answers . He managed to find the necessary information in the database by indirect signs in order to give a detailed answer. Alas, in some cases it was not only deliberately wrong, but could also lead to tragic consequences if the AI advice was followed. This means that in the near future AI will not replace medical workers – but can become an indispensable assistant for them, as a source of useful information.
