Affordable Access

Publisher Website

Automating Intended Target Identification for Paraphasias in Discourse Using a Large Language Model.

Authors
  • Salem, Alexandra C1
  • Gale, Robert C1
  • Fleegle, Mikala2
  • Fergadiotis, Gerasimos2
  • Bedrick, Steven1
  • 1 Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland.
  • 2 Department of Speech & Hearing Sciences, Portland State University, OR.
Type
Published Article
Journal
Journal of speech, language, and hearing research : JSLHR
Publication Date
Dec 11, 2023
Volume
66
Issue
12
Pages
4949–4966
Identifiers
DOI: 10.1044/2023_JSLHR-23-00121
PMID: 37931137
Source
Medline
Language
English
License
Unknown

Abstract

To date, there are no automated tools for the identification and fine-grained classification of paraphasias within discourse, the production of which is the hallmark characteristic of most people with aphasia (PWA). In this work, we fine-tune a large language model (LLM) to automatically predict paraphasia targets in Cinderella story retellings. Data consisted of 332 Cinderella story retellings containing 2,489 paraphasias from PWA, for which research assistants identified their intended targets. We supplemented these training data with 256 sessions from control participants, to which we added 2,415 synthetic paraphasias. We conducted four experiments using different training data configurations to fine-tune the LLM to automatically "fill in the blank" of the paraphasia with a predicted target, given the context of the rest of the story retelling. We tested the experiments' predictions against our human-identified targets and stratified our results by ambiguity of the targets and clinical factors. The model trained on controls and PWA achieved 50.7% accuracy at exactly matching the human-identified target. Fine-tuning on PWA data, with or without controls, led to comparable performance. The model performed better on targets with less human ambiguity and on paraphasias from participants with fluent or less severe aphasia. We were able to automatically identify the intended target of paraphasias in discourse using just the surrounding language about half of the time. These findings take us a step closer to automatic aphasic discourse analysis. In future work, we will incorporate phonological information from the paraphasia to further improve predictive utility. https://doi.org/10.23641/asha.24463543.

Report this publication

Statistics

Seen <100 times