Katherine Atwell

Katherine Atwell

Final-year Computer Science PhD candidate @ Northeastern University

News

  • 08.23.25: Our work, BASIL: Bayesian Assessment of Sycophancy in LLMs, was published at ArXiv
  • 07.31.25: I am co-organizing the fourth NLP for Positive Impact Workshop, co-located at ACL 2025!
  • 07.27.25: Our work, Measuring Bias and Agreement in Large Language Model Presupposition Judgments, was published at ACL Findings 2024
  • 07.27.25: Our work, Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI, was published at COLING Industry Track 2025
  • 06.25.25: My advisor, Malihe Alikhani, spoke with a Wall Street Journal reporter about our lab's work on AI sycophancy; see the article here!
  • 11.12.24: Our work, Studying and Mitigating Biases in Sign Language Understanding Models, was published at EMNLP 2024
  • 06.26.24: Our work, The Importance of Including Signed Languages in Natural Language Processing, was published at
  • 06.16.24: Our work, Generating Signed Language Instructions in Large-Scale Dialogue Systems, was published at NAACL 2024
See more

The Importance of Including Signed Languages in Natural Language Processing

Published in Sign Language Machine Translation, 2024

Recommended citation: Kayo Yin, Katherine Atwell, Julie A Hochgesang, Malihe Alikhani. 2024. The Importance of Including Signed Languages in Natural Language Processing. In Sign Language Machine Translation. Springer Nature Switzerland.

Identifying linguistic bias in text demands the identification not only of explicitly asserted content but also of implicit content including presuppositions. Large language models (LLMs) offer a promising automated approach to detecting presuppositions, yet the extent to which their judgments align with human intuitions remains unexplored. Moreover, LLMs may inadvertently reflect societal biases when identifying presupposed content. To empirically investigate this, we prompt multiple large language models to evaluate presuppositions across diverse textual domains, drawing from three distinct datasets annotated by human raters. We calculate the agreement between LLMs and human raters, and find several linguistic factors associated with fluctuations in human-model agreement. Our observations reveal discrepancies in human-model alignment, suggesting potential biases in LLMs, notably influenced by gender and political ideology.