Interpretable Semantic Axes for ‘Break' in Contextualized Embeddings

Presenter: Sujin Lee

Faculty Sponsor: Katrin Erk

School: UMass Amherst

Research Area: Computer Science

Session: Poster Session 6, 4:15 PM - 5:00 PM, Auditorium, A64

ABSTRACT

Large language models (LLMs) represent words as vectors in high-dimensional spaces called embeddings. Words that appear in similar contexts end up close together, forming geometric patterns that often mirror aspects of meaning. These patterns are remarkably systematic and thus the semantic terrain inside LLMs have long been investigated by researchers. Notably in the past Petersen and Potts (2023) have used the highly polysemous word break, with meanings ranging from literal physical breakage (“break a glass”) to emotional collapse (“break down”), and shown successful sense clustering in LLMs. However, I argue that clustering does not serve as sufficient evidence to answer what underlying semantic features drives the sense distinctions in LLMs and whether they align with meanings that humans recognize. My project aims to go beyond the prior findings of Petersen & Potts by implementing interpretable semantic axes, explicit directions in embedding space that correspond to psycholinguistic features. Using RoBERTa-large, I construct these axes from human feature-norm data (Binder et al., 2016) and project contextual embeddings of break onto them. I then analyze how different uses of the word separate along these dimensions. By implementing these axes to investigate whether an LLM “represents meaning” with cognitively real distinctions like causative vs. inchoative, my research connects computational patterns in LLMs with long-standing theoretical work in lexical semantics and cognitive science. 

Petersen, E., & Potts, C (2023), Lexical Semantics with Large Language Models: A Case Study of English “break”. Findings of EACL

Binder, J. R., et al. (2016), Toward a brain-based componential semantic representation. Cognitive Neuropsychology. Cogn Neuropsychol.

RELATED ABSTRACTS