Examining How LLM/LM’s Reasoning Capability Affects the Task Performance

Presenter

Ishita Kakkar

Campus

UMass Amherst

Sponsor

Mari Castañeda, Commonwealth Honors College, UMass Amherst

Schedule

Session 2, 11:30 AM - 12:15 PM [Schedule by Time][Poster Grid for Time/Location]

Location

Poster Board A30, Campus Center Auditorium, Row 2 (A21-A40) [Poster Location Map]

Abstract

This project investigates the impact of rationale quality on the task performance of Language Models (LMs)/Large Language Models (LLMs). The study aims to understand how the reasoning capability of LLMs influences their performance across various Natural Language Processing (NLP) tasks. Recent advancements have highlighted the effectiveness of Chain-of-Thought (CoT) prompting in enhancing LLMs' performance by generating intermediate reasoning steps. Furthermore, the distillation of LLM-generated rationales into smaller models has shown promise in improving performance. However, the significance of rationale quality remains underexplored. The primary objective of this research is to systematically quantify the extent to which LLM-generated rationales and their quality contribute to improvements in NLP task performance. The study will explore whether performance enhancements are primarily due to the synergistic effect of combining similar tasks in a Multi-Task Learning (MTL) framework, the relevance of the rationales, their truthfulness, or a combination of these factors. Additionally, the effectiveness of distilling LLM-generated rationales into smaller models to enhance their performance will be assessed, alongside the development of a methodology to evaluate the impact of rationale quality on various NLP tasks. Hypotheses posit that the quality of LLM-generated rationales significantly impacts the performance of smaller language models in NLP tasks and that a combination of synergistic task alignment, rationale relevance, and truthfulness is critical for optimizing task performance.

Keywords

Natural Language Processing, Prompt Engineering, Task Performance

Research Area

Computer Science

SIMILAR ABSTRACTS (BY KEYWORD)