I designed and evaluated ten human annotation workflows for conversational speech summarization, exploring how differences in presentation modality, order, and editing affect summary quality. After hiring and training 18 annotators and analyzing the resulting summaries, I confirmed that transcript-based summaries are more informative than audio-based ones and found that peer editing largely closes this gap. These results suggest that useful speech summarization datasets can still be collected even in settings where transcripts are limited
I improved automatic speech recognition for Hawaiian by rescoring Whisper outputs with a Hawaiian language model, establishing the new state-of-the-art for Hawaiian ASR at the time of publication. Since Hawaiian text is more available than labeled Hawaiian speech, rescoring approach was a practical alternative to finetuning and importantly also adhered to indigenous data sovereignty principples: my collaborators at Oxford and the University of Hawaiʻi trained the language model that I used to rescore Whisper, ensuring that Hawaiian data never left indigenous control
I built and evaluated the embedding component of a retrieval-augmented generation system for a compliance-checking chatbot that helps employees determine whether products align with company policy. To identify the best-performing model, I compared multiple open-source embeddings using both human judgments and automated metrics for context relevance, answer relevance, and faithfulness. This evaluation informed model selection and improved the system’s ability to support internal decision-making.