Shuyang Cao

selected publications

1. COLM

   Evaluating the Retrieval Robustness of Large Language Models

   Shuyang Cao, Karthik Radhakrishnan, David S Rosenberg, Steven Lu, Pengxiang Cheng, Lu Wang, and Shiyue Zhang

   In Third Conference on Language Modeling 2026

   Abs arXiv Bib PDF

   Retrieval-augmented generation (RAG) generally enhances large language models’ (LLMs) ability to solve knowledge-intensive tasks. But RAG could also lead to performance degradation due to imperfect retrieval and the model’s limited ability to leverage retrieved content. In this work, we evaluate the robustness of LLMs in practical RAG setups (henceforth retrieval robustness). We focus on three research questions: (1) whether RAG is always better than non-RAG; (2) whether more retrieved documents always lead to better performance; and (3) whether document order impacts results. To facilitate this study, we establish a benchmark of 1,891 samples spanning five datasets across three task categories, each with documents retrieved using both sparse and dense retrievers. We introduce three robustness metrics, each corresponding to one research question. Our experiments across 11 LLMs show that models achieve generally high retrieval robustness, but robustness varies substantially across tasks, suggesting that the decision to adopt RAG remains a case-by-case consideration. We further examine four additional prompting strategies that vary how models interact with retrieved documents. We find that Qwen and GPT models suffer notable robustness declines when reasoning is disabled, even on single-hop QA tasks, and that providing retrieved documents as tool responses improves Claude models but hurts Qwen and GPT models, highlighting potential issues of the GPT models regardless of their best overall robustness under vanilla prompting.

   @inproceedings{cao-etal-2026-retrieval,
     title = {Evaluating the Retrieval Robustness of Large Language Models},
     author = {Cao, Shuyang and Radhakrishnan, Karthik and Rosenberg, David S and Lu, Steven and Cheng, Pengxiang and Wang, Lu and Zhang, Shiyue},
     booktitle = {Third Conference on Language Modeling},
     year = {2026},
   }

2. ICLR

   ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists

   Jie Ruan, Inderjeet Nair, Shuyang Cao, Amy Liu, Sheza Munir, Micah Pollens-Dempsey, Yune-Ting Chiang, Lucy Kates, Nicholas David, Sihan Chen, Ruxin Yang, Yuqian Yang, Jihyun Gump, Tessa Bialek, Vivek Sankaran, Margo Schlanger, and Lu Wang

   In International Conference on Learning Representations 2026

   Abs arXiv Bib PDF Website

   This paper introduces ExpertLongBench, an expert-level benchmark containing 11 tasks from 9 domains that reflect realistic expert workflows and applications. Beyond question answering, the application-driven tasks in ExpertLongBench demand long-form outputs that can exceed 5,000 tokens and strict adherence to domain-specific requirements. Notably, each task in ExpertLongBench includes a rubric, designed or validated by domain experts, to specify task requirements and guide output evaluation. Furthermore, we propose CLEAR, an evaluation framework that supports accurate evaluation of long-form model outputs in our benchmark. To achieve fine-grained, expert-aligned evaluation, CLEAR derives checklists from both model outputs and references by extracting information corresponding to items in the task-specific rubric. Checklist items of model outputs are then compared with corresponding items of reference outputs to assess their correctness, enabling grounded evaluation. We benchmark 15 popular large language models (LLMs) and analyze components in CLEAR, showing that (1) existing LLMs, with the top performer Gemini-2.5-Pro achieving only a 33.4 F1 score, require significant improvement for expert-level tasks; (2) models can generate content corresponding to the required aspects, but far from correct; and (3) accurate checklist extraction and comparison in CLEAR can be achieved by open-weight models for more scalable, reproducible, and low-cost usage.

   @inproceedings{ruan-etal-2026-expertlongbench,
     title = {{ExpertLongBench}: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists},
     author = {Ruan, Jie and Nair, Inderjeet and Cao, Shuyang and Liu, Amy and Munir, Sheza and Pollens-Dempsey, Micah and Chiang, Yune-Ting and Kates, Lucy and David, Nicholas and Chen, Sihan and Yang, Ruxin and Yang, Yuqian and Gump, Jihyun and Bialek, Tessa and Sankaran, Vivek and Schlanger, Margo and Wang, Lu},
     booktitle = {International Conference on Learning Representations},
     year = {2026},
   }

3. EMNLP

   SYNC: A Synthetic Long-Context Understanding Benchmark for Controlled Comparisons of Model Capabilities

   Shuyang Cao, Kaijian Zou, and Lu Wang

   In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing Nov 2025

   Abs Bib PDF Website

   Recently, researchers have turned to synthetic tasks for evaluation of large language models’ long-context capabilities, as they offer more flexibility than realistic benchmarks in scaling both input length and dataset size. However, existing synthetic tasks typically target narrow skill sets such as retrieving information from massive input, limiting their ability to comprehensively assess model capabilities. Furthermore, existing benchmarks often pair each task with a different input context, creating confounding factors that prevent fair cross-task comparison. To address these limitations, we introduce SYNC, a new evaluation suite of synthetic tasks spanning domains including graph understanding and translation. Each domain includes three tasks designed to test a wide range of capabilities–from retrieval, to multi-hop tracking, and to global context understanding that that requires chain-of-thought (CoT) reasoning. Crucially, all tasks share the same context, enabling controlled comparisons of model performance. We evaluate 14 LLMs on SYNC and observe substantial performance drops on more challenging tasks, underscoring the benchmark’s difficulty. Additional experiments highlight the necessity of CoT reasoning and demonstrate that SYNC poses a robust challenge for future models.

   @inproceedings{cao-etal-2025-sync,
     title = {{SYNC}: A Synthetic Long-Context Understanding Benchmark for Controlled Comparisons of Model Capabilities},
     author = {Cao, Shuyang and Zou, Kaijian and Wang, Lu},
     editor = {Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet},
     booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
     month = nov,
     year = {2025},
     address = {Suzhou, China},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2025.emnlp-main.1707/},
     doi = {10.18653/v1/2025.emnlp-main.1707},
     pages = {33627--33648},
     isbn = {979-8-89176-332-6},
   }