Cappy: Outperforming and boosting large multi-task language models with a small scorer
360M parameters. That's all Google's new Cappy model needs to outperform language models 500x larger.
Why it matters
Google Research introduced a lightweight scoring model that matches or exceeds the performance of massive multi-task LLMs while requiring dramatically fewer parameters and enabling adaptation without access to proprietary model weights—a significant efficiency breakthrough for enterprise AI deployment.
The key facts
8 to knowCappy uses only 360 million parameters vs OPT-175B (175B) and OPT-IML-175B
Trained on 160 million instruction-response pairs with correctness annotations
Matches accuracy of T0-11B and OPT-IML-175B on PromptSource benchmarks
Outperforms on BIG-Bench: shows large margin improvements over FLAN-T5 baselines across 45 complex generation tasks
Works with closed-source LLMs via API—no parameter access required
Eliminates backpropagation through LLM parameters, reducing memory requirements
Presented at NeurIPS 2023
Continual pre-training built on RoBERTa using weak supervision (Rouge-L similarity scoring)
Go to the source
Google Research Blogblog.research.google
Publisher excerpt: Posted by Yun Zhu and Lijuan Liu, Software Engineers, Google Research Large language model (LLM) advancements have led to a new paradigm that unifies various natural language processing (NLP) tasks within an instruction-following framework. This paradigm is exemplified by recent multi-task LLMs,…