How to Train Your Vicuna - Finetuning, Evaluating, and Serving LLMs in the Wild - Hao Zhang
alpa.ai
lmsys.org
vllm.ai
Background: LLMs and Open Source LLMs
Topics
- How we trained Vicuna
- One core problem after Vicuna: Evaluation
- Evaluation
- LLM-as-judge
- Chatbot arena
Background
Meta Llama weights
Used ShareGPT to collect conversations from API and train Vicuna
Why is ShareGPT small yet effective?
ShareGPT data contains the user preferences implicit in the ChatGPT conversations that were trained using RLHF (Reinforcement Learning with Human Feedback)
Llama + ShareGPT = Vicuna
Strongly preferred by users than base models
Does not improve, sometimes decreases academic NLP metrics
”For better or worse, benchmarks shape a field” - David Patterson, Turing Award 2017
LLMs are extremely hard to evaluate
Unreliable: Benchmarks cannot tell human preference
- Requires expertise to evaluate
- Data contamination
Cost: Very expensive to evaluate
How to evaluate human preferences?
Use humans to rate chatbots
Human Evaluation
Chatbot answers interpreted by humans, so it makes sense for humans to be ultimate arbiters
Ideally, for every question we want to rank all LLMs
Ranking N choices is hard:
- Easier to pick best of N
- Even easier to pick best of two!
How to scale human evaluation
Tournament
ELO rating
Benchmark platform for LLMs that features anonymous, randomized battles in a crowdsourced manner
How to evaluate it?
- Humans take long and are expensive
- How about we use the strongest LLM to evaluate?
Evaluation: MT-Bench
Our initial method: LLM Exams and LLM-as-a-Judge
- Let two chatbots generate responses to the same set of questions
- Use GPT-4 to assess the model responses
- Simultaneously, let human raters assess the model responses
- Study the alignment between GPT-4 and human raters
Can we really trust LLM as judge?
Limitations: not unlike humans
Position bias: prefer first position
Verbosity bias: prefer long answers
Self-enhancement bias: prefer answers from itself
Limited reasoning: not good at grading math questions
High agreement despite limitations
Agreement between GPT-4 and humans over 80%: same/better as human-human agreement
Summary
- LLM evaluation is extremely hard
- Cracking this problem requires new techniques
- LLMs as judges
- Scalable human evaluation
- Many challlenges remain
- Contamination: generating unique exams difficult
- Diversity: most questions are easy; need hard questions to differentiate between LLMs