Lab 3: Answer Relevancy
Does the answer actually address what was asked? An answer can be faithful but still miss the point.
🎯
Answer Relevancy: Does the answer address the question?
We generate questions FROM the answer, then measure cosine similarity between those generated questions and the original. If the answer directly addresses what was asked, the generated questions will be very similar to the original.
How It Works
1
Original Question
The user's actual question
2
Generate 3 Questions
GPT generates questions the answer would respond to
3
Embed All
text-embedding-3-small converts all 4 questions to vectors
4
Cosine Similarity
Average similarity = Answer Relevancy score
Question
“How do I apply for paternity leave?”
Answer (Relevant)
Submit Form W-77B to the Culture Team at least 2 weeks before your leave starts. NovaCorp provides 14 days of fully paid paternity leave that must be taken within 3 months of the child's birth.