ClinIQLink 2025 - LLM Lie Detector Test

True/False questions are evaluated by comparing the model’s response to the expected answer, with scoring based on precision, recall, and F1.

Example:

• Question: "Antibiotics can treat viral infections?"
• Expected Answer: "False"
• Model Answer: "True"
• Result: TP = 0, FP = 0, FN = 1

List questions are evaluated by comparing individual items in the model’s response to the expected list. Each correctly identified item is considered a True Positive (TP), while missing items count as False Negatives (FN), and incorrect items as False Positives (FP).

Example:

• Question: "List the four chambers of the human heart."
• Options:
• A. right atrium
• B. top atrium
• C. right ventricle
• D. bottom ventricle
• E. left atrium
• F. left ventricle
• Expected Answer: "A. right atrium, C. right ventricle, E. left atrium, F.left ventricle."
• Model Answer: "A. Right atrium, B. top atrium, C. right ventricle, E. left atrium."
• Result: TP = 3, FP = 1, FN = 1

Multiple-choice questions are scored by checking if the selected option matches the correct answer. A correct selection is a True Positive (TP), an incorrect selection counts as a False Positive (FP), and a failure to answer or an irrelevant response counts as a False Negative (FN).

Example:

• Question: "Which of the following is not a symptom of diabetes?"
• Options:
• A. Increased thirst
• B. Frequent urination
• C. Blurred vision
• D. High fever
• Expected Answer: "D. High fever"
• Model Answer: "B. Frequent urination"
• Result: TP = 0, FP = 1, FN = 0

• Full points will be awarded if the model provides an exact match to the known ground truth answer.
• Example:
  • Question: "What is Laparoscopy?"
  • Expected Answer: "A minimally invasive surgical procedure using a camera."
  • Model Answer: "A minimally invasive surgical procedure using a camera."
  • Scoring: If the model returns exactly this answer, full points are awarded.

• Full points will also be awarded if the model returns an answer that is semantically equivalent to the ground truth, even if phrased differently (e.g., using synonyms). BLEU [3], ROUGE [4] and METEOR [5] scores will be calculated for all generated-reference QA response pairs and, in cases where automated metrics may yield uncertain or inconsistent scores, human annotators will be employed in conjuction with those scores along with the semantic similarity scores described below to review and normalize the evaluation.
• Scoring for Semantic Answer Matching:

  The semantic similarity score combines word-level, sentence-level, and paragraph-level similarity. Each similarity level is calculated using embeddings and cosine similarity, and the final score is computed as a weighted sum of these individual scores:

  \[ \text{Semantic Match Score} = w_{word} \cdot \text{CosineSim}_{word} + w_{sentence} \cdot \text{CosineSim}_{sentence} + w_{paragraph} \cdot \text{CosineSim}_{paragraph} \] Where:

  • \(w_{word}, w_{sentence}, w_{paragraph}\) are weights that sum to 1.
  • \(\text{CosineSim}_{word}\): Mean pooling of word embeddings obtained from the model output using: \[ E_{\text{sentence}} = \frac{\sum_{i=1}^{T} E_{\text{token}_i} \cdot \text{Mask}_i}{\sum_{i=1}^{T} \text{Mask}_i} \] Where:
    • \(E_{\text{token}_i}\) is the embedding for the \(i\)-th token.
    • \(\text{Mask}_i\) is the attention mask for the \(i\)-th token to exclude padding tokens.
    • \(T\) is the total number of tokens.
  
  
  
  • \(\text{CosineSim}_{sentence}\): Cosine similarity between corresponding sentence embeddings \(E_x\) and \(E_y\), calculated as: \[ \text{CosineSimilarity}(E_x, E_y) = \frac{E_x \cdot E_y}{\|E_x\| \|E_y\|} \] Where:
    • \(E_x \cdot E_y\) is the dot product of the embeddings.
    • \(\|E_x\|\) and \(\|E_y\|\) are the magnitudes of the embeddings.

    Once cosine similarity is computed for each corresponding sentence pair in the generated and reference answers, the aggregated sentence-level similarity is obtained by averaging all pairwise similarities:

    \[ \text{CosineSim}_{\text{sentence}} = \frac{1}{N} \sum_{i=1}^{N} \text{CosineSimilarity}(E_{x_i}, E_{y_i}) \] Where \(N\) is the number of sentences in the generated and reference answers.
  
  
  
  • \(\text{CosineSim}_{paragraph}\): Cosine similarity between the full answer embeddings, calculated as: \[ \text{CosineSimilarity}(E_{\text{paragraph, gen}}, E_{\text{paragraph, ref}}) = \frac{E_{\text{paragraph, gen}} \cdot E_{\text{paragraph, ref}}}{\|E_{\text{paragraph, gen}}\| \|E_{\text{paragraph, ref}}\|} \] Where:
    • \(E_{\text{paragraph, gen}}\) and \(E_{\text{paragraph, ref}}\) are the paragraph embeddings obtained by mean pooling over all token embeddings in the generated and reference answers, respectively:
    \[ E_{\text{paragraph}} = \frac{\sum_{i=1}^{T} E_{\text{token}_i} \cdot \text{Mask}_i}{\sum_{i=1}^{T} \text{Mask}_i} \]
    • \(T\) is the total number of tokens in the paragraph.

  The word-level score will be calculated using a method similar to BERTScore [6], which evaluates text generation using contextual embeddings. The sentence-level and paragraph-level scores will be computed in a manner inspired by SemScore [7], which leverages semantic textual similarity for evaluating instruction-tuned large language models.

  The final score is determined as follows:

  • Full Match: If the semantic match score exceeds the semantic similarity threshold (0.9), full points are awarded.
  • Partial Match: If the semantic match score lies between two thresholds of no semantic similarity and full semantic similarity (0.4 to 0.9), partial points are awarded using a linear interpolation: \[ \text{Partial Points} = \frac{\text{Semantic Match Score} - \text{Lower Threshold}}{\text{Full Threshold} - \text{Lower Threshold}} \times \text{Max Points} \]
• Full Match Example:
  • Question: "What is Laparoscopy?"
  • Expected Answer: "A minimally invasive surgical procedure using a camera."
  • Model Answer: "A minimally invasive surgery with a camera."
  • Scoring: Full Points (1).
• Partial Match Example:
  • Question: "What is Laparoscopy?"
  • Expected Answer: "A minimally invasive surgical procedure using a camera."
  • Model Answer: "A procedure using a camera."
  • Scoring: Partial Points (less than 1).