Paper Summary: LLM Internal Encoding of Truthfulness and Hallucinations

#

The objective of this paper is to gain a deeper understanding of errors produced by large language models (LLMs) by examining their internal representations. The authors aim to reveal how information about the truthfulness of LLM outputs is encoded internally, going beyond extrinsic, behavioral analysis. They also seek to investigate the relationship between these internal representations and the external behavior of LLMs, including their tendency to produce inaccuracies or “hallucinations”. Furthermore, the paper intends to explore whether internal representations can be used to predict the types of errors LLMs make and to detect the correct answer even when the model generates an incorrect one.

Archive Download

The key findings of the paper are:

#

• Truthfulness information is concentrated in specific tokens, particularly the exact answer tokens within the generated response. Leveraging this insight significantly improves error detection performance.
• Error detectors trained on LLMs’ internal representations show limited generalization across different tasks and datasets. Generalization is better within tasks requiring similar skills, suggesting that truthfulness encoding is not universal but rather “skill-specific” and multifaceted, with LLMs encoding multiple, distinct notions of truth.
• The internal representations of LLMs can be used to predict the types of errors the model is likely to make. This suggests that LLMs encode fine-grained information about potential errors, which can be classified based on the model’s responses across repeated samples.
• The study reveals a discrepancy between LLMs’ internal encoding and their external behavior. In some cases, the model’s internal representation may identify the correct answer, yet the model consistently generates an incorrect response. This indicates that LLMs may possess the knowledge to produce the correct answer but fail to do so during generation.
• Using a probing classifier trained on error detection and applied to a pool of generated answers can enhance the LLM’s accuracy by selecting the answer with the highest predicted correctness probability. This is particularly effective in cases where the LLM does not show a strong preference for the correct answer during generation.

The paper suggests the following techniques based on their findings:

#

• Enhancing error detection strategies by focusing on the internal representations of exact answer tokens. This localized approach yields significant improvements in identifying errors.
• Developing tailored mitigation strategies by leveraging the ability to predict error types from internal representations. Understanding the types of errors can guide the application of specific mitigation techniques like retrieval-augmented generation or fine-tuning.
• Further research into harnessing the existing knowledge within LLMs, as indicated by the discrepancy between internal encoding and external behavior, to reduce errors. This could involve exploring mechanisms to better align internal truthfulness signals with the generation process.
• Using probing classifiers as a diagnostic tool to identify when an LLM internally encodes the correct answer but fails to generate it. This can help in understanding the limitations of the model’s generation process.

Key takeaways from the paper include:

#

• LLMs possess more information about the truthfulness of their outputs internally than is evident from their generated text.
• The location of the token being analyzed significantly impacts the ability to detect errors, with exact answer tokens being particularly informative about truthfulness.
• Truthfulness encoding in LLMs is likely not a single, universal mechanism but a collection of “skill-specific” representations, which has implications for the generalization of error detection methods across different tasks.
• LLMs internally encode information that correlates with the types of errors they are prone to making, opening possibilities for targeted error mitigation.
• A notable misalignment exists between what LLMs internally “know” to be correct and what they actually generate, suggesting potential for improving accuracy by better leveraging the models’ internal knowledge.
• Adopting a model-centric perspective, by examining internal representations, offers valuable insights into the nature of LLM errors and can guide future research in error analysis and mitigation.