Page History
...
The first method focuses on extending the pre-training procedure. The term pretraining objectives describes the techniques that guide the learning process of a model from its training data. In the context of pre-training large language models, various methods have been employed based on the model's architecture. Decoder-only models such as GPT-4 usually use Casual Causal Language Modelling (CLM), where the model is presented with a sequence of tokens and learns to predict the next token in the sequence based solely on the preceding tokens (Wang et al., 2022). Integrating KGs into training objectives involves extending the standard llmLLM's pre-training objective of generating coherent and contextually relevant text by designing a knowledge-aware pre-training.
...
RAG methods aim to enhance the capabilities of LLMs by providing real-time information and domain-specific knowledge that may not be present in their training data. Despite its advantages over standalone LLMs, Conventional conventional RAG has the following limitations:
...
Consistency-checking frameworks are also essential for enhancing reasoning accuracy. Systems like KONTESTKONTEST evaluate LLM outputs against KG facts, ensuring logical coherence and flagging inconsistencies. This method reduces the logical errors LLMs might otherwise produce in reasoning tasks by cross-referencing generated answers with verified KG knowledge. Furthermore, GraphEval is used to assess the factuality of LLM responses, leveraging a judge model that systematically aligns generated answers with KG-derived facts [4, 13, 14].
...
