Page History

...

The training of large language models typically employs unsupervised methods on extensive datasets. Despite their impressive performance on a range of various tasks, these models often lack the practical, real-world knowledge required for certain specific applications. Furthermore, since domain-specific data is not included in the public domain datasets used for pre-training or fine-tuning large language models (LLMs), the integration of integrating knowledge graphs (KGs) becomes fundamental for the injection of injecting proprietary knowledge into LLMs, especially for enterprise solutions. In order to To infuse this knowledge into LLMs during training, many techniques have been researched in recent years, resulting in three main state-of-the-art methods (Pan et al., 2024): 

1. Integration of KGs into training objectives (See answer 1)
2. Verbalization of KGs into LLM inputs (See answer 2)
3.  Integrate KGs by Fusion Modules: Joint training of graph and language models (See answer 3)

...

The first method focuses on extending the pre-training procedure. The term pretraining objectives is used to describe describes the techniques that guide the learning process of a model from its training data. In the context of pre-training large language models, a variety of various methods have been employed based on the architecture of the model itselfmodel's architecture. Decoder-only models such as GPT-4 usually use Casual Language Modelling (CLM), where the model is presented with a sequence of tokens and learns to predict the next token in the sequence based solely on the preceding tokens (Wang et al., 2022). Integrating KGs into training objectives consists in involves extending the standard llm's pre-training objective of generating coherent and contextually relevant text by designing a knowledge-aware pre-training.

The second method involves integrating KGs directly into the LLM's input by verbalising verbalizing the knowledge graph into the prompt, thereby transforming structured data into a text format that from which the LLM can process and learn from. Data from the knowledge is either prepended or postpended to the user's question as contextual information in the prompt. Within this approach, the standard llm LLM pre-training objective of generating coherent and contextually relevant text remains untouched, and the knowledge augmentation task is modeled as a linguistic task.

...

Brief description of the state-of-the-art

First draft to be created until 11 October 2024

Proposed solutions:

Answer 1:

...

Integrate KGs into

...

LLM Inputs (verbalize KG for LLM training) – Before pre-training enhancement

Contributors:

• Diego Collarana (FIT)

Short definition/description of this topic: please fill in ...

• Content ...
• Content ...
• Content ... 

Answer 2: integrate KGs into LLM Inputs (verbalize KG for LLM training)

Contributors:

• Diego Collarana (FIT)
• Daniel Baldassare (doctima) – Lead
• Michael Wetzel (Coreon)
• Rene Pietzsch (ECC)
• ... 

• Daniel Baldassare (doctima) – Lead
• Michael Wetzel (Coreon)
• Rene Pietzsch (ECC)
• ... 

Draft from Daniel Baldassare :

Short definition/description of this topicDescription: Verbalizing knowledge graphs for LLM is the task of representing knowledge graphs as text so that they can be written directly in the prompt, the main input source of LLM. Verbalization consists of finding textual representations for nodes, relationships between nodes, and their metadata. Verbalization can take place at different stages of the LLM lifecycle, during training (pre-training, instruction fine-tuning) or during inference (in-context learning), and consists in:

• Mark boundaries of graph data using special tokens, like already for SQL-Queries: Improving Generalization in Language Model-Based Text-to-SQL
  Semantic Parsing: Two Simple Semantic Boundary-Based Techniques
• Encoding strategies for nodes, relationship between nodes, nodes communities and metadata Talk like a graph: Encoding graphs for large language models (research.google)
• What needs to be verbalized and where? System prompt for static information like KG-schema, user prompt for data instances

Answer 3: Integrate KGs by Fusion Modules

Contributors:

• Diego Collarana (FIT)

• Techniques
• Encoding strategies for nodes, relationship between nodes, nodes communities, and metadata Talk like a graph: Encoding graphs for large language models (research.google)
• What needs to be verbalized and where? System prompt for static information like KG-schema, user prompt for data instances

Answer 2: Integrate KGs into the LLM Training Objective – during pre-training

Description: The methods learn knowledge directly during training by improving the LLM's encoder and training tasks.

• Incorporate knowledge encoders
• Insert knowledge encoding layers
• Add independent adapters
• Modify the pre-training task

Considerations:

Standards:

• Content ...
• Content ...
• Content ... 

Answer 3: Integrate KGs during Fine-Tuning – Post pre-training enhancement

Description:

Considerations:

Standards:Short definition/description of this topic: please fill in ...

• Content ...
• Content ...
• Content ... 

References:

• S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, und X. Wu, „Unifying Large Language Models and Knowledge Graphs: A Roadmap“, IEEE Trans. Knowl. Data Eng., Bd. 36, Nr. 7, S. 3580–3599, Juli 2024, doi: 10.1109/TKDE.2024.3352100.
• T. Wang u. a., „What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization?“, in Proceedings of the 39th International Conference on Machine Learning, PMLR, Juni 2022, S. 22964–22984. Zugegriffen: 3. Oktober 2024. [Online]. Verfügbar unter: https://proceedings.mlr.press/v162/wang22u.html

...

The emerging field of Graph RAG develops methods to exploit the rich, structured relationships between entities within a KG to retrieve more precise, factually relevant context for LLMs [9]. Graph RAG methods encompass graph construction, knowledge retrieval, and answer-generation techniques [1,2,5]. We find methods that leverage existing open-source KGs [3] to methods for automatically building domain-specific KGs from raw textual data using LLMs [6]. The retrieval phase focuses on efficiently extracting pertinent subgraphs, paths, or nodes relevant to a user query with techniques like embedding similarity, pre-defined rules, or LLM-guided search. In the generation phase, retrieved graph information is transformed into LLM-compatible formats, such as graph languages, embeddings, or GNN encoding, to generate enriched and contextually grounded responses [4]. Recently, significant attention has been given to hybrid approaches combining conventional RAG and Graph RAG strengths [7,8]. HybridRAG integrates contextual information from traditional vector databases and knowledge graphs, resulting in a more balanced and effective system that surpasses individual RAG approaches in critical metrics like faithfulness, answer relevancy, and context recall. 

We describe various solutions for integrating knowledge graphs into RAG systems to improve accuracy, reliability, and explainability. 

...