...

• [1] Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, Siliang Tang: Graph Retrieval-Augmented Generation: A Survey. CoRR abs/2408.08921 (2024)
• [2] Diego Collarana, Moritz Busch, Christoph Lange: Knowledge Graph Treatments for Hallucinating Large Language Models. ERCIM News 2024(136) (2024)
• [3] Junde Wu, Jiayuan Zhu, Yunli Qi: Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation. CoRR abs/2408.04187 (2024)
• [4] Sen, Priyanka, Sandeep Mavadia, and Amir Saffari. Knowledge graph-augmented language models for complex question answering. Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations - NLRSE (2023)
• [5] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, Xindong Wu: Unifying Large Language Models and Knowledge Graphs: A Roadmap. IEEE Trans. Knowl. Data Eng. 36 (7) (2024)
• [6] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Jonathan Larson: From Local to Global: A Graph RAG Approach to Query-Focused Summarization. CoRR abs/2404.16130 (2024)
• [7] Bhaskarjit Sarmah, Benika Hall, Rohan Rao, Sunil Patel, Stefano Pasquali, Dhagash Mehta: HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction. CoRR abs/2408.04948 (2024)
• [8] Jens Lehmann, Dhananjay Bhandiwad, Preetam Gattogi, Sahar Vahdati: Beyond Boundaries: A Human-like Approach for Question Answering over Structured and Unstructured Information Sources. Trans. Assoc. Comput. Linguistics (2024)
• [9] Juan Sequeda, Dean Allemang, Bryon Jacob: A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases. GRADES/NDA (2024)

First draft to be created until 11 October 2024.

ADD NEW TOP LEVEL SECTION: LLM TRAINING

How do I enhance LLM explainability by using KGs? (2.2 – Answer Verification) – length: up to one page

Lead: Daniel Burkhardt

Problem Statement (one paragraph) 

KG-Enhanced LLM explainability focuses on integrating structured knowledge from Knowledge Graphs (KGs) to improve the explainability of large language models (LLMs). The primary goal is to allow LLMs to generate outputs that are more transparent and justifiable. This integration involves aligning LLM outputs with verifiable facts and structured data, enabling improved trust and factuality. The combination of KGs with LLMs ensures that the output remains grounded in known data, reducing issues [1,2]. 

Explanation of concepts 

• KG Alignment with LLMs: This refers to ensuring that the representations generated by LLMs are in sync with the structured knowledge found in KGs. For example, frameworks like GLaM fine-tune LLMs to align their outputs with KG-based knowledge, ensuring that responses are factually accurate and well-grounded in known data [3]. By aligning LLMs with structured knowledge, the explainability of model predictions is improved, making it easier for users to verify how and why certain information was provided [1].

• KG-Guided Explanation Generation: KGs assist in generating explanations for LLM outputs by providing a logical path or structure to the answer. By referencing entities and their relationships within a KG, LLMs can produce detailed, justifiable answers. Studies like those in the education domain use KG data to provide clear, factually supported explanations for LLM-generated responses [2,5].

• Factuality and Verification: Factuality in LLM outputs is critical for trust, and KGs play a crucial role in verifying the truthfulness of LLM answers. Systems like GraphEval [6] analyze LLM outputs by comparing them to large-scale KGs, ensuring that the content is factual. This verification step mitigates hallucination risks and ensures outputs are reliable [6,7].

Brief description of the state of the art (one paragraph) 

The integration of KGs into LLM explainability is currently focused on improving transparency through KG alignment and post-hoc verification methods. Existing models like GPT-3 and GPT-4 demonstrate high linguistic ability but suffer from opacity and factual inaccuracies. Research efforts, including KG-guided explanation generation and factual verification techniques (e.g., FACTKG [5]), aim to address these challenges by incorporating KGs during and after LLM inference [1,6,5].

References

1. Zhang et al., 2024, " Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering" 
2. Zhao et al., 2023, "Explainability for Large Language Models: A Survey" 
3. Dernbach et al., 2024, "GLaM: Fine-Tuning Large Language Models for Domain Knowledge Graph Alignment via Neighborhood Partitioning and Generative Subgraph Encoding" 
4. Rasheed et al., 2024, "Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendations" 
5. Kim et al., 2023, "FACTKG: Fact Verification via Reasoning on Knowledge Graphs" 
6. Liu et al., 2024, "Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs" 
7. Hao et al., 2024, "ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings" 
8. Jiang et al., 2023, "Efficient Knowledge Infusion via KG-LLM Alignment" 

Answer 1: Measuring KG Alignment in LLM Representations

Measuring the alignment between LLM representations and KGs involves comparing how well the LLM’s output matches the structured knowledge in the KG. For example, in GLaM [3], fine-tuning is performed to align LLM outputs with KG-derived entities and relationships, ensuring that responses are not only accurate but also interpretable. The alignment helps reduce issues like hallucinations by grounding responses in verifiable data. This method was used to improve performance in domain-specific applications, where LLMs need to accurately reflect relationships and entities defined in KGs [3, 4, 5].

Answer 2: KG-Guided Explanation Generation

KGs can be used to guide the explanation generation process, where the LLM references structured data in the KG to justify its output. For instance, in the educational domain, explanations are generated using semantic relations from KGs to ensure that recommendations and answers are factually supported [4]. This method not only provides the user with an understandable explanation but also reduces hallucination risks by ensuring that every output can be traced back to a known fact in the KG. Studies on KG-guided explanation generation in various fields confirm its utility in making LLM outputs more transparent and understandable to non-experts [4,5].

Answer 3: KG-Based Fact-Checking and Verification

KG-based fact-checking is an essential method for improving LLM explainability. By cross-referencing LLM outputs with structured knowledge in KGs, fact-checking systems like GraphEval ensure that generated responses are accurate and grounded in truth [6, 7]. This is especially useful for reducing hallucinations. GraphEval automates the process of verifying LLM outputs against a KG containing millions of facts, allowing for scalable and efficient fact-checking that improves both explainability and user trust [6].

• First, the generated output is analysed regarding the knowledge graph and key entities and relationships are extracted to create a graph representation of the LLM answer.
• Next, this graph representation is then analyzed regarding the knowledge graph used for retrieval and any knowledge models in the background are also included. The analysis retrieves a graph representation of an explanation or justification and is returned as a (sub)graph or graph traversal with any additional information added, like RDF* weights.
• Finally, the explanation is then returned to the user in a human-readable way to be cross-checked with the LLM generated answer.

Considerations:

• Limited input data: a short LLM generated answer poses a challenge to effectively backtrack sufficient information in the knowledge graph for a high-quality explanation.
• Presentation: the explanation is graph-based data and difficult to explain or present to non-experts.

Standards and Protocols:

Query languages

• https://www.w3.org/TR/sparql11-query/
• https://neo4j.com/product/cypher-graph-query-language/

Path retrieval

• https://graphdb.ontotext.com/documentation/10.7/graph-path-search.html
• https://neo4j.com/docs/graph-data-science/current/algorithms/pathfinding/

How do I enhance LLM reasoning through KGs? (2.3 – Answer Augmentation) – length: up to one page

Lead: Daniel Burkhardt

Problem Statement (one paragraph) 

KG-Enhanced LLM Reasoning improves the reasoning capabilities of LLMs by leveraging structured knowledge from KGs. This allows LLMs to perform more complex reasoning tasks, such as multi-hop reasoning, where multiple entities and relationships need to be connected to answer a query. Integrating KGs enhances the ability of LLMs to make logical inferences and draw conclusions based on factual, interconnected data, rather than relying solely on unstructured text [9, 10].

Explanation of concepts 

• Multi-hop Reasoning with KGs: This involves connecting different pieces of information across multiple steps using relationships stored in KGs. By structuring queries through KGs, LLMs can reason through several layers of related entities and provide accurate answers to more complex questions [11, 10].

• Tool-Augmented Reasoning: LLMs can use external tools, such as KG-based queries, to retrieve relevant data during inference, allowing for improved reasoning. ToolkenGPT [13] demonstrates how augmenting LLMs with such tools during multi-hop reasoning helps them perform more logical, structured reasoning by accessing real-time KG data [7, 13].

• Consistency Checking in Reasoning: KG-based consistency checking ensures that LLMs maintain logical coherence throughout their reasoning processes. Systems like KONTEST [12] systematically test LLM outputs against KG facts to ensure that answers remain consistent with established knowledge, reducing logical errors [12, 13].

Brief description of the state of the art (one paragraph) 

The use of KGs to enhance reasoning is advancing rapidly, with multi-hop reasoning and retrieval-augmented generation (RAG) methods emerging as key techniques. These methods allow LLMs to perform reasoning tasks that require connecting multiple pieces of information through structured KG paths [9, 11]. Furthermore, systems like ToolkenGPT [13] integrate KG-based tools during inference, allowing LLMs to access external factual data, improving their reasoning accuracy [10, 13].

References

9. Liao et al., 2021, "To hop or not, that is the question: Towards effective multi-hop reasoning over knoweldge graphs" 

10. Schick et al., 2023, "Toolformer: Language Models Can Teach Themselves to Use Tools" 

11. Bratanič et al., 2024, "Knowledge Graphs & LLMs: Multi-Hop Question Answering " 

12. Rajan et al., 2024, "Knowledge-based Consistency Testing of Large Language Models" 

13. Hao et al., 2024, "ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings" 

Answer 1: KG-Guided Multi-hop Reasoning

Multi-hop reasoning refers to the process of connecting multiple entities or facts across a KG to answer complex queries. Using KGs in this way allows LLMs to follow logical paths through the data to derive answers that would be challenging with unstructured text alone. For instance, the Neo4j framework enhances LLM multi-hop reasoning by allowing the LLM to query interconnected entities efficiently [11]. This method improves LLM performance in tasks requiring stepwise reasoning across multiple facts [9, 11].

Answer 2: KG-Based Consistency Checking in LLM Outputs

LLMs have demonstrated impressive capabilities in generating human-like responses across diverse applications. However, their internal workings remain opaque, posing challenges to explainability and interpretability [2]. This lack of transparency introduces risks, especially in high-stakes applications, where LLMs may produce outputs with factual inaccuracies—commonly known as hallucinations—or even harmful content due to misinterpretation of prompts [6, 9]. Consequently, there is a pressing need for enhanced explainability in LLMs to ensure the accuracy, trustworthiness, and accessibility of model outputs for both end-users and researchers alike [2, 4].

One promising approach to improving LLM explainability is integrating KGs, which provide structured, fact-based representations of knowledge. KGs store relationships between entities in a networked format, enabling models to reference explicit connections between concepts and use these as reasoning pathways in generating text [10]. By aligning LLM responses with verified facts from KGs, we aim to reduce hallucinations and create outputs grounded in reliable data. For example, multi-hop reasoning over KGs can improve consistency by allowing LLMs to draw links across related entities—a particularly valuable approach for complex, domain-specific queries [11]. Additionally, retrieval-augmented methods that incorporate KG triplets can further enhance the factuality of LLM outputs by directly integrating structured knowledge into response generation, thereby minimizing unsupported claims [3].

However, the integration of KGs with LLMs presents unique challenges, particularly in terms of scalability and model complexity. The vast number of parameters in LLMs makes interpreting and tracing decision paths challenging, especially when the model must align with external knowledge sources like KGs [8]. Traditional interpretability methods, such as feature attribution techniques like SHAP and gradient-based approaches, are computationally intensive and less feasible for models with billions of parameters [2, 12]. Therefore, advancing KG-augmented approaches is essential for creating scalable, efficient solutions for real-world applications.

The need for KG-augmented LLMs is especially critical in domain-specific contexts, where high-fidelity, specialized information is essential. In fields such as medicine and scientific research, domain-specific KGs provide precise and contextually relevant information that general-purpose KGs cannot match [1]. Effective alignment with these KGs would not only support more accurate predictions but also enable structured, explainable reasoning, thereby making LLMs’ decision-making processes transparent and accessible for both domain experts and general users alike.

Explanation of concepts 

• KG Alignment with LLMs: This refers to ensuring that the representations generated by LLMs are in sync with the structured knowledge found in KGs. For example, frameworks like GLaM fine-tune LLMs to align their outputs with KG-based knowledge, ensuring that responses are factually accurate and well-grounded in known data [3]. By aligning LLMs with structured knowledge, the explainability of model predictions is improved, making it easier for users to verify how and why certain information was provided [1].

• KG-Guided (post-hoc) Explanation Generation: KGs assist in generating explanations for LLM outputs by providing a logical path or structure to the answer. By referencing entities and their relationships within a KG, LLMs can produce detailed, justifiable answers. Studies like those in the education domain use KG data to provide clear, factually supported explanations for LLM-generated responses [2,5]. Some approaches involve equipping LLMs with tools, such as fact-checking systems, that reference KG data for verifying outputs post-generation. Through this process, known as post-hoc explanation, LLMs can justify or clarify responses by referencing relevant facts from KGs, enhancing user trust and transparency. This augmentation allows LLMs to provide clearer justifications and improve the credibility of their outputs by aligning with trusted knowledge sources [7].

• Domain-Specific Knowledge Enhancement: In specialized fields like medicine or science, domain-specific KGs provide high-fidelity information that general-purpose KGs cannot offer. Leveraging these specialized KGs, LLMs can generate responses that are both contextually relevant and reliable, meeting the specific knowledge needs of domain experts. This alignment with specialized KGs is critical to ensuring that outputs are appropriate for expert users and rooted in precise, authoritative knowledge [1].

• Factuality and Verification: Knowledge Graphs (KGs) provide structured, factual knowledge that serves as a grounding source for LLM outputs. By referencing verified relationships between entities, KGs help reduce the occurrence of hallucinations and ensure that responses are factually accurate. This grounding aligns LLM outputs with established knowledge, which is essential in high-stakes fields where accuracy is critical. Systems like GraphEval [6] analyze LLM outputs by comparing them to large-scale KGs, ensuring that the content is factual. This verification step mitigates hallucination risks and ensures outputs are reliable [2,6,7].

Brief description of the state of the art

Recent research in integrating KGs with LLMs has produced several frameworks and methodologies designed to enhance model transparency, factuality, and domain relevance. Key initiatives include KG alignment and post-hoc verification techniques, both of which aim to improve the explainability and reliability of LLM outputs.

For KG alignment, approaches such as the GLaM framework fine-tune LLMs to align responses with KG-based knowledge. This ensures that model outputs remain factually grounded, particularly by embedding KG information into the LLM’s representation space. GLaM has demonstrated that aligning model outputs with structured knowledge can reduce factual inconsistencies, supporting applications that require reliable, fact-based answers [3].

In post-hoc explanation generation, frameworks like FACTKG leverage KG data to verify model responses after generation, producing detailed justifications that reference specific entities and relationships. This KG-guided approach has shown efficacy in fields like education, where models need to generate clear, factually supported answers to complex questions. FACTKG’s methodology enables LLMs to produce explanations that are both traceable and verifiable, thereby improving user trust in the generated content [5].

In domain-specific contexts, specialized KGs provide high-fidelity information that general-purpose KGs cannot. For instance, in the medical domain, projects like KnowPAT have incorporated domain-specific KGs to enhance LLM accuracy in delivering contextually appropriate responses. By training LLMs with healthcare-specific KGs, KnowPAT enables models to provide precise, authoritative responses that align with expert knowledge, which is crucial for sensitive fields where general-purpose knowledge may be insufficient [1].

Further, initiatives such as GraphEval underscore the role of KGs in factuality and verification. By analyzing LLM outputs through comparisons with large-scale KGs, GraphEval ensures that model responses align with known, structured facts, helping mitigate hallucination risks. This comparison process has proven valuable in high-stakes fields, as it enables verification of LLM-generated information against a vast repository of established facts, making outputs more reliable and reducing potential inaccuracies [6].

Answer 1: Measuring KG Alignment in LLM Representations

Description

Measuring the alignment between LLM representations and KGs involves comparing how well the LLM’s output matches the structured knowledge in the KG. For example, in GLaM [3], fine-tuning is performed to align LLM outputs with KG-derived entities and relationships, ensuring that responses are not only accurate but also interpretable. The alignment helps reduce issues like hallucinations by grounding responses in verifiable data. This method was used to improve performance in domain-specific applications, where LLMs need to accurately reflect relationships and entities defined in KGs [3, 4, 5].

Considerations

• Data Quality and Coverage: The quality and completeness of the KG significantly impact alignment. If the KG lacks comprehensive data, the LLM might still produce hallucinations or incomplete outputs despite alignment efforts.
• Alignment Metrics: Measuring alignment between LLM and KG representations requires specific metrics. These may include entity coverage, similarity scores between LLM-generated and KG-retrieved responses, or accuracy in matching relational paths in the KG.
• Computational Complexity: Aligning LLM representations with extensive KGs is computationally intensive, especially with large-scale KGs. Efficient alignment techniques and resource allocation are crucial for scalable implementations.

Standards, Protocols and Scientific Publications

• Embedding Alignment: Techniques like TransE and BERT-based entity alignment support embedding alignment between LLMs and KGs.
• KG Ontologies: Ontologies help structure KG data and provide a common format, such as OWL (Web Ontology Language) and RDFS (RDF Schema).
  • OWL: https://www.w3.org/OWL/
  • RDFS: https://www.w3.org/TR/rdf-schema/

Answer 2: KG-Guided Explanation Generation

Description

KGs can be used to guide the explanation generation process, where the LLM references structured data in the KG to justify its output. For instance, in the educational domain, explanations are generated using semantic relations from KGs to ensure that recommendations and answers are factually supported [4]. This method not only provides the user with an understandable explanation but also reduces hallucination risks by ensuring that every output can be traced back to a known fact in the KG. Studies on KG-guided explanation generation in various fields confirm its utility in making LLM outputs more transparent and understandable to non-experts [4,5].

Considerations

• Interpretability for Non-Experts: Presenting KG-based explanations to lay audiences can be challenging if the KG structure is complex. Simplified language or visual aids may be necessary to enhance comprehension.
• Semantic Completeness: The KG needs to encompass the relevant relationships to generate comprehensive explanations. Missing relationships may lead to partial or insufficient justifications.
• Consistency Across Domains: Cross-domain use cases require consistent explanations, which may be complex if the KG includes overlapping or domain-specific relations that vary significantly.

Standards, Protocols and Scientific Publications

• Explanation Frameworks: Frameworks such as SHACL (Shapes Constraint Language) help ensure that KGs meet structural standards, which supports high-quality, consistent explanations.
  • SHACL: https://www.w3.org/TR/shacl/
• Query Languages for Explanation Generation:
  • SPARQL for RDF-based KGs: https://www.w3.org/TR/sparql11-query/
  • Cypher for Neo4j: https://neo4j.com/product/cypher-graph-query-language/
• Publications:
  • Rajabi and Etminani (2022), “Knowledge-graph-based explanable AI: A systematic review". 
  • Kim et al., 2023, "FACTKG: Fact Verification via Reasoning on Knowledge Graphs"

Answer 3: KG-Based Fact-Checking and Verification

Description: 

KG-based fact-checking is an essential method for improving LLM explainability. By cross-referencing LLM outputs with structured knowledge in KGs, fact-checking systems like GraphEval ensure that generated responses are accurate and grounded in truth [6, 7]. This is especially useful for reducing hallucinations. GraphEval automates the process of verifying LLM outputs against a KG containing millions of facts, allowing for scalable and efficient fact-checking that improves both explainability and user trust [6].

• First, the generated output is analysed regarding the knowledge graph and key entities and relationships are extracted to create a graph representation of the LLM answer.
• Next, this graph representation is then analyzed regarding the knowledge graph used for retrieval and any knowledge models in the background are also included. The analysis retrieves a graph representation of an explanation or justification and is returned as a (sub)graph or graph traversal with any additional information added, like RDF* weights.
• Finally, the explanation is then returned to the user in a human-readable way to be cross-checked with the LLM generated answer.

Considerations:

• Limited Input Data: Short LLM-generated answers may lack sufficient information for thorough fact-checking. In such cases, additional context or auxiliary information may be required to validate the response accurately.
• Presentation for Non-Experts: Graph-based explanations can be challenging to interpret for lay audiences. Translating complex graph structures into user-friendly formats or summaries can improve accessibility.
• Data Complexity and Scale: Fact-checking across large KGs can be resource-intensive, requiring efficient query algorithms and substantial computational power for high-speed verification

Standards, Protocols and Scientific Publications:

• Query Languages: Specialized query languages facilitate efficient and precise retrieval of relevant KG data for fact-checking purposes.
  • SPARQL for RDF databases: https://www.w3.org/TR/sparql11-query/
  • Cypher for Neo4j-based KGs: https://neo4j.com/product/cypher-graph-query-language/
• Path Retrieval Algorithms: Pathfinding and traversal algorithms enable effective fact-checking by retrieving relevant knowledge paths that align with LLM-generated content.
  • Ontotext GraphDB: https://graphdb.ontotext.com/documentation/10.7/graph-path-search.html
  • Neo4j Pathfinding Algorithms: https://neo4j.com/docs/graph-data-science/current/algorithms/pathfinding/

References

1. Zhang et al., 2024, " Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering" 
2. Zhao et al., 2023, "Explainability for Large Language Models: A Survey" 
3. Dernbach et al., 2024, "GLaM: Fine-Tuning Large Language Models for Domain Knowledge Graph Alignment via Neighborhood Partitioning and Generative Subgraph Encoding" 
4. Rasheed et al., 2024, "Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendations" 
5. Kim et al., 2023, "FACTKG: Fact Verification via Reasoning on Knowledge Graphs" 
6. Liu et al., 2024, "Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs" 
7. Hao et al., 2024, "ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings" 
8. Jiang et al., 2023, "Efficient Knowledge Infusion via KG-LLM Alignment" 
9. Weidinger, L., et al., 2021. "Ethical and social risks of harm from Language Models". arXiv preprint arXiv:2112.04359
10. Liao et al., 2021, "To hop or not, that is the question: Towards effective multi-hop reasoning over knoweldge graphs" 
11. Bratanič et al., 2024, "Knowledge Graphs & LLMs: Multi-Hop Question Answering" 
12. Sundararajan, M., et al. (2017). "Axiomatic Attribution for Deep Networks." International Conference on Machine Learning.
13. Bordes et al. (2013). "Translating Embeddings for Modeling Multi-relational Data".  

How do I enhance LLM reasoning through KGs? (2.3 – Answer Augmentation) – length: up to one page

Lead: Daniel Burkhardt

Problem Statement 

KG-Enhanced LLM Reasoning improves the reasoning capabilities of LLMs by leveraging structured knowledge from KGs. This allows LLMs to perform more complex reasoning tasks, such as multi-hop reasoning, where multiple entities and relationships need to be connected to answer a query. Integrating KGs enhances the ability of LLMs to make logical inferences and draw conclusions based on factual, interconnected data, rather than relying solely on unstructured text [9, 10].

Explanation of concepts 

• Multi-hop Reasoning with KGs: This involves connecting different pieces of information across multiple steps using relationships stored in KGs. By structuring queries through KGs, LLMs can reason through several layers of related entities and provide accurate answers to more complex questions [11, 10].

• Tool-Augmented Reasoning: LLMs can use external tools, such as KG-based queries, to retrieve relevant data during inference, allowing for improved reasoning. ToolkenGPT [13] demonstrates how augmenting LLMs with such tools during multi-hop reasoning helps them perform more logical, structured reasoning by accessing real-time KG data [7, 13].

• Consistency Checking in Reasoning: KG-based consistency checking ensures that LLMs maintain logical coherence throughout their reasoning processes. Systems like KONTEST [12] systematically test LLM outputs against KG facts to ensure that answers remain consistent with established knowledge, reducing logical errors [12, 13].

Brief description of the state of the art 

The use of KGs to enhance reasoning is advancing rapidly, with multi-hop reasoning and retrieval-augmented generation (RAG) methods emerging as key techniques. These methods allow LLMs to perform reasoning tasks that require connecting multiple pieces of information through structured KG paths [9, 11]. Furthermore, systems like ToolkenGPT [13] integrate KG-based tools during inference, allowing LLMs to access external factual data, improving their reasoning accuracy [10, 13].

Answer 1: KG-Guided Multi-hop Reasoning

Description: 

Multi-hop reasoning refers to the process of connecting multiple entities or facts across a KG to answer complex queries. Using KGs in this way allows LLMs to follow logical paths through the data to derive answers that would be challenging with unstructured text alone. For instance, the Neo4j framework enhances LLM multi-hop reasoning by allowing the LLM to query interconnected entities efficiently [11]. This method improves LLM performance in tasks requiring stepwise reasoning across multiple facts [9, 11].

Considerations

Standards and Protocols and Scientific Publications

Answer 2: KG-Based Consistency Checking in LLM Outputs

Description: 

KG-based consistency checking ensures that LLMs produce logically coherent and accurate outputs by comparing their answers with facts from a KG. KONTEST is an example of a system that uses KGs to systematically generate consistency tests, ensuring that LLM outputs are verified for logical consistency before being returned to the user [12]. This reduces errors in reasoning and improves the reliability of the model’s conclusions [12, 13].

Considerations

Standards and Protocols and Scientific Publications

References

9. Liao et al., 2021, "To hop or not, that is the question: Towards effective multi-hop reasoning over knoweldge graphs" 

10. Schick et al., 2023, "Toolformer: Language Models Can Teach Themselves to Use Tools" 

11. Bratanič et al., 2024, "Knowledge Graphs & LLMs: Multi-Hop Question Answering " 

12. Rajan et al., 2024, "Knowledge-based Consistency Testing of Large Language Models" 

13. Hao et al., 2024, "ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings" KG-based consistency checking ensures that LLMs produce logically coherent and accurate outputs by comparing their answers with facts from a KG. KONTEST is an example of a system that uses KGs to systematically generate consistency tests, ensuring that LLM outputs are verified for logical consistency before being returned to the user [12]. This reduces errors in reasoning and improves the reliability of the model’s conclusions [12, 13].

How do I evaluate LLMs through KGs? (3) – length: up to one page

...