KI-Startseite

Weitere Themen
KI-Interessierte auf DIN.ONE

Versions Compared

Old Version 27

changes.mady.by.user Diego Collarana

Saved on

compared with

New Version 28

changes.mady.by.user Robert David

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

How do I use KGs for Retrieval-Augmented Generation (RAG)? (2.1 – Prompt Enhancement) – length: up to one page

Lead: Diego

Contributors:

  • Daniel Burkhardt (FSTI)
  • Robert David (SWC)
  • Diego Collarana (FIT)
  • Daniel Baldassare (doctima)
  • Michael Wetzel (Coreon)

Explanation of concepts

  • Retrieval-augmented generation (RAG) methods combine retrieval mechanisms with generative models to enhance the output of LLMs by incorporating external knowledge. By grounding the generated output in specific and relevant information, RAG methods improve the quality and accuracy of the generated output.

  • Types of RAG:

    • Conventional RAG has three components: 1) Knowledge Base, typically created by chunking text documents, transforming them into embeddings, and storing them in a vector store. 2) Retriever searches the vector database for chunks that exhibit high similarity to the query. 3) Generator feeds the retrieved chunks, alongside the original query, to an LLM to generate the final response.
    • Graph RAG integrates knowledge graphs into the RAG framework, allowing for the retrieval of structured data that can provide additional context and factual accuracy to the generative model. 
      Basically, the retrieval can be done on any source that has a semantic representation, e.g. documents with semantic annotations or relational data via OBDA or R2RML, thereby ingesting structured and unstructured source information into the Graph RAG.
  • RAG is used in various natural language processing tasks, including question-answering, information extraction, sentiment analysis, and summarization. It is particularly beneficial in scenarios requiring domain-specific knowledge.

...

  1. Struggles to answer queries that require the intricate interconnectedness of information and global context crucial for generating comprehensive summaries.
  2. It cannot integrate structure and unstructured data, a use case typically required in industrial applications.
  3. Limited accuracy due to context loss during text chunking and its reliance on text similarity search.
  4. It has limited reasoning capabilities, especially with abstract questions that require reasoning, inference, or the synthesis of new information not explicitly stated in the source material.
  5. The answers cannot be backtracked to the information sources (factual grounding).
  6. The external knowledge, while consistent, can still lead to inconsistencies in the generated answer.

We describe various solutions for integrating knowledge graphs into RAG systems to improve accuracy, We describe various solutions for integrating knowledge graphs into RAG systems to improve accuracy, reliability, and explainability. 

...

Description: KG-Guided Retrieval Mechanisms involve using, for example, knowledge graphs or vector databases to enhance the retrieval process in RAG systems. Knowledge graphs provide a structured representation of knowledge, enabling more precise and contextually aware information retrieval. This approach can directly query knowledge graphs or use them to augment queries to other data sources, improving the relevance and accuracy of the retrieved information.

Draft Robert David:

  • Initial RAG idea: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • RAG is commonly used with vector databases.
    • can only grasp semantic similarity represented in the document content
    • only unstructured data
    • vector distance instead of a DB search limits the retrieval capabilities
  • Graph RAG uses knowledge graphs as part of the RAG system
    • KGs for retrieval (directly), meaning the database is storing KG data
    • KGs for retrieval via a semantic layer, potentially retrieving over different data sources of structured and unstructured data
    • KGs for augmenting the retrieval, meaning the queries to some databases are modified via KG data
  • Via Graph RAG, we can
    • ingest additional semantic background knowledge (knowledge model) not represented in the data itself
      • additional related knowledge based on defined paths (rule-based inference)
      • focus on specific aspects of a data set for the retrieval (search configuration)
      • personalization: represent different roles for retrieval via ingesting role description data into the retrieval (especially important in an enterprise environment)
    • reasoning
    • linked data makes factual knowledge related to the LLM-generated knowledge and thereby provides a means to check for correctness
    • explainable AI: provide justifications via KG
    • consolidate different data sources: unstructured, semi-structured, and structured (enterprise knowledge graph scenario)
    • doing the actual retrieval via KG queries: SPARQL
    • hybrid retrieval: combine KG-based retrieval with vector databases or search indexes

Answer 3: Hybrid Retrieval Combining KGs and Dense Vectors

Draft from Daniel Burkhardt: 

Short definition/description of this topic: Hybrid Retrieval combines the strengths of knowledge graphs and dense vector representations to improve information retrieval. This approach leverages the structured, relational data from knowledge graphs and the semantic similarity captured by dense vectors, resulting in enhanced retrieval capabilities. Hybrid retrieval systems can improve semantic understanding and contextual insights while addressing scalability and integration complexity challenges. 

  • First, the user's question is processed to extract key entities and relationships using entity linking and relationship extraction techniques as a (semantic) graph representation of the question. (Natural Language Understanding)
  • Next, the graph representation is executed against the knowledge graph database, which first retrieves information from the knowledge graph and then retrieves the associated mapped data source.
    Data sources can be of different kinds:
    • Knowledge graph data
    • Non-knowledge graph data with a graph representation:
      • Tabular and relational data, e.g. via OBDA or R2RML.
      • Semi-structured data, e.g. XML or DITA.
      • Unstructured natural language, e.g. via semantic annotations.
  • Then, the retrieved (different kinds of) results are consolidated (preprocessed) to be ingested into the LLM prompt. (Data consolidation)
  • Finally, the consolidated results are passed to the LLM for summarization or further processing to generate the final answer. (Response generation)

Considerations:

  • Limited input data: a short user's question poses a challenge to effectively create a graph representation sufficiently expressive for a high-quality retrieval of information.
  • Knowledge model: a high-quality graph representation of both the user question and the actual information in the database is very likely to need a knowledge model with sufficient expressivity in the background.
  • Graph representation: doing graph-based retrieval of (heterogeneous) data sources needs an established graph representation for each source.
  • Consolidation architecture: Setting up a system architecture for consolidated data sources needs different kinds of integration components.
  • Semantic gap: there is the risk of a gap of semantic information between the retrieved information and the LLM-generated answer, because any semantics contained in the knowledge graph and any knowledge model cannot be preserved during ingestion into the LLM generation.

Standards and Protocols:

  • Compliance with Data Standards: Ensure the knowledge graph adheres to relevant data modeling standards. Where applicable, utilize standardized vocabularies and ontologies.
  • Interoperability: Design the system for various graph databases and query languages. Support integration with external data sources and systems.

Answer 3: Hybrid Retrieval Combining KGs and Dense Vectors

Draft from Daniel Burkhardt: 

Short definition/description of this topic: Hybrid Retrieval combines the strengths of knowledge graphs and dense vector representations to improve information retrieval. This approach leverages the structured, relational data from knowledge graphs and the semantic similarity captured by dense vectors, resulting in enhanced retrieval capabilities. Hybrid retrieval systems can improve semantic understanding and contextual insights while addressing scalability and integration complexity challenges. 

Key Challenges

  • Knowledge Graph Construction and Maintenance: Creating and updating high-quality knowledge graphs for specific domains can be challenging and resource-intensive.
  • Scalability and Efficiency: Retrieving information from large and complex knowledge graphs while maintaining acceptable response times remains challenging.
  • Evaluation Standardization: The lack of widely accepted benchmarks and evaluation metrics hinders progress and comparability across Graph RAG approaches. The quality of KG is crucial.
  • Human Element, we need knowledge engineers and domain specialists. 

References

Key Challenges

  • Knowledge Graph Construction and Maintenance: Creating and updating high-quality knowledge graphs for specific domains can be challenging and resource-intensive.
  • Scalability and Efficiency: Retrieving information from large and complex knowledge graphs while maintaining acceptable response times remains challenging.
  • Evaluation Standardization: The lack of widely accepted benchmarks and evaluation metrics hinders progress and comparability across Graph RAG approaches. The quality of KG is crucial.
  • Human Element, we need knowledge engineers and domain specialists. 

References

...

Short definition/description of this topic: KG-Based Consistency Checking involves using knowledge graphs to ensure the consistency of LLM outputs. By comparing generated content with the structured data in KGs, this method can identify inconsistencies and improve the coherence of LLM-generated information.

literature:https://www.researchgate.net/publication/382363779_Knowledge-based_Consistency_Testing_of_Large_Language_Models

  • Content ...
  • Content ...
  • Content ... 

...

and improve the coherence of LLM-generated information.

literature:https://www.researchgate.net/publication/382363779_Knowledge-based_Consistency_Testing_of_Large_Language_Models

  • Content ...
  • Content ...
  • Content ... 

How do I evaluate LLMs through KGs? (3) – length: up to one page

Target applications that can not be evaluated with gold/reference data because the gold standard is changing over time (grounded knowledge), similarity-based methods are failing (hallucination), or to enhance test data with RAG (Bias detection).

  • When are KG needed in LLM evaluation:
    • analyzing grounding capabilities of LLMs (knowledge coverage and factuality)
    • analyzing hallucination of LLMs
    • analyzing inherent bias from training data
  • General Methods:
    • extracting KG triplets from LLM outputs and evaluate the results
    • enhancing example inputs for the LLM and evaluate biased results

Answer 1: Using KGs to Evaluate LLM Knowledge Coverage

...

literature: https://www.amazon.science/publications/grapheval-a-knowledge-graph-based-llm-hallucination-evaluation-framework

...


  • Content ... 

Answer 2: Analyzing LLM Biases through KG Comparisons

...