First draft to be created until 11 October 2024.
Question Answering
Contributors:
- Diego Collarana (FIT)
- Michael Wetzel (Coreon)
- Daniel Baldassare (doctima)
Draft from Daniel Baldassare
Definition/Description: Question answering is one of the main application field of large language models. Knowledge graphs have been shown to be useful for providing answers to questions about specific domains and improving the performance of LLM. This chapter covers:
- Methods to retrieve relevant data in the KG
- Methods to rerank the retrieved data from the KG
- Advanced prompt techniques like Chain-of-Thought, Tree of Thoughts (ToT) and Graph of Thoughts (GoT) to handle complex questions
Literature:
Towards Improving the Performance of Question Answering System using Knowledge Graph - A Survey
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Fact Checking
Contributors: please fill in ...
- Diego Collarana (FIT)
- Daniel Baldassare (doctima)
- Rene Pietzsch (ECC)
Problem statement
Large Language Models (LLMs) are powerful models capable of generating coherent text and completing a wide range of tasks, even if they haven't seen the task during training. However, despite their remarkable linguistic abilities, they still face significant challenges in ensuring factual accuracy, especially when knowledge is derived from domain-specific scenarios. LLMs often struggle to produce information based on reliable external sources, leading to the generation of content that may be linguistically correct but not true.
Explanation of concepts:
Factuality in large language models refers to their ability to generate content that aligns with accurate information, encompassing both general world knowledge and specific domain-related facts, which can be verified through reliable external sources (Wang et al., 2023).
Factuals errors have differents roots:
- Caused by model:
- Deficit of domain knowledge, especially in enterprise solutions
- Outdated information present in the training data
- Reasoning errors
- Caused by retrieval:
- Distracted by the amount of context retrieved
- Misunderstood the retrieved information
- Failed to recognize misinformation in contradictory extracted text passages
State-of-the-art
To check factual errors different methodologies:
- Rule-based metrics: lexical metrics like BLEU or ROUGE
- Custom trained evaluator models lke BARTSCore
- LLM-as-Judge
Proposed solutions
Two main methods in order to overcome the problem of generating false information and improve the factuality of LLMs:
Training
- Extending unsupervised text corpora at pretraining incorporating knowledge from KGs
- Finetuning a model with labeled dataset or KGs
- In case of domain-adoption: either train from scratch or finetune with external KG
RAG
- Knowledge retrieval from KG
- Advanced Prompt techniques like CoT
To more details about extending the knowledge of an LLM through training or RAG in combination with KGs see chapter 3.
References
- Wang, Cunxiang, u. a. Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity. arXiv:2310.07521, arXiv, 16. Dezember 2023. arXiv.org, http://arxiv.org/abs/2310.07521.
Draft from Daniel Baldassare
Definition/Description: Fact checking deals with the assertion of llm output's truthfulness.
Two main methods:
- Retrieve facts from KG (RAG- Approach)
- Train custom checker model augmented with knowledge from KG
Literature:
Idea: we first work on the other two main documents/pages and then bootstrap the content of this page with the applications listed in LLMs for KGs and KGs for LLMs.
Multilingual Knowledge Graphs
Contributor:
- Michael Wetzel (Coreon)
Definition/Description: please fill in ...
Another application ...
Contributors:
- Please add yourself if you want to contribute ...
- Please add yourself if you want to contribute ...
- Please add yourself if you want to contribute ...
- ...
Definition/Description: please fill in ...
