...

[8] Jens Lehmann, Dhananjay Bhandiwad, Preetam Gattogi, Sahar Vahdati: Beyond Boundaries: A Human-like Approach for Question Answering over Structured and Unstructured Information Sources. Trans. Assoc. Comput. Linguistics 12: 786-802 (2024)

[9] Juan Sequeda, Dean Allemang, Bryon Jacob:
A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases. GRADES/NDA 2024: 5:1-5:12

How do I enhance LLM explainability by using KGs? (2.2 – Answer Verification) – length: up to one page

...

Short definition/description of this topic: This involves using knowledge graphs to identify and analyze biases in LLMs. By comparing LLM outputs with the neutral, structured data in KGs, this approach can highlight biases and suggest ways to mitigate them, leading to more fair and balanced AI systems.

First Version:  In the second process, the inputs, i.e., the evaluation samples, are enhanced with information from a KG to provide helpful or misleading context. KG nodes must first be extracted from the samples using, for example, RAG. Then, based on the extracted KG nodes, the top k nodes can be determined from the KG using an arbitrarily efficient retrieval method. These nodes can then be used to enhance the input. For example, the nodes can be displayed as “superior knowledge” in the prompt in order to carry out adversarial attacks to obtain biased responses from open- and closed-source LLMs. Finally, the output of the model is analyzed. Again, different evaluation methods and metrics can be applied in the final step.

• Content ...
• Content ... 

literature: https://arxiv.org/abs/2405.04756