Page History

...

Draft from Daniel Burkhardt: 

Description:  This involves using knowledge graphs to analyze and evaluate various aspects of LLMs, such as knowledge coverage and biasesfactuality. KGs provide a structured framework information for assessing how well LLMs' knowledge capture and represent knowledge representation across different domains. This involves assessing the extent to which LLMs cover the verifying the knowledge represented in an LLM using KGs. By extracting and comparing knowledge or facts from LLM outputs with the structured data in KGs, this approach can identify gaps in knowledge and areas for improvement in LLM training and performance.

(First Version): The first evaluation process can be divided into two parts. Those can be executed through various techniques, which this section will not discuss. First, the LLM generates output sequences based on an evaluation set of input samples. Specific KG triplets are then identified and extracted from the generated output sequence. The variants for extraction and identification can be found in other subchapters of this DIN SPEC. The extracted KG triplets are usually domain or task-specific. These KG triplets are used to generate a KG. 
In the second step, the KG can now be analyzed. For instance, factuality can be checked by analyzing each KG triplet in the generated KG, given the context provided. Alternatively, the extracted KG triplets can be compared with an existing, more extensive KG to analyze the knowledge coverage of an LLM.

...