Page History
...
Standards and Protocols and Scientific Publications:The
- RDF (Resource Description Framework) is a W3C-standardized method for modeling graph data. It encodes information as triples: subject, predicate, and object, where the subject and object are nodes, and the predicate is an arc linking them, often identified by URIs; objects may also be literals.
- LAMA [4]: Seminal work demonstrating the evaluation of knowledge represented in LLMs using KGs (the use of KGs for generating evaluation datasets has, since then, been
...
- employed in various further scientific publications [1, 2
...
- , 5, 6, 7]).
Answer 2: Using KGs to Evaluate LLM Factuality
...
- Meaningful graph representations: Meaningful graph representations formally represent semantics that capture a sentence's meaning in a natural language. Various meaningful representations can be used to describe the meaning of a sentence and, therefore, have to be well-defined before evaluating an LLM on factuality using KGs. Target and objective KG should be mapped onto the same meaningful graph representations.
- Information Extraction: Any evaluated LLM output must be encoded into the pre-defined KG meaning representation. These process concepts are versatile and multiple solutions have been used and tested in research. Text-to-Graph Generation Models, KG construction prompt, or multi-component extraction where entities, coreference resolutions, and relations are detected and extracted in multiple stages.
- KG similarityfactuality: Depending on the KG generation strategy, the target and objective KG can be compared and analyzed at different levels and granularities. The general idea is to check whether each triple in the target KG is factually consistent given an objective KG (or context). For instance, a graph neural network (GNN) that encodes edge representations derived from the corresponding entity nodes can be trained on binary classification of factuality or non-factuality of each encoded edge.
...
- For meaningful graph representations, the standard protocols are, for instance, Abstract Meaning Representation (AMR) or Open Information Extraction (OpenIE). AMR is a semantic representation language generated as rooted, directed, edge-labeled, and leaf-labeled graphs. In AMR, the edges are semantic relations, and the nodes are concepts. AMR has a fixed relation vocabulary of approximately 100 relations and the inverse of each relation. In OpenIE, on the other hand, relation triples are represented as a subject, an open relation, and the object of the open relation. An open relation means that OpenIE does not contain a fixed relation vocabulary. Therefore, each sentence is represented as a directed acyclic graph, and an extractor is used to enumerate all word pairs and make a parallel prediction of the relation.
- Extracting information from a text and generating or enhancing a KG from it will be discussed in Chapter 4.2. NLP tasks like named entity recognition, coreference resolutions, and relation extraction are well-established problems in this field of research that are solved using either generative LLMs or fine-tuned language models. The third option of using prompting for generating a KG is based on two techniques: in-context learning and chain-of-thought reasoning (explained in Section 4).
- KG similarity ..factuality The standard protocol for checking the factuality of a generated KG from an LLM output sequence would be to encode the KG using an LLM or a GNN and predict the factuality using binary classification. For both models, context can be provided in addition to the generated KG for higher precision in the prediction. For this task, the GNN has to be fine-tuned to factuality prediction. When using an LLM for the prediction, prompting can be used to predict the factuality of KG triples. The prompt can be enhanced with in-context learning examples or the context of factual KG relations.
Answer 3: Analyzing LLM Biases through KG Comparisons
...
