The number of claims in an answer can be used as a proxy for how comprehensive it is.

Claim extraction is important because it unlocks the ability to evaluate long-form content generated by language models.

Claimify includes context—a window of text around each sentence—to ensure accurate interpretation of sentences during claim extraction.

Selection in the Claimify pipeline means filtering out sentences that do not contain any verifiable claims.

Claimify breaks down claim extraction into three parts: selection, disambiguation, and decomposition.

Evaluating hallucination detection is difficult when answers are paragraph or multi-paragraph length because it's hard to systematically verify all the information.

An example of a sentence that would be filtered out in the selection stage is 'companies should embrace AI' because it is an opinion, not a factual claim.

Claimify takes text of any length and decomposes it into high-quality claims.

Language models in question-answering applications may generate hallucinations where they make up information rather than basing answers on source documents.

Claim extraction enables evaluation beyond hallucination detection, including assessment of answer relevance to questions.

Extraction is the process of breaking down a text into claims.

Subjective qualifiers like 'notable' cannot be included in claims because they are not verifiable as true or false.

Claims should be decomposed into the simplest possible independent statements rather than kept as compound claims.

Individual factual claims can be assessed independently for relevance and then aggregated into a composite relevance measure for the entire answer.

Disambiguation in the Claimify pipeline is the process of detecting ambiguity in sentences and determining whether ambiguity can be resolved using context or must be flagged as unresolvable.

For the sentence 'countries like Afghanistan and Sudan have experienced similar challenges to those of Libya', the baseline never specified what 'those' refers to, while Claimify made specific claims about the issues (public health crisis, flooding, contaminated water) based on context.

Distilling multi-paragraph answers into simple standalone factual statements makes it easier to check hallucinations independently.

Ambiguity in text means there are multiple plausible interpretations, and depending on which interpretation is chosen, different sets of claims will be extracted.

The ability to determine whether ambiguity in text can be resolved using context is a unique and powerful aspect of Claimify.

Decomposition in the Claimify pipeline takes the disambiguated sentence and breaks it down into simple standalone factual statements.