Evaluation and lexicon¶
Evaluate¶
mcaligner evaluate PRED.jsonl GOLD.jsonl [--annotated-only] [--min-confidence 0.8] [--per-sentence] [--json]
With A the predicted links, S the gold sure links and P all gold links:
sure precision/recall/F1: predicted sure links against S;
all precision/recall/F1: A against P, ignoring S/P;
AER = 1 − (|A∩S| + |A∩P|) / (|A| + |S|) (Och & Ney 2003).
Only pairs in the gold file are scored, because gold often covers part of a
text; a gold pair without predictions counts as empty. --annotated-only
scores only pairs with predicted links (a partial run). --min-confidence
ignores weaker predicted links, to see precision at a confidence level. Totals
are micro-averaged.
Lexicon¶
mcaligner lexicon $SRC -a pred=ar,grc,out/pred.jsonl -o lexicon.yaml [--min-confidence 0.8] [--min-count 2] [--top 5]
ArabGr1-Arabic-Ephesians-Parsed:
رَسُول:
part_of_speech: NOUN
occurrences: 2
unaligned: 0
alignments:
ἀπόστολος: {count: 2, part_of_speech: NOUN}
A .json output path writes JSON. Only links at least --min-confidence
confident count (default 0.8). On the silver set this reproduces the old
ArabicLexicon.yaml.