# Evaluation

```bash
uv run mcparser evaluate pred.conllu gold.conllu          # table
uv run mcparser evaluate pred.conllu gold.conllu --json   # machine-readable
uv run mcparser evaluate pred.conllu gold.conllu --annotated-only  # only parsed sentences
```

`--calibration` adds a table of accuracy by the confidence recorded in MISC
(`Conf=`), pooled over UPOS, XPOS, lemma, FEATS, HEAD and DEPREL cells. On
about 70,000 cells from ten treebanks, cells at 0.95 or above were 99.4% right,
0.8–0.9 91.8%, and 0.5–0.7 57%, so the scorer ranks cells well; use the table to
choose `--threshold`.

`--predicted-only` ignores gold sentences missing from the predictions;
`--annotated-only` also leaves out predicted sentences that have not been
parsed yet (some HEAD still `_`), e.g. the rest of a file after
`run --max-sentences 30`. Without it, unparsed sentences count as wrong
attachments and pull UAS/LAS down.

The metrics are those of the CoNLL 2018 shared task, ported from the official
`conll18_ud_eval.py` and tested against it (`tests/official/`, 150 random
sentence sets with different tokens, multiword tokens and cells, and 50 whole
files): the numbers agree with the official script. Sentences are matched by
`sent_id` instead of by position, so a partly annotated file can be scored.

| Metric | A word counts as correct when … |
|---|---|
| Tokens | the surface token has the same characters as a gold token |
| Words | it is aligned with a gold word (segmentation) |
| UPOS / XPOS / Lemmas | the column matches |
| UFeats | the **universal** features match (language-specific ones such as `Clitic` are ignored; order is ignored) |
| AllTags | UPOS, XPOS and UFeats all match |
| UAS | its head is the aligned gold head |
| LAS | UAS, and the relation matches **without its subtype** (`obl:arg` = `obl`) |
| CLAS | like LAS, over content words only (precision and recall differ) |
| MLAS | CLAS, plus UPOS, UFeats and the same functional dependents |
| BLEX | CLAS, plus the lemma |

`--strict` adds `UFeatsAll`, `AllTagsAll` and `LASFull`, which count all features
and the full relation. They are lower where a treebank uses subtypes and
language-specific features (Coptic UFeats 95.8 → 95.0, Arabic LAS 81.2 → 80.7).
Numbers in older notes were strict.

`--official PATH` also runs the official script itself (download
<https://universaldependencies.org/conll18/conll18_ud_eval.py>) on the sentences
that were predicted. The official script refuses a tree with a missing HEAD,
several roots or a cycle, so such sentences are listed (`would refuse N sentences`)
and repaired for the official run only (missing heads become the root, extra
roots and cycles are attached to the first root as `dep`). Those repairs are
wrong heads, so the scores are not flattered. A difference to the official
script can only come from whole-file alignment when surface tokens differ at a
sentence boundary; with the tokens of the input it does not occur.

**Alignment.** Surface tokens are aligned by their character span in the
running text. Words inside a multiword token are then matched by form, in
order. A prediction that leaves *del* unsplit therefore loses the words *de*
and *el*, but everything after it still aligns.

Each metric reports **precision** (over predicted words), **recall** (over
gold words) and **F1**. With identical segmentation, all three equal accuracy.
Gold sentences missing from the predictions count against recall, and are
listed.

Because each cell carries a confidence, you can also check calibration. Cells
the scorer rates highly should be right more often; in practice, the lowest
scores pick out the cells most worth checking by hand.