Results¶
Two held-out test sets, never used for tuning (the dev subsets were chosen first, stratified by length with a fixed seed, and the prompt, guidelines and columns were chosen on those). Pipeline: rules, an Opus 5.5 aligner and a Sonnet 5.5 scorer for 2 rounds, cold start (no evidence sets), the corpus’s own guidelines. One run per configuration.
Test set |
Gold |
Guidelines |
Pairs |
Links (pipeline / gold) |
F1 (all links) |
AER |
Cost |
|---|---|---|---|---|---|---|---|
BICROSS (New Testament manuscripts, Greek-Latin) |
2 human annotators |
|
79 verses |
1,412 / 1,371 and 1,404 |
0.947 and 0.975 |
0.053 and 0.025 |
$2.64 |
DFHG (Greek fragments, Latin translation) |
merged human gold |
|
80 fragments |
1,215 / 1,284 |
0.922 |
0.042 |
$2.34 |
Both runs: 86-88 requests, no failed tasks, no retries, no tool calls; about $0.03 per pair.
BICROSS against the two annotators¶
The two annotators agree with each other at F1 0.946 (AER 0.054) on these 79 verses.
Compared with |
Sure P / R / F1 |
All links P / R / F1 |
AER |
|---|---|---|---|
Kreinecker (1,371 links) |
0.956 / 0.940 / 0.948 |
0.933 / 0.961 / 0.947 |
0.053 |
Tilke (1,404 links) |
0.984 / 0.946 / 0.965 |
0.972 / 0.978 / 0.975 |
0.025 |
Consensus (links of either annotator) |
0.944 / 0.970 / 0.956 |
0.986 / 0.952 / 0.969 |
0.013 |
Human against human |
0.946 |
0.054 |
The pipeline is at the level of human agreement (equal to it against Kreinecker, above it against Tilke, who links a little more, as the pipeline does). The consensus is a lenient reference, so compare the per-annotator rows with the human agreement.
DFHG¶
All links P/R/F1 0.948 / 0.897 / 0.922, AER 0.042; sure links 0.967 / 0.877 / 0.920. The annotators agree at a Dice of about 0.927 (from the counts in the gold README, over all 100 fragments, not only these 80). The remaining errors are verbs and participles linked to relatives or subordinators, prepositions the Latin renders by case, and adverbs.
What matters, and what does not¶
The guidelines. On the BICROSS dev verses the same pipeline scored F1 0.905 with the DFHG-style guidelines and 0.978 with the BICROSS ones: the annotators align the Greek article with its noun, DFHG leaves it unaligned. Give the agents the rules your annotators followed (
--guidelines).The parse columns. Words only, words + lemma + UPOS, and the default columns with all tools differ by at most 0.005 F1 on both test sets; words only is about 35% cheaper.
Prompt shows |
DFHG F1 / cost |
BICROSS F1 (Kreinecker / Tilke) |
BICROSS cost |
|---|---|---|---|
words only |
0.927 / $1.55 |
0.948 / 0.974 |
$1.76 |
+ lemma, UPOS |
0.922 / $1.88 |
0.946 / 0.973 |
$1.95 |
+ head, deprel, all tools (default) |
0.922 / $2.34 |
0.946 / 0.975 |
$2.64 |
Caveats¶
One run each; sampling noise on 79-80 pairs is about ±0.015 F1.
The guidelines were chosen on dev after seeing errors there. The DFHG test errors have been seen: tune on dev only.
The evidence path (lemma hints from accumulated links) and lower-resource languages are not tested here. The Greek-Arabic, Greek-Coptic, Greek-Syriac, Latin-Arabic, Syriac-Arabic and Gothic-Latin guidelines are unvalidated drafts: no human gold alignment exists for them yet.