paired-eval

它评什么What it evaluates

三个对象问的是三个不同的问题，各自固定不同的变量；混着评什么都得不到。The three objects ask three different questions and hold different things constant; mixing them up yields nothing.

怎么验：能用程序验的先验，验不过的再判How it verifies: programmatic checks first, judges only for the rest

有唯一真值 → exact 或程序判定；只有事实来源 → retrieval / trajectory（逐条 claim grounding）；只有质量标准 → rubric（自身要过 canary）。层可叠加：程序做 gate，rubric 做 score——验不过直接 0 分且不调用评委。A unique ground truth → exact or a programmatic check; only sources of fact → retrieval / trajectory (per-claim grounding); only a quality bar → rubric (which must pass a canary). Layers stack: the program gates, the rubric scores — failing the gate scores 0 and never calls the judge.

task = {"id": "sum-explained", "instruction": "12×12 等于多少, 并说明过程; 以「答案: 」开头给出结果",
        "verification": {"class": "gated",
                         "gate":  {"class": "exact", "gold": "144", "kind": "numeric"},
                         "score": {"class": "rubric", "criteria": [{"text": "说明了计算过程", "weight": 1}]}}}
r = pe.evaluate(task, response=answer, llm=judge)
print(r["score"], r["verdict"])        # 答错 -> 0.0 'gated_out'(judge 一次都没调用); 答对 -> rubric 分

task = {"id": "sum-explained", "instruction": "What is 12×12? Explain, and give the result starting with 'Answer: '",
        "verification": {"class": "gated",
                         "gate":  {"class": "exact", "gold": "144", "kind": "numeric", "marker": "Answer"},
                         "score": {"class": "rubric", "criteria": [{"text": "explains the calculation", "weight": 1}]}}}
r = pe.evaluate(task, response=answer, llm=judge)
print(r["score"], r["verdict"])        # wrong -> 0.0 'gated_out' (judge never called); right -> the rubric score

十秒钟看效果（统计层的离线演示）Ten seconds (an offline demo of the statistics layer)

这里的 strict / bare 是两个桩函数，不是模型——它们复刻一种真实观察到的失败机理（裸指令下正确答案被包进 markdown 围栏），只为在没有 API 时展示报告的形状。输出是真的（由测试与当前代码逐行比对）。strict / bare here are two stub functions, not models — they replay one real failure mechanism (a bare prompt wraps the correct JSON in markdown fences), only to show the shape of a report without any API. The output is real (a test keeps it identical to the current code).

$ python3 -m paired_eval
有效样本: 2/4 题有信息(恒过 2, 恒败 0)
拒答: {'strict': 0, 'bare': 0}
触顶(1.000): strict —— 该系统无余量, 它作为参照时效应会被压缩
bare vs strict: Δ=-0.500 CI95=[-1.000,+0.000] | 逐题p=0.506 逐轮McNemar=0.00781 Holm=0.00781 | 不一致对 0:8 集中度=0.50 | 显著: Δ=-0.500 CI95=[-1.000,+0.000] p=0.0078 (n=16)

$ python3 -m paired_eval --lang en
informative sample: 2/4 tasks informative (always-pass 2, always-fail 0)
refusals: {'strict': 0, 'bare': 0}
at ceiling (1.000): strict — no headroom; effects measured against it as the reference are compressed
bare vs strict: Δ=-0.500 CI95=[-1.000,+0.000] | per-task p=0.506 per-round McNemar=0.00781 Holm=0.00781 | discordant 0:8 concentration=0.50 | significant: Δ=-0.500 CI95=[-1.000,+0.000] p=0.0078 (n=16)

两题两系统全对、不携带信息（有效样本 2/4）；8 个不一致对全部偏向 strict，分布在 2 题上（集中度 0.50，非单题异常）；逐题置换 p=0.506 是因为只有 2 道有信息题，它的最小可能 p 就是 0.5——而逐轮 McNemar 用上全部 16 个配对单元。末段是可直接写进报告的结论。Two tasks were solved by both systems every time and carry no information (informative sample 2/4); all 8 discordant pairs favour strict, spread over 2 tasks (concentration 0.50); the per-task permutation p is 0.506 because with 2 informative tasks its minimum attainable p is 0.5 — per-round McNemar uses all 16 paired units. The last clause is a verdict you can paste into a report.

特性Features

四种结论，而不是一个 p 值Four verdicts, not one p-value

“p > 0.05” 有三种完全不同的含义，处方各异。interpret() 把每次比较翻译成其中一种：“p > 0.05” means three very different things with different remedies. interpret() translates every comparison into one of these:

用真实模型：三个对象各做一次 A/BWith real models: one A/B per object

任务按“有什么可验”混搭验证类；bench_tasks 把 judge 评分的题按显式阈值二值化，进同一套配对流水线；三个维度各固定各的变量。Mix verification classes by what can be verified; bench_tasks binarises judge-scored tasks at an explicit threshold so they flow through the same paired pipeline; each axis holds its own variable constant.

import paired_eval as pe
from examples.adapter_openai_compat import make_call, make_llm     # 只用标准库的 OpenAI 兼容适配器

my_tasks = [   # 按"有什么可验"混搭: 有真值 -> exact; 有真值又要看质量 -> gated; 只有资料 -> trajectory
    {"id": "date", "instruction": "把 2024年3月5日 写成 ISO 8601 日期; 以「答案: 」开头给出",
     "verification": {"class": "exact", "gold": "2024-03-05"}},
    {"id": "sum-explained", "instruction": "12×12 等于多少, 并说明过程; 以「答案: 」开头给出结果",
     "verification": {"class": "gated",
                      "gate":  {"class": "exact", "gold": "144", "kind": "numeric"},
                      "score": {"class": "rubric", "criteria": [{"text": "说明了计算过程", "weight": 1}]}}},
]
call_a, call_b, judge = make_call(model="model-a"), make_call(model="model-b"), make_llm(model="judge-model")
tasks = pe.bench_tasks(my_tasks, threshold=1.0, llm=judge)        # judge 评分的题也进同一套配对流水线

STRICT = "严格按要求输出, 不要任何多余内容。要求: "
runs = {
    "model":   pe.run_interleaved({"A": pe.make_model(call_a), "B": pe.make_model(call_b)}, tasks=tasks, n=6, prompt_prefix=STRICT),
    "harness": pe.run_interleaved({"strict": pe.make_model(lambda p: call_a(STRICT + p)), "bare": pe.make_model(call_a)}, tasks=tasks, n=6, prompt_prefix=""),
}
for axis, run in runs.items():
    print(axis, pe.report(run["reports"], refusals=run["refusals"])["text"], sep="\n")

import paired_eval as pe
from examples.adapter_openai_compat import make_call, make_llm     # standard-library-only OpenAI-compatible adapter

my_tasks = [   # mix by what can be verified: ground truth -> exact; truth + quality bar -> gated; only sources -> trajectory
    {"id": "date", "instruction": "Write 5 March 2024 as an ISO 8601 date, starting with 'Answer: '",
     "verification": {"class": "exact", "gold": "2024-03-05", "marker": "Answer"}},
    {"id": "sum-explained", "instruction": "What is 12×12? Explain, and give the result starting with 'Answer: '",
     "verification": {"class": "gated",
                      "gate":  {"class": "exact", "gold": "144", "kind": "numeric", "marker": "Answer"},
                      "score": {"class": "rubric", "criteria": [{"text": "explains the calculation", "weight": 1}]}}},
]
call_a, call_b, judge = make_call(model="model-a"), make_call(model="model-b"), make_llm(model="judge-model")
tasks = pe.bench_tasks(my_tasks, threshold=1.0, llm=judge)        # judge-scored tasks flow through the same paired pipeline

STRICT = "Follow the format exactly, no extra text. Task: "
runs = {
    "model":   pe.run_interleaved({"A": pe.make_model(call_a), "B": pe.make_model(call_b)}, tasks=tasks, n=6, prompt_prefix=STRICT),
    "harness": pe.run_interleaved({"strict": pe.make_model(lambda p: call_a(STRICT + p)), "bare": pe.make_model(call_a)}, tasks=tasks, n=6, prompt_prefix=""),
}
for axis, run in runs.items():
    print(axis, pe.report(run["reports"], refusals=run["refusals"], lang="en")["text"], sep="\n")

agent 维度同理：同一模型同一脚手架，比“单遍”与“出草稿后自检修正”。还能做：事前样本量规划（required_tasks / required_pairs）、筛出有区分力的题（screen_tasks / screen_graded）。完整示例见 README。The agent axis works the same way: same model, same scaffold, one pass vs draft-then-self-correct. Also: sample-size planning (required_tasks / required_pairs) and screening for tasks that discriminate (screen_tasks / screen_graded). Full example in the README.

与其他工具的关系How it relates to other tools

它们运行评测、产出逐题分数；paired-eval 接在下游，回答“这个差别可信吗、还差多少样本”。描述取自各项目自己的 README。They run evaluations and produce per-task scores; paired-eval sits downstream and answers “is this difference credible, and how many more samples would it take?”. Descriptions come from each project's own README.

文档Documentation