Imagine checking twelve pieces of text before using their labels. Two files say their reviewers agree completely. That sounds reassuring until you ask how many pieces both files actually labelled. If it was only one, eleven records are still outside that agreement calculation. This is the setup of the bundled, deliberately constructed partial-agreement example, not a result from real annotators.
Annotation means attaching a label to a record. Before labels can support an experiment, it helps to see where they came from, which records were left unfinished, and where two sets disagree. I built this desk to put those questions beside the headline percentage. Both checkpoints must contain the same complete inventory; the tool will not quietly compare only their overlapping records.
The twelve-record demo makes the different gaps visible. Four records have labels from both roles, and two of those agree: 50% conditional agreement with 4/12 both-labelled coverage. Across the inventory, disagreements, explicit abstentions and work not yet done stay visible. Those states stay separate, and all twelve appear in the comparison matrix.
Even an agreement starts unresolved in the manual review. A reviewer must record a separate decision; the original A and B annotations remain in the saved file. This avoids silently turning two matching assertions into a final label. It does not establish that the reviewer is right. A and B are file roles, and may even come from the same person.
The other practical problem is keeping the work. This app has no autosave: refreshing loses its memory. The intended handoff is explicit—record a decision, download the complete snapshot, then preview and verify it when restoring. The file includes source text and notes, so it needs the same care as the material you reviewed. The exact counts, recovery workflow and limitations follow below.
I built this tool to compare two complete annotation checkpoints and record a separate manual review. Open both Local Label Queue 0.1 checkpoints, inspect the disagreement, then decide one record at a time. Original role annotations stay in the exported review.
The static app runs in browser memory. It loads its code and, when requested, two fixed synthetic demo files. There is no model, account, server submission, local storage or autosave. A downloaded JSON snapshot contains the complete source texts, both annotations and review notes. Review the contents before sharing.
Use a recent browser with ES modules, File/Blob support and Web Crypto. HTTPS or localhost is required for SHA-256.
python3 -m http.server 8000 --bind 127.0.0.1
# Open http://127.0.0.1:8000 in a browser.
Preparing a download captures recorded decisions at that revision. Form edits have to be recorded first. The page cannot confirm where a download was saved. Notes are preserved exactly, including whitespace; they are limited to 2,000 UTF-16 units with valid Unicode scalar values.
Both files must use exactly the same complete inventory, taxonomy and order. Pairing uses the original inventory IDs after each checkpoint is validated. A missing record, altered text, different group, taxonomy change or order mismatch rejects the pair. No intersection-only subset replaces the complete inventory.
Each record belongs to one bucket:
| Bucket | Meaning |
|---|---|
| same_label | Both labelled, same label |
| different_label | Both labelled, different labels |
| both_abstained | Both explicitly abstained |
| one_abstained | Both reviewed: one labelled, one abstained |
| both_unlabelled | Neither reviewed |
| one_unlabelled | Exactly one unlabelled; the other labelled or abstained |
Conditional label agreement is same_label / (same_label + different_label). Its denominator excludes abstained and unlabelled records and is displayed beside the total inventory and both-labelled coverage. Zero both-labelled records gives N/A (null in JSON). The matrix includes every record, with distinct positional states for each label, abstained and unlabelled. A literal label named abstained cannot collide with the status.
The scripted 12-record demo has two records in every bucket. Only 4/12 are labelled by both roles, and 2/4 share a label: 50% conditional agreement. A separate partial-agreement-*.json example has 1/12 both labelled and 100% agreement. no-labels-*.json has a zero label-agreement denominator. These are manufactured counting examples, not annotation measurements or gold labels.
Final decisions always start unresolved. Agreement is never copied into a final label automatically. A reviewer can choose any taxonomy label, abstain, or reopen. Each actual change increments a monotonic revision; no-op actions do not. Undo restores one previous decision while incrementing the revision. Notes and decisions are assertions supplied by the operator.
Export keeps both canonical input checkpoints, source checksums, complete matrix and per-ID comparison, all decisions and a review checksum. Import recomputes the source checksums and comparison before verifying the review checksum. Swapping source roles changes the review binding. A saved review is self-contained and intentionally restores its own source pair after preview.
SHA-256 detects inconsistency with declared content; it does not authenticate an annotator or protect against someone changing a file and recalculating its hashes. Roles A and B may come from the same person, or even the same checkpoint. Identical checkpoint content gets a warning. Matching labels do not establish accuracy, semantic validity, independence or readiness for training. There is no chance-correction statistic, significance test, consensus algorithm or adjudication-quality score.
Inputs use JSON.parse last-key semantics for duplicate object keys. This is not a duplicate-key detector. Exact schemas and canonical field order are then applied; annotation and decision arrays normalize to inventory order. BOM, invalid UTF-8, invalid scalar strings, unsupported versions, extra fields, missing/foreign/duplicate IDs, altered summaries and inconsistent hashes reject. Checkpoint limits are 4 MiB each; review import and the exact pretty-printed export are limited to 12 MiB (oversized snapshots are rejected; shorten notes or start with a smaller inventory); inventory limits and taxonomy follow the vendored Local Label Queue 0.1 reader (1–1,000 records,2–9 labels). These limits do not promise instantaneous performance at every maximum combined.
No package installation is needed for Node 24 and Python 3.14 used during development. Code does not pin or promise every older runtime.
node test_review.mjs
PYTHONDONTWRITEBYTECODE=1 python3 verify_reference.py
node cli.mjs demo-a.json demo-b.json NEW-review.json
The CLI creates an unresolved review with exclusive new-file creation; it does not overwrite an existing file. The core checks cover all 25 combinations of three labels and the two other annotation states, 75 decision roundtrips, malformed and rehashed files, canonical order, manual initialization, no-op and undo behavior. The separately written Python reference recounts 33 inventories (including 1,000 records) and 5,280 actual state transitions, including retained notes and revisions.
Real browser acceptance is recorded outside the release candidate. It checks native downloads and file-picker restoration rather than a simulated file-input setter. Local acceptance does not establish deployment on Hugging Face or real user adoption.
准备使用十二条文字的标签时，你看到两份标注“100%一致”。先别急着放心：如果双方只给同一条记录打过标签，另外十一条还没有进入这个一致率的分母。工具里的partial-agreement演示正是这样构造的，不是真人标注研究的成绩。
标注就是给记录附上类别。这个工具把两份同批资料的标注摆在一起，让你同时看到整批有多少记录、双方都打过标签的有多少，以及这个子集中的一致率。两份文件必须包含同一份完整清单；它不会悄悄只取有交集的记录来比较。
默认十二条合成样例中，双方都给出标签的有四条，其中两条相同，所以条件一致率是50%，双方有标签的覆盖率是4/12。双方标签不同、主动弃权和还没做完的状态分别保留，全部十二条都进入比较矩阵。
人工复核从每条“未解决”开始，双方相同也不自动变成最终标签。你要单独记录裁定，原始两份标注仍保存在导出文件里。裁定也可能有错；A/B只是文件角色，不证明来自两个独立的人。
页面没有自动保存，刷新就会丢失内存。实际工作流是先记录裁定，再下载完整快照；恢复时先验证和预览，再明确应用。文件包含全部原文、标注和备注，分享前要检查。下面保留具体操作、完整计数和能力限制。
我用这件工具对照两份同批记录的标注，再单独记录人工复核结果。输入中的原始标签保留不改，每条裁定从“未解决”开始，双方相同也不会自动通过。
页面同时展示全体记录、双方都有标签的数量，以及这个子集的一致率。弃权和未完成分开计数：只标过一条且双方一致，也可能显示100%一致率，不能据此说整批标注可靠。A/B只是文件角色，不证明来自两个独立的人；例子全是合成演示。
保存要先记录当前裁定，再准备并下载完整JSON。刷新会清空内存；恢复时先验证、预览，再明确替换。错误文件保留当前工作。文件包含原文与备注，分享前检查内容；摘要用于一致性核对，不能证明签名或标签正确。
Code and original synthetic examples: Apache-2.0, by Renchao Wu. vendor/queue.mjs is an unchanged copy from the accepted Local Label Queue 0.1 candidate; exact source identity and SHA are in SOURCE.json. This review workflow is a separate work, work:annotation-pair-review:v1. No real human study, model fit, external dataset or adoption claim is attached to it. See NOTICE, LICENSE and CONTRACT.md.