Introduction
Today we are releasing the SquarePact Formatting Benchmark, a public dataset for evaluating how well AI tools edit real Word documents without damaging their formatting. To our knowledge, it is the first widely available benchmark of its kind.
The last few years brought real advances in legal and professional AI, and a wave of benchmarks to measure it. Most score reasoning, retrieval, drafting, or citation accuracy. Almost none ask the question that decides whether a tool can be trusted on a live document: can it make the change you asked for without breaking everything around it?
Ask a general-purpose assistant to revise one clause and the output often arrives with a broken numbered list, a font that drifted, a heading that lost its style, or track-changes that no longer make sense. For the documents SquarePact is built for, such as contracts, proposals, reports, and policies, this is not cosmetic. It is the difference between shipping a document and re-doing the work by hand.
We built this benchmark to measure that capability directly, and we are publishing the dataset so anyone can inspect it, reproduce our procedure, and test their own tools against it.
TL;DR / FAQ
In short: we are releasing the first public dataset for formatting-preserving AI edits in Word, comparing four Word add-ins. It keeps only the cases where tools measurably differ, is free to download with attribution, and is versioned over time. Scores live in Section 8.
It is produced by SquarePact (Actualization AI), a Tampa, FL company spun out of the AI research lab at the University of South Florida. It is led by Dr. John Licato, a professor of AI with more than 100 peer-reviewed publications and NSF SBIR research funding in language models and document reasoning, and Dr. Animesh Nighojkar.
They typically test a different capability (reasoning or drafting rather than formatting-preserving edits), are not transparent, or are not adaptive. We unpack all three in Why formatting is the missing benchmark, and the exact measures we score instead are in What we test.
Several reasons. Some vendors' terms of use prohibit competitors from using their software purely to benchmark it. We did not have access to many of the tools. Licensing every product solely to test it is also very expensive. We judged the more respectful path to be publishing the benchmark openly, see Access & license, and inviting those teams to run it against their own systems using our procedure.
We expect it, and we plan for it: the set is dated and versioned, and we add to it regularly. The full policy is in Versioning & maintenance, and it follows directly from the difference-only criterion.
Through full transparency. The dataset is downloadable so you can run it yourself, the methodology is public and reproducible, the measures are defined explicitly, and every number in the results is dated and tied to a build. Anyone who doubts the findings can reproduce them with the data we provide.
Permissive. Use it for research or product work; we only ask that you give credit and refer people back to this page. Details are in Access & license.
Why formatting is the missing benchmark
Formatting-preserving editing is under-measured for three distinct reasons. Existing resources tend to fail on at least one of them.
Capability mismatch
Most benchmarks reward answer quality or drafting fluency. They rarely test whether an edit preserves the structure and formatting of a document that already exists, which is exactly where professional tools tend to break.
Lack of transparency
The few datasets that do touch this capability are not openly available. They cannot be downloaded, inspected, or independently verified, so claims about them cannot be checked.
No adaptivity
Static benchmarks go stale as soon as tools optimize against them, and nothing keeps pace as the underlying systems improve. A benchmark for a moving target has to move too.
The dataset described here is built to address all three: it is public, inspectable, and versioned over time.
Systems evaluated
We evaluate four add-ins that operate inside Microsoft Word, where the editing work actually happens. Each receives an identical edit instruction on an identical document portion.
- SquarePact Ours
- Microsoft Copilot
- Claude for Word
- Grok for Word
What we test
Each edited portion is scored on five formatting-preservation measures. Every measure targets a concrete way a document degrades when an edit is applied carelessly.
| Measure | What it checks | Example failure |
|---|---|---|
| Character & paragraph formatting | Run- and paragraph-level properties survive the edit. | Pasted text drops to Calibri 11 mid-paragraph. |
| Styles & headings | Named styles and heading levels persist. | A "Heading 2" silently becomes body text. |
| Tables & list structure | Cell boundaries and list nesting hold. | A three-level list collapses to a single level. |
| Numbering & cross-references | Automatic numbering and references stay valid. | "See Section 4" now points to the wrong clause. |
| Track changes & comments | Revision marks and comments survive the edit. | An accepted change wipes an open comment thread. |
How we built it
The construction procedure is deliberately simple so that it can be reproduced. Some specifics are being finalized for the v1.0 write-up and are marked below.
-
Source corpus
We begin with a subset of the SuperDoc
docx-corpus, an openly available collection of real Word documents. See github.com/superdoc-dev/docx-corpus. -
Random sampling Details to follow
From that corpus we draw a random subset of documents, avoiding selection that would favor any particular system.
-
Portion selection Details to follow
Within each sampled document we select specific portions to edit, isolating the formatting features under test.
-
Standard tests Details to follow
We apply a standard set of editing tests to each portion, run identically across every system in the comparison.
-
Scoring & reporting
Each portion is scored on the five measures in Section 5, and results are reported in Section 8. Only portions that satisfy the criterion in Section 7 are retained.
The difference-only criterion
This benchmark does not aim to be exhaustive. Because its purpose is to improve tools, we retain a document portion only when the evaluated systems measurably differ on at least one measure. Portions on which every system behaves identically are excluded: they add size without adding signal.
Results
Table 3 reports per-measure scores for each system on the v1.0 set, with the composite as the mean across measures. Higher is better. Full figures publish with the dataset; each score is dated and tied to the build identifiers in the released metadata.
| Measure | SquarePact | Copilot | Claude (Word) | Grok (Word) |
|---|---|---|---|---|
| Character & paragraph formatting | — | — | — | — |
| Styles & headings | — | — | — | — |
| Tables & list structure | — | — | — | — |
| Numbering & cross-references | — | — | — | — |
| Track changes & comments | — | — | — | — |
| Composite | — | — | — | — |
Access & license
The release bundles the document portions, the tests applied to each, and the scored results, so anyone can replicate our numbers or evaluate a system we did not include.
- Contents
- Source .docx portions, per-portion test definitions, scored results, methodology write-up.
- Version
- 1.0 (July 2026)
- License
- Permissive. Free to use for research or product work; attribution and a link back to this page are required.
- Corpus
- Subset of the SuperDoc docx-corpus
Download v1.0Available on release.
Built a tool you would like measured against this set? The full procedure is public, so you can run it yourself, or see what the dataset powers and try SquarePact free in Word.
Versioning & maintenance
We expect systems to improve against this benchmark over time, including by optimizing directly for it. That is a sign the benchmark is doing its job. To keep it meaningful, every test is dated, results are tied to a version, and new cases are added on a regular cadence. Reported numbers always reference a specific version and date.
Citation
If you use this benchmark, please cite it and link back to this page.
@techreport{squarepact_formatting_benchmark_2026,
title = {The SquarePact Formatting Benchmark},
author = {Licato, John and Nighojkar, Animesh and Vaidya, Darsh and Sajithkumar, Govind},
year = {2026},
note = {Version 1.0},
institution = {Actualization AI, Inc.},
url = {https://squarepact.com/formattingBench/}
}