This is the public record of changes to Scholar Sidekick’s deterministic pipelines. Two version tags let a client pin behaviour over time and detect drift:
x-scholar-transform-version - the normalisation, formatting, and resolver chain. Mirrored in /.well-known/sources.json.x-scholar-verify-version - the citation-verifier pipeline (claimed-vs-resolved comparison, thresholds, similarity primitives). Emitted on /api/verify routes only.A version tag is bumped only when a change could alter byte-identical output (transform) or a verdict (verify) for the same input. Each entry below is tagged:
For agents: record x-scholar-transform-version (and x-scholar-verify-version for /api/verify calls) alongside any cached output. If either value changes, re-run cached comparisons or re-baseline your expected outputs.
transform_version 2026-06-16Two builtin-formatter author-rendering bugs are fixed. APA no longer emits a doubled period when the final author ends in an initial (e.g. Aspelmeyer, M.. → Aspelmeyer, M.) — which affected most multi-author APA article citations — and no longer prepends a stray period in the degenerate title-only case. CSE now reduces a single-token given name to one initial instead of upper-casing it in full (e.g. Aspelmeyer MARKUS. → Aspelmeyer M.); already initials-shaped given names like RS are kept as-is. Output-affecting for anyone who pinned APA or CSE plain-text bytes — re-baseline. vancouver, ama, ieee, and all CSL output are unaffected.
* markers in plain-text outputtransform_version 2026-06-11The builtin citation styles (vancouver, ama, apa, ieee) previously emitted Markdown italic markers like *Journal* in their plain-text output, which leaked into the text/output fields of /api/format, /api/format-items, and /api/format/stream (and so into the MCP server, CLI, and extensions). Those fields are now clean plain text; italic intent is carried in a new html field (and renderHtml/outputHtml on the stream) as sanitized <em> markup. Output-affecting for anyone who pinned builtin plain-text bytes - re-baseline. cse and all CSL output are unaffected (they never contained markers). The /api/export plain-text (txt) export and the homepage “Copy as Plain Text” are likewise asterisk-free.
?mode=strict)no version bumpThe resolver’s fixed-order chain now has an explicit, assertable name: ?mode=strict sets x-scholar-resolver-mode: strict and (under ?provenance=1) adds _provenance.mode + selection_reason. Strict means deterministic chain ordering within the currently-deployed transform_version - it is not a version pin or a historical snapshot (see /engineering-principles “What it does not promise”). A best-effort mode (freshest-source selection) is reserved and not yet available; any value other than strict resolves to strict and the header reports strict. Behaviour and output are unchanged - this only names what already happened.
no version bumpThe resolver cache previously used one flat TTL for every record. It is now calibrated by which upstreamproduced the record (fast-cadence registries like Crossref/DataCite cache shorter than stable ones like doi.org or PubMed) and by the cited work’s publication recency - a current-year paper (most likely to pick up a correction) gets a short TTL, a decades-old one is cached far longer. Net effect: fresher data on recent works, fewer wasted upstream calls on stable old ones. The chosen band is surfaced as _provenance.sources[].ttl_band (recent / mid / frozen) under ?provenance=1. This changes cache lifetime only, never output bytes, so transform_version is unchanged; CACHE_TTL_CALIBRATION=0 reverts to the flat TTL.
_provenance, cached checks, faster arXiv checksno version bumpRetraction + open-access checks (opt-in via ?checks=retraction,oa) gain three refinements. (1) Results are now served from a two-tier cache (memory + KV) with their own TTLs (retraction long, open-access short), independent of transform_version - fewer upstream calls, faster repeat lookups. (2) When ?provenance=1 and ?checks= are both set, the _provenance object carries a new trust_signals summary (per signal: requested / evaluated / status, or skipped_reason: "no_doi"). (3) The open-access check now exposes a direct pdf_url (Unpaywall url_for_pdf). Separately, /api/retraction-check, /api/oa-check, and the ?checks= path now resolve an arXiv input via its deterministic DataCite DOI (10.48550/arXiv.*) instead of the slow arXiv API - removing a recurring timeout. All additive and opt-in; formatted output is byte-identical, so transform_version is unchanged. Retraction/OA remain opt-in (not baked into every resolve).
_provenance (Auditor V2)no version bumpThe opt-in ?provenance=1 object now carries several additive fields beyond transform_version / sources / resolved_at: request_id; fallbacks_tried (resolvers consulted before the winner, omitted when the primary resolved on the first try); per-source cache state (hit/miss/bypass) and upstream_fetched_at; fields_from_source / fields_absent; and normalization steps applied to your input. A new ?provenance=full mode (acknowledged via x-scholar-provenance: full) adds a full_metadata block (funders, ORCID iDs, ROR iDs, license, clinical-trial IDs). Every field is optional and opt-in; default payloads and formatted output are byte-identical, so transform_version is unchanged. Cross-source conflicts are deferred to a later sub-phase. Full shape: /.well-known/sources.json → provenance_schema.
verify_version 2026-06-01 → 2026-06-05A citation whose author is written surname-first with a comma - Chukwuocha, Uchechukwu Madukaku, the form used by many reference styles and by Dublin-Core dc:creator fields (e.g. WHO IRIS records) - is now compared on the surname before the comma. Previously the given names after the comma were folded into the surname comparison, so when the resolved registry record split the same name differently (Crossref stores some names as family Madukaku Chukwuocha) the author scored just below the match threshold and a correct citation was flagged as a high-confidence mismatch. The verifier now reads the comma as the surname boundary and matches the author. The change is additive and recall-safe: a genuinely wrong author before the comma still does not match, so fabricated-author detection is unaffected. transform_version is unchanged (resolver and formatter output bytes are identical).
transform_version 2026-05-26 → 2026-06-05A work published online-first and later in print carries two publication years, often 1–2 years apart (common at Springer, Elsevier, and other publishers). A citation that used the print year was previously scored against the online-firstyear and flagged with a low-confidence year mismatch - even when the title and author matched exactly. The DOI resolvers now carry the alternate year(s) from Crossref’s published-print / published-online, and the verifier accepts a claimed year matching any of them. A genuinely wrong year - matching neither the online nor the print year - still lowers confidence exactly as before, so erroneous-year detection is unchanged. transform_version is bumped because the resolver record now carries the alternate years (read by the verifier) and to cold-invalidate the resolver cache; the formatted citation output is byte-identical (formatters use the primary issued year only).
Old Wiley/SICI DOIs carry interior angle brackets - e.g. 10.1002/1521-4141(200205)32:5<1223::AID-IMMU1223>3.0.CO;2-5. The identifier detector stopped at the first <, resolving a truncated, non-existent DOI → not_found. The detector now accepts </> inside a DOI body, so these DOIs resolve to the correct record and verify as matched. HTML-adjacent bare DOIs (…/foo</p>) are unaffected - a SICI bracket is followed by a digit, an HTML tag by a letter, so trailing tags are still stripped. Rides the same-day transform_version / verify_version = 2026-06-05.
arXiv identifiers were resolved via export.arxiv.org (the Atom API), which is rate-limited (~1 req/3s per IP) and frequently returns 429/503 → a transient 502 on cold ids. They now resolve via the arXiv DataCite DOI (10.48550/arXiv.{id}) first - a robust, non-rate-limited registry that carries the same arXiv preprint title, the submission date, the authors, and the published-journal DOI link. The Atom API remains as a fallback for the rare id absent from DataCite or during a DataCite outage, and arXiv results are now cached like other identifiers. Rides the same-day transform_version / verify_version = 2026-06-05.
Some publishers (e.g. Dove Press) return titles wrapped in entity-encoded tags - <p>…</p>. The verifier strips raw HTML tags (<i>, <sup>) before comparison, but the entity-encoded form collapsed the brackets to spaces and leaked the tag-name letter (p) as a spurious token, dragging title similarity below the match threshold (a false mismatch). The angle-bracket entities </> are now decoded before the tag strip, so entity-encoded tags are removed cleanly. Rides the same-day verify_version = 2026-06-05.
A citing author often writes a Greek letter out in full (Calcineurin Aalpha, IFN-gamma) where the canonical title carries the symbol (Calcineurin Aα, IFN-γ). The symbol survived normalisation as a lone non-Latin token and dragged title similarity below the match threshold (a false mismatch). Greek letters are now transliterated to their standard Latin romanisation (α→alpha, β→beta, γ→gamma, …) on both sides before comparison, so the spelled-out and symbol forms compare like-for-like. Folding is recall-safe - it can only raise similarity when one side spelled the letter out; a fabricated title is not rescued by folding a single letter. Rides the same-day verify_version = 2026-06-05.
Two title-normalisation gaps found in a coverage sweep: a resolved title carrying a character as a numeric HTML entity (IFN-γ, γ) had the entity dropped to whitespace instead of decoded, so an entity-encoded Greek letter or accent never reached transliteration; and Latin-extended letters with no Unicode decomposition (ø, ł, æ, œ, ð, þ, ß) survived as non-ASCII tokens that dragged title similarity below the match threshold (worst in a leading word - Łukasiewicz, Sørensen). Numeric entities are now decoded to their character, and those letters are folded to their conventional ASCII romanisation (ø→o, æ→ae, ß→ss, …). Both are recall-safe - they only raise similarity when the two sides spell a character differently. Rides the same-day verify_version = 2026-06-05.
not_found / 404 instead of a server errorverify_version 2026-05-26 → 2026-06-01A DOI that no registry can resolve - a fabricated or unregistered identifier, or a real DOI whose registrant returns a body the registration agencies cannot serve as JSON (e.g. some non-Crossref prefixes) - is now classified as a definitive not-found rather than an upstream error. On /api/verify such an identifier now yields a not_found (or ambiguous when a title search finds the real work under a different identifier) verdict instead of a 502 RESOLVE_ERROR - the expected answer for a fabrication check. On /api/format, /api/export, and /api/format-items, a request whose every identifier is unresolvable now returns a clean 404 NOT_FOUND instead of a 500/502. Genuine upstream outages (timeout / 5xx / rate-limit / network) are unchanged - they still return 502/504 so a client can retry. Note: a not_found verdict means “no usable record was found”, which is not by itself proof of fabrication; the verifier runs a title search to separate a fabricated work from a real one cited under a bad identifier. transform_version is unchanged (formatted output bytes are identical).
codeAll 4xx/5xx responses from the API now include a stable code field in the JSON body and mirror it in the x-error-code response header. Previously the field was present only on classified errors (auth, upstream, verifier, maintenance); validation, body-parse, unknown-style, and payload-size errors omitted it. Integrators can now branch on a stable token without parsing the human-readable message. New tokens introduced: VALIDATION_ERROR, UNKNOWN_STYLE, INVALID_CONTENT_TYPE, EMPTY_BODY, MALFORMED_JSON, and PAYLOAD_TOO_LARGE. The OpenAPI ErrorResponse schema now requires code. Formatted citation output is unchanged, so transform_version is not bumped.
When a work resolves with only an arXiv identifier (no DOI), the CSV, EndNote (Refer and XML), RefWorks, Zotero RDF, and MEDLINE/NBIB exporters now emit an arXiv carrier so the record imports cleanly into reference managers. The five URL-based formats add https://arxiv.org/abs/<id> as the URL when no explicit URL or DOI is present; MEDLINE emits AID - arXiv:<id> [arxiv] per NLM convention. This matches the behaviour already shipped for CSL-JSON, BibTeX, and RIS. Output for items that already carried a DOI or URL is byte-identical, so transform_version is not bumped.
verify_version: 2026-05-25 → 2026-05-26Two further false-accusation fixes for correctly-cited papers whose first-author field is stored in reference-list shorthand:
“P Giral”, “MJ Fila”, “Kristine Sørensen”), or a bare surname against an unsplit resolved full name (“kim” vs “Hongryel Kim”) - both common in real reference lists - now match. The claimed surname is recognised as a leading-or-trailing token run of the resolved name in either order. The check is additive and recall-safe: an unrelated first author is neither the leading nor the trailing run, and the title comparison gates fabrications regardless.“Collaborators” (e.g. the GBD “… Collaborators” consortium form) are now recognised as group authors, alongside the existing investigators / group / consortium / collaboration markers.Verdicts are deterministic at verify_version=2026-05-26 going forward. If you pinned verify output against the previous baseline, re-baseline against the current version.
transform_version: 2026-05-25 → 2026-05-26ISBN resolution via Google Books. The ISBN resolver consults Open Library first and Google Books on miss. The Google Books fallback is now authenticated and scoped to country=US; previously the keyless request was quota-limited from server egress and silently failed, so an ISBN that Open Library does not hold could come back unresolved. Affected ISBNs now resolve. The API key is never included in returned source links or error messages.
Output is deterministic at transform_version=2026-05-26 going forward.
verify_version: 2026-05-24 → 2026-05-25Further verifier-accuracy fixes for correctly-cited papers, the retirement of the group-author confidence cap, and a new point-of-use caveats field:
“NETTER-1 trial investigators. Phase 3 trial of…” or “…on behalf of the NETTER-1 trial investigators”) is no longer false-flagged - the group affix is stripped before the title comparison. The strip is additive (scored alongside the unstripped title, best score wins), so it can only rescue a correct citation, never let a fabricated title through.PMID now matches, paired with the resolver change below.“Title. A study of…” or “Title: A study of…”) now matches when the source structures the subtitle separately from the title (Crossref): the verifier recombines the canonical title and subtitle to compare, rather than scoring against the title alone.matched at the confidence the rest of the comparison earns, with a caveat (below) noting the group author itself could not be cross-checked. This removes an asymmetry that keyed off a registry storage quirk._provenance.caveats - a list of deterministic limitations of the verdict (resolved via title-search, an ISSN-only container identifier, a non-Latin-script title, a publication-year gap, or an unconfirmed group author). Additive metadata, omitted entirely on a clean high-confidence match.Verdicts are deterministic at verify_version=2026-05-25 going forward. If you pinned verify output against the previous baseline, re-baseline against the current version.
transform_version: 2026-05-22 → 2026-05-25PubMed collective authors. A paper whose PubMed record stores a collaboration/group author (ESummary authtype: “CollectiveName”, e.g. RECOVERY Collaborative Group) now renders the group name intact in formatted citations and exports, instead of mangling it into a personal name ({family:"RECOVERY", given:"Collaborative Group"}). Affects records resolved via PMID.
Output is deterministic at transform_version=2026-05-25 going forward.
verify_version: 2026-05-22 → 2026-05-24Reduced false accusations on correctly-cited papers whose author or identifier metadata is messy at the source. Crossref and PubMed reference lists routinely store authorship in ways that previously tripped the verifier; five fixes:
“Sørensen Kristine”, “Borchetia S., Das S.C., …”) now matches via the leading surname run when the resolved surname is a token-prefix.{family:"Philip R.", given:"Cohen"}) are now recognised when the claimed surname matches the resolved given name.ambiguous; ambiguous is reserved for a genuinely different work matching the claim.RECOVERY Collaborative Group, PIOPED II Investigators) is no longer false-flagged. When the record carries the group it matches directly (after normalising a leading The); when the record lists only the individual authors, the work is confirmed by title + identifier and returned as matched at medium confidence - signalling that the group author itself could not be cross-checked.Verdicts are deterministic at verify_version=2026-05-24 going forward. If you pinned verify output against the previous baseline, re-baseline against the current version.
transform_version: 2026-05-04 → 2026-05-22Re-baselined to reflect resolver and normalisation refinements shipped since the previous baseline. Output for affected identifiers may differ from the 2026-05-04 baseline:
“Family, Given”) explicitly, fixing a family/given swap and a trailing-comma artifact for arXiv items.<i>, &) is stripped from titles and container titles for similarity comparison and clean display.Output is deterministic at transform_version=2026-05-22 going forward. If you pinned expected output against the previous baseline, re-baseline against the current version.
verify_version: 2026-05-11 → 2026-05-22not_found with skipped_reason: “insufficient_claim” instead of a misleading high-confidence verdict.transform_version)First published transform_version. Established the byte-identical-output guarantee, the unconditional x-scholar-transform-version response header, the /.well-known/sources.json resolver-chain manifest, and the copy-paste verification kit.
transform_version is bumpedtransform_version