Field Notes — AI and engineering

The Chinese open-weights release timeline, dated

Fri, 09 Oct 2026 00:00:00 GMT

This is a list of dates, not an argument. Every row is a model whose weights were published for anyone to download, with a date I could trace back to the organization that published it, a parameter count as stated by that organization, and a link.

Most timelines of this kind are written from memory or from coverage, which is why they drift. I built this one from vendor pages, papers and Hugging Face’s API. This revision leaves the table at 67 rows: seven are new, one of them because an entry I dropped in the first pass turns out to be open-weights after all. The dropped list at the end has eleven entries, each with a reason.

How to read this timeline

Open weights only. A release is in the table only if the organization published downloadable weights. API-only and hosted-only models are out, including flagship ones: Qwen3-Max, the served ERNIE models, Kimi’s hosted variants, DeepSeek’s API-only models.

A date I could trace. The Date column has two kinds of value. A plain date is the date on the organization’s own dated page — a research blog post, a news post, a release page, an accepted paper. A date marked † is the Hugging Face repository creation timestamp in UTC, read from the Hugging Face API on 2026-10-09, used when the vendor had no dated page I could read. The two can differ by a few days, and that is usually preparation rather than a different release: DeepSeek-V3’s weights repository was created 2024-12-25 12:52 UTC and the announcement is dated 2024-12-26.

Parameter counts come from the vendor, or they are blank. Total parameters with activated parameters for mixture-of-experts models, as stated on the announcement, the model card, or the repository holding the same weights. A family that ships many sizes gets a range. Where none of the vendor’s pages for that release states a count, the cell is — rather than a number carried over from a sibling model, even when the lineage makes the number obvious.

One link per row, to the vendor or the weight repository. No aggregator links, no news coverage. That makes most rows single-source by construction: each cites the vendor’s own statement about its own release, and every † row is single-source by definition.

Scope. Language models and the flagship multimodal lines that carry the same brand and weights: Qwen3-VL, Qwen3-Omni, MiniMax-M3. Pure image, video and speech models are out of scope.

When the vendor’s page is client-rendered

Two of the labs whose releases I date most often publish announcements as a page that ships no text to a plain HTTP request. z.ai’s blog returns a 598-byte HTML shell that loads a JavaScript bundle; qwen.ai’s blog returns a shell that fetches its content after load. Both render fine in a browser, which is what a reader clicking the link sees.

For z.ai I did read something: the date string sits inside the page’s own bundle, so the bundle is where I checked GLM-4.5 (2025-07-28), GLM-4.6 (2025-09-30), GLM-4.7 (2025-12-22) and GLM-5 (2026-02-12). That is weaker evidence than a rendered page, and it is why GLM-5.2 stays out: its bundle contains 2026-06-16 and no other date, and a date visible only inside a build artifact is not one I will print as sourced.

Qwen’s older blog carries the full text and a date; the newer qwen.ai copies do not. So the three Qwen3-* rows dated from qwen.ai in the first pass now cite the weight repository and carry a †.

2023

Q1–Q2

Model	Date	Parameters	Source
ChatGLM-6B	2023-03-13 `†`	6.2B	THUDM/chatglm-6b
Baichuan-7B	2023-06-13 `†`	7B	baichuan-inc/Baichuan-7B
ChatGLM2-6B	2023-06-24 `†`	`—`	THUDM/chatglm2-6b

Q3–Q4

Model	Date	Parameters	Source
Qwen-7B	2023-08-03 `†`	7B	Qwen/Qwen-7B-Chat
Baichuan 2	2023-09-19	7B, 13B	arXiv:2309.10305
ChatGLM3-6B	2023-10-25 `†`	`—`	THUDM/chatglm3-6b
Yi-34B	2023-11-01 `†`	34B	01-ai/Yi-34B
DeepSeek Coder	2023-11-01 `†`	1.3B–33B	deepseek-ai/deepseek-coder-33b-instruct
DeepSeek LLM	2023-11-29 `†`	7B, 67B	deepseek-ai/DeepSeek-LLM-67B-Chat

Baichuan 2 is the only 2023 row dated by a paper: arXiv:2309.10305 is dated 2023-09-19, while its weight repositories appear earlier, on 2023-08-30 †, so the paper is the single source for that date. The two ChatGLM cells are blank because their cards do not state a count: 6.2 billion is on the ChatGLM-6B card and not on the cards of its two successors, and copying it across is the habit that makes timelines of this kind drift.

2024

Q1

Model	Date	Parameters	Source
Qwen1.5	2024-02-04	0.5B–110B	Qwen blog

Qwen1.5 was in the dropped list in the first pass, because I looked for it on the new blog and found nothing readable. The old blog still carries the post, dated 2024-02-04, listing six sizes plus 110B.

Q2–Q3

Model	Date	Parameters	Source
DeepSeek-V2	2024-05-07	236B total, 21B active	arXiv:2405.04434
GLM-4-9B	2024-06-04 `†`	9B	zai-org/GLM-4-9B-Chat
Qwen2	2024-06-07	0.5B–72B	Qwen blog
Qwen2-VL	2024-08-29	2B, 7B, 72B	Qwen blog
DeepSeek-V2.5	2024-09-05 `†`	`—`	deepseek-ai/DeepSeek-V2.5
Qwen2.5	2024-09-19	0.5B–72B	Qwen blog

Q4

Model	Date	Parameters	Source
Hunyuan-Large	2024-10-22 `†`	389B total, 52B active	tencent/Hunyuan-Large
Qwen2.5-Coder	2024-11-06 `†`	0.5B–32B	Qwen/Qwen2.5-Coder-32B-Instruct
DeepSeek-V2.5-1210	2024-12-10	`—`	DeepSeek changelog
DeepSeek-V3	2024-12-26	671B total, 37B active	DeepSeek news

The DeepSeek point releases inside a model line are dated on DeepSeek’s own changelog, one page per release. Hunyuan-Large’s 389B/52B comes from arXiv:2411.02265, submitted 2024-11-04, three weeks after the weights appeared.

2025

Q1

Model	Date	Parameters	Source
MiniMax-Text-01	2025-01-14	456B total, 45.9B active	arXiv:2501.08313
DeepSeek-R1	2025-01-20	671B total, 37B active	DeepSeek news
Qwen2.5-VL	2025-01-27 `†`	3B, 7B, 72B	Qwen/Qwen2.5-VL-72B-Instruct
QwQ-32B	2025-03-06	32B	Qwen blog
DeepSeek-V3-0324	2025-03-25	671B total, 37B active	DeepSeek changelog

The two DeepSeek updates here do not restate a parameter count on their own changelog pages; the cells carry the count from the repositories that hold those weights, which are the V3 and R1 repositories listed above.

Q2

Model	Date	Parameters	Source
Qwen3	2025-04-29	0.6B–235B	Qwen blog
DeepSeek-R1-0528	2025-05-28	671B total, 37B active	DeepSeek changelog
MiniCPM4-8B	2025-06-05 `†`	8B	openbmb/MiniCPM4-8B
MiniMax-M1	2025-06-16	456B total, 45.9B active	arXiv:2506.13585
Hunyuan-A13B	2025-06-27	80B total, 13B active	Tencent-Hunyuan/Hunyuan-A13B
ERNIE 4.5	2025-06-30	0.3B–300B; flagship 300B/47B	ERNIE blog

Q3

Model	Date	Parameters	Source
Kimi K2	2025-07-11 `†`	1T total, 32B active	moonshotai/Kimi-K2-Instruct
Qwen3-Coder	2025-07-22	480B total, 35B active	Qwen blog
GLM-4.5	2025-07-28	355B total, 32B active	z.ai blog
Step-3	2025-07-31	321B total, 38B active	StepFun research
Seed-OSS-36B	2025-08-21	36B	ByteDance Seed blog
DeepSeek-V3.1	2025-08-21 `†`	`—`	deepseek-ai/DeepSeek-V3.1
LongCat-Flash-Chat	2025-08-29 `†`	560B total, 18.6B–31.3B active	meituan-longcat/LongCat-Flash-Chat
Qwen3-Next-80B-A3B	2025-09-09 `†`	80B total, 3B active	Qwen/Qwen3-Next-80B-A3B-Instruct
Ling-flash-2.0	2025-09-17 `†`	100B total, 6.1B active	inclusionAI/Ling-flash-2.0
Qwen3-Omni-30B-A3B	2025-09-20 `†`	30B total, 3B active	Qwen/Qwen3-Omni-30B-A3B-Instruct
Qwen3-VL	2025-09-22 `†`	235B total, 22B active	Qwen/Qwen3-VL-235B-A22B-Instruct
DeepSeek-V3.2-Exp	2025-09-29	`—`	DeepSeek news
GLM-4.6	2025-09-30	`—`	z.ai blog

Q4

Model	Date	Parameters	Source
Ling-1T	2025-10-02 `†`	1T total, 50B active	inclusionAI/Ling-1T
DeepSeek-OCR	2025-10-17 `†`	`—`	deepseek-ai/DeepSeek-OCR
LongCat-Flash-Omni	2025-10-23 `†`	560B total, 27B active	meituan-longcat/LongCat-Flash-Omni
MiniMax-M2	2025-10-27	230B total, 10B active	MiniMax blog
Kimi K2 Thinking	2025-11-04 `†`	`—`	moonshotai/Kimi-K2-Thinking
DeepSeek-V3.2	2025-12-01	`—`	DeepSeek news
GLM-4.7	2025-12-22	`—`	z.ai blog

2026

Q1

Model	Date	Parameters	Source
GLM-4.7-Flash	2026-01-19 `†`	30B total, 3B active	zai-org/GLM-4.7-Flash
Step-3.5-Flash	2026-02-01 `†`	196B total, 11B active	stepfun-ai/Step-3.5-Flash
MiniMax-M2.5	2026-02-12	`—`	MiniMax news
GLM-5	2026-02-12	744B total, 40B active	z.ai blog
Qwen3.5-397B-A17B	2026-02-16	397B total, 17B active	Alibaba press release

Q2

Model	Date	Parameters	Source
MiniMax-M2.7	2026-04-09 `†`	`—`	MiniMaxAI/MiniMax-M2.7
Kimi K2.6	2026-04-14 `†`	1T total, 32B active	moonshotai/Kimi-K2.6
DeepSeek-V4 (Pro and Flash)	2026-04-24	Pro 1.6T total / 49B active; Flash 284B total / 13B active	DeepSeek news
Step 3.7 Flash	2026-05-23 `†`	198B total, 11B active	stepfun-ai/Step-3.7-Flash
MiniMax-M3	2026-06-01	~428B total, ~23B active	MiniMax blog

The DeepSeek-V4 row now carries both counts; the Flash model’s 284B total is stated on the same announcement page as the Pro figures. Step 3.7 Flash moved to a † date: its GitHub repository has no releases, and the page I first cited carries a creation timestamp rather than a release date.

Q3

Model	Date	Parameters	Source
LongCat-2.0	2026-07-05 `†`	1.6T total, ~48B active	meituan-longcat/LongCat-2.0
Hunyuan Hy3	2026-07-06	295B total, 21B active	Tencent newsroom
Qwen3.8	2026-08-05 `†`	27B; 2.4T total / 95B active	Qwen/Qwen3.8-2.4T-A95B
DeepSeek-V4-Pro GA	2026-08-13	1.6T total, 49B active	DeepSeek changelog
DeepSeek-V4-Flash-Vision-Exp	2026-08-21	`—`	DeepSeek changelog
DeepSeek-V4.1-Flash	2026-09-10	552B total; 8B active for input, 16B for output	DeepSeek changelog

Hy3 is the entry whose status changed in this revision. In the first pass I dropped it because the page I found did not say whether weights were published. It does: Hy3 is open-sourced under Apache 2.0 with weights on Hugging Face, and Tencent’s newsroom names the parameter counts. The V4-Pro GA row repeats the count from the April preview announcement, because the GA page does not restate it. The Qwen3.8 row is dated by the repository of the 27B model, the first of the line to appear; the flagship 2.4T-A95B repository was created three days later, and Qwen’s own announcement page is one of the client-rendered ones described above.

Two rows elsewhere disagree with third-party timelines for the same reason: LongCat-2.0’s repository was created 2026-07-05 while secondary coverage puts its launch at the end of June, and DeepSeek-V4-Flash’s repository was created 2026-04-22, two days before the announcement page.

What changed in this revision

Seven rows were added and three cells changed, listed because the edit history is the point of a table like this:

Qwen1.5 promoted into 2024 Q1 (2024-02-04), after finding the dated post on the old Qwen blog.
Hunyuan Hy3 promoted into 2026 Q3 (2026-07-06). It was dropped in error.
Four DeepSeek point releases added: V2.5-1210 (2024-12-10), V3-0324 (2025-03-25), R1-0528 (2025-05-28) and V4-Pro GA (2026-08-13). All four are dated on DeepSeek’s own changelog, the same source as V3.2 and V3.2-Exp.
Qwen3.8 added (2026-08-05 †), which the first pass omitted entirely.
Two ChatGLM parameter cells blanked. The ChatGLM2-6B and ChatGLM3-6B cards do not state a count, so the cells are now —.
DeepSeek-V4-Flash gained its total. The row said “Flash 13B active” and dropped the 284B total, which the same page states.

Three rows changed source or date type for the same structural reason: the Qwen3-Next, Qwen3-Omni and Step 3.7 Flash rows now cite a weight repository with a †, because the pages cited in the first pass were either unreadable to a plain request or carried no release date.

How this table gets updated

The table is edited in place. Every quarter I re-read the vendor pages for the labs already listed, then check Hugging Face for new repositories under those organizations. A release enters the table when it has a date and a link; until then it sits in the dropped list, with the reason.

Two rules keep it stable: I do not replace a † date with a later one unless the vendor publishes its own dated page, and I do not remove a superseded row.

The current table has 67 entries. Twenty-three of them fall in the twelve months from October 2025 to October 2026. That is a count of this table and nothing more; it is not a claim about the field, because the table only holds releases I could date.

What I could not check

Eleven candidates were dropped, for these reasons:

Yi-1.5 (May 2024) — the weight repository (01-ai/Yi-1.5-34B) was created 2024-05-11 †, and I could not date the announcement itself.
InternLM2 and InternLM2.5 — the weight repositories (internlm/internlm2-7b, internlm/internlm2_5-7b-chat) look like renamed predecessors, so the creation timestamp does not mark the release.
MiniCPM 1 and 2 — same problem; the repository (openbmb/MiniCPM-2B-sft-bf16) is now a monorepo for a whole series.
Baichuan 3 and 4 — I could not find weight artifacts at all.
ERNIE 5.0 and ERNIE 5.1 — Baidu’s blog dates them 2026-02-06 and 2026-05-09 (ERNIE 5.0, ERNIE 5.1) and describes ERNIE 5.0 as a 2.4-trillion-parameter model, but I found no open-weight repository for either.
Kimi K3 — the model card (moonshotai/Kimi-K3) carries the full architecture table (2.8T total, 104B activated) and a license, but no release date. The repository timestamp is 2026-06-13 and secondary coverage says mid-July. I could not reconcile that.
Kimi K2.5 — same shape of problem: the weight repository timestamp (moonshotai/Kimi-K2.5) is 2026-01-01 while the launch is reported at the end of January.
GLM-5.2 — the post is client-rendered and the only date string in its bundle is 2026-06-16. I am not willing to print that as the release date.
DeepSeek-V3.1-Terminus — a mid-cycle rename with a 2025-09-22 † repository (deepseek-ai/DeepSeek-V3.1-Terminus); including it would have double-counted the V3.1 line.
Qwen-Image-2.1 — a dated weight repository (Qwen/Qwen-Image-2.1, 2026-09-14 †) but an image model, outside this table’s scope by the rule stated at the top.
The Step-Audio line — the same, for speech. The repositories exist (stepfun-ai/Step-Audio-R1.1) and I did not date them, because they would need their own table with its own scope.

Three other limits on this table:

The † timestamps are preparation times, not launch times. A repository can be created days before the announcement, and from outside I cannot see whether weights were uploaded at once or in pieces. Where the gap is large I noted it above the table.
The dates are the vendor’s own. No release date here has been confirmed by a second source, and one link per release is deliberate: a vendor correcting itself is the signal worth keeping, and coverage repeating the vendor is not a second source.
License differences are not reflected. Apache-2.0, MIT and the vendor-specific licenses Kimi, MiniMax and LongCat ship under are not the same permission. Each row links to the page that states its license, and for Hy3 that page states Apache 2.0.

Measuring error in support AI, and what accuracy hides

Fri, 09 Oct 2026 00:00:00 GMT

The way I test a support or ticketing system is against a task book, not a demo. The number people ask for afterwards is accuracy, and I have stopped handing that over on its own: on every realistic test set I have built, the average was the least informative thing in the results file.

What follows is the method I use now, plus a run on public data so the numbers can be checked.

Five failure modes, five different bills

A support model fails in at least five ways, and they do not cost the same.

Fabrication. The reply asserts something absent from the policy or the record: a fee that does not exist, a wrong refund window, an invented delivery date. This is what people call hallucination. The taxonomy I lean on, OlaBench, splits it into four sub-types — factual hallucination, misuse of retrieved results, relevance hallucination, and logical inconsistency (OlaBench, arXiv:2510.22143v3, §3.1; v1 25 Oct 2025, v3 25 May 2026, retrieved 9 Oct 2026). A sentence that reads well and is false costs money, and the cost usually arrives months later.

Missed detection. The ticket contains a legal threat, a regulator’s name, a safety issue or a fraud signal, and the system files it as routine. Nothing false was said. The failure is that no human ever reads it. On a scoring sheet this often counts as a correct classification, because the model did pick a queue that exists.

Misrouting. The reply is reasonable and the queue is wrong, so a differently-skilled team applies its own policy. The measurable cost is time to resolution and repeat contact.

Out-of-scope answers. The system answers a question it was never built for — tax, legal advice, another provider’s product. The content may be accurate. The exposure is regulatory.

Refusal. The system declines, loops, or hands off. There is no answer, so there is no error to score. This one gets tracked as a transfer or containment rate and drops out of the error count entirely.

Failure	What is wrong	Where the cost lands	Commonly reported as
Fabrication	the content	refunds, goodwill, disputes	hallucination rate
Missed detection	nothing	escalation, legal, safety	almost nothing
Misrouting	the destination	time to resolution, repeats	accuracy
Out of scope	the remit	compliance	rarely measured
Refusal	nothing	churn, containment	transfer rate

One accuracy figure blends these five at whatever ratio the sample happens to contain, and that ratio is set by whoever picked the sample.

What I ran

I needed a routing task with a ground-truth label and text a model would really see. Public complaint data has both.

The US CFPB publishes its Consumer Complaint Database under CC0 (CFPB Consumer Complaint Database, retrieved 9 Oct 2026); the search API’s own metadata reports 18,274,023 records, license CC0, last updated 2026-10-08 (API metadata, retrieved 9 Oct 2026).

The JSON search endpoint returns 15 fields and no narrative (field reference, retrieved 9 Oct 2026). That is new: the September 2026 release removed complaint narratives from the database (CFPB release notes, Release 24, September 2026). So I took the text from a Hugging Face mirror instead, BEE-spoke-data/consumer-finance-complaints (dataset card, CC0, mirror last modified 29 Dec 2025); its has-text config holds 1,689,573 rows (split sizes, retrieved 9 Oct 2026).

Task: read the narrative only, predict the product queue. Ground truth: the product label the CFPB publishes. I drew 3,894 complaints from it. The oldest is dated 2015-03-20, the newest 2024-01-04.

The label set changed under the data

The raw product field holds 17 distinct strings in this sample, and they are not 17 products. The date ranges are what give it away.

Product string as published	Rows	Received, first → last in this sample
Credit reporting, credit repair services, or other personal consumer reports	1,892	2017-04-24 → 2023-08-24
Debt collection	547	2015-03-20 → 2023-12-18
Credit card or prepaid card	267	2017-04-24 → 2023-08-21
Mortgage	248	2015-04-21 → 2023-11-02
Credit reporting or other personal consumer reports	245	2023-08-26 → 2024-01-04
Checking or savings account	195	2017-04-24 → 2023-11-21
Credit reporting	92	2015-04-17 → 2017-04-09
Student loan	85	2015-10-06 → 2023-11-13
Money transfer, virtual currency, or money service	79	2017-08-08 → 2023-11-13
Vehicle loan or lease	75	2017-04-24 → 2023-11-01
Credit card	68	2015-04-06 → 2023-12-22
Payday loan, title loan, or personal loan	41	2017-04-24 → 2023-06-20
Bank account or service	34	2015-04-04 → 2017-01-31
Consumer Loan	19	2015-03-25 → 2017-02-03
Payday loan	3	2015-10-08 → 2017-02-24
Money transfers	2	2015-07-17 → 2015-12-05
Payday loan, title loan, personal loan, or advance loan	2	2023-09-20 → 2023-10-06

Read the dates down the table. Bank account or service last appears here on 2017-01-31, and Checking or savings account first appears on 2017-04-24. Three more strings end in 2017 and their successors start the same year. The three credit-reporting strings sit end to end: 2015 to 2017, 2017 to 2023, 2023 into 2024. Same category, renamed.

I collapsed the 17 strings into 9 categories before scoring anything. The mapping is in the script, applied in one place rather than inferred per row. Without that step an accuracy compares a 2016 row and a 2023 row whose label sets do not line up, and the model gets credit or blame for the taxonomy.

The live database disagrees with the mirror about which names are current. Aggregated over complaints received in 2026, the CFPB API returns 11 product strings, with Credit card and Prepaid card listed separately and no Credit card or prepaid card at all (product aggregation, complaints received 2026-01-01 onward, retrieved 9 Oct 2026). I did not reconcile the two taxonomies.

Three ways one accuracy figure misleads

Imbalance. After collapsing, the sample looks like this:

Collapsed category	Rows	Share
Credit reporting	2,229	57.2%
Debt collection	547	14.0%
Credit card or prepaid card	335	8.6%
Mortgage	248	6.4%
Checking or savings account	229	5.9%
Student loan	85	2.2%
Money transfer or virtual currency	81	2.1%
Vehicle loan or lease	75	1.9%
Personal / payday / title loan	65	1.7%

A predictor that answers “credit reporting” for every ticket scores 0.572 here. That is a constant, and it sits close enough to a real model’s score to be mistaken for one.

Sample selection. Same data, same model, two splits:

Split	Test n	Accuracy	Macro F1	Credit-reporting share of test set
Random 60/40	1,558	0.756	0.428	57.1%
Chronological, split at 2022-04-21	1,560	0.832	0.377	74.0%

The chronological run scores 7.6 points higher while getting worse on most categories. The test set became more concentrated in the biggest class, and the dominant-class predictor rides that concentration up. Anyone reporting the 0.832 as an improvement after retraining would be reading their own sampling as progress.

Coverage. Sort the test set by the model’s confidence and score only the top slice:

Coverage	Tickets scored	Accuracy on that slice
100%	1,558	0.756
75%	1,168	0.854
50%	779	0.904
25%	389	0.900
10%	155	0.897

The 0.904 at half coverage is a real number computed on held-out data, and published alone it misdescribes the system by 15 points. Note where it stops moving: from 50% coverage down to 10%, accuracy drifts from 0.904 to 0.897, so the model’s confidence and its correctness come apart on the tail. That tail is where the misroutes are.

Underneath the average, the per-category picture on the random split:

Category	Test n	Recall	Precision
Credit reporting	889	0.879	0.886
Debt collection	225	0.631	0.637
Credit card or prepaid card	129	0.760	0.430
Mortgage	114	0.860	0.737
Checking or savings account	93	0.548	0.607
Money transfer or virtual currency	32	0.000	0.000
Student loan	29	0.276	0.889
Vehicle loan or lease	29	0.000	0.000
Personal / payday / title loan	18	0.000	0.000

Three categories are never once predicted correctly. Accuracy is 0.756. Macro F1 is 0.428. The distance between those two is the honest headline for this model.

Pulling the high-risk errors out of the average

The useful published idea here is OlaBench’s risk axis, Critical Business Risk Rate. It counts only responses that inappropriately assert one of four things, each a high-stakes failure that may trigger compliance exposure, user disputes or reputational damage: admitting platform liability (618 test cases), misidentifying the ICS role (228), overcommitting (138), and disparaging individuals or merchants (16) (OlaBench, arXiv:2510.22143v3, §3.1 and Table 1; 25 May 2026, retrieved 9 Oct 2026). It is reported as its own number, beside the hallucination rate.

The shape of that is what to copy: choose failures whose consequences differ in kind, count them separately, publish both, and never average them.

For the routing test I defined three tiers, and I chose them:

critical — a complaint about funds or account access (checking or savings, money transfer) lands in the reporting queue
material — a credit or loan dispute lands in a different loan or reporting family
routine — a misroute inside the same family

Tier	Random split	Chronological split
Critical	7 / 1,558 = 0.4%	11 / 1,560 = 0.7%
Material	43 / 1,558 = 2.8%	28 / 1,560 = 1.8%
Routine	330 / 1,558 = 21.2%	223 / 1,560 = 14.3%

The two splits swap places depending on which tier you read: the one with the prettier accuracy hides a critical error rate almost twice as high.

Published work also shows how far the same named metric can move when the population changes. The OlaBench paper reports a critical business risk rate of 8.7% for its OlaMind-Stage-2 model on the benchmark, and a critical business risk rate below 0.05% in daily manual annotation of live dialogues after launch (OlaBench, arXiv:2510.22143v3, §5.2 and Table 3; 25 May 2026, retrieved 9 Oct 2026). Both are the same metric, measured on different populations with different annotators. Quoting one without the other is a choice.

How these numbers were produced

Data. 3,894 complaints from a CC0 mirror of the CFPB Consumer Complaint Database (config has-text, 1,689,573 rows). The draw is 40 pages of 100 consecutive rows at random offsets in the split, deduplicated by complaint ID, seed 20261009. Rows with a narrative under 80 characters were dropped. It is a clustered draw, not an independent one.
Label hygiene. 17 published product strings collapsed to 9 categories using the date ranges in the table above. The mapping is written out in the script, not inferred per row.
Model. Multinomial naive Bayes over word unigrams and bigrams, Laplace smoothing α = 1, features seen at least twice in training. Python standard library only.
Splits. Random 60/40 with seed 7; chronological split at the 60th percentile of the received-date order, which lands on 2022-04-21.
Selective accuracy. Test rows sorted by the model’s maximum posterior probability, top k scored.
Every number in the tables above is printed by that one script over that sample.
Deliberate choice. No language model appears in this pipeline, including the scoring. A measurement instrument that is itself a model carries its own error, and OlaBench publishes the size of its own: its judge agrees with human annotators at 91.7% on risk identification and 82.6% on hallucination detection, over 5,000 human-annotated instances (OlaBench, arXiv:2510.22143v3, §3.3 and Table 2; 25 May 2026, retrieved 9 Oct 2026). A judge that disagrees with people 8% of the time on the risk axis puts a floor under everything else you can measure with it.

What I could not check

Whether the mirror’s product strings match the live database’s current taxonomy. The mirror’s rows use names the live API no longer returns for complaints received in 2026. I collapsed by date range and did not reconcile the two.
Whether the mirror is complete. Its files are timestamped December 2025 and the newest complaint in my 3,894-row sample is dated 2024-01-04. I did not verify whether later records exist in it.
Whether complaints published with a narrative are representative of all complaints. Narratives appeared only with consent, so every number above sits on that consenting subset.
Whether this sample’s class mix reflects the population. Credit reporting is 57.2% of the sample and about 82% of the live database: 14,978,722 of 18,274,023 records carry one of the three credit-reporting strings (product aggregation, retrieved 9 Oct 2026). Most of the gap is the mirror’s date cap.
How noisy the product label is. It is chosen by the consumer at submission, not assigned by an expert, and I treated it as ground truth because companies route on it. If it is noisy, my error rates are a floor, not a ceiling.
Whether a published tiering exists for complaint-routing consequences. I could not find one, so the three tiers are mine. The OlaBench categories are a different kind of thing: they score what a reply asserts about liability and commitment, and they do not classify misroutes.
Whether error rates vary with company size, state, or complaint age. A stratified rate is the next thing to compute.
The naive Bayes model is a floor. A trained transformer would score higher on the same task. The run was to see the shape of the error distribution, and a better model changes the level without changing the shape.

How an eval set is built, and why it rots

Fri, 09 Oct 2026 00:00:00 GMT

An eval set is an instrument. It has a purpose, a scoring rule, a calibration record and an expiry date. Treating it as a file you download is where the trouble starts: the file already made those decisions without writing them down.

What follows is the build document I would want for a set I have to defend six months from now. Each rule is traceable to a public source or marked as my practice.

Where the samples come from

Four sources, each importing a different bias.

Source	What you get	The bias it imports	Documented where
A public benchmark, reused	Comparability with published numbers	Contamination; a task choice someone else made	Sainz et al., 2023-10-27
Public administrative data	A real distribution, a public licence, dated updates	A sample of who reports, not of who is affected	CFPB, read 2026-10-09
Synthetic generation	Volume you can afford, labels you control	Generator artifacts in the surface form	Gururangan et al., ACL 2018
Your own production traffic	Items that match what you ship	A convenience sample that drifts with the product	Lipton et al., ICML 2018

Reuse. LiveBench’s authors state the problem in one sentence: test set contamination “can quickly render benchmarks obsolete” (LiveBench, v2 2025-04-18). Their answer was questions from recent competitions, arXiv papers and news, added monthly. LiveCodeBench did the same for code: 400 problems published between May 2023 and February 2024 (LiveCodeBench, arXiv v1 2024-03-12). A static benchmark decays without anyone editing it.

Administrative data. The CFPB complaint database is the best example I know. Its search API reported 18,274,023 records under a CC0 licence on 2026-10-09, with a last_updated of 2026-10-08 (CFPB complaint search API, read 2026-10-09). Complaints are published only after the company responds or after 15 days, whichever comes first, and complaints referred to other regulators, including depository institutions with less than $10 billion in assets, are not published (CFPB API docs, read 2026-10-09). The agency states that the database “is not a statistical sample of consumers’ experiences in the marketplace” (CFPB database page, read 2026-10-09). A random draw samples the people who complained and whose complaints survived publication, not the people who were harmed.

Synthetic. Gururangan et al. showed that a text classifier reading only the hypothesis, with the premise deleted, reached about 67% on SNLI and 53% on MultiNLI (Annotation Artifacts in NLI Data, ACL 2018). The label was recoverable from phrasing because the crowd workers followed a template. Every generator has one. My test: delete the part of the input meant to carry the answer, score again, and compare with the majority-class rate. If accuracy stays well above it, the set is partly measuring its own generator. (my practice)

Production sampling. This is the source I trust least and use most. It produces items that match what the system sees, and its composition changes when the product does. Label shift is the named version of that failure: the marginal p(y) moves while p(x|y) does not, measurable without test labels using Black Box Shift Estimation (Lipton et al., ICML 2018).

What counts as correct

The match rule is a parameter, and the same predictions score differently under each choice. Four conventions:

Convention	The rule, as the source writes it	Source
Exact match against any reference	Counts predictions that match any one ground truth answer exactly; punctuation and articles ignored	SQuAD, arXiv v3 2016-10-11
Partial credit against the best reference	F1 over bags of tokens, “the maximum F1 over all of the ground truth answers”, then averaged	SQuAD, arXiv v3 2016-10-11
A threshold on localisation	IoU thresholds `[.5:.05:.95]`, 10 of them; 101 recall thresholds; `maxDets` of 1, 10 and 100	cocoapi `cocoeval.py`, read 2026-10-09
A graded rubric with an exclusion flag	0–3 for how well specified the issue is; 0–3 for whether the tests are scoped to it; a separate 0/1 question for any other reason the sample should not be used; a 1–5 confidence rating	SWE-bench annotation instructions, undated

Three things this table taught me.

Say who the ceiling belongs to. In SQuAD the human number comes from the data: the second annotator’s answer is treated as the prediction, the rest as ground truth, and humans score 77.0 exact match and 86.8 F1 on the test set (SQuAD, arXiv v3 2016-10-11). A ceiling quoted without its rule says as much about the rule as about the task.

One published number can hide ten thresholds. COCO’s headline detection score averages over ten IoU thresholds, three detection budgets and a recall grid. A single figure without a named threshold is a claim nobody can re-check. Pick the threshold, publish it, keep it fixed across versions.

The exclusion question is the part people skip. SWE-bench Verified is 500 samples “verified to be non-problematic by our human annotators”, drawn from 1,699 randomly sampled items that 93 Python developers annotated, with the full annotation set and the rubric released alongside (OpenAI, updated 2025-02-24). The rubric’s last section asks, 0/1, whether there is any other reason not to use the sample. My handling of it, (my practice):

A closed list of exclusion reason codes, decided before annotation starts.
Two raters must agree before an item is dropped; one rater’s “unclear” is not an exemption.
The dropped item stays in the file with its reason code, so the exemption stays auditable.
The exemption rate is reported next to every result; if it moves between releases, the set changed even when the questions did not.

Ground truth built by pooling is biased by construction. Test collections are built by judging only a sample of documents. When the pool is small relative to the collection, the judgment set “can be biased in that they favor relevant documents that contain topic title words”, a bias that depends on collection size and not on the number of relevant documents (Buckley et al., NIST, 2007-07-17). In a modern eval, if your labels come from checking only what your current systems returned, the label set favours items phrased the way those systems phrase things. My rule: draw a random slice from outside the candidate pool. (my practice)

Whether the raters agree

Agreement is a property of the pair and the setup.

Krippendorff’s alpha is 1 − Do/De, and it accepts any number of coders and missing entries (Krippendorff, 2011-01-25). For two raters, Cohen’s kappa, with Landis and Koch’s 1977 bands (reproduced in an AHRQ report, 2010).

Here is why the setup has to travel with the number. First-turn results on MT-bench, in the paper’s two setups: S1 counts ties and inconsistent votes, S2 excludes ties (MT-Bench, v4 2023-12-24):

Pair (first turn)	S1, R = 33%	S2, R = 50%	Source
GPT-4 pairwise vs human expert	66% (n = 1,343)	85% (n = 859)	MT-Bench, v4 2023-12-24
GPT-4 single-answer vs human expert	60% (n = 1,280)	85% (n = 739)	MT-Bench, v4 2023-12-24
Human vs human	63% (n = 721)	81% (n = 479)	MT-Bench, v4 2023-12-24

The headline everyone quotes is 85% against 81% for human agreement (MT-Bench, v4 2023-12-24). The same table holds 66% for the same judge under the other tie rule. An agreement figure quoted without its setup is not comparable to anything.

Measure agreement per stratum, not pooled. ChaosNLI collected 464,500 annotations, 100 per example, over 3,113 SNLI and MNLI examples and 1,532 ANLI examples (Nie et al., 2020-10-08). Models are near-perfect where humans agree and “can barely beat a random guess” where they do not, and those low-agreement items are most of the errors. A pooled score mixes two measurement regimes whose mixing ratio changes as models improve.

Sizing. For an agreement study, start from the kappa minimum-sample-size tables (Bujang and Baharum, EBPH 14(2), 2017). For comparing two systems, use a power analysis: Miller gives the MDE formula and shows that raising answers per question from 1 to 10 moved the MDE from 13.2% to 7.5%; in his illustrative table, HumanEval’s 164 questions at a fictional 83.6% carry a standard error of 3.2% (Miller, 2024-11-01). If items arrive in groups, such as the same document or template, compute clustered standard errors (Miller, 2024-11-01, Table 4); on DROP they were 1.34 against 0.44 naive.

My rule: size the calibration slice for the agreement interval I need, the scoring slice for the effect I want to detect, and refuse to report a gap below the MDE. (my practice)

Why the set rots

Decay	Symptom	What I do
Contamination	One set improves while unrelated sets do not	Keep item dates and prompt hashes; treat a stale set as historical
Label shift	Scores move while the system did not	Estimate the label mix and re-weight or re-cut the slice
Taxonomy and rubric drift	The same question means something different	Version the labels; freeze the rubric text per set version
Saturation	Every system sits near the ceiling	Retire the set and say when

Contamination’s extent “is unknown, as it is not straightforward to measure”, and where it exists it overestimates performance (Sainz et al., 2023-10-27). I cannot audit training corpora, so I keep the dates and admit the exposure window.

Saturation is not a failure of the set: GLUE’s performance passed the level of non-expert humans, which is what prompted SuperGLUE (Wang et al., 2019-05-02). Benchmark choice is fragile on its own: swapping tasks can reorder methods even when the methods do not change (Dehghani et al., 2021-07-14). Both are reasons to keep more than one set.

Taxonomy drift has a public paper trail. The CFPB’s September 2026 release notes record that “consumers’ complaint narratives … have been removed from the database”; the June 2026 release removed “Consumer disputed” and “Consumer consent provided” from exports, long after those filters went away; the May 2023 release re-based the ZIP code field on 2019 census estimates (CFPB release notes, read 2026-10-09). A set built on those narratives became unmaintainable one month ago. The same page explains the general rule: the database shows “the consumer’s original products, sub-products, issues, and sub-issues selections consistent with the options available on the form at the time the consumer submitted the complaint” (CFPB database page, read 2026-10-09). Labels frozen at submission, against a taxonomy that keeps moving.

Keeping it maintainable

(my practice, with borrowed pieces named)

Version the set. Additions and exclusions get a version bump and a changelog line; never edit an item silently.
Two slices. A frozen slice for comparability, a rolling window for freshness. LiveCodeBench and LiveBench are public versions of the same idea.
Write the metadata down. A datasheet (Gebru et al., 2018, CACM 2021) or a Croissant record (2024-03-28), so the next person need not infer the collection process.
A desensitisation rule that is checkable. The CFPB’s ZIP rule is a good template: publish the 5-digit ZIP unless the census area has under 20,000 people, then 3 digits only if the wider area has more than 20,000, otherwise nothing (CFPB field reference, read 2026-10-09). A rule stated as a threshold can be re-run; “we removed personal information” cannot.
Watch the exemption rate. It is the leading indicator that the rubric stopped fitting the data.
State the blind spot in one line. SQuAD’s metrics ignore punctuation and articles; COCO’s headline averages ten thresholds. Both are fine as long as the limitation is on the label.
Plan the retirement. When a set saturates or its taxonomy moves, freeze it, date it, and build the next one from the same seed items.

What I could not check

How many SWE-bench samples were dropped to leave 500. The announcement gives the annotated total (1,699) and the rubric, not the drop count (OpenAI, updated 2025-02-24).
Whether any specific model saw any specific benchmark item. Sainz et al. state the extent is unknown; I have no method that resolves this, so I do not claim cleanliness.
The CFPB’s narrative scrubbing standard. The field is gone and the current field reference no longer documents it; I could not reach a dated scrubbing specification.
The CFPB’s historical consumer_consent_provided semantics. The export field and its filter are gone; I found the removal dates, not the values.
Krippendorff’s own minimum sample size for alpha. I have the kappa tables from Bujang and Baharum; I found no equivalent table for alpha.
The Landis and Koch bands at first hand. I read a reproduction of the 1977 table in a 2010 AHRQ report, not the original.
LiveCodeBench’s current item count. The 400 figure comes from the March 2024 paper; the platform’s live count may differ.
The OpenAI SWE-bench Verified page at first hand. The live page served an automated-client challenge when I read it; the same version is readable in an Internet Archive copy, and the annotation instruction PDF is undated.
The CFPB record count above is a live API value and changes daily. It was 18,274,023 when I read it on 2026-10-09.

From token prices to task prices: three jobs, seven APIs

Fri, 09 Oct 2026 00:00:00 GMT

Every vendor publishes a price per million tokens. Almost none publishes what a job costs; the number that decides whether a feature ships is left as arithmetic homework.

I did that arithmetic for three jobs, from first-party price lists read on 2026-10-09. The token counts are assumptions, not measurements, and they are written out below. The money is the vendor’s standard-tier list price, in USD.

The three jobs

Job 1, label 1,000 support messages. Each call sends a 400-token instruction block and a 120-token message, and asks for a 15-token answer. Billed per run: 520,000 input tokens, of which 400,000 are the instruction block sent 1,000 times, plus 15,000 output tokens.

Job 2, turn a 100,000-word draft into a structured summary. Google’s token guide says a token is about 4 characters and 100 tokens about 60-80 English words, putting 100,000 words at 125,000-167,000 tokens (Google token counting guide, read 2026-10-09). I used 140,000:

One shot: 140,000 tokens in, a 3,000-token structured summary out.
Map-reduce: 14 chunk calls of about 10,000 tokens each, 700 tokens of notes per chunk, then a 9,800-token reduce call. Totals: 149,800 in, 12,800 out.

Job 3, one round of tool-assisted bug fixing. Eight turns against a 20,000-token repository: system prompt, tool schemas, three files. The prompt grows about 2,500 tokens per turn as tool results come back, from 20,000 to 37,500. Cumulative billed input is 230,000 tokens, of which 192,500 are a re-sent prefix a cache can serve; output is 6,000 tokens including reasoning.

How I read the price lists

Five first-party pages, all read on 2026-10-09; the Gemini API page builds its tables client-side, so I read it in a browser.

OpenAI API pricing — standard tier
Anthropic pricing
Gemini API pricing — paid tier, introductory rates through 2026-12-31
DeepSeek Models & Pricing — off-peak column
xAI pricing

The rules I applied:

Standard tier only, unless a row says batch: the real-time tier.
Cache reads are billed at the published cached-input rate. Cache writes are billed where a vendor publishes a write price. OpenAI and Anthropic do: $2.50 per million tokens for the two frontier models here, $0.125 for the two small ones, on Anthropic’s 5-minute cache. Google, DeepSeek and xAI publish no write price, so I charged zero, which flatters those three in every cached row.
In the cached rows I charge a write for every input token that is not a cache read: 400 tokens in Job 1, 37,500 in Job 3.
Reasoning tokens are output tokens. OpenAI states they are billed as output tokens (OpenAI reasoning guide, read 2026-10-09). Gemini’s output price is quoted “including thinking tokens” and DeepSeek’s V4.1-Flash runs in thinking mode by default.
Tool and search fees sit outside the main table.
No prompt hits a long-context threshold; the prompt-length step that does apply, Anthropic’s 100,000-token rule, is below.
No aggregator site was used for any price.

What each job costs

Task	Model	Estimated cost	Assumptions	Price source (read 2026-10-09)
Job 1, no cache	gpt-6.1-sol	$1.19	standard tier; 520k in / 15k out	OpenAI
Job 1, no cache	gpt-6-luna	$0.060	standard tier; 520k in / 15k out	OpenAI
Job 1, no cache	Claude Sonnet 5.5	$1.19	standard tier; 520k in / 15k out	Anthropic
Job 1, no cache	Claude Haiku 5.5	$0.060	standard tier; 520k in / 15k out	Anthropic
Job 1, no cache	Gemini 3.8 Flash	$0.45	standard tier, introductory rate; 520k in / 15k out	Google
Job 1, no cache	DeepSeek V4.1-Flash	$0.087	off-peak window; 520k in / 15k out	DeepSeek
Job 1, no cache	Grok 4.7	$1.13	standard tier; 520k in / 15k out	xAI
Job 1, instructions cached	gpt-6.1-sol	$0.43	400k of input served from cache; one 400-token write	OpenAI
Job 1, instructions cached	gpt-6-luna	$0.024	400k of input served from cache; one 400-token write	OpenAI
Job 1, instructions cached	Claude Sonnet 5.5	$0.43	400k cached, 5-minute TTL and write	Anthropic
Job 1, instructions cached	Claude Haiku 5.5	$0.024	400k cached, prompts under 100k, 5-minute TTL	Anthropic
Job 1, instructions cached	Gemini 3.8 Flash	$0.18	400k of input served from cache, storage excluded	Google
Job 1, instructions cached	DeepSeek V4.1-Flash	$0.028	off-peak; cache hits free of a write fee	DeepSeek
Job 1, instructions cached	Grok 4.7	$0.53	400k of input served from cache	xAI
Job 2, one shot	gpt-6.1-sol	$0.31	140k in / 3k out, one call	OpenAI
Job 2, one shot	gpt-6-luna	$0.016	140k in / 3k out, one call	OpenAI
Job 2, one shot	Claude Sonnet 5.5	$0.31	140k in / 3k out, one call	Anthropic
Job 2, one shot	Claude Haiku 5.5	$0.078	140k in / 3k out, above the 100k step	Anthropic
Job 2, one shot	Gemini 3.8 Flash	$0.12	140k in / 3k out, one call	Google
Job 2, one shot	DeepSeek V4.1-Flash	$0.023	off-peak; 140k in / 3k out	DeepSeek
Job 2, one shot	Grok 4.7	$0.30	140k in / 3k out, under the 200k threshold	xAI
Job 2, map-reduce	gpt-6.1-sol	$0.43	14 chunk calls + 1 reduce call; 149.8k in / 12.8k out	OpenAI
Job 2, map-reduce	gpt-6-luna	$0.021	14 chunk calls + 1 reduce call	OpenAI
Job 2, map-reduce	Claude Sonnet 5.5	$0.43	14 chunk calls + 1 reduce call	Anthropic
Job 2, map-reduce	Claude Haiku 5.5	$0.11	15 calls, each prompt under 100k	Anthropic
Job 2, map-reduce	Gemini 3.8 Flash	$0.16	14 chunk calls + 1 reduce call	Google
Job 2, map-reduce	DeepSeek V4.1-Flash	$0.030	off-peak window, 15 calls	DeepSeek
Job 2, map-reduce	Grok 4.7	$0.38	14 chunk calls + 1 reduce call	xAI
Job 3, agent loop	gpt-6.1-sol	$0.25	8 turns; 230k in (192.5k read, 37.5k written) / 6k out	OpenAI
Job 3, agent loop	gpt-6-luna	$0.013	8 turns; 230k in (192.5k read, 37.5k written) / 6k out	OpenAI
Job 3, agent loop	Claude Sonnet 5.5	$0.25	8 turns; 192.5k reads and 37.5k 5-minute writes / 6k out	Anthropic
Job 3, agent loop	Claude Haiku 5.5	$0.013	8 turns, every prompt under 100k	Anthropic
Job 3, agent loop	Gemini 3.8 Flash	$0.065	8 turns; 230k in (192.5k cached) / 6k out, storage excluded	Google
Job 3, agent loop	DeepSeek V4.1-Flash	$0.0098	off-peak window; 8 turns	DeepSeek
Job 3, agent loop	Grok 4.7	$0.21	8 turns; 230k in (192.5k cached) / 6k out	xAI

Two pairs of rows are identical on purpose. gpt-6.1-sol and Claude Sonnet 5.5 both list $2.00 per million input tokens, $10.00 output, $0.10 cached input and a $2.50 cache write (OpenAI pricing and Anthropic pricing, read 2026-10-09); gpt-6-luna and Claude Haiku 5.5 both list $0.10, $0.50, $0.01 and $0.125.

The same jobs with each vendor’s biggest discount

Batch is the biggest published discount: 50% on OpenAI (pricing page, read 2026-10-09), on Anthropic (“a 50% discount on both input and output tokens”, pricing page, read 2026-10-09) and on Google’s paid tier (“Batch API (50% cost reduction)”, pricing page, read 2026-10-09), where the Batch column halves the cached-input rate too. Not everywhere: xAI’s batch section gives 20% to grok-4.3 and the grok-4.20 variants and says unlisted models have no batch discount, which leaves Grok 4.7 at full price (xAI pricing, read 2026-10-09). DeepSeek publishes no batch tier.

Model	Job 1, cached	Job 2, one shot	Job 3, loop	Batch discount	Price source (read 2026-10-09)
gpt-6.1-sol	$0.22	$0.16	$0.12	50%	OpenAI
gpt-6-luna	$0.012	$0.0078	$0.0067	50%	OpenAI
Claude Sonnet 5.5	$0.22	$0.16	$0.12	50%	Anthropic
Claude Haiku 5.5	$0.012	$0.039	$0.0067	50%	Anthropic
Gemini 3.8 Flash	$0.088	$0.058	$0.033	50%	Google
DeepSeek V4.1-Flash	$0.028	$0.023	$0.0098	none published	DeepSeek
Grok 4.7	$0.53	$0.30	$0.21	none	xAI

Claude Haiku 5.5’s one-shot cell is the outlier: batch brings it to $0.039, still three times its cached Job 1 of $0.012, because a 140,000-token prompt sits above the 100,000-token step (Anthropic pricing, read 2026-10-09).

What the numbers say

Every figure here is arithmetic on the rows above, whose rates are linked (OpenAI, Anthropic, Google, DeepSeek, xAI, read 2026-10-09).

Output tokens decide the bill at the frontier tier. In cached Job 1 on gpt-6.1-sol, $0.15 of the $0.43 is output (35%) and $0.04 is cached instruction tokens (9%).

The spread inside one vendor is bigger than the spread between vendors. Cached Job 1 is $0.024 on gpt-6-luna and Haiku 5.5, $0.43 on gpt-6.1-sol and Sonnet 5.5, $0.53 on Grok 4.7: 22x from dearest to cheapest, 18x between the cheap pair and the frontier pair. At the same tier, the vendor changes nothing.

The agent loop is the cheapest job here. Job 3 is $0.25 on gpt-6.1-sol against $0.31 for the one-shot summary and $1.19 for Job 1. On the small models the order flips: Haiku 5.5 pays $0.078 for the one-shot summary against $0.013 for the loop, because 140,000 tokens crosses its threshold and 230,000 over eight turns does not.

Map-reduce costs more than one shot. On the two $2/$10 models it is $0.43 against $0.31, and the gap is all output tokens, 12,800 against 3,000. Chunking buys reliability and pays for it in output.

The absolute numbers are small. The main table spans $0.0098 to $1.19; batch pulls the floor to $0.0067. At this scale, a cost problem is usually a volume problem.

The variables that break the conversion

Thresholds re-price the whole request. Claude Haiku 5.5 charges $0.10 and $0.50 per million input and output tokens up to 100,000 prompt tokens, and $0.50 and $2.50 above it; the length counts cache reads and writes, and each request is priced on its own (Anthropic pricing, read 2026-10-09). Job 2’s 140,000-token prompt pays the higher rate on everything: $0.078, against $0.016 under the cap. xAI applies long-context rates to all tokens in a request once the prompt passes 200,000, where Grok 4.7 costs $4.00 input, $1.00 cached input and $12.00 output instead of $2.00, $0.50 and $6.00 (xAI pricing, read 2026-10-09). Gemini 3.1 Pro Preview steps from $2.00 to $4.00 input per million above 200,000 tokens (Gemini Enterprise Agent Platform pricing, read 2026-10-09). OpenAI’s line is 272,000 input tokens, published in a column tooltip; above it gpt-6.1-sol’s input doubles to $4.00 and its output to $15.00 (OpenAI pricing, read 2026-10-09). All three jobs stay under every line; Job 2 is halfway to OpenAI’s.

Tokenizer differences. Anthropic notes that Claude 4.7 and later use a tokenizer that “produces approximately 30% more tokens for the same text” (Anthropic pricing, read 2026-10-09). A comparison built on one words-per-token ratio, as mine is, is off by that much for Anthropic.

Time of day. DeepSeek’s peak hours are 01:00-04:00 and 06:00-10:00 UTC, Monday to Friday, excluding Chinese public holidays; every other hour is off-peak at half price (DeepSeek Models & Pricing, read 2026-10-09). Job 1 uncached is $0.087 off-peak and $0.174 at peak. Off-peak is 133 of the week’s 168 hours.

Cache economics differ in shape as well as in rate. Anthropic’s cache hit is 10% of the input price on most models, 5% on Opus 5.5 and Sonnet 5.5, 2.5% on Fable 5.1 and Mythos 5.1; writes cost 1.25x for five minutes and 2x for an hour, so a 5-minute cache pays for itself after one read (Anthropic pricing, read 2026-10-09). DeepSeek’s hit price is $0.003 per million against $0.15 for a miss (DeepSeek pricing, read 2026-10-09). Google charges no write fee but charges storage, $0.50 per million tokens per hour on Gemini 3.8 Flash through 2026-12-31 and $4.50 on Gemini 3.1 Pro (Gemini pricing and Gemini Enterprise Agent Platform pricing, read 2026-10-09); a cached 140,000-token document runs $0.07 an hour on Flash and $0.63 on Pro.

Platform fees sit outside the token meter. OpenAI bills $2.50 per 1,000 tool calls and $10 per 1,000 web searches, so Job 3’s eight calls cost $0.02, and Anthropic’s tool-use prompt adds 286 to 675 tokens (OpenAI pricing and Anthropic pricing, read 2026-10-09). xAI bills X search per item fetched: $5 per 1,000 posts, $10 per 1,000 profiles (xAI pricing, read 2026-10-09).

Latency and geography carry multipliers. OpenAI’s Fast mode, renamed from Priority on 2026-07-30, is 2x, and data-residency endpoints add 10%; Gemini 3.8 Flash’s Priority tier is $1.35 input against $0.75 standard (OpenAI pricing and Gemini pricing, read 2026-10-09). xAI’s priority tier is 2x and its US regional endpoint 1.1x; Anthropic charges 1.1x for US-only inference; regional Bedrock and Google Cloud endpoints carry a 10% premium (xAI pricing and Anthropic pricing, read 2026-10-09).

The list price itself moves. Google quotes its Flash prices “through December 31, 2026” and doubles them on 2027-01-01: $0.75 to $1.50 input, $3.75 to $7.50 output, $0.075 to $0.15 cached input, $0.50 to $1.00 per million tokens per hour of cache storage (Gemini pricing, read 2026-10-09). Anthropic’s $2/$10 for Claude Sonnet 5 was launched as introductory pricing through 2026-08-31 and is now standard, with the rise to $3/$15 cancelled (Anthropic pricing, read 2026-10-09). OpenAI’s promotional pricing for GPT-5.6 Sol runs at least through 2026-11-21, on a model not in this table (OpenAI pricing, read 2026-10-09). Any per-task table is a snapshot with an expiry date.

What I could not check

I ran none of these three jobs. The token counts are stated assumptions, and a measured run would move every number here.
Cache-write pricing for Google, DeepSeek and xAI. I found no published write price for any of the three, so I charged zero. xAI’s prompt-caching page lists a billing rate per token type with no write line; Google’s context-caching price is a read rate plus hourly storage; DeepSeek’s table has only a hit price and a miss price. If any of them does charge for writes, its cached rows are too low.
Whether the Haiku 5.5 threshold behaves as written at the boundary. The page counts cache reads and writes in a prompt’s length, which would mean a cached 140,000-token document pays the higher rate. I did not test it.
Real token counts. I used Google’s words-per-token ratio across every vendor; Anthropic’s own note about a 30% tokenizer difference is a warning that this is approximate.
Volume, committed-use and negotiated enterprise pricing, free tiers and trial credits are outside this table. The price on the page is the list price.
Resale through cloud platforms. Anthropic bills Bedrock and Google Cloud in consumption units, and Gemini Enterprise Agent Platform prices are not the Gemini API prices: on 2026-10-09 the two pages disagreed on cache storage, $0.50 per million tokens per hour (Gemini pricing) against $1.00 for one model (Gemini Enterprise Agent Platform pricing). The route you buy through can change the number.

Why there is no Western mini program

Fri, 09 Oct 2026 00:00:00 GMT

The usual version of this question is about super apps, and it has been answered at length by people with better data than mine; mine is narrower. A mini program is a small bundle of code that runs inside somebody else’s application, and the question is which parts of that arrangement exist in what Apple and Google ship.

I read the vendors’ own documentation for WeChat Mini Programs, Apple App Clips, Google Play Instant and Progressive Web Apps, and compared them on five dimensions: install and size, identity, payment, entry point, distribution and review.

One mechanism carries most of the answer. A mini program brings neither its own user account nor its own checkout. It borrows both from the application hosting it.

The five dimensions, side by side

	WeChat Mini Program	Progressive web app	Native app (iOS/Android)	App Clip (iOS)
Install and size	Runs inside WeChat. Main package 2 MB maximum, all packages together 30 MB (20 MB for programs developed by a third-party provider) (WeChat subpackages, read 2026-10-09)	No package at all. Chrome offers install after HTTPS plus a manifest carrying a name, 192 px and 512 px icons, `start_url` and a display mode (web.dev install criteria, read 2026-10-09)	Full binary from the store. Apple states the App Store limits the size of apps installable over a mobile connection and gives no number on that page (Apple, read 2026-10-09); Google states a compressed download limit of 200 MB for apps published as app bundles (Google, read 2026-10-09)	10 MB uncompressed on iOS 15 and earlier, 15 MB on iOS 16 and earlier, 100 MB on iOS 17 or later under conditions (Apple, read 2026-10-09)
Identity	`openid` per mini program, `unionid` shared across every app under one Open Platform account. Both come from `wx.login` plus `code2Session`, which WeChat documents as returning the UnionID “without user authorization” (login, UnionID, read 2026-10-09)	Each origin keeps its own account. The browser hands nothing to a site silently	Account per app. Sign in with Apple where the app offers it, as an explicit step by the user (Apple, read 2026-10-09)	Sign in with Apple, the keychain and CloudKit, shared with the parent app (Apple, read 2026-10-09)
Payment	WeChat Pay. A merchant number is bound to the mini program’s AppID, at most 50 AppIDs per merchant number (WeChat Pay, page updated 2025-07-02). Virtual goods must go through the mini program virtual-payment channel, where “the platform charges a technical service fee” (WeChat, read 2026-10-09)	Card and wallet APIs called by the merchant, on the merchant’s own acquiring contract	In-app purchase is required to unlock features. Apple’s guidelines name QR codes among the mechanisms that may not be used instead (Apple, read 2026-10-09)	Apple’s in-app purchase. Hosts in the Mini Apps Partner Program “earn 85% of qualifying In-App Purchase sales within qualifying mini apps” (Apple, read 2026-10-09)
Entry	Documented entry paths carry IDs: 1011 scan a QR code, 1012 long-press an image, 1013 pick a code from the album, 1023 the Android home-screen icon, 1005 and 1006 search boxes, 1026 the nearby list, 1183 search inside the PC WeChat mini program panel (WeChat scene values, read 2026-10-09)	A URL, a search result, or a home-screen shortcut the user adds by hand. On iOS there is no install prompt at all (web.dev, read 2026-10-09)	Store search and the home-screen icon	An App Clip Code, NFC tag or QR code at a physical location; Siri suggestions; Maps; Messages; a Smart App Banner on a website in Safari (Apple, read 2026-10-09)
Distribution and review	Published from the mini program console. Virtual payment requires a certified entity: an enterprise, institution or individual merchant (WeChat, read 2026-10-09)	Nothing is submitted. It is HTTPS plus a manifest	Store review against the App Review Guidelines	No listing of its own. It ships inside a full app, which the App Clip documentation says “must include the same functionality as the App Clip” (Apple, read 2026-10-09); Guideline 2.5.16(a) adds that all App Clip features must be in the main app binary and that “App Clips cannot contain advertising” (Apple, read 2026-10-09)

The Android equivalent is gone. “Starting December 2025, Instant Apps cannot be published through Google Play, and all Google Play services Instant APIs will no longer work” (Google Play Instant, last updated 2026-06-24).

Identity arrives before the first screen

The WeChat login flow is one call on the client and one on the server. wx.login returns a code, the server exchanges it through code2Session, and the reply carries an openid for that user in that mini program, a unionid and a session key (WeChat, read 2026-10-09).

The UnionID page says what this costs the user: the developer gets it “without user authorization” (WeChat, read 2026-10-09). No phone number, no password, no consent sheet. The openid is scoped to one mini program; the unionid is identical across every mobile app, website app, official account and mini program bound to the same Open Platform account, so a merchant with several surfaces sees one person.

Compare the alternatives. On the web, an origin cannot read an identifier out of the browser; it asks the user to sign in and then owns a credential to protect. On iOS, an App Clip can use Sign in with Apple and the keychain, which Apple’s documentation points developers at sharing with the full app, but those are technologies an app opts into rather than an identifier handed over at launch.

That is why the first screen of a mini program is the product. Onboarding was paid for at install time, by the host.

Payment is a binding between two accounts

Paying inside a mini program does not mean the mini program holds money. WeChat Pay binds a merchant number to the mini program’s AppID. One merchant number can bind at most 50 AppIDs, the binding starts on the merchant side and is confirmed from the mini program side, and once established cannot be unbound (WeChat Pay, page updated 2025-07-02). The rail and the code package are joined by an account relationship neither side can create alone.

Virtual goods get a second layer. The documentation is blunt: virtual currency, unlocked features, subscription content, paid services, tips and virtual gifts “require integration with Mini Program’s virtual payment system for both purchase and payment”, and the platform charges a technical service fee on the amount (WeChat, read 2026-10-09). Physical goods go through ordinary WeChat Pay; digital ones go through the host’s channel, with the host’s cut.

Apple’s version of that cut is recent and documented. The Mini Apps Partner Program was announced on 13 November 2025 (Apple, 2025-11-13): host apps carrying mini apps can join, earn 85 percent of qualifying in-app purchase sales inside those mini apps, and must use Apple’s in-app purchase system for them (Apple, read 2026-10-09). Admission is not automatic: a host needs the Advanced Commerce API and a manifest describing itself and its mini apps.

Apple’s App Review Guidelines still require in-app purchase to unlock features or functionality, and the same sentence lists QR codes among the mechanisms that cannot substitute for it (Apple, read 2026-10-09). What changed, on 1 May 2025, is that Guidelines 3.1.1, 3.1.1(a), 3.1.3 and 3.1.3(a) were updated for the United States storefront after a court decision about buttons and external links (Apple, 2025-05-01).

The QR code is the offline switch

A WeChat developer does not have to choose between a printed code and a link shared in a chat. The official scene-value table lists both, with IDs: 1011 scanning a QR code, 1012 long-pressing an image, 1013 picking one from the album, 1023 the Android home-screen icon, 1005 and 1006 the two search boxes, 1026 the nearby list, 1183 search inside the PC WeChat mini program panel (WeChat, read 2026-10-09).

Two details make the printed code cheap. Codes come from an API, and WeChat states every generated mini program code is permanently valid. The “one item, one code” interface has no quantity limit and needs a smaller printed area than an ordinary URL code (WeChat, read 2026-10-09). A code already printed and pointing at a URL can be configured to open a chosen page inside a mini program (WeChat, read 2026-10-09). The person scans what is already on the table.

App Clips have the same physical invocations: an App Clip Code, an NFC tag, a QR code at a physical location (Apple, read 2026-10-09). Two constraints pull the other way.

The first constraint trades size against physical entry. The 100 MB ceiling for iOS 17 or later carries four conditions: the App Clip “only supports digital invocations” and not “physical invocations such as App Clip Codes, QR codes, or NFC tags” (Apple, read 2026-10-09), it is used where a reliable connection is likely, and it “doesn’t support iOS 16 and earlier”. The exception Apple names is a demo link from App Store Connect, which may use the 100 MB limit and still support App Clip Codes, NFC tags and QR codes. An App Clip with a counter code stays inside the smaller limit.

The second concerns persistence. Apple’s Human Interface Guidelines say App Clips “remain on the device for a limited amount of time”, and that only App Clip Codes produced in App Store Connect or with Apple’s own code generator are approved for use (Apple, read 2026-10-09). A merchant cannot print its own.

Who reviews it, and who is answerable

Reviews differ in kind as well as in strictness. A PWA is submitted to nobody; Chrome installs it once its manifest and HTTPS satisfy the criteria (web.dev, read 2026-10-09). An App Clip cannot be submitted alone. It ships inside a full app that must already contain the same functionality, cannot carry advertising, and is reviewed as part of that app (Apple, read 2026-10-09). A mini program is submitted inside WeChat, and the entity requirement arrives with payment rather than with the code: to sell virtual items, the program must be a certified enterprise, institution or individual merchant (WeChat, read 2026-10-09).

The commercial consequence: in the App Store, a mini-app-like product reaches users through a host that already has an account relationship and a reviewed app behind it. In WeChat, the party needing the corporate identity, the certification and the merchant binding is the mini program author.

Where the four pieces sit

In WeChat, four things a mini program depends on sit with one owner: the identity system, the payment rail, the entry surface (camera, search, chat, home screen, desktop) and the distribution channel with its review. An author plugs into a host that has all four and pays for the parts it uses.

In the Apple and Google stacks those four are held by parties with different interests. The operating system vendor issues the platform account and runs the payment system; the browser owns the entry surface for web content; the stores own distribution and review; the rail for physical goods sits with the merchant’s bank relationship.

No single party there can hand all four to a third party. Apple’s answer in November 2025 was to supply the missing pieces itself and keep the commission: hosts get Apple’s in-app purchase for mini apps and keep 85 percent of qualifying sales, in exchange for approval, a manifest and Apple’s review.

The mechanism that makes mini programs work is therefore an ownership arrangement rather than a technology gap. Reproducing it outside WeChat means finding one party that holds the users, the identity, the payment rail and the entry surface, and getting it to open all four to third-party code at a price the third party will pay. In the West, the two parties that hold those pieces chose to be the host themselves.

How I checked this

Every figure and quote comes from documentation published by the vendor it describes: WeChat’s mini program and WeChat Pay docs, Apple’s developer documentation, guidelines and news items, Google’s Android documentation, and web.dev.
All pages were fetched on 2026-10-09. Most carry no publication date, so a date beside a source is my reading date unless the page states its own: the WeChat Pay AppID binding page (updated 2025-07-02), the Google Play Instant page (last updated 2026-06-24), Apple’s updated-guidelines news item (2025-05-01), Apple’s Mini Apps Partner Program news item (2025-11-13) and Tencent’s write-up of the 2022 WeChat Open Class (2022-01-07).
Size limits and scene-value IDs come from the pages that state them: WeChat’s subpackage page for 2 MB and 30 MB, Apple’s App Clip page for the 10/15/100 MB table, WeChat’s scene-value table for the ID numbers.
Google Play Instant’s page would not load in a plain HTTP client, so I read it through the Internet Archive and then through a rendering fetch that returned the live page.

What I could not check

The fee WeChat charges on virtual payments. The page says a fee is charged on the payment amount and gives no percentage.
The standard WeChat Pay rate for physical goods. I found no official rate page; what I reached describes partner fee schemes and refers to a merchant rate without the base number.
The extra steps for virtual payment on iOS. WeChat’s virtual payment page says iOS “requires additional activation and adaptation procedures” and points at separate documentation; I could not load that page, so I do not know whether an Apple-side charge is involved.
Whether an App Clip can take payment for physical goods through Apple Pay. Not stated on the App Clip pages I read.
Any current count of mini programs or mini-program daily users. Tencent’s release discloses Weixin monthly accounts and names Mini Games and Mini Shops, but carries no mini-program count. The most recent first-party figure I found: more than 450 million daily users at the end of 2021, from Tencent’s write-up of the 2022 WeChat Open Class (Tencent, 2022-01-07). That figure is four years old.
What “certified Mini Program” requires in practice, beyond the entity types named on the virtual payment page, and how long WeChat’s review takes.

Twenty hot lists, no API keys

Thu, 08 Oct 2026 00:00:00 GMT

Every night at 04:30 a script on this machine collects the current front page of 20 platforms (Zhihu, Bilibili, V2EX, Hupu, Maoyan, Hacker News, Lobsters, GitHub, Reddit, YouTube, arXiv and a few tech news feeds), translates the English headlines into Chinese, builds a static site and pushes it. There is no server and no database. The output is a set of JSON files and a single-page app that reads them.

I have no API keys for most of these platforms, and for the ones that do offer keys, I did not want to create accounts for a hobby project. That constraint produced the most useful part of the system: a catalogue of what each source actually gives you, and a record of how each one broke. This is that record. It is more useful than the feature list.

The sources that broke, and what the failure was really about

Reddit, first attempt. There was a trick where you request translate.google.com/translate?u=<reddit url> and read the translated page, which passes through Reddit’s HTML. It worked for months and then started returning a 302 to translate.goog, which serves a shell with no posts in it. The failure looked like a parser bug. It was a removed feature.

Reddit, second attempt. The official Atom feed, r/popular/hot/.rss, works and needs no key. Two traps: geo_filter=GLOBAL has to be present, or the feed comes back localised to the country your request exits from, and mine exits from a node that made the whole page German. And the feed carries no score and no comment count, so those fields are simply absent from my data rather than zero.

Reddit, third attempt, which is where it ends. Scores are available through the JSON API if you have OAuth credentials. I opened the app-creation form, submitted it, and got {"success": true} back, while the applications list stayed empty. Their documentation has since moved new legacy-Data-API apps to an application process for moderation use cases. Unauthenticated .json returns 403 from anything that looks like a datacenter address: direct, through a proxy, and with a browser TLS fingerprint that makes the request look like Chrome. Two public mirrors were dead ends as well, one of which now wants to be paid and explicitly refuses automated clients, and the other of which returns /r/popular posts from 2015.

There is a path that works: send the request with a logged-in browser’s cookies. I measured it, got a real score back, and decided not to use it. The data is not worth building the pipeline on something the platform’s terms do not allow.

arXiv. The API started answering with a 14-byte body, 429 "Rate exceeded.", for every query, from every route. Not occasional throttling: the endpoint was closed to my traffic entirely. The official RSS feeds for cs.AI, cs.LG and cs.CL serve the same listing without a key. The endpoint is still in the code as a fallback, and I expect to delete it.

36Kr. The page is entirely client-rendered, so the extractor’s basic mode returns {"results": [], "failed_results": [{"error": "Error fetching content"}]}. The old code read results[0] and raised an IndexError, which the pipeline caught as a platform failure and handled by keeping yesterday’s data. So for weeks the site showed a platform labelled “AI news” that was quietly a day or two out of date. The fix was one parameter, extract_depth: advanced. The bug was the exception handling that turned a hard failure into a silent stale one.

The night the whole thing hung

One night no platform updated. The cron entry reported a timeout after 3600 seconds and the captured output was empty.

Nothing was wrong with the network: I could see every platform’s connection being made in the proxy log at 04:30, and then no connections at all for the next 55 minutes. Nothing had reached the build step either, which I could tell from one log file’s modification time. The script was alive and doing nothing, and I had no idea where.

Two things had made that state undiagnosable. Python buffers stdout when it is not attached to a terminal, so when the process was killed, every line it had printed died with it. And urlopen(timeout=N) applies to each socket read rather than the whole request, so a slow response can stall for far longer than the timeout suggests.

The fix is unglamorous: exec </dev/null so no child can block on stdin, python3 -u so output survives a kill, a timeout around every step, a translation budget, and a step log that records a timestamp as each stage starts. The next failure will be somewhere specific in that log.

A successful job is not fresh data

This is the lesson I would keep if I could keep only one. “The task exited 0” and “the data is from today” are different claims, and a pipeline that only reports the first one will lie to you quietly for weeks. So there is a separate check: it reads every platform’s JSON, compares its date with today, and exits non-zero if any source is stale or empty. It does not care that the fetch succeeded.

The same principle shows up in the summary step. A model writes the “today’s focus” card, reading the top five items from each platform, and it is asked to return only item identifiers and a reason. The script then looks up each identifier and fills in the title and URL from the data it already has. The model never writes a link, so it cannot invent one. If the summary fails, the old one stays on the page marked stale.

Where the ordering is not what it looks like

Three caveats, since the site presents these lists as rankings:

Hupu’s “hot” list is a weighted stream rather than a sorted list of numbers. There is no score to sort by.
36Kr and TechCrunch are editorially ordered. The order is theirs, not a measurement.
V2EX’s public hot endpoint returns about nine items on a good day.

The site says which list is which on each platform page. A ranking whose method is unstated is just a list.

Reproducing this

The collector is Python, one function per platform, and the run prints a report. The interesting part to copy is not the fetching. It is the two checks around it: a freshness check that fails when data is old even though the job succeeded, and a generation step where the model returns only identifiers that the script resolves into real URLs.

Verifying an email signup end to end

Thu, 08 Oct 2026 00:00:00 GMT

This site has a signup form and a double opt-in. When you submit it, the mailing tool sends you a confirmation email, and until you click the link in that email you are not on the list.

I wanted to know whether that actually worked, so I ran the loop once, with a real address, and checked the state at each step. Here is what “checked” meant, and the failure that only showed up at the far end.

What the states look like

The tool has two states that matter: Unconfirmed and Active. A new signup lands in Unconfirmed, does not appear in the default list of subscribers, and does not count towards the group’s size. That is correct behaviour for double opt-in, and it is also the single most confusing thing about the setup: the list looks empty right after a successful signup.

So the check is not “did the count go up”. It is a sequence:

Submit the form on the live site, from the live site’s own page, and read the response. Mine came back {"success": true} and the page replaced the form with a thank-you message.
Open the subscriber record and read four fields: status Unconfirmed, the source line naming the form, the group it was added to, and the activity entry that says it was added.
Check the campaign report for the confirmation email and confirm its sent counter moved from 1 to 2. The number matters: a signup that produces no outgoing mail is a broken loop that looks like a working one.
Wait for the recipient to click the link, then read the record again and watch the status move to Active, the group count move with it, and the unconfirmed count go back to zero.

Only after step 4 is the pipeline verified. Steps 1 to 3 prove the plumbing; step 4 proves the part a reader actually performs.

The thing that failed was the inbox

The confirmation email was sent. It never arrived.

The address in question was a QQ Mail account, and QQ Mail silently drops mail from foreign senders. No bounce, no spam folder entry, nothing in the logs on my side. The mailing tool reported a successful send, and it was telling the truth.

This is worth stating plainly because it is invisible from the sending side. Every dashboard I can look at says the email left. The only way to learn otherwise is to ask the person at the other end, and a signup flow that depends on them telling you is a signup flow that will lose readers without ever showing an error. The practical fixes are ordinary ones: tell people which folder to check, use a mailbox you trust for your own tests, and expect some fraction of a list to be unreachable through no fault of your own.

Two smaller traps

The dashboard’s copy was arguing with the site. The embedded form ships with a default headline and the line “Signup for news and special offers!”, which is roughly the opposite of what this site promises. The form editor is a client-rendered app, and synthetic clicks from a scripted browser do not reach its text-editing state, so there was no clean programmatic fix. I hid those two elements in CSS and let the site’s own heading stand. That is a workaround and it is labelled as one in the stylesheet.

The status filter is not a native control. Selecting “Unconfirmed” means clicking a custom overlay, and clicking where the option was last time lands on nothing, because the menu re-renders. What works: send an Escape first in case the menu is already open, click the trigger, re-read the option’s coordinates after it opens, and then click. The confirmation that the filter is applied is a line of text reading which status is being shown, not the menu’s own state.

Cleaning up after yourself

The test created subscriber records in a real account, so the last step was deleting them, which is also the step that is easy to skip. I kept two addresses deliberately, for future sends: the one that can receive foreign mail and the one that cannot. Knowing which is which is useful the next time a confirmation email “sends successfully”.

What I would still verify

Payment. The support page has a working button and a payment page that returns 200, and I have never completed a real transaction through it. Until I do, the honest description of that button is “untested”, not “working”. The same standard applies here: the signup was verified end to end because a person on the other side clicked a link and the state changed. Anything less than that is a hope with a green tick next to it.

What this site actually ships, and the widget I can't remove

Thu, 08 Oct 2026 00:00:00 GMT

The front page of this site is a 2,130-byte HTML document containing 91 elements. I wrote 1,022 bytes of JavaScript for it, in two inline blocks, and I have no build step that could hide anything else. That is the whole page: text, a stylesheet, two typefaces.

Then you scroll to the signup form, and jQuery 3.7.1 arrives.

How I measured it

Not from the build output. Build output tells you what you shipped, not what a browser fetched. I opened the live site in headless Chrome, attached to it over the DevTools protocol, and read performance.getEntriesByType('resource') plus the navigation timing entry.

The page, from my line:

document: 2,130 bytes encoded, 91 elements in the DOM
same-origin requests: 4. Stylesheet 3,498 B, Inter 48,556 B, Newsreader 132,300 B, favicon 469 B
JavaScript I wrote: 1,022 B, in two inline <script> blocks (255 B to set the theme before first paint, 767 B for the toggle)
there is also a 309-byte inline <script type="application/ld+json">, which is data for search engines rather than code

And the signup widget, which is a third-party embed:

9 requests to MailerLite’s CDN
225,273 bytes downloaded, of which 223,312 bytes is JavaScript: jQuery 87,532 B, the widget’s own loader 52,683 B, an input-mask bundle 70,970 B, a form renderer 12,127 B
the remaining ~2 KB is two small stylesheets

So the form costs about 100 times my own HTML in JavaScript, on every page that shows it.

Two honest caveats on those numbers. Cross-origin responses report transferSize: 0 unless the server sends Timing-Allow-Origin, so the widget’s bytes came from curl against the six asset URLs, not from the browser’s own accounting. And the timings came from my machine, over a tunnel: document response started at 1,593 ms, DOM ready at 2,151 ms, and the load event at 9,526 ms. That last number is the widget too. None of these three timings are what you would see.

Why the typefaces are mine to serve

The two font files are the heaviest thing I send you, and they are the part I am most confident about. They are subsets: Inter at 48 KB, Newsreader at 132 KB, latin ranges only, no third-party request, no DNS lookup to someone else’s CDN, and a 6-month Cache-Control from my own host.

I could have linked to a font CDN instead. I decided not to, for a reason I can state precisely: I cannot verify that CDN from where my readers are. I also cannot verify it from where I am. When I benchmarked a list of hosts from this machine, Google Fonts answered in about a second, which looks like proof that nothing is blocked, except that my machine routes everything through a tunnel: a request to a foreign host leaves from a Tokyo exit node, and only requests to Chinese hosts leave from my actual ISP address. Two different addresses are answering me depending on which host I ask, and both of them are reported as “direct”.

That is the useful lesson here, and it cost me a wrong sentence earlier in this project. A test that does not tell you which way the packets left is not a reachability test. So the font files ship with the site, where the only network path involved is the one that already delivered the page.

The failure worth writing down

I styled the widget to match the site and pushed it. The button stayed black. In dark mode the email field stayed white.

MailerLite’s stylesheet writes its rules with the form’s container id and !important: #mlb2-46888919.ml-form-embedContainer ... .ml-form-embedSubmit button { background-color: #000 !important }. My override was .newsletter .ml-form-embedSubmit button, which has two classes. An id plus four classes beats two classes no matter how many !important flags I add, because !important only settles ties inside the same specificity tier.

The fix is boring: repeat the same id in my selector and add one more class, so my rule wins on specificity and then on !important. The interesting part is that the screenshot looked fine. A black button on a light page looks like a design choice. I only caught it because I stopped looking at pixels and asked the browser for computed values:

getComputedStyle(document.querySelector('.ml-embedded button[type=submit]')).backgroundColor
// "rgb(0, 0, 0)"  -> my stylesheet was not applying at all

Both rounds after that, light and dark, I asserted on computed colours, and kept the screenshots for layout only. Pixels are for judging whether something looks right. They are a bad instrument for finding out whether your CSS is in the cascade.

What I would do about the 220 KB

Defer it. The widget only has to exist when a reader scrolls to it or clicks it, and the page can load the embed then instead of in the <head>. That is the next change I want to make here, and when it is done I will re-run the same measurement and put both numbers in this post, including the case where it does not help.

Until then the honest statement is: this site has no framework, no bundler, no client-side router, no analytics, and about a kilobyte of JavaScript that I wrote. The one third-party script on it is a newsletter form, and it is heavier than everything else on the page put together.

Reproducing this

The measurement is a CDP script: open the page in headless Chrome, enable the Network domain, read performance.getEntriesByType('resource'), then curl -o /dev/null -w '%{size_download}' the asset URLs to get the cross-origin bytes the Resource Timing API refuses to report. If you want to compare your own site, the part that matters is the last step. Browser-side byte counts for third-party assets are usually 0, and it is easy to read that as “the widget is free”.

What goes on this site, and what does not

Wed, 07 Oct 2026 00:00:00 GMT

Most writing about AI tooling is written by people who watched a demo. This site is written by someone who runs the same task book against five tools, then reads the 39 acceptance checks line by line to find out which ones passed for the wrong reason.

That difference is the whole point, and it comes with a cost: it is slower. So the rules here.

One long post every two weeks, not three short ones a week. Each post has to contain something a reader can re-run, copy, or argue against — a task book, a scoring rule, a failure taxonomy, the script that computed the number.

The three posts published alongside this one are an opening batch. The fortnightly rhythm starts after them.

Numbers come from scripts, not from impressions. If a post says a tool passed 39 out of 39, there is a script that produced that 39, and I will say what it could not see. Where I have not verified something, the post says so in those words.

No employer internals. The method is mine to publish; the data is not. Anything under an NDA, any customer ticket, and any internal number stays out.

If it breaks on the third run, the post says so. The failure mode is usually the most useful sentence in the whole piece.

If that sounds like the kind of thing you want in your inbox, the signup form is on the front page. Until then, the posts page is where everything lands.