On 10 November 2026, Daniel Lenton of unify posted a preprint, Continual-ARC: Measuring Skill Acquisition over a Working Lifetime, on arXiv (2611.14505). The paper is marked under review. It puts agent harnesses, which keep notes and skills, and methods that update a model's weights on the same scale, using ARC puzzles that return with fresh inputs. A companion blog post dated 11 November 2026, "Why we built Continual-ARC," covers the same design. The project site lists Lenton with Yusha Arif and Haris Mahmood.
This is a self-published preprint. No peer review or independent replication is reported. Lenton writes that a result is verified by rerunning the learner on 50 private puzzles.
What ARC is, and what continual means here
ARC, introduced by Francois Chollet in 2019, is a set of grid puzzles. Each puzzle shows a few example grids, each an input and its output, and the solver has to infer the rule and apply it to one more input. The puzzles are built to need only core priors such as objectness and counting, not specialist knowledge.
Standard ARC serves a puzzle once. Continual-ARC is a lifelong stream built from the three ARC-AGI editions. The same tasks return after dormant gaps, each time with a fresh instance, so a cached answer does not fit. New tasks keep arriving through the stream.
The results below are from the ARC-AGI-1 track at seed 0, which Lenton ran. The stream uses 25 of that edition's 400 training tasks, eight instances each, for 200 rounds at one user. Fresh instances come from Michael Hodel's Re-ARC generators. The blog says four puzzles are active at a time, and that the last new puzzle arrives in round 97 of 200. After the stream, the learner is frozen and given two tests of 100 instances each. The seen test, which measures memorization, serves four fresh instances of each of the stream's 25 tasks. The held-out test, which measures generalization, serves 100 tasks from ARC-AGI-1's evaluation set that the stream never shows. Each learner and its reset play each test four times, with the learned state held fixed. The paper and the blog both note that these repetitions do not measure variation across independent learning runs.
How a visit is scored
On each instance the learner may pay for a demonstration, submit an answer, or act inside a harness. A wrong answer costs as much as a demonstration: one unit of a budget of 16. A visit can ask for up to eight demonstrations. It ends on a right answer or the eighth wrong one. An unsolved instance costs the full 16. The score is the share of this feedback budget used. Lower is better.
A raw score would mainly measure how strong the model already is. Lenton therefore reads each learner against a reset: the same system on the same stream and the same tests, with its state wiped before each instance. The share learned on the stream, L, is the budget saved as a fraction of the most any memory can save (the reset's spend minus a perfect-memory spend that pays on first exposure and nothing on returns). 0 means the learner keeps nothing. 1 means it answers every returning task at no cost. A negative value means it does worse than the reset. The share saved on the seen test, Sseen, and on the held-out test, Sheld, is the share of the reset's spend on that test that the learner no longer pays. 0 is no change. 1 is the test for free.
What was tested, and what weight training means
Lenton ran Qwen3.5-9B as the student (Q) and GLM-5.3 Flash as the teacher (G), plus GPT-6 Luna at low, medium, and high reasoning effort. The main harness for Qwen and GLM is prime-agent (Karten et al., 2026). On GPT-6 Luna he also ran hermes-agent (Nous Research, 2026) and openclaw (OpenClaw contributors, 2026), each with persistence on as shipped (H) and with that state wiped (H0).
Two prompting floors sit beside the harnesses. Stateless (C0) has no memory between instances. Demo-cache (C) stores paid demonstrations and replays them for free later, without learning the rule. A program-library reference (P) asks the model for a Python program, stores it if it reproduces the demonstrations, and runs it first when the task returns.
In QT, QTE, and QETE, GLM plays the stream in prime-agent and writes the notes, and Qwen then plays from the teacher's state. QETE gives expert guidance, a procedure that tells the system to write, check, save, and reuse one program per task, to both the teacher and the student. A separate prompt (A, or H-A on GPT-6 Luna) asks the harness, after every verdict, to save liberally anything that could help on later instances.
The weight-training method, QDE, is not the 9B model training only on its own outputs. Between rounds, still inside prime-agent, DAgger updates a LoRA adapter of Qwen3.5-9B so that it imitates GLM-5.3 Flash's replies at states the student met. The teacher reads expert guidance that is withheld from the student. The paper also reports policy-gradient and self-distillation variants as unsuccessful (Appendix H). DAgger also ran in a collective regime of two or four users, each with 25 non-overlapping tasks and one shared state.
Results on Qwen3.5-9B at one user
All figures below are on the ARC-AGI-1 track at seed 0, against each method's reset, with 95% intervals over tasks.
With notes in prime-agent as shipped (Q), L was -0.02 [-0.09, 0.04], Sseen was -0.01 [-0.04, 0.03], and Sheld was -0.01 [-0.05, 0.03]. The paper states: "Q's persistence saved nothing measurable against its reset." Asking Qwen to store liberally (QA) gave Sseen of +0.02 [-0.02, 0.07] and Sheld of 0.00 [-0.05, 0.04]. Expert guidance alone (QE) gave L of +0.05 [-0.06, 0.16], Sseen of +0.03 [-0.08, 0.13], and Sheld of -0.01 [-0.05, 0.03].
When the teacher wrote the notes without guidance (QT), leaving 106 notes and files, Sseen was +0.06 [-0.01, 0.14] and Sheld was +0.03 [-0.02, 0.07]. With guidance only for the teacher (QTE), which left 104 notes and files, Sseen was +0.01 [-0.05, 0.07] and Sheld was -0.02 [-0.07, 0.03]. Other work on what agents keep, including WikiTrace and Memory Tax, also studies stored lessons. Here, those teacher-written notes did not produce a measurable held-out gain.
Only when both were given guidance (QETE) did Qwen memorize: L was +0.29 [0.19, 0.40] and Sseen was +0.34 [0.26, 0.42], but Sheld was -0.05 [-0.10, 0.00]. The paper says GLM solved 98% of the instances, which Qwen failed to put to use on the held-out test.
DAgger (QDE) at one user reached L of +0.43 [0.33, 0.54], Sseen of +0.57 [0.48, 0.67], and Sheld of +0.29 [0.23, 0.35]. In absolute terms (Table 10), the seen-test budget used fell to 36.0% from 84.0% for the reset, the held-out budget to 60.4% from 84.7%, and the held-out solve rate rose to 49.0% from 19.2%. The 11 November blog says the trained weights "saved more than half of what the untrained model spent on the puzzles it had met, and more than a quarter on the 100 new ones." At two users, Sheld was +0.40 [0.34, 0.47]. At four users it was +0.39 [0.33, 0.46]. The paper says, "Seemingly, DAgger reached diminishing returns for generalization beyond two users."
The program library (QP) had the strongest seen-test saving among these Qwen methods, Sseen of +0.65 [0.46, 0.86], but Sheld of +0.03 [-0.02, 0.09], which the interval does not separate from zero. The demo-cache floor (QC) saved Sseen of +0.37 [0.26, 0.51] and Sheld of -0.01 [-0.05, 0.03], the saving from not paying for the same demonstrations twice. Untrained Qwen in prime-agent (Q0) solved 18 of the 100 held-out tasks, against 31 in the stateless loop (QC0). DAgger solved 47 to 52. The paper says part of what DAgger learned was to drive the harness, and that it also "learned something fundamental about ARC itself."
Results on GLM-5.3 Flash and on three harnesses with GPT-6 Luna
For GLM-5.3 Flash at one user, Table 1 reports that notes as shipped (G) had L of +0.12 [-0.16, 0.36], Sseen of +0.12 [-0.03, 0.27], and Sheld of 0.00 [-0.11, 0.10]. That Sseen interval includes zero. The Results section separately states that "G's saved a bit, with Sseen = 0.25 [0.17, 0.33] over three runs." Asking G to store (GA) gave Sseen of +0.26 [0.13, 0.39]. With expert guidance (GE), Sseen was +0.74 [0.64, 0.84] and Sheld was +0.07 [-0.02, 0.16]. The program library (GP) reached Sseen of +0.81 [0.59, 1.00]. The demo-cache (GC) reached Sseen of +0.44 [0.34, 0.57]. The paper says, "Neither harness (Q, G) beat its demo-cache floor (QC, GC)."
Even G, "which solved nearly every instance, still spent 13.6% of the budget per solved instance after every task had been met, against 17.5% for G0" (Figure 4). Appendix E.2 says G kept a rule in words for every task, but prime-agent showed six entries per instance, chosen before the task was known, so the rule reached G on only 40 of 154 returns.
On GPT-6 Luna at high effort, Table 12 reports the share saved against each method's own reset. As shipped (H) on the seen test, hermes-agent had +0.03 [-0.05, +0.10], openclaw had -0.04 [-0.12, +0.03], and prime-agent had -0.05 [-0.17, +0.04]. The paper says the harnesses as shipped did not save more than chance on the seen test. Asked to store (H-A), prime-agent reached Sseen of +0.42 [0.29, 0.55], while hermes-agent's +0.18 [-0.01, +0.39] and openclaw's +0.07 [-0.02, +0.17] were "within noise of nothing." With expert guidance (H-E), the three harnesses saved 0.62 to 0.72 on the seen test: hermes-agent +0.62 [0.45, 0.81], openclaw +0.64 [0.44, 0.82], and prime-agent +0.72 [0.63, 0.82].
Appendix E reports that, with GPT-6 Luna driving, prime-agent kept an entry for later sessions only when the agent saved it as global, and the agent never did; hermes-agent loaded its skills on every return, but they held general advice which did not affect spend; and openclaw never promoted a memory, so an earlier correct answer never came back on any of its returns. On the held-out test, "none of the 30 readings of the methods that keep something saved more than chance once each effort's eleven learners are judged together."
The paper's summary is: "Left to their own prompting, only G saved measurably on either test across three harnesses, four models, and GPT-6 Luna's three efforts." No harness-based method in the reported grid gained on unseen tasks. Weight-based learning did.
Limits the sources state
In a section titled "Limitations and what's next," Lenton's 11 November 2026 blog post states that the stream comes from ARC-AGI-1's public training set, which every model developer knows, and that a model may know some of these rules already; that notes and weights are compared on one open student, with one teacher and one seed; that none of the models is the strongest available, and a stronger one might keep notes that answer a returning puzzle; and that the stream's answers and Re-ARC's solvers are public, so a learner could bring them along and score without learning. The post says they verify a result by rerunning it on 50 private puzzles.
The paper's Discussion says, "How well they learn in other domains remains open." It also states: "One weakness of this study is that most of its runs used relatively small models on ARC-AGI-1, the easiest of our three tracks." DAgger, it adds, "needs a stronger teacher whose reasoning tokens are available." Early results on ARC-AGI-2 and ARC-AGI-3 are in Appendix K, left open for later methods. The ethics statement reports that prime-agent's Python skills, refine among them, did not load in these runs. The benchmark code and run records were reachable at their GitHub URLs on 12 November 2026.
Sources
- Daniel Lenton, "Continual-ARC: Measuring Skill Acquisition over a Working Lifetime," arXiv 2611.14505, 10 November 2026, preprint under review: https://arxiv.org/abs/2611.14505
- Dan Lenton, "Why we built Continual-ARC," unify, 11 November 2026: https://www.unify.ai/blog/why-we-built-continual-arc
- Continual-ARC site, listing Daniel Lenton, Yusha Arif and Haris Mahmood: https://continualarc.org/
- Benchmark code, checked 12 November 2026: https://github.com/unifyai/continual-arc
- Run records, checked 12 November 2026: https://github.com/unifyai/continual-arc-runs
