Fitting the examples is not enough: the generalization challenge in F3-AGI v4.0

1. Two examples can support incompatible rules

Imagine a system observing two pairs: 1 → 2 and 2 → 4. “Double the input” seems like a natural rule. Yet the function x² − x + 2 produces exactly the same two answers. For input 3, the first returns 6; the second, 8. Both explain what was shown and disagree about what has not yet been observed.

This arithmetic example was constructed for this article. It is neither an ARC task nor an F3-AGI result. It isolates a fundamental problem: with few demonstrations, a perfect fit to the available data does not, by itself, determine how to generalize.

A third demonstration, 3 → 6, would rule out hypothesis B. This still would not make rule A the only possible function: other expressions could match all three points and diverge later. The system needs additional selection criteria, such as constraints on the type of transformation, concept composition and simplicity relative to its vocabulary.

This choice is a modeling trade-off. Preferring short programs can be useful, but the shortest program depends on the available operations. A “short” program in a library with a specialized primitive may be “long” in another. Expression length alone does not certify that it captures the intended rule.

2. What ARC adds to the problem

In ARC-AGI-1, each task provides input–output grids as demonstrations and new inputs for testing. Constructing a solution includes choosing the output dimensions and filling every cell correctly. The official repository separates 400 training tasks from 400 evaluation tasks and advises against using the latter to adjust the algorithm during development. Its README describes three attempts in the original interface; F3-AGI separately reports pass@1 and pass@2 under its own local evaluation protocol. [1]

Two meanings of “training” need to be distinguished. A task’s demonstrations are legitimate evidence for inferring its transformation during solving. The system’s development set, meanwhile, informs operators, hyperparameters and search strategies. If examples from a task manually influence those decisions, that task participates in development even if its test outputs remain hidden.

In On the Measure of Intelligence, François Chollet distinguishes demonstrated task skill from the efficiency of acquiring new skills, emphasizing prior knowledge and experience. This perspective suggests a useful question when reading results: how much competence came from a rule inferred at solving time, and how much was already built into the system? [2]

This does not require eliminating all prior knowledge. A system needs a representation to recognize objects, relations and possible transformations. What matters is making that repertoire visible and controlling which information entered it. Otherwise, a score improvement can mix progress in inference with adaptation to the reference set.

3. Programs as hypotheses: strengths and limits

A library of symbolic operations makes it possible to express hypotheses as verifiable compositions. Beyond producing a final grid, a system can represent a transformation as a sequence: identify components, select an object, apply a spatial relation and construct the output. This sequence makes some decisions easier to inspect.

Lake and colleagues argue that composition, causal models and learning to learn are relevant directions for systems that acquire knowledge from few examples. Their argument helps situate the interest in structured representations; it does not validate F3-AGI or demonstrate that any symbolic architecture achieves human learning. [3]

DreamCoder provides a concrete example of another line of program-learning research: it combines a language of concepts with networks that guide search and a cycle that expands reusable abstractions. The connection here is conceptual: the hypothesis language can itself be learned. We conduct no experimental comparison between DreamCoder and F3-AGI in this essay. [4]

The library’s advantage comes with a boundary: the system can explore only what it can express or construct from its operations. If the necessary concept is absent and cannot be discovered through the available compositions, additional search may repeatedly traverse an unsuitable space.

This creates two distinct failure modes for developers. In one, an expressible solution exists, but search cannot find it within budget. In the other, the intended solution cannot be expressed in the current repertoire. More runtime may help the first; the second requires changing the representation or learning new abstractions. Useful diagnosis should distinguish these cases.

4. How F3-AGI organizes search

In the static study, F3-AGI v4.0 combines perceptual invariants, a parametric library of 30 operators, early rejection of incompatible candidates and a residual patch synthesis mechanism. The reported execution is offline, on CPU and without LLM calls. The full description is available in the institutional research record and preprint. [5]

Checking the smallest demonstration pair aims to reduce computation before evaluating candidates on every pair. The benefit is operational: spend less time on incompatible hypotheses. It does not make a surviving hypothesis correct for every future input.

The selection criterion warrants similar care. Structural and complexity constraints can reduce spurious candidates but cannot remove insufficient evidence. Final selection must still choose between programs that may coincide on demonstrations. Test performance is where that choice is examined.

The preprint’s three studies must also remain distinct. Continual learning uses a synthetic stream; static evaluation uses public ARC-AGI-1 tasks; interactive exploration uses local games inspired by ARC-AGI-3. A result from one study should not automatically be transferred to another. Here, our focus is generalization in the static study.

5. Why 57/400 needs to appear alongside 19/50 and 38/350

The aggregate pass@2 result of 14.25% correctly describes the 400 evaluated tasks. However, it combines two subsets with different methodological histories. Demonstrations from the first 50 tasks participated in library development: the manuscript reports ten new operators built after inspecting cases from this subset. The author describes the remaining 350 tasks as strictly uninspected. [5]

The 38% figure shows that the developed repertoire solves some tasks that helped guide its design. The 10.86% figure asks a different question: how much does that repertoire transfer to tasks that, according to the reported protocol, were not inspected? Publishing both keeps a development measure from standing in for transfer.

The difference between rates does not, by itself, identify a single cause. These subsets are not a paired comparison of the same population under two controlled conditions. Task composition, operator vocabulary and development decisions can all contribute. Treating the gap as a causal estimate of “how much the system memorized” would exceed the available evidence.

Public documentation also differs from independent audit. The repository provides per-task results for inspecting the counts, but this essay is written by the study’s author. Verifying the absence of prior inspection and reproducing results in new contexts require further evidence. A stronger next evaluation would freeze the version and test a set that has not informed new decisions. [6]

6. When 99% of pixels still means a failed task

Consider another hypothetical illustration: a grid with 100 cells, 99 background cells and one colored cell that must be placed correctly. A program returning only the background could match 99% of cells while completely missing the relation defining the task. Pixel accuracy recognizes visual overlap but does not certify the intended transformation.

ARC’s exact-success criterion requires the correct grid, including its dimensions. The F3-AGI preprint reports 57/400 exact pass@2 solutions and, under a different measure, 128/400 tasks with at least 90% test pixel accuracy. These indicators are not interchangeable and should not be added together. The latter can help locate failures; the former establishes which tasks were actually solved under the protocol. [1] [5]

Simplicity-based selection also encounters a limit here. “Fill everything with the background” is a very short program. If the fit criterion primarily rewards the dominant background, an additional preference for short programs may favor precisely this degenerate solution. Representation, fit and complexity penalties need to be examined together.

The difference between pass@1 and pass@2 adds another observation. On the full set, the second attempt produced just one additional exact solution. The manuscript acknowledges that this rescue came from incidental candidate diversity, rather than systematic calibration of the second position. An extra attempt’s gain should not be presented as evidence of reliable hypothesis ranking. [5]

A more useful error analysis preserves these distinctions: wrong dimensions, wrong object, wrong spatial relation, a correct global transformation with an incorrect local detail, or a suitable candidate ranked poorly. A single mean accuracy hides which problems dominate the system.

7. Does a patch repair the error or learn an exception?

Residual correction aims to complete a global transformation already close to the solution. The study attributes seven solutions in the uninspected subset to Cycle-3. This is relevant evidence for investigating local repair, but the count alone does not tell us whether the correction captured a general relation or a narrower regularity. [5]

A patch hypothesis can fit every demonstration while relying on incidental details: an object’s size, a boundary’s position or a particular combination of elements. The initial ambiguity returns within a repair step.

We propose that a future evaluation record the program before patching, the correction applied and its behavior when the same relation appears in another configuration. The unit of analysis would be the pair “global transformation + patch,” preserving successes and regressions. This protocol is not yet a result of the study.

Counterfactual tests can help if they respect task semantics. Consistently renaming colors may be valid when a rule depends only on relations between colors; it may be invalid when a particular color has its own meaning. Rotating an input does not automatically preserve a task either. Before testing invariance, we must justify which relation should remain unchanged and what output is expected.

These tests do not replace new tasks, but they may reveal hidden dependencies in an apparently correct explanation. Their value lies in formulating ways to refute a hypothesis instead of accumulating only favorable examples.

8. Turning the discussion into a stronger evaluation

Publishing code makes inspection possible. Turning openness into cumulative evidence also requires a protocol that limits decisions made after observing results. The following roadmap proposes future rounds; it does not claim that F3-AGI has already completed every step.

1. Freeze the artifact.

   Record the commit, library version, dependencies, parameters and execution budget before opening the new set. Treat any change as a separate experimental condition.

2. Separate information channels.

   Distinguish tasks used to build the system, tasks used to select parameters and tasks reserved for evaluation. Record human inspection as well as automated training.

3. Separate inference from scoring.

   Generate predictions from demonstrations and test inputs. Only then compare them with expected outputs in an evaluation stage that does not modify the candidate program.

4. Make the budget explicit.

   Maintain comparable conditions for attempts, time, memory, search and external calls. Publish what happened when the budget expired, including unanswered tasks.

5. Preserve per-task evidence.

   Record selected candidates, produced output, exact success, error diagnosis and time. An aggregate spreadsheet cannot replace the history needed to understand each failure.

6. Test the robustness of conclusions.

   Run additional seeds for stochastic components. Repeat timing measurements, document hardware and investigate whether an improvement persists on new sets and in suitable ablations.

The current version offers a concrete starting point: an identified release, implementation, configurations and public records. These materials support reproducible questions. Release v4.0.0 and the software DOI identify the artifact; they do not replace peer review or independent confirmation. [7]

A future comparison with another solver would also need to declare the evaluated population and use the same budget. Juxtaposing rates from different task sets, or comparing a local run without LLMs to a system given a different amount of computation, is insufficient. A useful result describes which capabilities change under comparable conditions.

9. The next advance must withstand a new question

F3-AGI v4.0 provides an inspectable architecture and initial results in bounded contexts. For this discussion, its most productive contribution is making the connection between vocabulary, search and evaluation visible: a library can narrow the hypothesis space, but its performance depends both on what it expresses and on how it was built.

The next round should investigate at least three directions: transfer to tasks that did not guide development; distinguishing search failure from representational limits; and the stability of residual corrections under semantically justified changes. These are research questions, rather than results assumed in advance.

For the community, we propose treating a candidate solution as a refutable explanation. What evidence supports it? Under what conditions should it fail? What would justify abandoning the operator, patch or ranking used? These questions help turn a good demonstration into a research program.

The materials are available for this discussion. Our F3-AGI institutional research page brings together the scientific summary, preprint, software DOI and code. Comments on failure cases and reproductions are particularly useful when they include the version, protocol and observed outputs.