In F3-Swarm’s post-hoc audit, functions preserved verbatim in logs passed 27 of 28 unseen cases, but only 6 of 7 tasks passed every case. Including a reconstructed function changes the denominators. The distinction is central to interpreting the 28/30 aggregate and designing better evaluations.
1. Passing examples is evidence, not a specification
Two functions can return the same answers for four inputs and disagree on the fifth. The first may implement the intended rule; the second may recognize only the format or an accidental pattern in the examples. A verifier cannot distinguish them if its tests do not include an input on which they diverge. This multiplicity of solutions compatible with available evidence is the central issue.
F3-Swarm used four development tests per code task to accept, repair or escalate a solution. They are useful as an initial gate and a source of counterexamples. Once input, expected output and actual output guide repair, that case is part of development. It remains useful for regression testing but no longer serves as independent generalization evidence.
2. What a boundary case reveals
In T27, parsing Brazilian dates, one solution used the standard-library date parser while another recognized the format with a regular expression. Both passed visible examples after an execution-environment adjustment. Nonexistent dates such as February 29, 2025 and April 31, 2026 revealed the difference: recognizing DD/MM/YYYY does not validate the calendar. In the audit, the 0.5B candidate passed 4/4; the 1.5B candidate passed 2/4.
In T25, sanitizing a PIX payment key, code approved in development extracted digits and treated inputs containing eleven as Brazilian tax identifiers. An alphanumeric random key containing eleven digits exposed the defect: stripping letters destroys part of the identifier. The candidate passed three of four unseen cases, but the task failed. The example shows the value of testing inputs that cross category boundaries.
3. Choose cases that separate hypotheses
When two implementations pass the known examples, ask where their hypotheses diverge. For dates, vary calendar rules, month boundaries and leap years in addition to string shape. For identifiers, test inputs whose superficial appearance resembles another class. For bracket rules, use the exact threshold, a value below and one above. Every case should have a reason to be there.
Relations between inputs and outputs can supplement isolated expected values. For instance, valid normalization may need to preserve the result when applied twice; adding outer spaces may leave an identifier’s interpretation unchanged if the specification allows it. These properties are test hypotheses: verify that they belong to the contract first. More tests do not help if they repeat the same distinction or impose a rule the task does not require.
4. The expected output can be wrong too
A test compares a function’s behavior against an oracle: an expected rule or value. If that reference is wrong, a correct implementation can fail. In the modulo-11 task (T26), the preprint documents two incorrect expected values in the original tests. Patched P5b corrected the oracle and added an algorithm hint to the prompt. It would be easy to attribute the whole gain to the model or hint if these changes appeared as one result row.
The audit recovered the logged 3B code from before the hint and evaluated it on four unseen inputs. It passed all four. This is evidence about that original function on those cases; it does not show every attempt without a hint will find the same solution. When a reference changes, preserve the old one, document why, and distinguish rescoring the same code from generating a new candidate.
5. Logged code and reconstruction are different evidence
For seven tasks, evaluated code could be recovered verbatim from logs. For T29, the log recorded the outcome but did not serialize the selected function’s code. The audit includes a reconstruction separately. The distinction matters because reproducing expected behavior with reconstructed code does not prove it was the function that ran the first time.
| Subset | Cases passed | Tasks with 4/4 |
|---|---|---|
| Verbatim code in logs | 27/28 · 96.4% | 6/7 · 85.7% |
| Includes reconstructed T29 | 31/32 · 96.9% | 7/8 · 87.5% |
Explore Table 4
Choose an audit subset
This display uses the published totals; it does not run the functions.
The totals answer different questions. 27/28 counts individual cases in the verbatim subset; 6/7 counts tasks passing all four cases. With T29 those totals become 31/32 and 7/8. A single failure carries different weight in the two denominators, and four cases for one function are not four independent tasks. Case-level rates cannot replace task-level rates.
6. How to interpret 29/30 and 28/30
P5b’s 29/30 (96.7%) is development-set performance. The adjusted 28/30 (93.3%) adds 11/12 triage and 10/10 extraction results from the original set to 7/8 functions passing the post-hoc audit, including reconstructed T29. Only the code component was evaluated on unseen cases. The precise phrasing is “aggregate adjusted by the code audit.”
The reported Wilson interval for 28/30 is 78.7%–98.2%. It conveys the scale of numerical uncertainty, but does not fix design choices: cases created after development, classes evaluated differently and reliance on a reconstruction. The audit tests whether selected functions withstand new cases; it does not measure performance on an external set of thirty unseen tasks.
7. Unseen by the function does not mean specified in advance
Audit cases were designed after Labs 01–05. The preserved functions already existed and were not repaired using these new cases for the reported measurement. This makes the audit informative about weaknesses of selected candidates. At the same time, post-hoc case selection is neither a random sample of future use nor an evaluation protocol frozen before development.
In a next round, specify case families and scoring rules in advance, keep data hidden and reserve an external set of new tasks. Report exploratory results, prespecified evaluation results and post-repair regressions separately. This keeps a number reused in different contexts from answering questions the test was never designed to address.
8. How to design a stronger audit
Before generating code, record the rule each task should implement and decide which results are acceptable. Write ordinary and boundary cases; review expected outputs against an independent reference. Reserve a hidden set unavailable to both the model and the repair team. Whenever a hidden case goes back into the prompt, move it to the regression suite and refresh the set measuring generalization.
Preserve the exact selected code, its hash, model version, prompt, configuration and per-test result. The trail should support re-evaluating the same artifact without generating another response. Record code changes and oracle changes separately. When data is missing, declare the observed subset and report reconstructions on their own line.
9. What to publish alongside the percentage
A reproducible report starts with the question: is the unit of success a case, a function, a task or an entire workflow? Then report how many candidates were generated and passed development, which were eligible for audit and why any were excluded. In tables, place numerator and denominator alongside the percentage. Give every test and reference version an identifier.
Where possible, also publish failing counterexamples, function provenance and the configuration needed to repeat scoring. A “verbatim,” “reconstructed” or “unavailable” column helps readers separate direct evidence from interpretation. If results disagree, another researcher should be able to identify whether the disagreement lies in the artifact, oracle or counting rule.
10. Functional correctness is not execution isolation
A function can return the correct output and still act outside its contract: consume excessive resources, read files or attempt network access. The four tests observe outputs; the prototype harness does not provide security isolation for untrusted code. Using one Python namespace addresses a lab execution issue; it does not constrain process permissions.
If the decision is to run generated code, also evaluate CPU, memory and time limits, file and network access, and stopping criteria. Assess security separately from accuracy. The study’s lesson is to treat acceptance as a decision supported by identifiable evidence: development examples, unseen cases and execution conditions answer different questions.
Questions to keep exploring
Do 27/28 and 6/7 contradict each other?
No. One counts cases; the other requires four correct cases to pass each task.
Is the 28/30 aggregate entirely out of sample?
No. It combines original triage and extraction results with the function audit.
Why present T29 separately?
The exact code was not serialized in the log. The reconstruction has different provenance.
Does passing all cases guarantee safety?
No. Functional accuracy and execution isolation need separate evaluations.
Sources and references
The sources below document the experiments. Editorial examples are identified separately from measurements.
- Polybee AI ResearchF3-Swarm v1.5.2 · technical preprint
Manuscript, experimental design and limitations.
- ZenodoF3-Swarm v1.5.2 · DOI 10.5281/zenodo.24095260
Persistent research record.
- Polybee AI Researchreproduce_all_results.py
Script documenting subsets, counting rules and the post-hoc audit.
- Polybee AI Researchprecise_tier3_results.json
Record of the specialist code before the patch.
- Polybee AI Researchlab04_aacr_results.json
Cascade records, including the provenance of T29 data.