A trading system takes shape long before its first order. An idea starts the work; decisions give it structure. Which data should it use? What job belongs to the model? How will you measure risk, and where can AI agents help? Learning to connect those decisions gives you a skill you can carry into your next project.
The research behind Quantic Eagle began before the company was formed. Roughly ten years of experiments, development and decisions have produced practical knowledge about organising research, comparing alternatives and making completed work reusable. Before Capital draws on those lessons.
If we were starting again, we would want a map of the work: what to define first, what to build in stages and how to give AI agents useful, manageable assignments. This edition brings selected experience into visual cases, procedures, worksheets and tools you can adapt to your own lab. That research journey is separate from the company’s age and from any live performance record.
From data to decisions: what you need to be able to reconstruct
A model is one part of the system. This teaching map connects each stage to a possible failure, a check and a record to retain. It does not describe Quantic Eagle’s proprietary architecture.
1 Data available at the time
- Risk
- Using revisions or information that arrived after the decision.
- Check
- Compare the observation time with the availability time.
- Keep
- Sources, versions and availability timestamps.
2 Features and outcomes
- Risk
- Building a feature with future information or using an outcome not yet known.
- Check
- Define each feature window and when each outcome ends.
- Keep
- Feature definitions and label intervals.
3 Time-based splits
- Risk
- Mixing past and future or overlapping outcomes across training and evaluation.
- Check
- Set the boundaries and check exclusions before fitting.
- Keep
- Split membership and reasons for exclusions.
4 Training and selection
- Risk
- Choosing models, thresholds or transformations after seeing the final test.
- Check
- Compare a baseline and alternatives using only data reserved for those choices.
- Keep
- Trials, parameters, transformations and selection criteria.
5 Portfolio and execution simulation
- Risk
- Turning identical scores into different orders or leaving out costs.
- Check
- Trace selection, sizing, constraints and costs through the same decision path.
- Keep
- Decision traces and execution assumptions.
6 Equity and observable risk
- Risk
- Treating balance as equity or a deposit as a return.
- Check
- Account for open positions, sampling frequency and external cash flows.
- Keep
- Series, adjustment method and included or excluded costs.
7 Held-out evaluation and stress
- Risk
- Treating a test already consulted during selection as independent evidence.
- Check
- Evaluate the fixed candidate and record every analysis consulted.
- Keep
- OOS results, stress tests and selection history.
8 Decision and new observations
- Risk
- Changing the system without separating earlier and later versions.
- Check
- Document the version, limitations and next evaluation conditions.
- Keep
- A dated decision and a separate prospective record.
The arrows show research dependencies, not permission to trade. Fit preprocessing and parameter estimates only on the permitted data. OOF means that a prediction comes from a model not trained on that fold; this alone does not establish temporal validity. An OOS period repeatedly used to make choices is no longer an untouched final test.
Where are you starting from?
A practical fieldbook for people developing a quantitative project who already know the basic ideas of backtesting, risk and return.
To read the fieldbook, you need a browser and familiarity with the basic concepts. For the Python exercises, you need to work with CSV files and use a terminal. If backtests, drawdown or training data are new to you, introductory study should come first.
The distinctive material is a selection of cases, decisions and practices drawn from internal research: useful experiments, abandoned approaches, corrections and findings that changed the design. The fieldbook explains the transferable reasoning and gives you ways to practise it.
Six decisions to start with
Shape the idea. Define the behaviour you want to investigate and choose a simple reference for comparison. A well-formed question also makes the first AI assignment more useful.
Design around the data. Choose the universe, sources and availability times around the decision your system will make. Your dataset becomes an explicit part of the design.
Give each model a purpose. Start with a comparison you can explain. Add complexity when you can say what contribution you are looking for and how to measure it on consistent terms.
Build risk into the research plan. Decide how to observe equity, costs and open positions. Read the economic result alongside the path taken to reach it.
Organise the AI work. Set bounded tasks, inputs and delivery criteria. Measure completed work and review time: both help you decide what is worth automating.
Prepare the next experiment. Preserve an identifiable version, record your decisions and decide what to observe next. Each experiment can then become a foundation for the next one.
The complete fieldbook connects these choices to chapters, worksheets and workshops. Here, three cases take you inside the lab: measuring risk, selecting models and understanding data quality. You can already try part of the method with the free CSV check, locally and without providing an email address.
The curve that left something out
Imagine an index that starts at 100 and ends at 104. Closed trades never take it backwards. Reported drawdown is zero.
Now include open positions in the same observations. Equity reaches 101, falls to 87 and finishes at 104. Observed drawdown is 13.86%. The endpoint is unchanged; the path reveals something else.
Two curves. The same endpoint.
This is a synthetic example, not Quantic Eagle performance. It shows why a metric's definition matters as much as its value. In practice, you need to establish observation frequency, costs, cash flows and coverage of open positions. Daily observations can miss an intraday low.
Start by asking whether your report measures the risk you want to understand or only the risk its available data can describe.
The interesting candidate missed the first shortlist
In an internal comparison covering the first 192 backtests of a campaign, nine of the ten candidates ranked highest on the economic measure were outside the upper proxy quartile of the reference population. Rank correlation between the preliminary proxy and the economic ranking was about 0.27.
Who would have missed the shortlist?
This was a selected cohort in adaptive research. It does not establish a universal rule, a superior model family or future returns. It raises a practical question: how much useful work might your preliminary filter exclude?
A proxy can measure one part of the problem. A portfolio also involves costs, concurrent positions, risk and concentration. Comparing both rankings under consistent conditions makes the difference visible.
Before choosing a model, decide what it should learn
Are you teaching a system to recognise direction, or to make an economically useful decision? The answer changes how you should read its metrics.
The loss function guides training. AUC (area under the ROC curve) measures how well a score ranks positive examples above negative ones; it does not directly measure the value of a decision, its costs or its capital requirements. A loss designed to approximate an economic objective can give different weight to mistakes that a directional metric treats alike.
A strong AUC is therefore insufficient to choose a trading system. A weak AUC is no evidence of economic intelligence either. Training AUC describes data used to fit the model: an extreme value calls for checks on separability, overfitting and possible leakage. Choosing an economic loss does not turn an in-sample result into evidence of future performance.
In our research, we distinguish the training objective, the score used to screen candidates and the portfolio backtest result. Even an economic proxy can rank candidates poorly once the complete system applies costs, constraints and execution rules. The 192-candidate comparison described here concerns a proxy and a backtest ranking. It is not an AUC study and does not establish that the loss caused the disagreement.
Four simultaneous opportunities have gross outcomes of +1, +8, −1 and −8, giving directional labels of 1, 1, 0 and 0. Model A assigns scores of 0.9, 0.7, 0.8 and 0.6. Model B assigns 0.6, 0.9, 0.7 and 0.8. A orders three of the four positive-negative pairs correctly, giving AUC 0.75. B orders two correctly, giving AUC 0.50. Both follow the same rule: take only the highest-scoring opportunity. A selects +1; B selects +8. Subtract 0.10 in costs from each trade. Outcomes become available after selection; they are not model inputs. This establishes that the objectives can disagree. It neither demonstrates that B will perform better on future data nor shows that an economic loss produced it.
Training against economic objectives is also established in academic research. Lim, Zohren and Roberts describe networks trained to optimise the Sharpe ratio, with a turnover penalty to account for transaction costs. Our educational contribution is the process of designing and testing a system, not a claim to have invented this category. Original paper: Enhancing Time Series Momentum Strategies Using Deep Neural Networks.
The useful question is: what behaviour am I rewarding, and how closely does it match what the system must execute?
Separate the dates, then check when the outcomes become known
Synthetic example: you want to evaluate decisions in periods 5–6. A training row starts in period 4, but its label uses an outcome that becomes known only in period 6. Fitting before period 5 with that label would introduce future information.
Naïve random split
- 1T
- 2V
- 3T
- 4T
- 5V
- 6T
- 7—
- 8—
Uses later observations to evaluate earlier decisions. It does not reproduce the task of predicting the future from the past.
Walk-forward without checking labels
- 1T
- 2T
- 3T
- 4T
- 5V
- 6V
- 7—
- 8—
Training starts before evaluation, but the outcome for row 4 arrives in period 6. Sorting rows by their start time does not resolve this.
Walk-forward with the overlap removed
- 1T
- 2T
- 3T
- 4×
- 5V
- 6V
- 7—
- 8—
Row 4 is excluded. In this example, outcomes for rows 1–3 are known before period 5; a real dataset requires checking every interval.
Purging, embargo and a gap before evaluation
Purging removes training observations whose label intervals overlap evaluation. An embargo can exclude a period after the test when the cross-validation design allows later training data. It aims to limit residual dependencies, such as overlapping information windows. Strictly causal walk-forward does not fit on those future observations; an exclusion interval before the test is a gap. Its length and boundaries depend on labels, data availability and feature windows. There is no universal number of days.
These exclusions do not fix late-published data, normalisation fitted on the entire dataset or repeated selection against the test. Keep excluded rows and reasons, include a broken example the check must detect, and record which results you had seen before deciding.
Try it on paper: a row starts in period 4 and its outcome ends in period 6. Can it train a model making a decision at the start of period 5? No. Now move the outcome end to period 4. That particular overlap disappears; feature availability and selection history still need checking.
For chronological splitting and the gap parameter, see the official TimeSeriesSplit documentation. For selection risk across many backtests, see Bailey and co-authors. Neither reference certifies this project or guarantees future performance.
scikit-learn: TimeSeriesSplit · Bailey et al.: The Probability of Backtest Overfitting
An identical file can still be incomplete
An integrity check can confirm that you have the expected file. It can also confirm, perfectly, that you have reproduced the same information gap.
Our audits required separate checks for file identity, data coverage and the history actually available to calculate indicators. Those are three different questions.
Three conditions to keep separate
— Expected prefix · ✓ Available · ! Unexpected gap
An indicator's earliest observations may be missing for an expected reason. An unexplained gap months later calls for investigation. Removing every incomplete row can hide the distinction and change the sample.
Making experience usable
An AI assistant can explain these concepts. To reconstruct the decisions, experiments and checks in our private archive faithfully, it needs access to those materials. The work behind this edition is to select, explain and turn part of that experience into usable methods while keeping proprietary configurations private.
Each case follows the same sequence: what we wanted to learn, what appeared to be true, which check changed the picture and what procedure the reader can use. Decisions to stop are explained too.
Starting Again takes the next step: how to organise a small lab for learning and experimentation today, with an idea, a baseline, usable data, bounded agent assignments and verifiable results. The aim is to avoid unnecessary trial and error. Actual time saved will depend on your starting point and how you use the material.
Six answers before another experiment
Three episodes explain why we chose to open part of the lab.
The probabilities matched. The orders could differ.
An audit had verified a replay's models, inputs and predictions. A later check found a difference in the component resolving directional conflicts. The replay actually reached that branch. Its status as an exact replica was withdrawn, while the result was retained as diagnostic evidence.
Where the paths diverge
The practical point is striking: identical models can produce different portfolios. Priorities, capacity limits and position management sit between prediction and order. A slot occupied now changes which opportunities can be admitted later. Checking a replica means following the decision through the whole chain.
The fieldbook explains how to locate the first divergence and decide which calculations can be reused. Repeating everything is not always necessary; retaining everything without tracing dependencies can be equally mistaken.
The agent reviewed the same objects repeatedly
An AI component received the global candidate list in every small batch. Responses were concatenated without complete scope and uniqueness checks. Duplication began in the data contract, before any assessment of reasoning quality.
With twelve objects in four batches, the test is straightforward: each object belongs in its own batch, once. You do not need to pay a model to check that property. You need identifiers, simulated responses and a parser that rejects out-of-scope output.
The economic question follows: did we retain enough information to establish whether the component helped? In the internal case, we had not. Spending was real and benefit could not be established from the retained evidence, so the component was suspended. That decision is as much part of the experience worth sharing as a successful solution.
The report was slowing down what it described
A research queue had caches and parallel processes. But generating its report inside the admission loop delayed replenishment. The system appeared organised for parallel work while an auxiliary function introduced waiting time.
For someone building a small AI-assisted lab, automation also means separating responsibilities, assigning a writer and verifying recovery without duplicated work. An elegant dashboard does not establish progress. Complete, checked and recoverable results do.
The check to take away
- What information was available when the decision was made?
- Which outcomes had already been inspected when the model was selected?
- Does the curve account for open positions, costs and cash flows consistently?
- Does the comparison change one thing or several conditions at once?
- Can someone else reproduce the result from the retained material?
- What outcome would make you abandon the hypothesis?
Missing answers identify the next piece of work. They do not grade your ability or certify a strategy.
From reading to your own research desk
Before Capital — Research Verification Protocol connects twenty chapters and laboratory cases to seven Python commands covering observed drawdown, timestamp contradictions, expanding splits with purging, rankings, decision traces, batches and observation windows. Each command takes CSV or JSON files in its documented format. It includes a reproducible notebook, tests, twelve worksheets and agent assignments. The planned delivery is a package in your chosen language: English, Italian or Spanish.
Each workflow starts with a problem, provides a command and shows the expected result. Reproduce the example, then connect a copy of your own data. The report retains input file hashes, the tool version and parameters. The agent assignment also asks for a counterexample to the conclusion: a persuasive explanation becomes a claim you can test.
You do not need to share your strategy with us. Explore the fieldbook and start with the case you recognise in your own work.
The intended outcome is a dossier you can understand, reproduce and challenge. The material does not include trading signals, proprietary models or guaranteed earnings.
The fieldbook is in preparation. The resources on this website are free to use. You can request one email when the fieldbook is ready; no purchase or reservation is available. Notify me when it is ready.
Sources and scope
The 192-backtest case comes from internal Quantic Eagle research dated 16 September 2026 and has not been independently audited. Other figures are conceptual or synthetic. The decade refers to the founder's stated research journey, which began before incorporation, not ten years of live results.
For selection across multiple backtests, see Bailey et al., The Probability of Backtest Overfitting. Quantic Eagle operates its own capital and does not accept or manage third-party funds. This is general methodological education, not an investment recommendation.
The False Green Museum
Two programs return identical probabilities. One selects A, the other B. Two batches return valid JSON but repeat the same objects. A feature is a finite number, yet its window has a gap. Three reassuring results; three unanswered questions.
In the False Green Museum, you can expose each defect and apply its correction. The synthetic figures are downloadable, so you can show a colleague why a passing check is different from a supported conclusion. No registration is required.
Often the most useful part of experience is knowing where a proof stops. Score parity can be exact without covering selection. A correct hash can identify an incomplete dataset. An agent can fulfil its task without its economic contribution having been measured.
The full protocol adds six workshops, three workflow checks and a dossier collecting findings, declared input identities and missing checks. Each counterexample has a corrected version. First require the check to distinguish them; then bring it to your data.
If your equity observations use different column names, the local mapper helps prepare them. If you only have closed trades, it does not invent the missing curve. That boundary is part of the product: recognising what the evidence cannot tell you before drawing a conclusion.