Profit improves. Does risk improve too?
We compare two implementations of the same risk rule in simulations of 672 model combinations. Members and periods are identical within each comparison. Profit improves in 439 comparisons and drawdown falls in 239, but both improve together in only 145. This shows why “more profit” and “less risk” are different conclusions.
Here OOS identifies a period not used to fit these models. It had already been consulted to select candidates, so it was not an independent final test. Drawdown measures the decline from an earlier peak in the simulated account-value curve.
| Relative to the previous simulation | Lower drawdown | Drawdown not lower | Total |
|---|---|---|---|
| Higher profit | 145 | 294 | 439 |
| Profit not higher | 94 | 139 | 233 |
| Total | 239 | 433 | 672 |
The four inner cells sum to 672: 145 + 294 + 94 + 139. The 439 cases with higher profit include the 145 that also reduce drawdown; the 239 with lower drawdown include those same 145. “Not higher” and “not lower” include unchanged results, so they do not necessarily mean deterioration. Improvement over a previous version does not establish profitability, robustness or readiness to trade. These are aggregates from an internal comparison already used to select candidates.
Internal research, snapshot dated 19 September 2026. OOS period: 1 August 2025–31 July 2026. Paired comparisons across all 672 ensembles; the three chart counts overlap and must not be added together. The aggregates cannot reproduce the proprietary replay. These are aggregates from internal simulations, calculated with the cost assumptions used at the time. They describe the recorded comparisons; they do not show net profitability or readiness for live trading.
FREE EXERCISES · INVENTED DATA
Three decisions to try in your own lab
You change the minimum score required to buy: which calculations need repeating? Preparing each report holds up the next calculation: how much waiting time can you remove by preparing reports in a separate process? You apply a new rule to past data: what can you write in the record of original decisions? Three invented exercises let you try these choices.
You changed a rule. Which results can you reuse?
You are testing a rule for selecting stocks to buy. For each stock and day, the program supplies past prices to the model. The model returns a score: a number used to rank stocks and compare them with a threshold. After the first run, you instruct the program to save a CSV with one row per date and stock, such as “5 January, A, 0.65”. Keep scores for stocks that were not bought too. Here 0.65 is not a probability of profit. A is excluded by “score at least 0.70” but becomes eligible under “at least 0.60”. With the same model and prices, must you produce the score CSV again?
- Data and model Prices read by the model
- Scores One score per stock and date
- Choices and amounts Which stocks to buy and how many
- Trades and reports Costs and account value
If the table is wider than your screen, scroll horizontally. With a keyboard, focus the table and use the arrow keys.
| Change | Keep | Repeat |
|---|---|---|
| Signal threshold | data, model, scores | selected stocks, share quantities, order and cost record, profit and account declines, report |
| Rule for conflicting signals | data, model, scores | selected stocks, share quantities, order and cost record, profit and account declines, report |
| Amount per trade | data, model, scores, selected stocks | share quantities, order and cost record, profit and account declines, report |
| Cost charged | data, model, scores, selected stocks, share quantities | order and cost record, profit and account declines, report |
| Members of the combined model | data | model, scores, selected stocks, share quantities, order and cost record, profit and account declines, report |
No. This example calculates scores solely from prices, which you have not changed. A still receives 0.65, so you can read that value from the first run’s CSV. Selection does change: A now meets the minimum score. Start from the same initial account and simulate the decisions again in date order. Apply the stated rules for purchase amount, entry price and sale timing, then record quantities, proceeds and fees. These steps reconstruct holdings and cash in the second run; changing a threshold in the final report is not enough.
The table describes a simplified program: the model receives only prices, purchase amounts do not depend on remaining cash, and fees are subtracted after the orders have been selected. That is why changing fees alone leaves the order list unchanged in the table. A program that rejects unaffordable positions needs another check: a €200 order with a €1 fee cannot be funded with only €200. If you change the fee from zero to €1, you must repeat the decision about executing that order as well.
You lower the minimum score required to buy and the new simulation buys €800 of shares, leaving €200 before costs. Can you reuse the saved score for the next decision? Compare the inputs that produced it with those supplied by the new simulation. If the model reads only unchanged prices, the new purchase does not alter those inputs. If it also reads available cash, it now receives 200 where it previously received 1,000: the old score has not been checked for this situation. Run the same model with the new inputs; its result may stay the same or change. This does not automatically require retraining. Then check whether cash covers the order, including costs. The table’s final row assumes that replacing a model requires retraining the program combining the models: verify that dependency in your own system.
Ten minutes to calculate. Why fourteen between results?
You have three tests ready and one compute process. Each test takes 10 minutes, but results arrive at minutes 10, 24 and 38. Before starting more processes, you want to locate the eight minutes of waiting accumulated before the third result. Which start and finish times would distinguish slow computation, no ready work and waiting for a report?
Open the case and explanation
If the table is wider than your screen, scroll horizontally. With a keyboard, focus the table and use the arrow keys.
| Design | First result | Second result | Third result | Last report |
|---|---|---|---|---|
| Computation waits for reporting | 10 min | 24 min | 38 min | 42 min |
| Reporting runs separately | 10 min | 20 min | 30 min | 34 min |
The case log shows that the process waits four minutes to prepare each report before starting the next test. With a separate reporting process, the next calculation can start immediately. The third result is ready at minute 38 in the first design and minute 30 in the second. Computation still takes 30 minutes in total: you have removed 8 minutes of waiting before the third result. The last report is ready at minute 42 or 34. A finished result and a finished report are different events.
The example assumes that writing reports does not slow the three calculations and that each result is saved before its report is prepared. On a real computer, the processes can compete for processor time, memory or disk access and run more slowly. Measure timings with and without reporting. If writing takes longer than calculation, also check how many reports remain queued.
The decision then and the simulation now
In this invented example, at 10:00 UTC on 5 January the model gives stock A a score of 0.62. The program compares that number with its minimum score required to buy. Rule v1 requires at least 0.65, so A is not bought. The next day, you lower the threshold to 0.60 and simulate 5 January again. The model reads the same prices and still gives A 0.62. What can you write in the decision record?
If the table is wider than your screen, scroll horizontally. With a keyboard, focus the table and use the arrow keys.
| Record | Rule | Outcome |
|---|---|---|
| 5 January decision log | v1 · ≥ 0.65 | No purchase |
| New simulation on 6 January | v2 · ≥ 0.60 | Simulated purchase |
Keep both: “v1 did not buy” and “v2, applied to those data today, would have bought”. Do not replace the original log. If you chose 0.60 after seeing the 5 January outcome, that day has already influenced selection; it is no longer a fresh test.
The log documents what you retained. It does not by itself prove that the data were available then, that an order would have filled or that the new threshold will work in future.
Download inputs and results · JSONDownload the program · JavaScript
To reproduce the calculations, save the program and run it with Node.js. The JSON file lists the assumptions and expected results.
node QE-WORK-01.jsQE-WORK-01 / 1.0. These examples check only the system described. They do not reconstruct your architecture or assess whether a trading strategy is profitable.
From data to decisions: what you need to be able to reconstruct
A model is one part of the system. This teaching map connects each stage to a possible failure, a check and a record to retain. It does not describe Quantic Eagle’s proprietary architecture.
1 When the information became available
- Risk
- The test buys at 10 am using news released at noon.
- Check
- Compare the purchase time with the earliest time you could receive the news.
- Keep
- The news source, file version and timestamps.
2 What the model reads and learns
- Risk
- A price average includes tomorrow, or the model learns from a sale that has not happened yet.
- Check
- List the prices used for each input. For each outcome used in training, record when the sale makes it known.
- Keep
- The formula, input rows and time the outcome becomes known.
3 Which data train the model and which evaluate it
- Risk
- You simulate a Monday purchase using a model that has already learned from the following Friday’s outcome.
- Check
- Before training, separate the rows and check when each outcome becomes known.
- Keep
- The rows assigned to each period and reasons for exclusions.
4 How you choose a version to test
- Risk
- You look at the final test result and change the minimum score required to buy to improve it.
- Check
- Use one period to choose between versions. Reserve another for subsequent evaluation and do not consult it to adjust the choice.
- Keep
- Every version tried, its settings and the results already consulted.
5 From scores to orders
- Risk
- Two programs give A and B identical scores, but one buys A and the other buys B.
- Check
- Compare which stocks may be bought, their priority order, the final choice and quantities. Then check simulated prices and costs.
- Keep
- Decision records, orders and assumptions about execution prices.
6 What the account is worth throughout the test
- Risk
- The balance hides a loss on shares still held, or a deposit looks like a gain.
- Check
- Add cash and the value of open positions. Record deposits and withdrawals separately, along with how often you observe the account.
- Keep
- Account values, external cash movements, costs and the method used to calculate returns.
7 What happens on data not used to choose
- Risk
- You present the period that helped you pick the winner as fresh confirmation.
- Check
- Save the chosen version before the next evaluation. Also examine stated adverse scenarios, distinguishing them from actual observations.
- Keep
- New evaluation results, scenario assumptions and a record of past choices.
8 How you preserve decisions and continue
- Risk
- You rerun the past with a new rule and lose the record of what the program decided at the time.
- Check
- Keep both versions and link each decision to the rule that produced it. Define the next evaluation before seeing its results.
- Keep
- Dated versions, original decisions, new simulations and checks still needed.
The map uses invented cases. When standardising prices for training, estimate the mean and standard deviation from permitted training data only. A prediction for a row excluded from that particular training split is called out-of-fold (OOF). Also check preprocessing and choices that used its outcome; training on other rows does not remove future information when those rows come from later dates. A period excluded from weight fitting, called out-of-sample (OOS), may still have been consulted to select the system.
The curve that left something out
Equity is the total value of the account at a given time, including unrealised profits and losses on open positions. Balance excludes those unrealised results. Treatment of credit, fees and other components depends on the source. In the comparison below, equity and balance are observed at the same times. The calculation uses only those values: it does not reconstruct what happened between observations or verify how the source calculated them.
To compare curves for an invented account, we set their starting value to 100. The chart numbers are an index, not euro amounts. Looking only at the balance after closed trades, the index never falls and ends at 104. The report therefore calculates zero balance drawdown.
Now include open positions in the same observations. Equity reaches 101, falls to 87 and finishes at 104. Observed drawdown is 13.86%. The endpoint is unchanged; the path reveals something else.
Two curves. The same endpoint.
This is a synthetic example, not Quantic Eagle performance. It shows why a metric's definition matters as much as its value. In practice, you need to establish observation frequency, costs, cash flows and coverage of open positions. Daily observations can miss an intraday low.
Start by asking whether your report measures the risk you want to understand or only the risk its available data can describe.
The preliminary ranking differed from the backtest ranking
When you have many models to test, a quick preliminary calculation can help choose which receive a complete simulation. This preliminary measure is a proxy for the result you ultimately care about. In an internal campaign, the preliminary ranking and the backtest ranking ordered the same 192 candidates differently. This records disagreement between rankings; it does not establish that either selection method is superior.
The historical comparison of 192 candidates does not provide the preliminary ranking formulas, the backtest metric, period and cost conventions, the full population before selection or candidate-level results. Without these elements you cannot reconstruct that comparison. The candidates were also chosen during research that adapted to results already consulted, so that period is not an independent final test.
In your own comparison, record the same candidate IDs in both rankings, define each sorting measure and its period, and retain costs and selection rules. Test some candidates outside the preliminary shortlist too. Otherwise you cannot observe the results of the candidates that your filter excludes.
A proxy can measure one part of the problem. A portfolio also involves costs, concurrent positions, risk and concentration. Comparing both rankings under consistent conditions makes the difference visible.
Synthetic experiment · Entirely invented data
The top-ranked model loses on the period used for selection
You have two backtests covering periods of equal length, with the same capital and cost conventions. A earns 3,000 invented currency units on the training period and loses 20 on the selection period. B earns 200 and 100. Add 10% of the first result to 90% of the second: who ranks first?
A: 300 − 18 = 282. B: 20 + 90 = 110. A ranks first because the formula includes 300 units of profit made on the same data used to train the model; the second period contributes −18. On the second period alone, B comes first with 100 versus −20. That does not make it ready to trade: risk, activity and other periods still need evaluation. And once a period guides your choice, it is selection data, not an untouched final test.
Invented example. Neither the weights nor the results come from Quantic Eagle’s operational research. The formula illustrates a problem; it is not a recommended selection rule.
Explore the programme and exercises →Before choosing a model, decide what it should learn
You need to choose a model and the report favours the one that separates winning purchases from losing ones more accurately. Your system can buy only one stock: check which it chooses and how much that purchase gains or loses after costs. Imagine four purchases available at the same time. You may make only one, always for the same amount. Two invented models assign scores to set priorities; each chooses its highest score. The gains and losses in the table become known afterwards; they are not model inputs. Amounts and costs use the same invented currency unit.
On a small screen, scroll the table horizontally.
| Purchase | Gross outcome, known later | A score | B score |
|---|---|---|---|
| 1 | +1 | 0.9 | 0.6 |
| 2 | +8 | 0.7 | 0.9 |
| 3 | −1 | 0.8 | 0.7 |
| 4 | −8 | 0.6 | 0.8 |
A chooses purchase 1: a gain of 1 minus a cost of 0.10 leaves +0.90. B chooses purchase 2: a gain of 8 minus the same cost leaves +7.90. The scores are not probabilities, and no model was trained for this example.
Now compare each winning purchase against each of the two losing purchases: four pairs. A gives the winner a higher score in three pairs out of four; B does so in two. This fraction is the example’s AUC: 0.75 for A and 0.50 for B. A ranks winners and losers better, but B selects the purchase that earns more here. Ranking quality and purchase outcome answer different questions. The case does not show which model will work better in future. The pairs are 1–3, 1–4, 2–3, and 2–4. For A, 0.90 exceeds 0.80 and 0.60; 0.70 exceeds 0.60 but not 0.80. For B, 0.60 exceeds neither 0.70 nor 0.80; 0.90 exceeds both.
Are you teaching a system to recognise direction, or to make an economically useful decision? The answer changes how you should read its metrics.
The loss function guides training. AUC (area under the ROC curve) measures how well a score ranks positive examples above negative ones; it does not directly measure the value of a decision, its costs or its capital requirements. A loss designed to approximate an economic objective can give different weight to mistakes that a directional metric treats alike.
A good AUC is insufficient to select a trading system; a low AUC does not establish its economic merit either. If AUC is very high on training data, check whether examples really are easy to distinguish, whether the model memorised features of that sample, or whether it received information unavailable at decision time. Changing the loss function does not turn a training result into evidence on unseen data.
In our research, we distinguish the training objective, the score used to screen candidates and the portfolio backtest result. Even an economic proxy can rank candidates poorly once the complete system applies costs, constraints and execution rules. The 192-candidate comparison described here concerns a proxy and a backtest ranking. It is not an AUC study and does not establish that the loss caused the disagreement.
Training against economic objectives is also established in academic research. Lim, Zohren and Roberts describe networks trained to optimise the Sharpe ratio, with a turnover penalty to account for transaction costs. Our educational contribution is the process of designing and testing a system, not a claim to have invented this category. Original paper: Enhancing Time Series Momentum Strategies Using Deep Neural Networks.
The useful question is: what behaviour am I rewarding, and how closely does it match what the system must execute?
Separate the dates, then check when the outcomes become known
Invented example: you want to train a model before period 5 and use it to make decisions in periods 5 and 6. One training example buys in period 4 and sells only in period 6. Its gain or loss is the answer you want the model to learn to predict, called the label. Before period 5 you cannot know that answer: the sale has not happened. Sorting rows by purchase date is not enough; check when each outcome becomes known too.
Naïve random split
- 1T
- 2V
- 3T
- 4T
- 5V
- 6T
- 7—
- 8—
Trains the model on observations that come after some of the decisions being evaluated. It does not reproduce the task of predicting the future from the past.
Walk-forward without checking labels
- 1T
- 2T
- 3T
- 4T
- 5V
- 6V
- 7—
- 8—
Training starts before evaluation, but the outcome for row 4 arrives in period 6. Sorting rows by their start time does not resolve this.
Walk-forward with the overlap removed
- 1T
- 2T
- 3T
- 4×
- 5V
- 6V
- 7—
- 8—
Row 4 is excluded. In this example, outcomes for rows 1–3 are known before period 5; a real dataset requires checking every interval.
Purging, embargo and a gap before evaluation
Purging removes training observations whose label intervals overlap evaluation. An embargo can exclude a period after the test when the cross-validation design allows later training data. It aims to limit residual dependencies, such as overlapping information windows. Strictly causal walk-forward does not fit on those future observations; an exclusion interval before the test is a gap. Its length and boundaries depend on labels, data availability and feature windows. There is no universal number of days.
These exclusions do not fix late-published data, normalisation fitted on the entire dataset or repeated selection against the test. Keep excluded rows and reasons, include a broken example the check must detect, and record which results you had seen before deciding.
Try it on paper: a row starts in period 4 and its outcome ends in period 6. Can it train a model making a decision at the start of period 5? No. Now move the outcome end to period 4. That particular overlap disappears; feature availability and selection history still need checking.
In TimeSeriesSplit, gap is a number of samples, not automatically a number of days.
For chronological splitting and the gap parameter, see the official TimeSeriesSplit documentation. For selection risk across many backtests, see Bailey and co-authors. Neither reference certifies this project or guarantees future performance.
scikit-learn: TimeSeriesSplit · Bailey et al.: The Probability of Backtest Overfitting
Two identical files. Both lack the price for 3 March
Invented example: two copies of prices.csv match byte for byte, but both lack the 3 March close required by a moving average. Taking the last three rows present can replace that close with an older price, giving the model or rule a different average from the one required. In the Museum case, the current price of 10 exceeds the incorrect mean of 9 but not the correct mean of 11: a purchase is permitted that the rule would exclude. Comparing file fingerprints, called hashes, does not expose the gap; compare the sessions present with those required. Follow the missing input through to the purchase decision.
To calculate an indicator, check both the dates it requires and how many preceding observations it needs. A file may contain every date in the test period but too little history before that period begins.
Three conditions to keep separate
— Expected initial missing observations · ✓ Available · ! Unexpected gap
A three-session average cannot exist during the first two sessions because three prices are not yet available. That absence is expected. A missing price months later, when all three sessions should be present, needs investigation. Dropping every incomplete row indiscriminately hides the difference and changes which cases you evaluate.
From a worked case to your own project
The exercises showed how a rule change affects orders, why a report can delay the next job and why original decisions must be preserved. The fieldbook in preparation takes you through these checks on your own program, with instructions, worked examples and worksheets.
Explore the fieldbook programme →Sources and scope
The 192-backtest case comes from internal Quantic Eagle research dated 16 September 2026 and has not been independently audited. The teaching figures are diagrams or synthetic examples; QE-RESEARCH-02 instead presents aggregates from the internal comparison recorded on 19 September 2026, with its source and limitations beside the chart. The fieldbook draws on around ten years of accumulated research experience, with contributions from the founder and external collaborators, including work before the company was formed. This does not represent ten years of live trading results.
For selection across multiple backtests, see Bailey et al., The Probability of Backtest Overfitting. Quantic Eagle operates with its own capital and does not accept money to invest on behalf of others or manage their investments. This is general methodological education, not an investment recommendation.
Compare more faulty and corrected examples in the False Green Museum.