Education in research methods · Quantic Eagle operates only with its own capital. No client funds, investment advice or trading signals.
The test passes.
Did you check the right thing?
You are about to replace a backtest function, ask AI for a second review, or use an indicator to decide a purchase. What can go wrong even when a program returns a result? Follow three invented cases and the checks they help you build.
The cases and data are invented to show faults you can look for in your own tests. Each diagram starts on “Faulty case”. Click “Corrected case” to see the result after the correction described below it. Click “Faulty case” again to compare with the original output.
Each case explains the situation, the comparison to make and what still needs checking. The download button saves the diagram in the state you are viewing.
QE-CONTRACT-01 / 1.0
Two programs can agree and still be wrong
You want to choose a strategy to investigate further and look at the profit from a backtest, a simulation of purchases and sales using historical data. If the program deducts less commission than the assumed rate requires, a strategy can look better simply because its costs are wrong. Two programs can give the same result and still hide this problem if they share the function that converts the rate.
We start with a calculation you can do by hand. It helps you see what to check before trusting the calculations in a larger system. Work out the expected fee from the rate and the purchase amount, checking both separately from the program.
In this invented case, you buy shares worth €8,000 in total: this is the purchase amount, not a number of shares or lots. The assumed commission is 1 basis point, or 0.01% of that amount. The simulator records €0.008 and a second program returns the same number. Calculate the commission yourself before opening the answer.
Read the solution after writing your decision
To calculate 0.01% of €8,000, divide 0.01 by 100 and multiply: 8,000 × 0.0001 = €0.80. The €0.008 reported by both calculations is a hundred times too small. Agreement does not establish that the rate was applied correctly: two calculations can repeat the same incorrect conversion. Compare the cost recorded in the backtest with the €0.80 calculated by hand from the assumed rate. For a later sale, use that sale’s amount and the rate applicable to it.
The free file contains the calculation, expected result and two further comparisons involving windows and open positions. These are invented examples that run in Python without additional libraries. The fieldbook’s final project connects these checks to data, a model, orders, costs and a dossier.
Download the program and check the calculationsSave the file as QE-CONTRACT-01.py. Open a terminal in the same folder and run the following command. The result contains units, expected amounts and calculations.
python QE-CONTRACT-01.pyThe date passes. The value comes from the future.
This case uses invented data.
You want to test a strategy that uses an economic indicator to decide when to buy. You download a historical file from its publisher. That file may contain values the publisher corrected after the first release, for example after receiving more data. Using today’s corrected value to simulate a decision made before the correction gives the system information it could not have had then. Checking only the first publication date misses that error.
This case is invented: a statistical agency estimates the change in January industrial production relative to December in an imaginary country. It publishes an initial estimate of +2%, then revises it to −1% after collecting more responses. The exercise strategy permits a purchase only when the change is positive. Your program imports each release into its data archive; “Import into the archive” is the time when the value enters the archive used by the program. January is the measured month; the values become knowable only on the February dates shown. All times are UTC.
Case table: scroll horizontally if needed.
| Version | Change · % | Publication | Import into the archive |
|---|---|---|---|
| 1 | 2 | 2026-02-01 09:00 | 2026-02-01 09:05 |
| 2 | −1 | 2026-02-03 09:00 | 2026-02-03 09:01 |
In your notes, on paper or in a file, write one row for D1, D2, D3 and D4 in the second table. For each, record the value already imported into the archive at that moment and whether the rule permits a purchase. If no release of the January indicator has reached the archive yet, write “data unavailable: decision blocked”. Zero would mean an actual measured change of zero; it does not represent missing input. Then open the solution.
| Decision | UTC time |
|---|---|
| D1 | 2026-02-01 09:02:00 |
| D2 | 2026-02-01 09:10:00 |
| D3 | 2026-02-03 09:00:00 |
| D4 | 2026-02-03 09:02:00 |
Open the solution after deciding
Case table: scroll horizontally if needed.
| Decision | UTC time | Usable change · % | Action |
|---|---|---|---|
| D1 | 2026-02-01 09:02:00 | — | Unavailable |
| D2 | 2026-02-01 09:10:00 | 2 | Purchase permitted |
| D3 | 2026-02-03 09:00:00 | 2 | Purchase permitted |
| D4 | 2026-02-03 09:02:00 | -1 | No purchase |
At D1 the first estimate is public but has not reached the program’s archive: the decision must be blocked. D2 and D3 can use +2%; at D3 the −1% revision has been published but is imported one minute later. D4 uses −1%, so the rule does not permit a purchase. Using today’s −1% at D2 and D3 would exclude purchases permitted then. Attaching the old release date hides the fault: link each value to the release that contained it.
Retain every version for each observation period. At each decision admit only versions already published and received, then choose the latest eligible version. Equality is allowed here because receipt events at the same time precede the decision; uncertain clocks or event order require a more conservative rule.
Save the download as QE-VINTAGE-01.py and run it with Python 3.10 or later. Data and calculations are included; no registration, external service or additional library is required.
python QE-VINTAGE-01.pyThis check requires the necessary versions and timestamps; it does not measure a real provider’s latency, predictive ability or profit. Another test appends future data and checks that past decisions remain unchanged. That is useful, but may miss this fault: if the initial file already paired the revised value with an old date, adding rows does not correct or necessarily reveal that wrong row.
For the distinction between observation period and available vintage, see FRED/ALFRED documentation on real-time periods. That source uses dates; the intraday times in this case are invented.
The scores are right. They are attached to the wrong stocks.
You want faster backtests and rewrite the function that selects a stock to buy. The model scores each stock: A = 0.7 and B = 0.6. It has not selected a purchase; the next rule must buy only the stock with the highest score. In the rewrite, you reorder the stocks from A, B to B, A, but keep the scores in their original order. The program pairs them by list position: it now attaches 0.7 to B and 0.6 to A.
The numbers stay the same. But 0.7 is attached to B, so the program buys B.
The model has not changed its prediction. The step linking numbers to stocks has swapped A and B. A check comparing only the list [0.7, 0.6] passes: the numbers are unchanged. But the simulated account holds B instead of A. Gains, losses and later purchases can change from that point.
The model returns a score per stock. The fault arises when the stocks are reordered but the scores stay in their original order.
The check you need
Before replacing the function, try the same stocks in a different order. Compare stock–score pairs and the selected stock. If both stocks are eligible, A must win in either order. Then start from the same account and compare decisions in sequence: a different purchase also changes the cash and positions available afterwards.
What this establishes. The supplied program reproduces this pairing fault with invented data. We do not attribute this cause to the historical laboratory case. The comparison does not simulate broker execution or establish profits.
In your own project
Try two variations before opening the program’s solutions: A is excluded from eligible stocks; or both stocks score 0.7 and the stated tie rule selects the first stock alphabetically. Must A always win? Write the expected stock for each case and identify the fields to compare to distinguish a pairing fault from a different choice consistent with the rules.
Two agents say “all clear”. Did you get two checks?
An agent has built your backtest and the result looks promising. Before spending more time on the strategy, you want a second check of the costs: profit inflated by incorrect commissions could lead you to pursue the wrong project. You ask two agents to review the calculation. Both receive backtest.py, the trade log trades.csv and the fee schedule broker-fees.txt. The first delivers its report. You start the second in the same folder, which now contains that report too: it can already see that the first found no issues.
The second agent also receives the first agent’s report. Two approvals do not establish two independent checks.
You want someone to challenge the result. If the second agent starts from the first agent’s “all clear” rather than reconstructing the costs, an incorrect commission may receive no fresh check. You might trust the profit because two agents approved it without knowing what checks their approvals rest on. The diagram shows a report made visible; it does not establish that the agent was influenced by it.
The check you need
Before starting each reviewer, check which files and earlier messages it can consult: writing “work independently” does not remove a report from the folder. Ask it to identify the step checked, the amounts reconstructed and the fee schedule used. Collect the reports separately, then return to the originals to assess both agreements and objections.
What this establishes. The diagram and program reconstruct assigned materials in an invented case; they neither call AI nor measure its bias. Keeping reports separate allows a fresh reading without the earlier verdict, but does not guarantee that the agents find every fault.
In your own project
You want an independent cost review. You have prepared backtest.py, trades.csv, broker-fees.txt and the report already written by the first agent. Which file do you keep out before starting the second? Which do you retain so it can redo the calculation? Try the other goal too: if you simply want to finish checking many CSVs sooner, assign distinct groups and check which files were examined. Repeating one group does not complete the group left out.
A price is missing. The average changes and triggers a different purchase.
You want to use a price average to decide a purchase. The program can return a plausible number even when the download is missing a session: check the dates used before passing the average to the model. At the opening of session 4, the available price is €10. The invented rule buys only if that price exceeds the mean closing price of the three previous sessions: 1, 2 and 3. Their original closes are €10, €11 and €12, giving a mean of €11 and no purchase. But session 2 is missing from the downloaded CSV. The function takes the last three prices present, using €5 from session 0 instead of the missing €11. The mean falls to €9.
Current price €10 > incorrect mean €9: the program decides to buy.
The function calculates (5 + 10 + 12) / 3 = €9. This is more than a wrong indicator in a report: because 10 exceeds 9, the program permits a purchase that the rule using the required three sessions would exclude. This can happen when a download loses a row and the calculation takes the last three available rows without checking their sessions.
The check you need
Before passing the mean to a rule or model, determine the required sessions using the market calendar. Check for exactly one close per session and that each was available before the purchase. The check must prevent an older session being used to complete the count. The supplied program shows both the erroneous purchase and the block that prevents it.
What this establishes. The rule and prices are invented to show a decision altered by its inputs. We do not propose this average as a profitable strategy or calculate the purchase return.
In your own project
Try a close present in the file but received only after the decision: can it enter the mean? Then try a duplicate row. These differ from a missing session, but counting three numbers must not hide any of them. Also distinguish the beginning of a series, when three closes do not yet exist, from a session missing later.
Try these faults before using the results in your system
Data checks belong before an average becomes a signal; pairing checks belong before a score becomes a purchase. The program below reproduces the three Museum mechanisms. First predict the first six variations: an ineligible stock, a tie, an earlier verdict shared with the second reviewer, a missing fee schedule, a late close and a duplicate row.
In the seventh exercise, M1, you must choose between two stocks: the missing price can change eligibility and then the selection while scores stay the same. Compare incomplete data, the active check and restored data. Explain in your notes why the first different decision does not by itself identify the cause.
Reproduce the cases, then combine the checks
Download the JavaScript and JSON into the same folder. With Node.js installed, open a terminal in that folder and run the commands below. The first shows the cases explained in the diagrams. The second displays variations: record your predicted outcome and reason first. The third shows outcomes to compare with your answers.
Invented inputs and variations · JSON · Standalone program · JavaScript
node QE-TRADING-CHECKS-01.js
node QE-TRADING-CHECKS-01.js --exercises --lang en
node QE-TRADING-CHECKS-01.js --solutionsIn the output, BUY means a purchase is permitted, NO_BUY means no purchase under the rule, and BLOCKED means the decision was stopped because required inputs are incomplete or unusable. With the missing price, the unchecked calculation decides BUY using a mean of 9; the check blocks the decision. After recovering the original price, the mean is 11 and the valid outcome is NO_BUY.
QE-TRADING-CHECKS-01 · 1.1. The program does not call AI, send orders or calculate returns. It reconstructs pairings, materials supplied to reviewers and decisions using the invented inputs above.
A shorter check: coverage and mean only
Download these two files into the same folder and run the command with Node.js. They contain the invented inputs from the window case: first incomplete, then with the missing original observation recovered.
Inputs and expected results · JSON · Readable check · JavaScript
node QE-WINDOW-01.js QE-WINDOW-01.jsonIn the faulty case, the mean of the last three available values is 9, but time 2 is missing: the check returns coverage_pass: false and mean: null. The corrected case returns true and 11. The program completes verification when it detects the expected fault: a passing test can confirm that an input must be rejected.
QE-WINDOW-01 · version 1.1. The program checks the declared calendar, duplicates and availability before decision time. It does not recover or invent data, or certify the file’s real provenance.
The fieldbook is in preparation. The resources on this website are free to use. You can request one email when the fieldbook is ready; no purchase or reservation is available.
Notify me when it is readyWritten by Quantic Eagle LTD · General education in research methods. About.