What a Mechanism Owes You That an Artifact Never Will

I spent an hour answering the same question over and over again, just in a different format each time.

“What is the difference between a true signal and noise that survived your backtests?”

I kept answering with my process. The hypothesis stated before data is touched. Parameters are never calibrated simply because they performed better. Validation runs forward. All of that is true, all of it is what I do. It just wasn't the answer to the question I was being asked.

It took me a bit to see the mismatch. He was asking about evidence and I was describing hygiene, which only certifies the process that produced the hypothesis while saying nothing about the hypothesis itself. A clean prior and a lucky one produce the same equity curve.

So let’s look at the honest version of the problem, and then the honest version of my answer.

The Record Cannot Answer It

I’ll take the worst-behaved version first simply because it will clear the ground. If a pattern turned up by searching a pile of variables and got an explanation bolted on afterward, there's no argument to have. Virtually everyone agrees that record is worthless, and DSR exists to say just that.

The interesting case is mine, done right. The hypothesis was written down before the data was touched, and the model was built to capture a relationship I had reasoned out in advance, not one the data handed me.

That does not mean the first construction was the finished one. Once a model meets a backtest, edge cases and anomalies surface that no amount of forethought could have enumerated. An edge case is not a trade that lost. It is a trade the model took outside the structural inefficiency it exists to trade, and the P&L is beside the point. The clearest edge cases in my log are winning trades. A trade that profits without the structural condition present is a false trade, and it gets coded out, which lowers the backtest score. That is an action a score-chasing process cannot produce. It is also the one claim in this paragraph you don't have to take my word for: the log of coded-out winners and the backtest score it cost are both artifacts, and the direction of the cost is checkable by anyone holding them. So calibration does happen, but the line that has to be held is the question driving each calibration decision. At no point am I asking how to move parameters to produce a better curve. That question never enters my process. What I am actually asking is how the parameters can better capture the structural inefficiency the model exists to trade. Any parameter moved for the curve rather than the reason shows up downstream, whether I confess the intent or not. Those two questions can land on the same choice in isolation and still mean opposite things: one fits the model to the sample, the other fits it to the reason. The difference doesn’t reveal itself in any single move, it shows up in whether the whole set of them chased the score, which is exactly what the diagnostics read. I don’t walk away without testing sensitivity, because a model one parameter away from breaking is not a sound one, but that asks whether the structure is fragile, not which setting scored best.

So the model survived forward, calibrated only ever toward the mechanism, and that still leaves it with two possible explanations.

Maybe the mechanism I described is real and operating. Or maybe I got the prediction right and the reason wrong. I pointed at some condition, the condition really does precede the move, but not for the reason I gave. The statistical relationship is real. My reasoning about why it exists isn't.

That might sound survivable to some readers, but it is the opposite. My reasoning about the mechanism is not post hoc rationalization. It is what I built the initial model out of, and every calibration afterward was steered by it. Before any data was collected, the hypothesis named a relationship between variables, and the whole model was constructed to capture that relationship. So every choice downstream of it inherits the reasoning. When I determine the conditions under which the model degrades, I am really asking where that relationship stops holding: where it is no longer present, or where it is still present but no longer produces what it did before the shift. I am not reading that off what the data says. I am reading it off the why.

When the reasoning is wrong, every one of those calls is getting made blindly and they can only possibly be correct when luck cooperates. Worse, every calibration made was aimed at a mechanism that was never there, spent with the same care I would spend on a real one, and the backtest could not tell me the difference. The observed edge might pay until the moment one of those blind calls matters, and then it breaks, and the record gave no warning, because the record never encoded the why to begin with.

None of this implies the process was flawed. You can be right about what the market will do and still be wrong about why it did it. Careful work doesn't prevent that. It only makes it easier to recognize when it happens. The care and discipline were real. The calibration was honest. The equity curve looked exactly as it should, all while the explanation was quietly wrong. That's what makes this the hard case. Nothing in the work tells you which world you're in.

The record can't separate those two, and the reason is almost tautological once you see it. On the question of which twin you're holding, every evaluation metric reduces to the return series, and the twins share that series by construction. So any two hypotheses that produce the same returns produce the same value on every one of them. Note what that disqualification is and isn't. On other questions, these tools consume more than returns: DSR consumes the breadth of your search, and search breadth is a fact about your process, not your P&L. That's exactly why they still work on the question of whether calibration chased the score: score-chasing and reason-driven calibration differ in an observable fact, the relationship between parameter selection and the sample, and that fact enters the calculation. What never enters it is the why. The twins agree on everything the tools can see. The real mechanism and the right-prediction-wrong-reason twin produce the same returns. Everything they disagree about is in the why, and the why never entered the return series to begin with, so no statistic taken from that series can recover it.

So I stop asking the record to settle it and start asking it to test something. Passive history cannot separate the twins, no matter how much of it you buy. But history I have staked a claim against can.

The reasoning that produced the prediction did not only predict returns. It made other commitments, and those I can check right now, against history I already hold. If the mechanism is what I claim, it names where the model should fail. If a model strictly trades in the small-cap momentum universe, a regime like 2022 has the exact conditions the edge should break. That is not a wait-and-see prediction. It is a claim I can test against a period already in the record. If I run that backtest over 2022 and it held up, that is not a relief. It is a contradiction. The thing I was most certain would kill the edge didn't. That means I do not actually know where the edge fails, which also means I don't understand the mechanism producing the returns. It fails somewhere, every real edge does. But my explanation of where it would fail was wrong, and a mechanism that cannot tell you where it breaks is not a mechanism. It is only a story attached to the results. The return series could never have told me that. The commitment could, and it did it against data I already had.

The one caveat is the fact that I lived through 2022. I could not have named it as the place the edge should break without already knowing, ambiently, that it was hostile to small-cap momentum. That does not void the test, but it grades the two outcomes differently. A confirmation, the edge correctly dying where I predicted, still counts for the mechanism; a stake that could have failed didn't. But part of that prediction was available to anyone who knew the era, so part of the confirmation is discounted, the part ambient hindsight could have supplied on its own. What is not discounted is the shape the mechanism predicted and hindsight could not: how fast the edge degrades, through which channel, whether sustained downtrends kill it differently than chop. On the other hand, the contradiction, the edge surviving a regime I was sure would kill it, keeps full force, because knowing the regime was hostile makes survival more surprising, not less. Confirmations against known history are graded by how much of them I could have called without the mechanism. Contradictions are not graded down at all.

Now, I can already guess what someone reading this is thinking: "Why don't you just run a 10+ year backtest and find the regime that kills the model? Why are you trying to guess it out of structural reasoning?"

The answer is simple. A found failure regime is a feature, not a commitment to be falsified. It's the same after-the-fact discovery that has already been disqualified. The mechanism that predicts the failure before looking stakes something that could have been wrong. The search that finds the failure cannot be wrong; it just reports.

Now, I don't want to leave this as an omission. What if the regime that kills the model has never occurred anywhere, not just in my sample, but in the entire history that markets have produced? That is more than possible. Decimalization, a historical moment I use to defend my own process, created a regime that had never physically existed until the rules made it exist. So that specific claim, about the edge degrading under a condition that has never happened, isn't falsifiable in advance.

The Tools Do Not Get You Out

The obvious reply is that statistics already handles this. Deflated Sharpe, PBO, CPCV, walk-forward. So before I get to what the commitments can do, this has to be cleared.

Start with what a single passing test actually licenses, because it is less than people read into it. A surviving Deflated Sharpe says one thing: this Sharpe is unlikely to have come from the search process that produced it. Look at what the calculation consumes. Your performance statistics and the breadth of your search. Nothing about the market besides your own return series enters it. So a real mechanism and a persistent accident submit the same Sharpe and the same trial count, and they receive the same verdict. That is correct behavior for a tool whose question is about your process, but nothing inside the calculation converts an answer about the search into a statement about the edge. A person does that, in the reading. The pipeline has no step that asks whether this is a mechanism, so search hygiene inherits the burden by default, and passing the last gate becomes proof because it is the last gate.

Stacking more tests does not fix it, because they are not independent. Deflated Sharpe prices the search, PBO estimates the probability the configuration is overfit, CPCV resamples folds into a distribution of out-of-sample performance, walk-forward preserves time ordering. Four different methods, and one shared input: each is a function of a single realized path of the market. However that path was unusual, all four inherited the unusualness together. A draw that flatters the hypothesis flatters all four in the same direction, so they can pass together for one reason and none of them can see it. Four green lights, one bulb.

There is one reading that comes close. When you read CPCV by its dispersion, it stops measuring search hygiene and starts measuring regime-contingency. A mean Sharpe of 3.0 with a standard deviation of 5.0 is a worse system than a 2.0 with a standard deviation of 1.0, and that is close to what I claim to know structurally. But it still resamples inside the history I was handed, so it certifies stability across the regimes my sample happens to contain, not the ones it never held. Closest anyone gets with a statistic, and still short of the turn that matters.

None of these get me out, and they were never built to. They audit the search, and at that they work, which is why the overfitting diagnostics still mean something when I come back to calibration. What they cannot do is ask whether a mechanism is there, and that question is the only one that separates the two explanations. So it has to be asked somewhere else, against facts the return series does not contain.

What a Mechanism Owes You

A weighted coin and a lucky coin favoring the same side will produce identical records. Only the weighted coin commits to a balance point, a wobble, and a waterline. The difference between them is real, but it does not exist in the outcomes alone.

A mechanism works the same way. It is a claim about how a piece of the machinery operates, so it carries more consequences than the one it was invented to explain. Which resting orders get consumed. What a halt does to the book. What shows up in the tape at the moment of the event if the thing is actually running.

Those commit in advance to facts the performance record does not contain:

1. When the edge should be absent, not only when it should be present.
2. Where it should break, and under which regime change.
3. What residue should appear if the mechanism is present.

An artifact satisfies exactly one thing. It kept working.

A mechanism has to satisfy all of them, and every one of them was an opportunity to be wrong. An artifact that also breaks precisely where the mechanism said it would, appears only under the conditions it specified, and leaves the residue it implied is no longer one coincidence. It is a conjunction of them, and the cost of believing that conjunction climbs with every term.

I am not accumulating proof. I am raising the number of independent coincidences the artifact explanation has to absorb.

Why Most Commitments Are Not Commitments

This is where the argument is usually lost, and it took an argument with someone building the opposite kind of system for me to see where.

He runs thousands of hypotheses compressed into features, with a model on top learning which ones matter. I said one of his hypotheses had an obvious failure condition sitting in its own text. It needs oscillation, it dies in sustained trends. Then I asked where that condition goes in his pipeline.

He said it goes in as a feature. Condition met, flag it, let the model evaluate the base signal against it.

They are completely different objects.

A feature cannot be wrong. It is a measurement. It asserts nothing, so nothing can contradict it. If the model discovers that the reversion works fine in trending conditions after all, the model simply learns that relationship and continues. No claim was made. No claim failed. Nothing was ever at risk.

A failure condition can be wrong, and being wrong is the entire point of it. Written down in advance, it stakes something. The mechanism says the edge dies here. If the edge does not die there, the mechanism is wrong, and it is wrong in a way that no amount of good performance repairs.

A feature is symmetric. A failure condition is not.

That asymmetry is what makes a ledger of commitments cost the artifact anything. A pile of measurements costs it nothing, no matter how large the pile, because measurements never had a chance to disagree.

His system carries measurements and learns relationships. Mine carries assertions that can be contradicted.

Residue, Not Identity

One class of commitment gets excluded, and it is the class that sounds most like understanding.

A model that depends on correctly naming who is on the other side and why they are there fails the moment I am wrong about them, and in markets I will frequently be wrong about them, because intent is unobservable. Other people's intent, that is. That is a different kind of claim than the one I made about my own. Modeling a counterparty is a claim about a fact in the world I cannot see. “Institutional accumulation is happening here” is a claim about intent that cannot fail cleanly. There is no check. Something else producing the same print satisfies it just as well.

So the commitments never name the participant or the motive. They measure the consequence the behavior leaves in the tape. If five different participants, acting for five entirely different reasons, all produce the same measurable condition, the model does not need to identify the cause. It trades the residue the cause leaves behind.

This is what converts a story into an actual falsifiable claim. Does this observable condition predict the outcome it claims to predict has an answer in the data, independent of any narrative about who or why. The model survives because the condition predicts, not because the story is persuasive to a human ear.

It also means a commitment can be about returns and still count. The edge should be absent in this state is a statement about returns, but the conditioning variable is not a return, so the artifact does not get it for free. What disqualifies an entry is not that returns appear in it. It is that you cannot check it without correctly naming somebody's intent.

Where This Argument Leaks

The conjunction only costs the artifact something if the terms are independent, and writing commitments down separately does not make them separate.

Two commitments are dependent when they can fail for the same reason without that reason appearing in either one. Every entry I write came out of one pipeline, one vendor's data, one classification scheme, one person's model of the machinery. A systematic error upstream does not corrupt one entry. It manufactures agreement across all of them.

This happens in my own book with my own model. The absence condition, the break condition and the tape signature for a given mechanism are all computed from the same reconstructed order book. A depth-estimation error that bites at thin quotes moves all three in the direction the mechanism predicts, and none of the three entries mentions depth estimation anywhere. Three commitments, one coincidence, and a bill I thought I had charged three times.

A shared upstream is invisible precisely because it is shared by everything you are looking at.

I do not have a clean solution to this and I am not going to pretend I do. What I have is the requirement that each commitment names what it depends on, and the habit of asking of any two entries what single upstream fact could satisfy both. The honest position is that a ledger of twenty entries is worth considerably less than twenty times one entry, and the gap is not measurable from inside.

Say that out loud, because the failure mode here is the one the whole method exists to prevent: a discipline that generates the feeling of rigor faster than it generates actual rigor.

What a Failure Buys

None of this makes the test come back positive more often. It changes what a negative result is worth, and that is where the compounding actually lives.

Two people hold the same belief that their model survives a regime change. One built structure-first and holds it as a set of specific structural reasons. The other built data-first and holds it because the model held across the data. Both run the test. Both get the same single bit back. No.

For the first, that no is a located contradiction. The system was supposed to survive for stated reasons, so a failure means one of those reasons is wrong, and you can go thread by thread. Which mechanism behaved as I reasoned it would, and which did not.

For the second, the no is a verdict with nothing to attach to. There was never a per-parameter account of why anything contributed, so there is nothing for the failure to localize against.

Data is only as useful as the structure it is handed to.

This is also why I run the tests at all. I avoid treating structural reasoning as an empirical result. The logic tells me what to test. The data tells me whether the logic held. Regime testing is not the line between the two approaches. What happens when the test comes back negative is the line.

Decay Is Not the Objection People Think It Is

The standard argument against all of this is alpha decay. Weather has fixed physical laws and forecasting skill persists. Market edges die. Therefore markets have no stable causality and mechanism talk is decoration.

Decay itself is a causal phenomenon, and it has more than one cause. An edge can be arbed away as other participants discover it and act on it. A rule change can remove the structural condition it fed on, decimalization did exactly that. Or it was never an edge at all, just nonstationary noise that has since reverted. None of these means there was never a mechanism. Competition is simply one of the things a real mechanism has to include.

From the return series alone these are indistinguishable: money was there, now it isn't. But a mechanism names the specific market condition its returns depend on, and that condition is observable in the tape right now, independent of whether the returns are still coming. Name it carefully, though, because this is where the whole check is decided. The condition was never just that the inefficiency forms. Its presence never paid me; capturing it did, so the commitment always included the capture, that the window stays wide enough to reach, the queue holds, the fills clear where they should.

So I check it directly, and the check splits two ways. If the condition is gone, and that includes its capturable half, the depth that used to be there, or fills that used to clear and now slip while someone faster takes the window, then the decay is explained and the edge died exactly where the mechanism said it would. Competition lives here, not in some junk drawer of unexplained deaths: a crowded edge shows a changed condition, the raw pattern intact while capture rate falls against an inefficiency count that has not moved, and that change is in the tape. If instead the condition is fully intact, pattern present, fills still clean, window still actionable, and the returns died anyway, then every mechanical part of the machine checked out and the output stopped regardless. Nothing broke, because nothing in the machine was driving it. This is the same contradiction as the test that fails to break, the model run through the exact regime it was supposed to die in, coming out intact: the condition you named as fatal is present and the thing you predicted would happen didn't. The correlation failed while every component verified working, which is exactly what an artifact looks like when the coin finally stops landing your way.

The correlation-holder cannot run any of this. He knows the returns stopped; he was never holding a claim about what they depended on, so a flat line is all he gets, and he cannot tell a competitor eating the edge from an artifact that was always noise.

The obvious objection is that this is a distinction without a difference. Both forks end the same way, an edge that used to pay and doesn't anymore, so who cares which death it was. That is right about the strategy that died and wrong about everything next to it. The dead strategy is the cheap loss; it is already gone either way. The fork is about the positions that have not died yet. If the condition is gone, the mechanism was real, and I know both whether it can return and what class of thing to look for next, because I understand what it fed on. If the condition is intact and the returns died anyway, the mechanism was probably never there, and the correct move is the opposite: stop trusting everything else built on that same reasoning, because those systems are still running only because their returns have not happened to fail yet. The correlation-holder cannot make either move. He buries the strategy and keeps trading everything adjacent to it, unable to tell whether the neighbor rests on the same rotten reason, because he never held a reason, only the curve. Same flat line, two opposite instructions for the rest of the book. Correlational methods tell you what stopped working, after it stopped. A mechanism tells you why, and comes with its own expiration condition attached at conception.

What This Actually Buys

Not proof. I cannot prove a mechanism is exogenous, and that is a property of the problem rather than a defect in my implementation. A structural story that survives every test I can construct could still be a coincidence that also happens to produce coincidentally consistent side-facts. I cannot rule that out. Neither can anyone.

What I can do is make the artifact explanation expensive.

There are three available stances. Certainty of presence: the edge is real and known. Certainty of absence: it is all coincidence. And the calibrated position: edges are usually noise, occasionally real, and the only honest verdict is a probability earned by subjecting a stated hypothesis to tests it could have failed and watching what survives.

So the honest description is never that the edge is real. What I have is a posterior belief. At some point, the coincidence explanation becomes less plausible than the mechanism, so I bet on the mechanism instead. The uncertainty never disappears, it just falls low enough to justify a trade, and position size comes from the mechanism itself and what the model was designed to exploit.

Coda

The question was what separates a true signal from noise that survived your backtests, and the answer is not a better test on the returns. Every instrument in that direction consumes the record where the two hypotheses agree by construction.

What separates them is that one of them owes you things beyond the record and the other owes you nothing. A mechanism has a balance point and a wobble and a waterline. An artifact only ever has the flips.

So write down what the mechanism owes before you look. Make the entries capable of being wrong, not merely capable of being measured. Keep them from resting on anybody's intent. Check whether two of them are secretly the same entry. When one eventually fails, take the located contradiction, which is the thing you actually paid for.

None of that yields certainty. It yields a bill the artifact has to pay, and a posterior low enough to act on instead of a certainty you pretend. What I act on it with, the size, comes from the mechanism.