Statistical Significance and Economic Value in Investment Research

A report says, “The p-value is only 0.001, proving the strategy works.” Lin notices that the reported advantage is about 0.20 percentage points per period, while assumed implementation costs are 0.25 percentage points. Even if the statistics are correct, what remains to invest in?

Statistical testing and economic value answer different questions. Read a small, explicitly conditioned test, then examine its costs. The useful skill is understanding what the calculation compared and what it left unproved, rather than memorizing a threshold.

What you will be able to do

• Explain a null hypothesis, effect estimate, interval, and p-value without calling them probabilities of investment success.
• Compare two test outputs while distinguishing insufficient evidence from evidence of no effect.
• Preserve the test's prior specification, sampling assumptions, costs, and remaining validation gaps.

State the question before looking at results

This independent synthetic exercise does not represent C2's historical backtest or alter the C4 ledger or C6 performance cards. Assume ten independent, identically distributed experiments of equal length. Each records strategy return minus a consistently defined benchmark return. For the simple t-test, also assume these differences come from a normal population with unknown mean and variance.

Those are model assumptions, not facts established by inspecting ten observations. They do not imply that financial returns are normally distributed or independent. All numerical differences below use percentage points, before the separately stated incremental implementation costs. They are not risk-adjusted alpha.

Before revealing the values, define the question: “Does the population mean gross active return differ from zero?” The null hypothesis sets that mean at zero; the alternative says it differs from zero. Use a two-sided test because either direction addresses the question. Do not choose a positive-only alternative after seeing positive data.

Set a 5% significance level in advance as this exercise's rejection rule. It is a methodological convention, not approval to invest. A question about net returns exceeding an economic hurdle would need its own prior definition. A test of gross return against zero cannot quietly answer it.

Read the effect before the significance label

Experiment A supplies ten observations: 0.10, 0.20, 0, 0.30, 0.10, 0.20, 0.40, 0.10, 0.20, and 0.40 percentage points. Their arithmetic mean is 0.20, sample standard deviation approximately 0.1333, and standard error approximately 0.0422 percentage points.

The standard deviation describes variation among observations. The standard error describes the sampling scale of the estimated mean under the stated model; here it is the sample standard deviation divided by the square root of ten. A smaller standard error does not mean a single investment outcome has become less variable.

The one-sample t statistic divides the difference between estimated mean and null mean by that standard error. It is about 4.7434: the mean is that many estimated standard-error units from zero. This number is neither a return nor a multiple of capital to invest.

Using nine degrees of freedom, the independently checked two-sided p-value is approximately 0.00105. The 95% confidence interval for the mean is about 0.1046 to 0.2954 percentage points per period. NIST describes the corresponding mean interval and test framework.[1]

Under the assumptions and predeclared 5% rule, the result is inconsistent enough with zero mean to reject the null. That addresses one population-mean question. It does not show that the strategy passed every research, cost, or execution requirement.

What the p-value does not measure

Here, the p-value is the probability, assuming the null and statistical model, of a t statistic at least as extreme as the observed one, counting both tails. It is not the probability that the null is true, or the probability that the result arose from chance alone. The ASA statement warns against those interpretations and distinguishes significance from effect size and practical importance.[2]

The rejection rule can make errors. If the true mean is zero and the model and single predeclared design apply, the 5% level concerns the frequency of erroneous rejections over repeated uses of the procedure. It does not assign a 5% probability of error to this particular conclusion.

Failure to reject also does not prove zero. An effect might be small, observations too few, or noise too large to distinguish it. Power asks how often the method would reject for a specified nonzero effect. It depends on effect size, count, variability, and design; it is not another name for the p-value.

Keep the estimate and interval beside either label. A p-value of 0.049 versus 0.051 does not divide extraordinarily reliable knowledge from useless information. Inspect design and uncertainty on both sides of the chosen threshold.

The 95% refers to repeated construction

Under the relevant conditions, repeatedly sample and build intervals using the same procedure. About 95% of those intervals would contain the fixed true mean. That long-run coverage describes the method.[1] A particular calculated interval either contains the mean or does not; this frequentist interpretation does not assign a 95% probability to the parameter lying inside it.

It is also not a 95% range for the next return. Uncertainty about a population mean differs from variability of an individual outcome. Positive endpoints for the mean do not guarantee a profit in every period.

Report “a confidence interval for the mean under the specified model,” preserving the sample and assumptions. Do not replace it with “future returns will be at least 0.1046 percentage points.” The interval has not capped tail losses, even if none appeared in the observations.

A significant gross effect can disappear after costs

Now apply a fixed incremental cost of 0.25 percentage points to every experiment. This represents extra implementation cost relative to the benchmark. It is a separate teaching assumption, not C2's fee schedule or a broker quote. The inputs were gross of this cost, so this is not a second deduction from already-net returns.

The gross sample mean of 0.20 minus 0.25 gives a net sample mean of −0.05 percentage points. Because every observation loses the same constant, its sample standard deviation is unchanged. The net mean interval shifts down by that amount, to approximately −0.1454 through 0.0454 percentage points.

The gross significance result has not established a positive sample-average net advantage. Nor does the net interval crossing zero prove certain loss. Retain the possible mechanism, then investigate costs, capacity, implementation, and additional evaluation. The old gross p-value is not permission to deploy.

Real costs can depend on order size, conditions, and execution rather than remain constant. Separate cost scenarios and missing evidence. Translating an interval by a fixed amount is a property of this controlled exercise, not a general solution to uncertain costs.

A larger mean can remain harder to identify

Use completely separate synthetic experiment B, with equally long gross active-return observations under the same definition. Its five values are −1, 0.2, 0.4, 0.6, and 1.8 percentage points. Retain the independent, identically distributed, normal-population and predeclared two-sided-test assumptions.

B's mean is 0.40 percentage points, above A's 0.20. Its sample standard deviation is 1.00 and standard error about 0.4472. The t statistic is approximately 0.8944, two-sided p approximately 0.4216, and 95% mean interval approximately −0.8417 to 1.6417 percentage points.

This is not a 42.16% probability of no effect. The design supplies insufficient evidence to reject zero under the chosen rule, and the interval remains wide. A separately assumed fixed cost of 0.05 leaves a net sample mean of 0.35 but does not erase its uncertainty.

Do not rank the two strategies by either p-value or mean alone. A's gross estimate is more precise but its stated cost consumes it. B's effect estimate is larger and much less precise. Neither card alone establishes an investment conclusion.

“A is significant and B is not” also does not establish that their true effects differ. Testing each against zero and testing the difference between them are separate questions. The latter needs information about pairing and shared conditions. No such direct test has been performed here.

Check which assumption the financial data could violate

Adjacent months may share a market regime. Annual holding windows rolled monthly overlap. Treating trades, daily rows, or duplicated samples as independent observations can understate standard errors and invalidate nominal interval coverage or error control.

First examine dependence, overlap, instability, extreme values, and selection. Then choose a suitable method. More advanced time-series inference is further study. “This independent-observation test is unsuitable; another analysis is needed” is a valid result without forcing a p-value from inappropriate data.

Ten independent demonstrations do not validate a long-term strategy. They have not addressed historical membership, signal availability, fills, costs, or evaluation outside development. With C2, build the specified ledger and establish the statistical question and data conditions. Today's ten values are not SPY history.

Many attempts create another limitation. Testing numerous windows, parameters, and directions before displaying only the smallest p-value changes the interpretation of a single predeclared test. Save all attempts. Multiple testing and selection receive further treatment in G42: overfitting and repeated trials; falling below 0.05 does not bypass them.

Explain the outputs yourself

Explanation question: Why does A's p-value of about 0.00105 not mean a 99.895% probability of future profit?

Application question: May a report describe B's nonsignificant result as “proof of no advantage”? Identify the test's object and missing evidence before checking the guide.

The first p-value describes a statistic's tails under zero mean and the assumed model. It does not directly estimate future profitability, skill, or implementation. For B, say that this design did not supply sufficient evidence to reject; consider its wide interval, observations, and power instead of turning nonrejection into proof of zero.

Change the cost assumption

Retain A's ten observations and statistical assumptions. In another predeclared cost scenario, use fixed incremental cost of 0.05 percentage points. Calculate net mean and its 95% mean interval, then name an investment condition that remains unproved even if both endpoints are positive.

The net mean is 0.15 and the interval approximately 0.0546 to 0.2454 percentage points. Within this constant-cost model it lies above zero. That has not established real costs, future persistence, tail risk, capital capacity, or executable orders. It is not a guaranteed minimum gain of 0.0546 per period.

Changing the cost is an explicit sensitivity exercise. Preserve the earlier 0.25 scenario and the reasons for both. Reporting only the easiest scenario after inspecting significance would hide a choice relevant to the conclusion.

Save a test-and-value interpretation card

Record the research question, null and alternative, direction, significance level, sample scope, and all attempts. Add mean, sample variability, standard error, interval, p-value, and assumptions. Missing fields stay missing rather than disappear behind a screenshot of stars.

In a second section, state whether costs were deducted, their assumptions, net effect, possible losses, and unfinished implementation checks. Explain in your own words what can be rejected and what remains unproved. If the data conditions fail, identify the method or conclusion that must wait.

Hide the word significant. Can the card still explain magnitude, uncertainty, and economic cost? Check that model assumptions have not become verified data facts. Those two checks reveal more understanding than repeating the threshold.

References

1. NIST: Confidence Limits for the Mean. Mean intervals, the t framework, and repeated-sampling coverage.
2. American Statistical Association: 2016 statement release on statistical significance and p-values. March 7, 2016; interpretation, thresholds, effect magnitude, and complete reporting. Not evidence of a financial strategy's effectiveness.

Sources checked October 30, 2026. All observations are independent original synthetic teaching inputs and the calculations were independently reproduced. AI-assisted draft; independent AI full-text and bilingual review is complete. Human financial review and target-reader testing have not been conducted and are not scheduled for this production round.