Collinear inputs: stable fitted values can hide unstable slopes
A regression can fit observed outcomes closely while providing weak evidence about the separate contributions of similar inputs. When predictors move together, changes in their coefficients can offset one another. Inspect the design and the intended claim before interpreting a stable fitted line as stable component attribution.
The deliverable is an auxiliary-regression calculation, a coefficient-and-fit sensitivity table and a statement of which combinations the data actually distinguish. A variance inflation factor is a diagnostic about the predictor design under a specified model. It is not a causal test or an automatic instruction to delete a variable.
Define the design rather than inspecting one coefficient
Consider two hypothetical quantitative workload measures, X1 and X2, in documented normalized units. The outcome Y is completion time in minutes for a fixed research task. These predictors are measurements, not randomly assigned independent treatments. No coefficient below should be read as the causal effect of changing workload while everything else stays fixed.
The statsmodels VIF reference describes variance inflation and its relationship to the explanatory-variable matrix. Record exactly which columns, observations and intercept convention enter the matrix. A diagnostic computed on another sample or without the intended intercept may not match the fitted model.
The following five rows are invented. The small size makes the arithmetic visible; it does not support precise population conclusions. Retain the original row identities and full numerical precision instead of inferring collinearity from a rounded chart.
| Row | X1 | X2 | Original Y, minutes | Added perturbation |
|---|---|---|---|---|
| A | 1 | 1.1 | 15.4 | +0.04 |
| B | 2 | 1.9 | 19.5 | −0.13 |
| C | 3 | 3.2 | 25.7 | +0.20 |
| D | 4 | 3.8 | 29.3 | −0.17 |
| E | 5 | 5.0 | 35.1 | +0.06 |
Rebuild the auxiliary regression
Regress X2 on an intercept and X1. Its fitted value is 0.09 + 0.97X1. The residuals are 0.04, −0.13, 0.20, −0.17 and 0.06, exactly the perturbation column in this teaching table.
The residual squared sum is 0.091. X2 has mean three and total centered squared sum 9.5. Thus the auxiliary R-squared is one minus 0.091 / 9.5, approximately 0.990421. With an intercept and the standard definition, VIF is one divided by one-minus-R-squared, approximately 104.3956.
This uses an auxiliary model for one predictor given the others; it is not the R-squared of Y on the full predictor set. With two nonconstant predictors and an intercept, the same squared correlation gives the VIF for either predictor. More predictors require the relevant auxiliary regression rather than substituting one pairwise correlation.
Under the usual homoskedastic linear-model variance calculation, VIF multiplies a coefficient’s variance relative to a design with the same predictor variation but no linear dependence on the other included columns. A VIF around 104 is a variance multiplier, not a 104-fold increase in the outcome’s error or the coefficient’s size. The associated standard-error multiplier is its square root, under that comparison’s assumptions.
Compare slopes and fitted values under a small perturbation
Fit Y on an intercept, X1 and X2. Then add the last column to Y and fit the same design again. The largest response change is 0.20 minutes, or twelve seconds.
| Quantity | Original response | Perturbed response |
|---|---|---|
| Intercept | 9.897802 | 9.807802 |
| X1 coefficient | 1.231868 | 0.261868 |
| X2 coefficient | 3.802198 | 4.802198 |
| Maximum fitted-value change at these rows | — | 0.20 minutes |
The X1 slope falls by 0.97 while X2 rises by one. The example deliberately chooses a sensitive direction: the response perturbation equals X2 − 0.09 − 0.97X1. Substituting that expression shows why the intercept and two slopes change by exactly −0.09, −0.97 and +1 while leaving the residual pattern unchanged.
This is a constructed sensitivity, not a statement that every random twelve-second measurement error produces these shifts. It demonstrates that the observed predictor geometry allows large reallocations between components alongside relatively small changes in fitted outcomes at the observed combinations.
An explicit joint contrast helps show what remains stable. If X1 increases by one and X2 increases by 0.97, following the auxiliary fitted direction, the original fitted change is approximately 4.92 minutes. The perturbed model gives the same 4.92. Algebraically, the coefficient changes contribute minus 0.97 plus 0.97 times one, which cancels.
That joint association is different from holding X2 fixed while changing X1 alone. The sample contains little independent variation for the latter interpretation. A researcher may therefore have more defensible evidence about movement along the workload pattern than about an isolated component. This is still an association under the specified model, not a causal intervention result. Report the actual contrast and units rather than calling every linear combination a component effect.
The original fitted values are not proven accurate for new tasks. At the unobserved combination X1 = 5 and X2 = 1.1, the two fitted models differ by 1.1 − 0.09 − 0.97 × 5 = −3.84 minutes. Both coordinates are individually within the observed ranges, but the combination is far from the pattern in the table. Stability on the training rows is not a general prediction guarantee.
Distinguish near dependence from exact non-identification
In the main example, X2 is close to a linear function of X1 but not exactly equal, so the full design has separate coefficient estimates. Their reliability still needs the sampling model and suitable uncertainty assessment. Five observations with three fitted parameters leave very little residual information.
As a short boundary, replace X2 with X1 exactly. Then a model using both slopes identifies their sum, not the two separate slopes. More repetitions of identical predictor combinations cannot create the missing contrast. A software package may drop a column or select a numerical solution, but that choice is not evidence for a unique substantive contribution.
Keep this exact case separate from an experiment that deliberately aliases factorial effects. Here the main audit concerns nearly redundant measured inputs and the stability of their fitted slopes. The design choice and substantive questions need to be recorded, not reduced to one shared mathematical label.
Do not mistake rescaling for new information
Changing minutes to seconds changes coefficient units. Standardizing predictors can improve numerical conditioning and make some comparisons easier, but ordinary linear rescaling does not change the usual auxiliary R-squared or VIF when an intercept is included. It does not create independent variation between the workload measurements.
For example, multiplying X2 by ten changes its auxiliary fitted equation to 0.9 + 9.7X1. Both its residual and total squared sums multiply by one hundred, leaving their ratio unchanged. Its outcome-model slope divides by ten while fitted outcomes remain the same. Different coefficient units are not additional identification.
A matrix condition number can be sensitive to units, and a changed basis or predictor set can alter diagnostics and coefficient meanings. Report those changes explicitly. A combined workload index may be a useful new target, but its coefficient is no longer the separate contribution originally claimed for X1 or X2.
The VIF documentation presents a threshold of five as one recommendation. Do not convert it into a universal deletion rule. Removing a genuine confounder can introduce bias, and low VIF does not establish valid causal adjustment. A regularized fit can stabilize estimates through additional assumptions or penalties without supplying missing component evidence.
Deliver a coefficient-identification worksheet
| Record | What to retain | Check |
|---|---|---|
| Design matrix | Columns, coding, intercept, sample | Same design as the fitted model? |
| Auxiliary models | Predictor target and R-squared | VIF calculated from the correct regression? |
| Response perturbation | Original and changed values | Structured scenario or random-error model? |
| Results | Coefficients and fitted-value changes | Attribution and prediction separated? |
| Intended claim | Association, prediction or causal effect | Does the design support it? |
| Remedy | More contrasts, combined target or model limits | What assumption or information changes? |
Seek observations where the predictors vary separately when feasible and relevant to the target population. Preserve measurement definitions and investigate duplicated constructs before collecting more of the same pattern. If separate attribution remains unsupported, report a joint association or a limited predictive result instead of choosing whichever individual slope is significant.
Plan new input contrasts without selecting rows for the outcome they produce. A dataset engineered to reduce a diagnostic by dropping inconvenient responses can introduce selection bias. More observations are useful only insofar as their design and measurement support the stated question.
Recheck after a measurement revision, a new predictor is added or new tasks change the joint input pattern. Save both coefficient and fitted-value comparisons. The conclusion should explain what the data can distinguish, rather than equating a stable fitted average with a reliable explanation of each component.
FAQ
Does a high VIF mean the model cannot predict anything useful?
No. Some predictions can remain stable within the observed joint pattern while individual slopes are fragile.
Will standardizing the columns remove collinearity?
Not by creating new information. Ordinary linear rescaling leaves the usual intercept-based auxiliary R-squared unchanged.