Learn

AI-Assisted Evidence Checks

Use AI to help read and compare material while independently checking citations, figures and uncertainty.

94 guides

• Serial dependence: repeated months are not independent evidence

  Calculate mean uncertainty under explicit serial-dependence assumptions, distinguish finite and long-run effective counts, and preserve actual time spacing.

• Backfilled data: test when observations became available

  Track observation periods and availability intervals separately to exclude late backfills and revisions from historical AI research cutoffs.

• Bilingual AI research: verify qualifiers before fluency

  Check bilingual AI research claim by claim: preserve speaker, modality, time, negation, units and defined terms before accepting a fluent translation.

• Censored data: below a threshold does not mean zero

  Preserve censoring thresholds and inequalities, distinguish censored values from missing or truncated records, and propagate bounds into averages.

• Chain indexes: explain why real components do not add up

  Explain chained-dollar nonadditivity with a reproducible Fisher quantity example, then choose compatible measures for shares and growth contributions.

• Calendar alignment: do not invent monthly observations

  Align quarterly and monthly evidence without inventing observations. Preserve covered periods, release dates and aggregation rules in mixed-frequency research.

• Citation chains: trace repeated claims to the origin

  Trace AI citations to original observations, distinguish repeated reporting from independent evidence, and inspect how qualifiers change along the chain.

• Separate AI interpretation from reproducible calculations

  Keep verified inputs, normalization rules, formulas, and interpretation separate so AI-assisted calculations remain reproducible and reviewable.

• Effect scales: reconstruct risk differences, ratios and odds

  Reconstruct risk differences, risk ratios and odds ratios from one binary table, checking baseline probabilities, event reversals and outcome-based sampling.

• AI citation checks: does the source support the sentence?

  Check whether each AI citation exists, supports the exact sentence, and matches its scope, then revise partially supported or overstated claims.

• Classifier alerts: recalculate precision at the target base rate

  Recalculate alert precision from sensitivity, specificity and the target base rate. Translate a filing classifier into review workload without treating alerts as findings.

• Cluster bootstrap: resample the independent unit

  Resample whole company clusters with replacement, preserve repeated cluster instances and paired outcomes, and distinguish simulation effort from sample information.

• Collinear inputs: stable fitted values can hide unstable slopes

  Rebuild an auxiliary regression and variance inflation factor, then perturb a small dataset to distinguish stable fitted values from fragile component slopes.

• Competing exits: estimate the event that actually interests you

  Reconcile competing terminal exits with cumulative state probabilities, distinguishing Aalen–Johansen updates from failure-only survival and cause-specific hazards.

• Compositional data: a rising share can hide unchanged activity

  Reconcile original amounts, closed shares and component ratios, checking lost scale, reference choices, structural zeros, detection limits and missing totals.

• Uncertainty propagation: keep covariance between inputs

  Propagate uncertainty through differences and ratios using covariance. Distinguish shared measurement error, first-order approximations, scenario ranges and unsupported precision.

• AI counterevidence: test a thesis with disconfirming questions

  Turn AI criticism into bounded evidence searches. Separate proposed challenges from observed facts, avoid biased search loops and record what could actually weaken a thesis.

• Collider checks: a control variable can create an association

  Trace how conditioning on a common effect creates an association, using complete and selected tables while keeping structural assumptions separate from causal estimates.

• Counterfactual comparisons: test the comparison group

  Test comparison groups through selection, pre-event paths, common shocks and spillovers before treating a difference-in-changes as causal.

• Dataset snapshots: freeze rows as well as documents

  Freeze research row populations, input objects, crosswalks and transformations with key-set and value checks that distinguish reproduction from refresh.

• Delayed entry: build the risk set when observation starts late

  Construct delayed-entry risk sets from actual observation intervals, checking event-time denominators, left truncation assumptions and population coverage.

• Unit conversion: enforce dimensions before aggregation

  Validate dimensions, scale factors and conversion direction before AI aggregates mixed units, currencies, time periods or per-share metrics.

• Cutoff studies: audit the assignment rule before a causal claim

  Audit cutoff rules, local outcome and receipt limits, fuzzy Wald ratios and identifying assumptions before interpreting a threshold comparison as causal.

• AI document comparison: verify changes against both versions

  Compare report versions without mistaking a moved table or changed denominator for business improvement. Classify edits, verify both sources and update dependent research.

• Duplicate rows: define the observation key before deleting

  Define observation keys before deleting duplicate financial rows. Separate equal values, repeated publications and revised facts while retaining an audit trail.

• Aggregate evidence: do not infer individual behavior

  Distinguish group-level evidence from individual behavior. Use feasible overlap bounds and a hypothetical cross-tabulation to expose what aggregate percentages cannot identify.

• Equivalence checks: define acceptable differences before testing

  Prespecify acceptable mean differences and apply paired equivalence tests, separating practical equivalence from nonsignificance and individual agreement.

• Event windows: identify overlapping explanations

  Design event windows with source timestamps, market-session conventions and overlapping-news registers before assigning outcomes to one disclosure.

• Evidence constrained prompts: request extraction before judgment

  Use a copyable evidence-first task specification: extract passages, classify claims, preserve unknowns and verify source boundaries before asking AI for interpretation.

• Company ID joins: use effective intervals, not tickers

  Join company data with dated entity-security crosswalks, interval checks and cardinality tests so reused tickers do not stitch unrelated histories.

• Evidence independence: count mechanisms, not documents

  Map shared measurements, assumptions and selection rules before counting AI research sources. Use an evidence-cluster worksheet without inventing independence scores.

• Event rates: count time at risk before comparing frequencies

  Build exposure-time ledgers for first and recurrent events, compare rates with consistent units and separate time-based frequencies from fixed-period probabilities.

• AI factual errors: triage invented details systematically

  Isolate AI-generated names, dates, documents, quantities, and explanations for source-based checks before they become approved research evidence.

• Aliased designs: identify effects a reduced test cannot separate

  Reconstruct a fractional factorial design, identify aliased effect columns and plan complementary runs instead of treating repeated combinations as identification.

• Factorial comparisons: test whether two changes work differently

  Rebuild four-cell research experiments to distinguish conditional effects, average main effects and interaction before claiming two workflow changes add up.

• Funnel asymmetry: investigate missing studies without counting them

  Connect study effects and precision to dated protocols and reporting status, distinguishing selective non-reporting from heterogeneity without inventing missing studies.

• Human review for AI research: assign checks by consequence

  Assign human review to specific claims and calculations by consequence, document completion criteria, and reopen checks when dependencies change.

• Interrupted series: distinguish immediate change from a new trend

  Reconstruct immediate, period-specific and average changes in interrupted series, checking centered time, slope parameters, residual dependence and measurement shifts.

• Research labels: stop future outcomes leaking into inputs

  Prevent outcome labels from leaking into research inputs. Track decision cutoffs, label maturity, source versions and preprocessing rules across evaluation splits.

• Histogram shapes: test bins before claiming two populations

  Test histogram widths, origins and normalization before interpreting business-data modes. Preserve raw observations and separate visual clusters from economic populations.

• Log transforms: preserve zeros, negatives and the target

  Check log-transform domains, offsets and units, preserve zero and negative observations, and distinguish geometric summaries from arithmetic-mean targets.

• Mediator controls: state whether you want total or direct effects

  Separate total and controlled direct effects in explicit mediator models, checking hidden common causes, interactions and the assumptions behind adjusted coefficients.

• Prediction intervals: keep one new case separate from the mean

  Compute separate intervals for a conditional regression mean and one new observation, checking leverage, residual variance and known versus estimated scale.

• Long document AI review: test coverage before trusting summaries

  Test long-report AI coverage with a document map and independently located facts. Separate extraction, retrieval and interpretation failures before using a summary.

• AI assisted research memo: a final evidence release check

  Release an AI research memo only after checking claims, source support, calculations, time and scope. Use a worksheet that preserves material gaps and honest reviewer identity.

• Measurement agreement: high correlation can hide unusable gaps

  Compare paired measurement differences with a prespecified tolerance, distinguishing individual agreement limits from correlation and mean-bias intervals.

• Missing data in AI output: preserve unknowns explicitly

  Distinguish missing values from reported zero in AI research. Preserve search scope, extraction failures and eligibility rules before averaging or comparing figures.

• Multiple tests: count every question behind the best result

  Record the full test family, distinguish FWER from FDR, and reproduce Bonferroni and Benjamini-Hochberg decisions under explicit assumptions.

• Missing observations: bound a result before imputing values

  Bound full-sample means using known sums and defensible outcome limits before imputing missing observations or trusting complete-case averages.

• Model support: marginal ranges can conceal joint extrapolation

  Check joint input combinations with a full-rank triangle example, reconstruct convex weights and distinguish an in-range coordinate from supported interpolation.

• Negative baselines: describe loss changes without false growth

  Compare narrowing losses and profit transitions using signed amounts. Explain negative and zero baselines, and label alternative percentage scales explicitly.

• OCR evidence checks: protect signs, decimals, footnotes

  Verify OCR signs, decimals, column placement, and footnotes against original page images, then recalculate every conclusion affected by corrections.

• Negative controls: test an explanation where it should fail

  Challenge a research explanation with negative controls. Preserve the control rationale, source versions and diagnostic limits without claiming automatic causal correction.

• Ordinal scores: preserve order without inventing equal distances

  Audit ordered research ratings through category counts, monotone recoding and explicit thresholds before interpreting average scores as comparable quantities.

• Outliers: preserve raw values and treatment consequences

  Investigate investment-data outliers before treatment. Preserve raw values, distinguish errors from unusual economics and disclose effects on the conclusion.

• Rolling windows: avoid treating overlap as fresh confirmation

  Identify shared observations behind rolling research metrics. Separate mechanical overlap from new evidence and preserve raw values, window endpoints and release timing.

• Panel entry and exit: separate growth from composition

  Separate continuing-company growth from panel entry and exit. Reconcile a revenue bridge and distinguish dataset coverage changes from economic business creation.

• Paired comparisons: preserve the match before measuring change

  Preserve matched observation keys, calculate variance of within-pair changes, and distinguish paired precision from causal evidence or arbitrary matching.

• Population drift: recheck what a validated workflow now sees

  Monitor the population entering a research workflow. Decompose aggregate success into input mix and conditional quality while preserving label coverage and unknown formats.

• Noisy predictors: test attenuation before interpreting a slope

  Derive predictor-error attenuation under an explicit classic model, then test replicate averaging, correlated-error counterexamples and validation needs.

• Private research with AI: minimize what leaves your notes

  Build a minimum AI research packet, remove unnecessary private fields, check attachments and track copies without assuming aliases or deletion guarantee anonymity.

• Common support: find comparisons the data cannot identify

  Audit treatment support against target-population weights, separating structural zeros from sparse samples and labeling restriction, bounds and model extrapolation.

• Probability claims: test calibration with outcomes and scores

  Evaluate probability calibration with matured labels, reliability tables and binary Brier scores. Separate calibration from ranking, accuracy and test-set tuning.

• Quantile checks: fix the interpolation rule before comparing

  Reproduce type 1 and type 7 sample quantiles, preserve ordering and missing-value rules, and distinguish interpolation from observed tail evidence.

• Study power: choose the difference worth detecting in advance

  Plan study size and detectable differences from power, error scale and practical targets, checking t-test requirements and expected recruitment losses.

• Randomization tests: permute assignments allowed by the design

  Enumerate legal randomized assignments, define a sharp null and reproduce exact tails, paired restrictions and Monte Carlo calculations from a small experiment.

• Rank association: monotonic order is different from a linear slope

  Rebuild raw values and average ranks to distinguish monotonic association from a linear relationship, handle ties and avoid interpreting zero correlation as independence.

• Reviewer agreement: measure disagreement before adjudication

  Measure independent reviewer agreement before adjudication. Build a label cross-tabulation, calculate Cohen kappa and preserve disagreements without treating consensus as truth.

• Nominal to real data: choose a defensible deflator

  Choose and document a defensible deflator, align periods and base years, and distinguish calculated purchasing-power growth from physical volume.

• Repeated measurements: separate within-run and between-day error

  Design repeated measurements across days, pool within-day variance and isolate day effects before choosing a precision budget for the reported result.

• Restatement panels: keep original and revised observations

  Keep original and revised financial observations in separate selected panels, with filing identities, scope checks and a worked growth-rate revision.

• Extreme baselines: separate regression to the mean from recovery

  Diagnose change after extreme-baseline selection with a repeated-measurement model, distinguishing expected regression to the mean from evidence of genuine improvement.

• Retrieval misses: audit relevant evidence outside the results

  Audit relevant disclosures omitted from a fixed research search pool. Diagnose ingestion, alias, filter and ranking misses without claiming open-web recall.

• Rounded data: calculate the interval behind a reported total

  Use rounding intervals to test whether displayed components can reconcile to a total, preserving precision limits through differences and ratios.

• Search precision: classify false positives before summarizing

  Measure research search precision with a fixed relevance rule, classify false positives and compare query changes without discarding contradictory evidence.

• Seasonal adjustment: compare compatible series variants

  Compare seasonally adjusted and unadjusted series using compatible definitions, transformations and vintages before treating different rates as contradictory evidence.

• Sample selection: record who drops out at each filter

  Audit company sample selection with a stage-by-stage membership funnel, exclusion reasons, coverage denominators and filter sensitivity.

• Specification sensitivity: report the choices that move results

  Test whether research conclusions depend on sample, weighting and cleaning choices. Report a specification matrix without treating its range as a confidence interval.

• Repeated checks: set a stopping rule before looking again

  Set inspection and stopping rules before evaluating research workflows repeatedly. Separate operational monitoring from fixed-horizon and time-uniform statistical claims.

• Small proportions: show interval uncertainty beside the rate

  Calculate Wilson intervals for small audit proportions, handle zero failures explicitly, and keep confidence claims tied to sampling and grading rules.

• Conflicting sources: make AI expose the disagreement

  Expose source disagreements through an identity, date, definition, and revision matrix instead of averaging incompatible values or forcing consensus.

• Aggregate reversals: test within-group evidence before a claim

  Rebuild subgroup counts and fixed-composition success rates to explain aggregate reversals without confusing workload change with causal performance.

• AI research source versions: keep a reproducible register

  Connect AI research claims to the exact source versions, extracted packets and run IDs used. Track corrections without confusing byte changes, new periods and revised facts.

• Study synthesis: separate a pooled mean from differing effects

  Rebuild inverse-variance study synthesis, between-study heterogeneity and prediction intervals without mistaking a precise pooled mean for consistent effects.

• Standard error versus spread: what the error bar describes

  Identify whether an error bar represents standard deviation, standard error or a confidence interval, and verify sample units and assumptions.

• Survey estimates: weights do not determine the standard error

  Separate survey point weights from sampling-design variance, preserve clusters and replicate conventions, and calculate derived-statistic uncertainty correctly.

• Survivorship bias: rebuild the historical company universe

  Rebuild historical company membership with entries, exits, identity continuity and outcome coverage before AI summarizes a survivor-only dataset.

• AI table extraction: preserve headers and units

  Extract table values with full header paths, units, periods, and footnotes, and verify compatibility before calculating changes.

• Table joins: detect many-to-many row multiplication

  Detect many-to-many row multiplication in investment datasets. Define join relationships, reconcile totals and avoid inventing segment allocations.

• AI research time scope: separate known then from known now

  Control AI research timing with event, disclosure, retrieval, and cutoff dates, keeping later corrections separate from the earlier information set.

• Weighted averages: preserve the denominator across entities

  Choose weights that match the research question. Preserve ratio denominators, disclose missing coverage and distinguish combined margins from issuer averages.

• Prompt injection in research documents: keep source boundaries

  Maintain boundaries between research documents and instructions, minimize private context, and review outputs after suspicious source content appears.

• XBRL facts: match contexts before choosing a number

  Select XBRL facts by entity, period, unit, dimensions and filing version. Reconcile machine contexts to original disclosures and distinguish API omissions from zero.

• Evaluate an AI research workflow with a small reference set

  Build a reference set for AI research tasks with checked answers, failure categories and release rules. Compare workflow changes without claiming a universal accuracy rate.