Odd Question — Methodology

Effective date: September 1, 2026 Last updated: September 1, 2026

Note to reviewer: This describes what the pipeline actually does today, not an aspiration. It will be revised as the weighting model expands.

What Odd Question is

Odd Question runs a daily house question — the same question, for everyone, every day — alongside personalized content each member opts into. The house question is what every "national finding" is drawn from. Personalized ("Pulse") content is never presented as a national or representative number; it's explicitly labeled as a segment or as exploratory.

Who answers

Respondents are self-selected app users, not a probability sample. This is true of essentially all consumer polling panels, including well-known commercial ones — the honest way to make a self-selected panel usable for research is to be precise about how it's corrected back toward the population, not to pretend the panel itself is a random sample. That correction is what the rest of this page describes.

Assigned vs. organic

Every vote is tagged assigned or organic. Assigned means the app placed the question in front of the member — not their own taste pulling it up. Only assigned votes, from members who pass our trust checks (below), ever count toward a representative finding. Organic votes (personalized/self-selected content) never do.

Trust and quality checks

Every vote carries a trust score. A vote's trust is lowered by automated anomaly detection — implausibly fast response times, or failing a periodic attention check (an obvious-answer question occasionally mixed into the daily drop, same format as any other question). Only votes at or above a trust threshold enter the pool used for representative findings; low-trust votes are excluded, not corrected or reweighted. Separately, sustained rate or speed abuse (not a single missed check) results in full exclusion from representative findings, not just a lowered score.

Weighting — joint raking on five demographic axes

Findings are weighted so the panel is designed to approximate the real US population, rather than reported as a raw average of whoever happened to answer — and this is joint weighting, not five independent corrections layered on top of each other. Concretely: every respondent who has self-reported all five axes below gets a single weight, computed by iterative proportional fitting (raking, 30 iterations) so that the weighted panel is brought into alignment with the real population's age, gender, race/ethnicity, household income, and political party distribution all at once — not just each one individually. Raking reduces bias on the margins it targets; it does not guarantee the result is unbiased on dimensions it doesn't weight for (see "opt-in bias" below).

Only respondents who've opted into and answered all five self-report questions enter the raked pool — gender, race, income, and party are each a separate, optional opt-in, so this pool is smaller than the raw respondent count for that question, and it is a self-selected subset of respondents (people willing to disclose income, race, and party), not a random slice of the panel. Raking corrects the disclosed group's demographic mix; it does not correct for whatever differs between people who disclose and people who don't.

A finding only uses the full raked weighting once the eligible pool clears two floors, not one: at least 300 respondents who self-reported all five axes, and an effective sample size (see below) of at least 150 after weighting. The second floor exists because raw respondent count alone doesn't guarantee stable weights — a small panel can hit a raw-N target while the weighting itself still produces a handful of extreme, high-influence weights that quietly gut the effective precision. Below either floor, the finding falls back to age-only weighting — the same post-stratification approach used before raking existed, weighting each age band's share of the answer by its real population share (with a 20-respondent-per-band floor) — and every deliverable states plainly which of the two was actually used for that specific number. We never present a thin raked result as if it were solid.

Weights are capped, and we report how often. After raking converges, any individual respondent's weight is capped at 4x the average and the panel is renormalized — a standard safeguard so one rare demographic combination (e.g. a respondent who's simultaneously in several under-sampled categories at once) can't single-handedly swing a finding when the panel is still small. Every raked deliverable states what percentage of that specific finding's respondents actually hit the cap — a cap that never binds is meaningless reassurance, and a cap binding on a large share of a small pool is a real signal, not something to bury in a footnote.

How confident is a given finding

Every finding reports which weighting mode actually produced it (full raked, or the age-only fallback), plus two numbers alongside the headline result:

What we don't do

Track record

Every prediction question, once its real-world outcome is known, gets scored against what actually happened — the crowd's Brier score (0 is a perfect call, 2 is the worst possible), computed the same way any member's personal forecasting record is, aggregated across everyone instead of one person. The full history, updated as each prediction resolves, is public at oddquestion.app/track-record.

Questions

If you're evaluating Odd Question as a data source and have a methodology question this page doesn't answer, contact us at hello@oddquestion.app.