Data Scientist Mock Interviews

Offline metrics (NDCG, Precision@k on time-split logs) rank candidates cheaply but inherit feedback-loop bias: you mostly evaluate on what the old system already showed users.

Why it matters: The eval question is where interviewers separate people who shipped models from people who trained notebooks.

Common mistake: A big offline lift proving real-world lift. Position bias and exposure skew routinely eat 30-50% of the gain.

Trade-off: Online wins trust but costs latency of iteration: name the guardrails and ramp speed you accepted as the price.

Ideal structure: Offline (NDCG, Precision@k, leakage checks) vs online (A/B, guardrails, ramp), and why offline can lie, with your own project as example.

Tip: Name the leakage check you ran.

Follow-up to expect: “Offline +12%, online +2%: list three plausible reasons”: exposure bias, novelty decay, and metric mismatch are the expected trio.

Not a tour. An audit trail: business question, data grain and mess, baseline, modeling choice with its cost, eval results, shipping decision, measured impact, and one honest regret.

Why it matters: It is the single most predictive DS interview question; everything the loop cares about (judgment, communication, provenance) shows up here.

Common mistake: Optimizing the story’s ambition over its traceability. Panels grade the second follow-up on your data story, not your headline.

Trade-off: Include the trade-off you actually made (complexity vs interpretability, speed vs rigor). Its absence reads as inexperience.

Ideal structure: Problem → data → baseline → modeling → eval → ship → impact (e.g., +3% CTR, -$12k/month), with a trade-off you made.

Tip: Quantify impact, not just accuracy.

Follow-up to expect: “What would you change today?” A specific upgrade (new label source, calibration layer, simpler model with features) proves the learning is real.

The per-department turn makes PARTITION BY mandatory, and “second highest value” versus “second row” is DENSE_RANK versus ROW_NUMBER. State which you are answering first.

Why it matters: Grain + semantics + performance in one question is exactly how DS SQL screens are written.

Common mistake: The two-MAX subquery still technically works for values but collapses into per-row scans without an index and confuses NULL departments.

Trade-off: Window scan (one pass, needs (dept, salary) index) vs self-join (reads twice). Name which plan EXPLAIN shows and why.

Ideal structure: DENSE_RANK vs ROW_NUMBER, window vs self-join, index on (dept, salary), and the tie semantics you chose.

Tip: State tie handling before writing SQL.

Follow-up to expect: “Only departments with 5+ employees, top 3 salaries.” HAVING versus WHERE placement on the rank filter is the expected follow-up.

Accuracy is meaningless at that imbalance. The decision moves to the cost matrix: missed fraud loses money, false declines lose customers, and the threshold sits where the marginal costs cross.

Why it matters: It tests whether you optimize business outcomes or leaderboard metrics; every production DS answer needs the cost sentence.

Common mistake: Tuning for best F1 by default. F1 bakes in equal costs you have not earned or checked.

Trade-off: Name whose review queue absorbs the false positives and at what labor cost. The threshold is an operations decision wearing a model hat.

Ideal structure: Precision-recall vs cost matrix, threshold via expected cost, and monitoring for drift.

Tip: Bring a cost number.

Follow-up to expect: “Approval rates drop and fraud barely moves: adjust or investigate?” The drift-before-tuning answer is the grade.

The structure is the answer: population and unit of randomization, primary metric with a pre-registered threshold, guardrails, minimum detectable effect driving the sample-size math, duration in full weekly cycles, and the stopping rule.

Why it matters: Experiment design appears in DS, analytics, and product-adjacent rounds alike. It grades causal hygiene under practical constraints.

Common mistake: Peeking daily and stopping at significance. Name sequential testing or a fixed horizon instead.

Trade-off: Shorter tests need larger effects; a 2% conversion lift demands days you may not have. State which constraint you traded.

Ideal structure: Hypothesis → metric + MDE + power → unit and splits → duration in cycles → guardrails → pre-registered decision rule.

Tip: Say “minimum detectable effect” once with a real number and the interviewer hears experience.

Follow-up to expect: “The test moves conversion but not revenue: what now?” Metric-hierarchy reasoning: which was the primary and why the mismatch is informative, not broken.

Quasi-causal reasoning: pre-post with seasonality controls, difference-in-differences across an unaffected segment or region, synthetic control with honest assumptions, and the sentence on what still confounds it.

Why it matters: The most common real job problem (no clean A/B available) and the strongest seniority signal in analytics interviews.

Common mistake: Presenting a trendline as impact; panels specifically probe what would have happened without the feature.

Trade-off: Every quasi-experiment trades internal validity for feasibility. Naming which assumption your result hangs on is the pass condition.

Ideal structure: State the counterfactual you cannot get, the best available proxy comparison group or time control, its threats, and the metric plus guardrails you still measure.

Tip: Volunteer the confounder interviewers would use to check you. It is the whole grade.

Follow-up to expect: “What evidence would make you downgrade the claim to inconclusive?” Pre-committing to a falsifier is the senior move.