Testing and verification

coreCommon atGoogleStripeAnthropicMetaRead firstEdge casesPractise in the Sandbox →

Verify in rising order of cost: trace one small example by hand, run the edge cases as a named table, then, when correctness is subtle, compare against a brute-force oracle on hundreds of random small inputs. Hand-picked tests check the cases you thought of. Random tests with an oracle find the ones you didn’t.

The idea in plain words

Testing in an interview isn’t QA. It is evidence. The interviewer wants to see that you know your code works for a reason, not because it compiled.

Three kinds of evidence, from cheapest to strongest:

• A hand trace. Walk one small input through the code, line by line, writing down the variables. It catches logic slips and shows the interviewer you can reason about your own code.
• Example tests. A short table of inputs with known answers: the example from the prompt, plus the edge cases. Each row is one idea, with a name.
• Property tests. Instead of an answer per input, state a rule that holds for every input (“the output is sorted”, “it matches the slow version”) and check it on many random inputs.

The last idea is the powerful one. Write a brute-force oracle: a slow, obviously correct version (try every pair, sort everything, recompute from scratch). Then generate small random inputs and compare. This is called differential testing, and it is how competitive programmers and library authors catch the bugs that hide from hand-written cases.

See it

Three small functions, each with a bug. The hand-written tests all pass, because they are the cases the author had in mind. Then run random tests against the oracle until one fails, and shrink the failure to the smallest input that still breaks.

Hand-picked tests vs random tests with an oracle

run it

function under test

second_largestmergeceil_div

1def second_largest(xs):

2 s = sorted(xs, reverse=True)

3 return s[1] if len(s) > 1 else None

Oracle (slow, obviously right): sorted(set(xs))[-2], or None

1. The tests you would write by hand

Run 3 hand-written tests

2. Random small inputs, checked against the oracle

Run 1Run until it fails↺ New seed

tests run0

statusnot started

Each random test builds a small input, runs the function and the oracle, and compares. Tiny sizes and a narrow value range make duplicates, empties and negatives show up quickly.

Notice two things. The counterexample usually appears within a few dozen tries, because the generator uses tiny sizes and a narrow value range (lists of 0 to 6 items, values 0 to 5), so duplicates, empties and ties show up all the time. And the shrunk input is small enough to trace by hand in under a minute.

When to reach for it

The situation…	Reach for…
you just finished the first version	A hand trace of the prompt’s example. Out loud, with a variable table.
the function has obvious edge cases	A table of named cases (empty, one, duplicates, boundaries).
the fast solution is clever (two pointers, greedy, DP)	A brute-force oracle plus random small inputs. The brute force is often the first thing you wrote anyway.
output has a structure you can check (“sorted”, “a permutation of the input”, “decode(encode(x)) == x”)	A property check, even without an oracle.
a multi-level problem gets level 2	Keep the level 1 tests and run them after every change. That’s your regression suite.
the interviewer asks “how would you test this in production?”	Unit tables, property tests with a library, plus integration tests for the I/O around it.

How it works

1. Trace the example by hand. Pick the smallest interesting input (3 to 5 elements). Write a table: one column per variable, one row per loop pass. Compare the final row with the expected answer.
2. Write the edge-case table. Each row is (name, input, expected). Run them all in one loop or one parameterised test, so adding a case is one line.
3. Write the oracle if correctness is subtle. Make it simple enough to be obviously right: nested loops, sort, recompute. Speed doesn’t matter.
4. Generate small random inputs with a fixed seed. Small sizes (0 to 8) and a narrow value range (say -3 to 3) are better than big random ones: they hit the edge cases often.
5. Compare and print the input on failure. The message must contain the exact input, so you can paste it into the table as a regression test.
6. Shrink before you debug. Remove elements and move values toward 0 while it still fails. A two-element failure is far easier to trace than a forty-element one. See debugging.

Table-driven tests without a framework

In an interview you may only have a plain file. A list of named rows and one loop is enough. The function here is ceil_div, chosen because negative numbers make it easy to get wrong.

Python

table_driven.py

def ceil_div(a: int, b: int) -> int:
    """Smallest integer >= a / b. b must be non-zero."""
    return -(-a // b)  # // floors, so negate, floor, negate = ceil


# 1. The table: (name, inputs, expected). Each row is one idea, named so failures explain themselves.
CASES: list[tuple[str, int, int, int]] = [
    ("exact", 6, 3, 2),
    ("rounds up", 7, 3, 3),
    ("zero numerator", 0, 5, 0),
    ("negative numerator", -7, 3, -2),   # -2.33 -> -2, not -3
    ("negative divisor", 7, -3, -2),
    ("both negative", -7, -3, 3),
    ("divisor one", 41, 1, 41),
]

if __name__ == "__main__":
    # 2. One loop runs every row; the message names the row and shows got vs want.
    for name, a, b, want in CASES:
        got = ceil_div(a, b)
        assert got == want, f"{name}: ceil_div({a}, {b}) = {got}, want {want}"
    print("ok")

TypeScript

table_driven.ts

import assert from "node:assert/strict";

/** Smallest integer >= a / b, for safe integers. b must be non-zero. */
export function ceilDiv(a: number, b: number): number {
  const q = Math.trunc(a / b); // JS division is float; trunc moves toward zero
  // Round up only when there is a remainder and the true quotient is positive.
  return q * b !== a && a > 0 === b > 0 ? q + 1 : q;
}

// 1. The table: [name, a, b, expected]. Each row is one idea, named so failures explain themselves.
const cases: [string, number, number, number][] = [
  ["exact", 6, 3, 2],
  ["rounds up", 7, 3, 3],
  ["zero numerator", 0, 5, 0],
  ["negative numerator", -7, 3, -2], // -2.33 -> -2, not -3
  ["negative divisor", 7, -3, -2],
  ["both negative", -7, -3, 3],
  ["divisor one", 41, 1, 41],
];

// 2. One loop runs every row; the message names the row and shows got vs want.
for (const [name, a, b, want] of cases) {
  assert.equal(ceilDiv(a, b), want, `${name}: ceilDiv(${a}, ${b})`);
}
console.log("ok");

C#

table_driven.cs

// 1. The table: (name, a, b, expected). Each row is one idea, named so failures explain themselves.
(string Name, int A, int B, int Want)[] cases =
[
    ("exact", 6, 3, 2),
    ("rounds up", 7, 3, 3),
    ("zero numerator", 0, 5, 0),
    ("negative numerator", -7, 3, -2),   // -2.33 -> -2, not -3
    ("negative divisor", 7, -3, -2),
    ("both negative", -7, -3, 3),
    ("divisor one", 41, 1, 41),
];

// 2. One loop runs every row; the message names the row and shows got vs want.
foreach (var (name, a, b, want) in cases)
{
    int got = CeilDiv(a, b);
    if (got != want) throw new Exception($"{name}: CeilDiv({a}, {b}) = {got}, want {want}");
}
Console.WriteLine("ok");

// Smallest integer >= a / b. b must be non-zero.
static int CeilDiv(int a, int b)
{
    int q = a / b;                       // C# integer division truncates toward zero
    bool hasRemainder = a % b != 0;
    bool positive = (a > 0) == (b > 0);
    return hasRemainder && positive ? q + 1 : q;
}

The same table in pytest, vitest and xUnit

With a test runner, each row becomes its own test with its own name in the report. These are the idiomatic forms (verified with pytest 9, vitest 2 and xUnit 2.9):

Python

test_ceil_div.py

import pytest


def ceil_div(a: int, b: int) -> int:
    return -(-a // b)


@pytest.mark.parametrize(
    ("a", "b", "want"),
    [(6, 3, 2), (7, 3, 3), (-7, 3, -2), (7, -3, -2)],
    ids=["exact", "rounds up", "negative numerator", "negative divisor"],
)
def test_ceil_div(a: int, b: int, want: int) -> None:
    assert ceil_div(a, b) == want


def test_zero_divisor_raises() -> None:
    with pytest.raises(ZeroDivisionError):
        ceil_div(1, 0)

TypeScript

ceilDiv.test.ts

import { describe, expect, test } from "vitest";

function ceilDiv(a: number, b: number): number {
  if (b === 0) throw new RangeError("division by zero");
  const q = Math.trunc(a / b);
  return q * b !== a && a > 0 === b > 0 ? q + 1 : q;
}

describe("ceilDiv", () => {
  test.each([
    { name: "exact", a: 6, b: 3, want: 2 },
    { name: "rounds up", a: 7, b: 3, want: 3 },
    { name: "negative numerator", a: -7, b: 3, want: -2 },
    { name: "negative divisor", a: 7, b: -3, want: -2 },
  ])("$name: ceilDiv($a, $b) = $want", ({ a, b, want }) => {
    expect(ceilDiv(a, b)).toBe(want);
  });

  test("zero divisor throws", () => {
    expect(() => ceilDiv(1, 0)).toThrow(RangeError);
  });
});

C#

CeilDivTests.cs

using Xunit;

public static class MathUtil
{
    public static int CeilDiv(int a, int b)
    {
        int q = a / b; // throws DivideByZeroException when b == 0
        return a % b != 0 && (a > 0) == (b > 0) ? q + 1 : q;
    }
}

public class CeilDivTests
{
    [Theory]
    [InlineData(6, 3, 2)]    // exact
    [InlineData(7, 3, 3)]    // rounds up
    [InlineData(-7, 3, -2)]  // negative numerator
    [InlineData(7, -3, -2)]  // negative divisor
    public void CeilDiv_matches_math(int a, int b, int want) =>
        Assert.Equal(want, MathUtil.CeilDiv(a, b));

    [Fact]
    public void CeilDiv_by_zero_throws() =>
        Assert.Throws<DivideByZeroException>(() => MathUtil.CeilDiv(1, 0));
}

Arrays in xUnit rows

[InlineData] only takes compile-time constants, so arrays of arrays don’t fit. Use [MemberData(nameof(Cases))] with a public static TheoryData<string, int[]> Cases => new() { { "empty", [] }, ... }; property. The edge case generator emits exactly that shape.

Differential testing against an oracle

Here the fast function is a merge of two sorted lists, and the oracle is “concatenate and sort”. 500 random cases with sizes 0 to 5 and values -3 to 3 cover empty lists, all-equal lists and interleavings in well under a second.

Python

oracle.py

import random


def merge_sorted(a: list[int], b: list[int]) -> list[int]:
    """The 'fast' function under test: merge two sorted lists in O(len(a) + len(b))."""
    out: list[int] = []
    i = j = 0
    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            out.append(a[i]); i += 1
        else:
            out.append(b[j]); j += 1
    return out + a[i:] + b[j:]  # leftovers: forgetting one of these is the classic bug


def oracle(a: list[int], b: list[int]) -> list[int]:
    return sorted(a + b)  # 1. obviously correct, too slow to ship, perfect for checking


if __name__ == "__main__":
    rng = random.Random(42)  # 2. fixed seed: a failure reproduces on every run
    for trial in range(500):
        # 3. small sizes and a tiny value range force duplicates, empties and ties
        a = sorted(rng.randint(-3, 3) for _ in range(rng.randint(0, 5)))
        b = sorted(rng.randint(-3, 3) for _ in range(rng.randint(0, 5)))
        got, want = merge_sorted(a, b), oracle(a, b)
        assert got == want, f"trial {trial}: a={a} b={b} got={got} want={want}"  # 4. print the input
    print("ok")

TypeScript

oracle.ts

import assert from "node:assert/strict";

/** The 'fast' function under test: merge two sorted arrays in O(a.length + b.length). */
export function mergeSorted(a: readonly number[], b: readonly number[]): number[] {
  const out: number[] = [];
  let i = 0, j = 0;
  while (i < a.length && j < b.length) out.push(a[i] <= b[j] ? a[i++] : b[j++]);
  return [...out, ...a.slice(i), ...b.slice(j)]; // leftovers: forgetting one is the classic bug
}

// 1. Obviously correct, too slow to ship, perfect for checking. Note the comparator:
//    the default sort compares as strings, so [-1, -2].sort() is wrong for numbers.
const oracle = (a: readonly number[], b: readonly number[]) => [...a, ...b].sort((x, y) => x - y);

// 2. JS has no seeded Math.random, so use a tiny seeded PRNG (mulberry32) to make failures reproducible.
function mulberry32(seed: number): () => number {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const rand = mulberry32(42);
const int = (lo: number, hi: number) => lo + Math.floor(rand() * (hi - lo + 1));
// 3. Small sizes and a tiny value range force duplicates, empties and ties.
const smallSorted = () => Array.from({ length: int(0, 5) }, () => int(-3, 3)).sort((x, y) => x - y);

for (let trial = 0; trial < 500; trial++) {
  const a = smallSorted(), b = smallSorted();
  // 4. On failure, the message carries the exact input.
  assert.deepEqual(mergeSorted(a, b), oracle(a, b), `trial ${trial}: a=${JSON.stringify(a)} b=${JSON.stringify(b)}`);
}
console.log("ok");

C#

oracle.cs

var rng = new Random(42); // 2. fixed seed: a failure reproduces on every run
for (int trial = 0; trial < 500; trial++)
{
    // 3. small sizes and a tiny value range force duplicates, empties and ties
    int[] a = SmallSorted(rng), b = SmallSorted(rng);
    int[] got = MergeSorted(a, b), want = Oracle(a, b);
    if (!got.SequenceEqual(want)) // 4. on failure, print the exact input
        throw new Exception($"trial {trial}: a=[{string.Join(",", a)}] b=[{string.Join(",", b)}] got=[{string.Join(",", got)}]");
}
Console.WriteLine("ok");

// The 'fast' function under test: merge two sorted arrays in O(a.Length + b.Length).
static int[] MergeSorted(int[] a, int[] b)
{
    var output = new List<int>(a.Length + b.Length);
    int i = 0, j = 0;
    while (i < a.Length && j < b.Length) output.Add(a[i] <= b[j] ? a[i++] : b[j++]);
    output.AddRange(a[i..]); // leftovers: forgetting one of these is the classic bug
    output.AddRange(b[j..]);
    return [.. output];
}

// 1. Obviously correct, too slow to ship, perfect for checking.
static int[] Oracle(int[] a, int[] b) => [.. a.Concat(b).Order()];

static int[] SmallSorted(Random rng) =>
    [.. Enumerable.Range(0, rng.Next(0, 6)).Select(_ => rng.Next(-3, 4)).Order()];

What it costs

Technique	Time to write	Finds	Misses
Hand trace (one input)	1-3 min	logic slips, wrong updates, off-by-one on that path	anything that input doesn’t exercise
Edge-case table	2-4 min	the cases you listed	the cases you didn’t think of
Property / oracle + random inputs	3-6 min	surprising inputs, often within 100 tries	bugs that need large inputs (overflow, timeouts), bugs shared by the oracle
Library property testing (Hypothesis, fast-check, FsCheck)	similar, outside interviews	the above, plus automatic shrinking and saved failures	the same blind spots as your properties

Random testing is cheap because the generator and the oracle are short. Its limit is that a wrong oracle agrees with a wrong solution. Keep the oracle so simple that you can check it by eye.

Variations you’ll meet

• Properties without an oracle. Round-trip (decode(encode(x)) == x), invariants (the output is sorted and is a permutation of the input), idempotence (f(f(x)) == f(x)), and metamorphic rules (doubling every price doubles the total).
• Stateful property tests. Generate a random sequence of operations (push, pop, peek) against your structure and against a simple model (a plain list). Compare after every step. This is how you test a heap, an LRU cache or a rate limiter.
• Assertions as tests. An assert of the invariant inside the loop (debug builds only) turns every test into a check of every step. See debugging.
• Libraries. Hypothesis for Python, fast-check for TypeScript, FsCheck or CsCheck for .NET. None of them is stdlib, so in an interview you write the ten-line version yourself.

Pitfalls

• Testing only the prompt’s example. It was chosen to explain the problem, not to break your code.
• Random inputs that are too big. Length 1000 with values up to 10⁹ almost never produces duplicates or empties. Go small.
• Unseeded randomness. A failure you can’t reproduce is a failure you can’t fix. Fix the seed, and print it.
• JS sort in the oracle. [...a, ...b].sort() sorts numbers as strings. Always pass (x, y) => x - y.
• Asserting on output you computed with the code under test. That’s a tautology, not a test.
• Comparing floats with ==. Use a tolerance, or keep the arithmetic in integers.

In the interview

• When you finish coding, say: “Let me trace the example first,” and do it with a small variable table. Interviewers rate self-verification highly, and it often catches a bug before they do.
• Then list edge cases and run two or three by hand. Mention the rest.
• If you wrote a brute force first, keep it: “I’d compare the two on random small inputs.” Offer to do it if there’s time.
• Expect “How would you test this?” Answer in layers: example tests, edge-case table, a property or oracle check, then integration and load tests if it’s a system.

Check yourself

Check yourself

pick one

Your random tester has run 10,000 cases with lists of length 500 and values in [0, 10⁹], all passing. A colleague's tester with lengths 0-6 and values 0-3 fails on the 12th case. Why is the small one better at finding bugs?

AIt is not; the large one simply got unluckyBSmall sizes and a narrow value range make empty inputs, duplicates and ties very likely, and those are where bugs liveCSmall inputs run faster, so it can try more cases

Check yourself

pick one

You test a sort function only by checking that its output is in non-decreasing order. Which buggy implementation still passes?

AOne that returns the input unchangedBOne that returns an empty list for every inputCOne that swaps the first two elements

Practice on your own

For each, write a brute force, then the fast version, then a 20-line random comparison:

• Maximum product of any two numbers in a list (negatives matter).
• Count pairs i < j with xs[i] + xs[j] == target.
• Remove duplicates from a sorted array in place; return the new length.
• Run-length encode a string and decode it back (round-trip property).
• A min-stack with push, pop and get_min, tested against a plain list model.

Go deeper

• Hypothesis: what you can generate and how, and David MacIver’s essays on shrinking and property choice.
• John Hughes, “QuickCheck Testing for Fun and Profit” (2007): where property-based testing came from.
• pytest: parametrizing tests, Vitest: test.each, xUnit: theories with data.
• Competitive Programmer’s Handbook (Laaksonen): the stress-testing habit, comparing a fast solution with a brute force on random inputs.

Related

Edge casesThe checklist that fills your test table.

DebuggingWhat to do once a test fails.

Complexity analysisThe brute force is your oracle; know why it's slow.

Adapting to changing requirementsTests as a safety net when level 2 arrives.