Skip to content
Hari KharelStudy

Math and number handling

coreCommon atGoogleStripeTrading firmsAmazonRead firstComplexity analysisPractise in the Sandbox →

Numbers in code are not numbers in math. C# ints wrap silently at ±2³¹, JS numbers are doubles that stop being exact past 2⁵³, and only Python ints never overflow. Integer division floors in Python but truncates in TS and C#, so -7 % 2 is 1 in one and -1 in the others. Floats can’t store 0.1 exactly: compare with a tolerance and keep money in integers.

The idea in plain words

Section titled “The idea in plain words”

A car’s odometer has six digits. Drive one mile past 999,999 and it shows 000,000. It didn’t crash; it just ran out of digits and kept the low ones. That’s integer overflow, and it’s what a C# int does by default.

A ruler marked only in eighths of an inch can’t show a tenth of an inch. It shows the nearest mark. That’s floating point: a double stores the nearest sum of powers of two, and 0.1 isn’t one.

Interview code breaks on numbers in four predictable places:

  • Size. Does every intermediate value fit in the type? (Sums of 10⁵ values up to 10⁹ reach 10¹⁴: too big for int, fine for long.)
  • Sign. Which way does division round with negatives? What sign does % return?
  • Exactness. Is this value a float that only looks exact?
  • Precision of the algorithm. Should this be integer arithmetic all the way (gcd, modular arithmetic, cross products) instead of a float?

See it

Section titled “See it”

Three labs. In each, change the numbers and watch where the languages disagree.

Number lab: the same arithmetic in Python, TS and C#
try your own numbers
Try a negative a or b. Then try a = 9, b = 3.
languageexpressionquotientremainderrounds
Pythona // b, a % b-41toward −∞ (floor)
TypeScriptMath.trunc(a / b), a % b-3-1toward 0 (truncate)
C#a / b, a % b-3-1toward 0 (truncate)
TypeScriptMath.floor(a / b)-4–floor, but via a float
-5-4-3-2-101a/b = -3.500● floor: Python //◌ truncate: TS, C#
The signs differ and b doesn't divide a, so the two rules disagree. Both still satisfy a = q·b + r: -7 = -4·2 + 1 = -3·2 + -1. Python's remainder takes the sign of b; TS and C# take the sign of a.

When to reach for it

Section titled “When to reach for it”
The problem says… What to think about
“values up to 10⁹”, “n up to 10⁵”, “sum of…” Multiply the bounds. Over 2.1·10⁹ needs 64 bits; over 9·10¹⁵ is unsafe for JS numbers.
“return the answer modulo 10⁹ + 7” Reduce after every + and *. In JS, products of two residues need BigInt.
negative numbers and % or / Rounding direction differs by language. Wrap indices with a floor-mod.
“fractions”, “ratios”, “slopes”, “same line” Avoid floats: reduce p/q by gcd, or cross-multiply to compare.
money, prices, percentages Integer minor units (cents) or a decimal type. See money arithmetic.
“every k days”, “both cycles line up”, “tile exactly” gcd and lcm.

How it works

Section titled “How it works”

1. Know your integer types

Section titled “1. Know your integer types”
Python TypeScript C#
default integer int, unbounded number, a 64-bit double int, 32-bit
exact integer range unlimited ±(2⁵³ − 1) = Number.MAX_SAFE_INTEGER int ±2.1·10⁹, long ±9.2·10¹⁸
on overflow never overflows rounds to the nearest double, silently wraps silently (unchecked); throws in checked
big integers built in bigint (10n), can’t mix with number System.Numerics.BigInteger, also Int128 (.NET 7+)
32-bit wrapping on purpose (x + 2**31) % 2**32 - 2**31 x | 0, Math.imul default
overflow.py
def to_int32(x: int) -> int:
    """Wrap an exact Python int the way a 32-bit two's-complement register would."""
    return (x + 2**31) % 2**32 - 2**31   # 1. shift, reduce mod 2^32, shift back




if __name__ == "__main__":
    INT_MAX, INT_MIN = 2**31 - 1, -(2**31)
    # 2. Python ints never overflow; they grow as needed.
    assert INT_MAX + 1 == 2_147_483_648
    assert 2**64 == 18_446_744_073_709_551_616
    # 3. What C# int (unchecked) or JS `x | 0` would give you instead:
    assert to_int32(INT_MAX + 1) == INT_MIN
    assert to_int32(50_000 * 50_000) == -1_794_967_296
    assert to_int32(-5) == -5
    # 4. Floats DO have limits: past 2^53, not every integer is representable.
    assert float(2**53) == float(2**53 + 1)
    # 5. Safe midpoint: in fixed-width languages lo + hi can overflow; this form cannot.
    lo, hi = 2_000_000_000, 2_100_000_000
    assert lo + (hi - lo) // 2 == 2_050_000_000
    print("ok")

2. Integer division and remainder

Section titled “2. Integer division and remainder”

All three languages promise a == q*b + r. They disagree on which way q rounds:

-7 / 2 (int) -7 % 2 rule
Python //, % −4 1 floor (toward −∞); r has the sign of b
TypeScript Math.trunc(a / b), % −3 −1 truncate (toward 0); r has the sign of a
C# /, % −3 −1 truncate (toward 0); r has the sign of a

The everyday consequence: stepping backwards around a ring of size n. In Python (i - 1) % n just works. In TS and C# it gives -1 for i = 0; use ((i - 1) % n + n) % n or a helper.

floor_mod.py
import math




def trunc_div_mod(a: int, b: int) -> tuple[int, int]:
    """What TS and C# do with / and %: quotient rounds toward zero, remainder takes a's sign."""
    q = abs(a) // abs(b) * (1 if (a < 0) == (b < 0) else -1)
    return q, a - q * b




if __name__ == "__main__":
    # 1. Python's // floors (rounds toward -infinity); % takes the sign of the divisor.
    assert (7 // 2, 7 % 2) == (3, 1)
    assert (-7 // 2, -7 % 2) == (-4, 1)
    assert divmod(-7, 2) == (-4, 1)
    # 2. Truncating semantics, as in TS / C#, for comparison.
    assert trunc_div_mod(-7, 2) == (-3, -1)
    assert math.fmod(-7, 2) == -1.0             # fmod follows C: sign of the dividend
    # 3. Both satisfy a == q*b + r; they only differ in which way q rounds.
    for a, b in [(-7, 2), (7, -2), (-7, -2)]:
        q, r = divmod(a, b)
        assert a == q * b + r
    # 4. Never write int(a / b) for integers: a / b is a float and loses precision.
    assert int(10**17 / 3) != 10**17 // 3
    assert (-1) % 5 == 4                        # wrap an index backwards around a ring of 5
    print("ok")

3. gcd and lcm

Section titled “3. gcd and lcm”

Euclid’s algorithm rests on one fact: any number dividing both a and b also divides a mod b. So gcd(a, b) = gcd(b, a mod b), and the numbers shrink fast: at most about 5 × (digits of the smaller number) steps (Lamé’s theorem), i.e. O(log min(a, b)).

  1. While b isn’t 0, replace (a, b) with (b, a mod b).
  2. When b reaches 0, a is the gcd.
  3. lcm(a, b) = a / gcd(a, b) * b. Divide first so the intermediate stays small.
gcd.py
import math




def gcd(a: int, b: int) -> int:
    """Euclid: gcd(a, b) = gcd(b, a mod b). O(log min(a, b)) steps."""
    a, b = abs(a), abs(b)
    while b:                      # 1. replace (a, b) by (b, a mod b) until b hits 0
        a, b = b, a % b
    return a                      # 2. gcd(a, 0) = a




def lcm(a: int, b: int) -> int:
    if a == 0 or b == 0:
        return 0
    return abs(a) // gcd(a, b) * abs(b)  # 3. divide first: in fixed-width languages a*b can overflow




if __name__ == "__main__":
    assert gcd(84, 36) == 12 and gcd(36, 84) == 12
    assert gcd(-12, 18) == 6 and gcd(0, 5) == 5
    assert lcm(4, 6) == 12 and lcm(0, 7) == 0
    assert gcd(84, 36) == math.gcd(84, 36) and lcm(4, 6) == math.lcm(4, 6)  # stdlib, 3.9+
    print("ok")

Stdlib: Python has math.gcd and math.lcm (3.9+). .NET has BigInteger.GreatestCommonDivisor but no Math.Gcd. JS has neither.

4. Modular arithmetic

Section titled “4. Modular arithmetic”

When an answer is huge (“count the ways…”), problems ask for it modulo a prime, usually 10⁹ + 7. The rules: (a + b) mod m and (a · b) mod m can be computed from a mod m and b mod m, so reduce after every operation and numbers never grow. Division is different: you multiply by a modular inverse, which for prime m is a^(m−2) mod m (Fermat’s little theorem).

Fast exponentiation reads the exponent’s bits: square the base each step, multiply it into the result when the bit is 1. That’s O(log e) multiplications instead of e.

mod_pow.py
MOD = 1_000_000_007




def mod_pow(base: int, exp: int, m: int) -> int:
    """base**exp % m by repeated squaring: O(log exp) multiplications."""
    result = 1
    base %= m
    while exp > 0:
        if exp & 1:                    # 1. this bit of exp is set: multiply it in
            result = result * base % m
        base = base * base % m         # 2. square for the next bit; reduce every step
        exp >>= 1
    return result




if __name__ == "__main__":
    assert mod_pow(2, 10, 1000) == 24
    assert mod_pow(3, 0, 7) == 1
    assert mod_pow(123456789, 10**18, MOD) == pow(123456789, 10**18, MOD)  # built in
    # 3. Modular inverse when m is prime (Fermat): a^(m-2). Python 3.8+: pow(a, -1, m).
    assert mod_pow(5, MOD - 2, MOD) * 5 % MOD == 1
    assert pow(5, -1, MOD) == mod_pow(5, MOD - 2, MOD)
    print("ok")

5. Floating point

Section titled “5. Floating point”

A double (IEEE 754 binary64, the type of every JS number, Python float and C# double) has 1 sign bit, 11 exponent bits and 52 fraction bits: about 15-17 significant decimal digits. Consequences:

  • 0.1, 0.2 and 0.3 aren’t exact. 0.1 + 0.2 is 0.30000000000000004. Compare with a tolerance.
  • Rounding errors add up. Ten additions of 0.1 give 0.9999999999999999.
  • Large numbers lose small ones. 1e16 + 1 == 1e16.
  • Special values. NaN != NaN; 0.0 == -0.0; dividing by zero gives Infinity, not an exception (for floats).
Python TypeScript C#
tolerant compare math.isclose(a, b) write your own (below) write your own (below)
exact decimal decimal.Decimal("0.1") none built in: use integer cents or a library decimal (0.1m), 28-29 digits
exact fraction fractions.Fraction(1, 3) none none
accurate sum math.fsum none none
floats.py
import math
from decimal import Decimal
from fractions import Fraction




def nearly_equal(a: float, b: float, rel: float = 1e-9, abs_tol: float = 1e-12) -> bool:
    """Relative tolerance for big values, absolute tolerance near zero."""
    return abs(a - b) <= max(rel * max(abs(a), abs(b)), abs_tol)




if __name__ == "__main__":
    # 1. 0.1 has no exact binary form, so errors appear in the last digit.
    assert 0.1 + 0.2 != 0.3
    assert repr(0.1 + 0.2) == "0.30000000000000004"
    # 2. Compare with a tolerance, never ==.
    assert nearly_equal(0.1 + 0.2, 0.3)
    assert math.isclose(0.1 + 0.2, 0.3)          # stdlib, rel_tol=1e-09 by default
    # 3. Exact decimal arithmetic: construct from strings, not floats.
    assert Decimal("0.1") + Decimal("0.2") == Decimal("0.3")
    assert Decimal(0.1) != Decimal("0.1")         # Decimal(0.1) captures the binary error
    # 4. Exact rationals.
    assert Fraction(1, 10) + Fraction(2, 10) == Fraction(3, 10)
    # 5. A running += drifts; math.fsum tracks the lost bits (3.12+ sum() compensates too).
    total = 0.0
    for _ in range(10):
        total += 0.1
    assert total != 1.0 and math.fsum([0.1] * 10) == 1.0
    print("ok")

What it costs

Section titled “What it costs”
Operation Cost Note
+ − * / on fixed-width ints or doubles O(1) one CPU instruction
Python int / bigint / BigInteger with d digits +: O(d); *: about O(d^1.585) (Karatsuba) for big d “free” until numbers get huge
gcd(a, b) O(log min(a, b)) Euclid
b^e mod m O(log e) multiplications repeated squaring
decimal (C#, Python) 10-100× slower than double software arithmetic

An interviewer can push on this: “your solution is O(n), but each step multiplies two growing Python ints”. If numbers grow to n digits, the “O(1)” arithmetic isn’t. Modular reduction keeps every number small and every step truly O(1).

Variations you’ll meet

Section titled “Variations you’ll meet”
  • Integer square root. math.isqrt(n) in Python. In other languages, Math.sqrt on a double can be off by one for values near 2⁵³: correct with while r*r > n: r-- and while (r+1)*(r+1) <= n: r++, or binary search.
  • Ceiling division. ceil(a / b) for positive ints is (a + b - 1) // b, or -(-a // b) in Python. Avoid the float Math.ceil(a / b) for large values.
  • Sieve of Eratosthenes. All primes up to n in O(n log log n).
  • Cross products instead of slopes. To test whether three points are collinear, check (x2-x1)*(y3-y1) == (y2-y1)*(x3-x1) with integers. No division, no floats.
  • Counting with combinations. C(n, k) mod p via precomputed factorials and inverse factorials.
  • Bit tricks live on their own page: bit manipulation.

Pitfalls

Section titled “Pitfalls”
  • Overflow in the middle, not the result. a * b / c overflows even when the answer fits. Reorder (a / gcd * b) or widen.
  • Math.abs(int.MinValue) throws in C#: -2³¹ has no positive counterpart in 32 bits. Same for int.MinValue / -1.
  • int(a / b) in Python goes through a float. For big ints it’s wrong. Use a // b.
  • parseInt and big JSON numbers in JS. Anything past 2⁵³ is already rounded when it’s parsed. Ids like that must travel as strings.
  • Comparing floats with ==, or using floats as hash keys. Round, scale to integers, or use a tolerance.
  • Money in floats. 0.1 + 0.2 dollars isn’t 30 cents. Use integer cents.
  • Sort comparators that subtract. (a, b) => a - b overflows in C# with large ints (wrong sign) and loses precision in JS past 2⁵³. Compare instead: a.CompareTo(b).

In the interview

Section titled “In the interview”
  • Ask for the bounds: largest value, longest input, can values be negative? Then say the arithmetic out loud: “n is 10⁵ and values are up to 10⁹, so the sum can reach 10¹⁴. I’ll use a long.”
  • In TypeScript, mention when you’re relying on number being exact: “these sums stay under 2⁵³, so plain numbers are fine.”
  • With negative operands, say which modulo you mean and use a helper rather than % directly.
  • If you see division with floats, ask yourself whether integers can do it: cross-multiply, reduce by gcd, or work in cents.
  • Expect follow-ups: “What if the values were 64-bit?” (overflow of the sum: BigInteger, or 128-bit), “What if this ran on money?” (decimal, rounding modes, banker’s rounding).

Check yourself

Section titled “Check yourself”
Check yourself
pick one
In C#, int a = 100_000, b = 100_000; long c = a * b; What is c?
Check yourself
pick one
You step backwards around a circular buffer of size 5 with next = (i - 1) % 5. What do Python and TypeScript give for i = 0?
Check yourself
pick one
In TypeScript you compute (a * b) % 1_000_000_007 with a and b both just under 10⁹ + 7, using plain numbers. What happens?

Practice on your own

Section titled “Practice on your own”

Classic textbook exercises, not gym questions:

  • Reverse the digits of a 32-bit integer, returning 0 if the result overflows, without using 64-bit types.
  • Add two non-negative integers given as decimal strings of any length.
  • Count primes below n with the sieve of Eratosthenes.
  • Compute C(n, k) mod 10⁹ + 7 for many queries with n ≤ 10⁶.
  • Given points with integer coordinates, find the largest number that lie on one line, using gcd-reduced slopes instead of floats.

Go deeper

Section titled “Go deeper”
Section titled “Related”
Money arithmeticInteger cents, rounding modes and currencies.
Bit manipulationMasks, shifts and two's complement.
Edge casesEmpty, zero, negative, MIN and MAX inputs.
Binary searchWhere the midpoint overflow bug lives.