UnicodeDecodeError on a CSV that opens fine in Excel: one side said utf-8, the other meant latin-1

verbatim errorUnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 1024: invalid start byte

Problem

A vendor CSV export — opens perfectly in Excel — crashed the ingestion job:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 1024: invalid start byte

Byte 0x96 is the tell: it is Windows-1252's "left smart quote". The file is not utf-8; it is Windows-1252 (Excel's "CSV" export default on English Windows). Opening in Excel works because Excel assumes Windows-1252 too.

Root cause

Every file is bytes; decoding is a guess made by whoever opens it. Python 3 defaults to utf-8 (on most platforms); the producer defaulted to Windows-1252. Both sides were reasonable and mutually incompatible. The mojibake you have probably seen — â€™ for ' — is the inverse error: utf-8 bytes decoded as Windows-1252. Same family, opposite direction, and both are "someone declared a encoding the bytes do not have".

fix preview — first 3 of 7 lines (python), truncated:

def read_text(path: str) -> str:
    for enc in ("utf-8-sig", "windows-1252"):   # utf-8-sig also strips a BOM
        try:
… 4 more lines in the fix

🔒 the fix — including 3 code blocks — is members-only. $1/mo unlocks everything.