Python 3.15 UTF-8 Default: Fix CSV Imports Without Losing Text

Python 3.15.0, released on 9 October 2026, enables UTF-8 mode by default. For teams importing CSV files, reading reports or passing text between automation scripts, that is a useful consistency improvement with an important catch: existing files are not automatically converted to UTF-8.

A legacy export that previously worked because of a machine’s locale may now fail to decode or produce incorrect text. The correct repair is to identify the encoding at each file boundary, not suppress every decoding error. Python’s final UTF-8 default specification describes the behaviour and its compatibility controls.

What changes, and what does not

The change affects default text encoding for files, standard input and output, and pipes under UTF-8 mode. Code that already specifies the correct encoding has a clearer contract than code relying on the workstation’s default.

The specification notes that Windows users are particularly affected because many Unix environments already use UTF-8. That does not make an import safe simply because its developer uses Linux: the file may have been generated by a Windows desktop application.

Nor does a .csv extension identify an encoding. CSV describes a record format, not a promise that the underlying bytes are UTF-8. Treat the producer’s documented export setting as part of your input contract.

Inventory the text boundaries first

List each recurring input and output: supplier CSVs, spreadsheet exports, log files, configuration files and subprocess output. For each, record its producer, declared encoding, representative sample and downstream consumer.

Search for open calls without an encoding argument and text-mode subprocess calls whose output encoding is implicit. Use Python’s optional default-encoding warning while running your tests:

python -X warn_default_encoding -m unittest

This example assumes your project uses unittest; use the corresponding invocation for your own test runner. Warnings identify code paths actually exercised, so a quiet run does not prove an untested monthly import is safe.

Keep original input files unchanged during investigation. Saving a problematic file through an editor may transform its bytes and hide the original failure.

A small CSV contract that preserves Hindi text

Suppose a fictional internal tool exchanges location records in UTF-8. Write that rule into both sides of the workflow instead of relying on the new default. The following is an illustrative local round-trip example, not a reported production test:

import csv

rows = [{"city": "जयपुर", "status": "ready"}]
with open("sample.csv", "w", encoding="utf-8", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["city", "status"])
    writer.writeheader()
    writer.writerows(rows)

with open("sample.csv", encoding="utf-8", newline="") as f:
    restored = list(csv.DictReader(f))

assert restored == rows

The assertion checks the decoded values, not merely whether a file exists. Extend this idea with your own cases: names in the scripts your customers use, punctuation, empty fields, embedded line breaks and the exact header names expected downstream.

If a producer explicitly supplies UTF-8 with a byte-order mark, an appropriate UTF-8-sig read can handle that marker. Do not apply that setting as a universal cure for an unknown file; it cannot turn a genuinely different character encoding into UTF-8.

Handle a known legacy export deliberately

If a supplier documents an older encoding, use that exact encoding at ingestion, then write your normalised internal output as UTF-8. A Western legacy encoding cannot represent every Hindi character, so a missing character may already have been lost in the producer’s export.

Do not automatically retry a failed UTF-8 read with whichever decoder happens to accept every byte. Successful decoding is not proof of correct text. Obtain an original sample and the export configuration, then compare expected names and symbols.

Avoid errors="ignore" as a migration fix. Silently dropping undecodable bytes can turn a visible failure into an incorrect customer name or identifier. Quarantine the input and report the filename, producer and failure location without exposing personal records in general logs.

Use compatibility switches as temporary diagnostics

Python permits disabling UTF-8 mode with PYTHONUTF8=0 or -X utf8=0. The specification recommends comparing behaviour, finding implicit encodings and then testing again with UTF-8 enabled. A compatibility switch is a bridge for investigation, not a durable declaration of the file format.

If you intentionally need the locale encoding, specify that choice explicitly and document the environment dependency. Otherwise use a named encoding agreed with the producer and consumer.

For automation teams, the workflow prototype and fallback guide offers complementary context for deciding how a failed import should stop, notify an owner and preserve recoverable input.

Finish the migration with samples from every real producer and a round-trip or expected-value check for each consumer. The goal is not a script that stops complaining; it is a text workflow whose data remains correct across machines and Python versions.

Make a failed import recoverable

Keep a sample-based acceptance record for each producer: expected headers, representative decoded values, agreed encoding and the consumer that approved them. When an import fails, preserve the original bytes and stop before updating downstream records. An owner can then request a corrected export without reconstructing a partially processed batch.

After changing the decoder, rerun against the unchanged original and compare expected values, not just the absence of an exception. This separates a genuine repair from a setting that merely hides the failure.