Operations & Systems

UTF-8 CSV Imports: Find Where Business Text Becomes Corrupted

A product label arrives as Café. Someone corrects the visible cell, saves the spreadsheet, and calls the import repaired.

Keep the original file before accepting that answer. The next export needs to preserve the intended text too.

Work backward from the damaged value until you find the first boundary where its representation changes. Record that boundary and the setting used there. Test the correction on a copy.

This article proposes a diagnosis worksheet for business imports. The records are fictional. No particular spreadsheet, database, or importer is assumed to use the illustrated settings.

Separate the File From Its Display

Give four things separate names on the worksheet: stored bytes, decoded characters, visible display, and import configuration.

Bytes are the file evidence. Decoded characters are the text produced when those bytes are interpreted. The display draws that text. Import configuration records the receiver's chosen interpretation.

Ask for evidence at each layer. A screenshot shows a display. It does not establish which bytes arrived or which decoder read them.

Start with a read-only original and a working copy. Record the original filename, receipt date, sender's declared encoding, and file version. Keep later exports under different filenames.

If the sender cannot identify the encoding, write unknown. Treat the filename extension as a file label rather than an encoding declaration.

Use the CSV data contract to document the handoff's agreed conditions. Its sample convention belongs to that example. Confirm the convention required by your receiver before applying it elsewhere.

Choose a small set of text values whose intended spelling the data owner can verify. Include characters outside plain English letters. Keep that sample attached to the import decision.

Follow a Synthetic Label Through the Boundary

Consider a fictional supplier file containing Café, Piñata, and 東京. These labels are test fixtures, not customer records or verified commercial products.

The following results come from encoding each label as UTF-8, then deliberately decoding those same bytes as ISO-8859-1. That wrong-decoder choice is part of the example.

Expected label Original UTF-8 bytes, hexadecimal Wrong ISO-8859-1 decoding
Café 43 61 66 C3 A9 Café
Piñata 50 69 C3 B1 61 74 61 Piñata
東京 E6 9D B1 E4 BA AC æ followed by U+009D, then ±äº¬

The last row names the control code explicitly because a visible rendering would conceal part of the result. Inspect character values when a screenshot leaves the result ambiguous.

The fixture supplies the expected text. The preserved file supplies the bytes. A separate UTF-8 decoding reproduces the expected labels. The fictional import preview produces the wrong-decoder results.

That evidence locates the first observed change at the preview's decoding boundary. It does not prove that every later stage preserves the corrected text.

For this trial, select UTF-8 in the fictional receiver's explicit decoder field. Reopen the working copy from the preserved original. Do not continue from an already modified preview.

Record the corrected preview, then inspect the stored value after a test import. Export that test record and compare its text again. Keep each observation tied to its own file or record version.

Use Signatures According to the Receiving Contract

Unicode's UTF and byte-order-mark FAQ identifies EF BB BF as the UTF-8 signature. UTF-8 has no byte-order ambiguity. Its initial BOM signals encoding rather than byte order.

The FAQ also explains that receiving protocols differ. Some require a signature; some receivers do not expect one. Absence of a BOM does not identify an otherwise unknown encoding.

Document the receiver's requirement. Avoid declaring every business CSV correct or incorrect based on BOM presence alone.

For the fictional trial, create two copies from the same verified text: one with the required signature and one without it. Change only that condition. Name the files distinctly and record which receiver setting each trial uses.

If a trial shows an unexpected character at the beginning of the first header, inspect the initial bytes and decoded header. Do not delete characters across the entire file to make one heading look right.

Keep the accepted convention with the file contract. A later operator should be able to select the intended input without repeating your investigation.

Stop When Replacement Characters Appear

Unicode's error-handling explanation describes U+FFFD REPLACEMENT CHARACTER as one possible response to ill-formed input. Other permitted responses include reporting an error or filtering an offending byte.

Preserve the original bytes when � appears. Determine whether that character already exists in the source or appears during the observed conversion.

Use a second fictional fixture: the bytes 43 61 66 E9. Under ISO-8859-1 these represent Café. A UTF-8 decoding configured to replace invalid input produces Caf� instead.

Saving that damaged result as UTF-8 produces 43 61 66 EF BF BD. The new file preserves the replacement character. It does not recover the original E9 byte or establish the intended label.

In this example, recovery starts from the preserved bytes and a verified source declaration. The fixture tells us which decoder fits. A real unknown file requires evidence from its owner or an authoritative export specification.

Do not infer missing letters from surrounding words and call that decoding. If manual correction is necessary, record it as a correction with an owner and evidence.

Mark an unresolved replacement as held. A plausible label is insufficient acceptance evidence when the original spelling remains unknown.

Build an Import Diagnosis Worksheet

Use one row per trial. Keep expected and observed samples side by side. The following fields form a proposed operating record, not a Unicode certification checklist.

Field What to record
File version Original name, receipt, and working-copy identifier
Declared encoding Sender's statement or unknown, with its source
Chosen decoder Exact receiver selection, application version, and trial
Signature condition Observed initial bytes and agreed requirement
Expected sample Owner-confirmed label and explicit code points where useful
Observed sample Preview, stored value, and export observations
Round-trip result Compared artifact, comparison method, and pass or difference
Disposition Accept, hold, or reject, with owner and reason

Add a short evidence location beside each observation. A screenshot may illustrate the symptom. Keep the underlying file or character sequence available for inspection.

For the Café fixture, the declaration is UTF-8. Trial one chooses ISO-8859-1 and shows Café. Its disposition is reject for this input.

Trial two chooses UTF-8 and shows Café. Keep its disposition provisional until stored-value and export checks also pass.

Record an application version even when the setting appears obvious. If the receiver exposes no explicit decoder choice, record that limitation. Ask its owner for the documented import path rather than inventing a hidden setting.

Keep one decision per row. A trial that changes the decoder, the file contents, and the import method leaves the cause unresolved.

Define What the Round Trip Must Preserve

Choose the comparison before running the trial. Byte equality and text equality answer different questions.

For a transfer-copy check, compare the copied file with the preserved original bytes. For an import-export check, compare the intended field values after decoding each artifact using its documented encoding.

Record both results when both matter. In the synthetic signature trial, adding EF BB BF changes file bytes. If the receiver recognizes the signature, the product-label text can still match.

Do not treat a changed file hash as proof that the label changed. Do not treat a matching screenshot as proof that file bytes stayed equal.

For the accepted UTF-8 trial, list three checkpoints: preview equals the fixture, stored field equals the fixture, and exported field equals the fixture. Attach the exact exported file to the worksheet.

Keep identifier preservation as a separate acceptance condition. Correct accents do not establish that a numeric-looking identifier survived import.

Set the boundary of your conclusion. Three passing sample labels establish the observed sample result under the recorded configuration. They do not establish that every value in an unexamined production file is correct.

Distinguish a Display Problem From Changed Text

When a character looks wrong, compare its recorded code points with the expected sample before replacing it.

In a fictional second receiver, suppose the stored field contains the expected sequence for 東京, while one screen shows empty boxes. Keep the stored sequence and the screenshot as separate observations.

Ask the application owner to check font support and its rendering environment. If another supported display draws the same sequence correctly, record that evidence. Avoid changing stored text merely to satisfy one screen.

If the stored sequence itself differs, return to the earlier checkpoints. Find whether the change appears in the preview, after storage, or during export.

Use precise disposition language. “Display unresolved; stored sample matches” tells the next operator what remains. “Encoding fixed” collapses several untested conditions into one claim.

Connect the result to the broader CRM data-quality review only after the text boundary is understood. Keep representation checks separate from decisions about whether records describe the same entity.

Accept the Handoff With Evidence

Finish the diagnosis with a short decision note. Name the accepted file version, decoder, signature convention, tested samples, and export result.

For the fictional UTF-8 file, the note can state: trial two preserves all three fixture labels through preview, storage, and export. The first trial fails at decoding. The original remains unchanged.

For the replacement-character file, keep the failed UTF-8 trial attached. Accept a corrected trial only after the source owner confirms the encoding and expected label.

If the original is missing, say that the original bytes are unavailable. Request a fresh export. A re-saved damaged file is evidence of the current damage, not proof of its cause.

Give the receiving owner the worksheet and the accepted artifact together. Record unresolved fields explicitly. Release only the scope that the evidence supports.

Keep the bytes. Name the boundary.

Sources

  • Unicode: UTF-8, UTF-16, UTF-32 and BOM FAQ. UTF-8 signatures, receiving-protocol differences, and ill-formed input handling. Checked 2026.10.28. Decoder fixtures are independently calculated synthetic examples. Worksheets and acceptance procedures are editorial proposals.

Topic map: Spreadsheets, CSV files, and data handoffs

Articles in this cluster, then the 3 clusters that share the most topics with it.

This cluster: Spreadsheets, CSV files, and data handoffs

Neighbor: Business reporting and statistics

Neighbor: Records, access, and continuity

Neighbor: APIs, webhooks, and automation