A Character-Encoding Fixture for Supplier CSV Uploads

Supplier spreadsheets often arrive as CSV, and a CSV file does not declare its own character encoding. A name like Crème Brûlée Set can reach you as Crème Brûlée Set, or with question marks or empty boxes where accents belong. This article proposes a small fixture and worksheet you could use to detect such corruption before importing. It is a design suggestion for your own workflow.

The idea: witness rows

A witness row is a product line with a known, correct name that you place at the top of the supplier template, or check for in their file. If the witness comes through intact, that is evidence, not proof, that the rest decoded properly. If it comes through damaged, the file should not be imported as it stands.

Suggested witness names, chosen to exercise different character families:

• Café Crème Ünïcode Test (Western European accents)
• Łódź Żółć Test (Central European letters)
• Ğüneş Işık Test (Turkish letters)
• Naïve Façade – Test (accents plus a dash and a typographic apostrophe, as in It’s)

Use a SKU pattern such as ZZ-WIT-001 so the rows are easy to find and exclude later.

The reject-or-confirm worksheet

One row per witness:

• witness_sku
• expected_name
• name_as_read: copied from your preview, not retyped
• exact match: yes or no
• suspicious sequences found: Ã, Â, â€, a replacement symbol, or a question mark where a letter belongs
• decision: confirm or reject

Then a second block for the real rows. Search the product name and description columns for the same suspicious sequences and record the count and the first few examples. Count zero is encouraging, not conclusive, since some corruption is silent.

Branching choices

If all witnesses match and no real row shows suspicious sequences, proceed to a small trial import. If witnesses are damaged in a recognisable pattern such as à followed by a symbol, the file was probably saved in one encoding and read in another. Ask the supplier to re-export, stating the encoding you expect, or re-open it yourself with an explicit encoding if you are able to do so safely. If witnesses show question marks, characters may have been lost before the file reached you, and re-reading will not restore them. That file should go back to the supplier. If only some rows are damaged, the file may have been merged from several sources; reject it whole rather than patching by hand.

Hypothetical worked example

A supplier file is opened in the preview. The witness Café Crème Ünïcode Test appears as Café Crème Ãœnïcode Test. The worksheet records no match and the sequence Ã. A search finds the same sequence in 40 of 300 real rows. The decision is reject. The team asks for a re-export with the encoding named. The new file shows all four witnesses matching and zero suspicious sequences, so the team confirms and runs a ten-row trial.

Stop conditions

Stop if any witness fails, if the preview cannot be shown to be reading the file in the encoding you intended, or if a re-export fails in a new way. Do not repair names by find and replace on live data, since a repair rule can mangle correctly encoded text.

Acceptance checks

• All witness rows match exactly.
• Suspicious-sequence search returns nothing, or each hit is explained.
• A ten-row trial import displays accents correctly on the product page, in search and in any exported copy.
• Witness SKUs are removed or hidden after the check.
• The worksheet is kept with the batch record.

The fixture cannot guarantee a perfect file. It makes the most common failure cheap to see before it spreads through a catalogue.

Sources