1.8 KiB
name, description
| name | description |
|---|---|
| data | Process, validate, or transform data. Use when parsing files, cleaning datasets, designing a data pipeline, or debugging a transform that produces wrong output. |
Data
Bad data fails silently and far away from where it entered. Validate at the boundary, keep the raw, and make every transform checkable.
Validate at the boundary
Parse and validate when data enters the system, not when it is used. A schema check at the edge turns a corrupt record into a clear rejection; skipping it turns the same record into a wrong answer three layers later. Reject loudly, with the record and the reason — never coerce and carry on.
Keep the raw
Store the untransformed input alongside the derived. When a transform turns out to be wrong, the raw lets you recompute; without it, the information is gone. Derived data is rebuildable; source data is not.
Transformations are pure and tested
A transform takes input and returns output with no hidden state, so it can be tested on a fixture and re-run safely. Test the edge cases that actually occur in data: the empty field, the wrong type, the unexpected null, the duplicate, the encoding that is not UTF-8.
Duplicates, nulls, and ranges are the usual corruption
Check for: unexpected duplicates on a key, nulls where a value is required, values outside a sane range (a negative age, a date in the future), and referential breaks (an id pointing at nothing). These four catch most real-world data problems before they reach a report.
Idempotent pipelines
A step that can be re-run without duplicating or corrupting its output is a step you can retry after a failure. Key on a stable id and upsert rather than blind-insert. A pipeline you cannot safely re-run is a pipeline you will one day have to fix by hand at 2am.