Retention is the policy for how long you keep data. Deletion is the mechanics of removing it when the clock runs out or a request arrives. This page goes deep on both as a single discipline. It is a sibling to the data lineage pillar, which is what makes deletion provable, and a child of the data governance pillar. The core truth: a value lives in more places than the row you deleted.
A retention and deletion discipline treats every data element as something with a lifecycle, not something you keep by default. Retention is the policy layer. It answers how long a given category of data should exist and on what lawful basis, mapped to a business need or a legal obligation. Deletion is the mechanics layer. It is the actual work of removing that data once the retention clock runs out or a subject request arrives.
The two are distinct on purpose. A retention schedule that no deletion process enforces is a document, not a control, and a deletion routine with no schedule behind it has no defensible answer to why data left or stayed. GDPR Article 5(1)(e), the storage limitation principle, is the clearest statement of the idea: personal data should be kept in identifiable form no longer than necessary for the purpose it was collected for. The discipline operationalizes that principle across every system that holds a copy.
Modern stacks make deletion hard because a single value rarely lives in one place. A row written to a primary database is copied into derived tables, materialized views, caches, search indexes, message queues, application logs, an analytics warehouse, and one or more backups within minutes. Deleting the original row leaves every downstream copy intact, so "delete the row" is not deletion, it is deletion of one representation among many. The value can still be read, joined, exported, or reconstructed from any surviving copy.
Backups sharpen the problem. Good backup design is immutable and append only precisely so that ransomware or a bad actor cannot alter history, and that immutability collides directly with an erasure obligation that says a specific person's data must go. You cannot surgically reach into a sealed, immutable backup set and excise one record without undermining the integrity guarantee the backup exists to provide. Resolving that tension is the central engineering problem of deletion, not an edge case.
A retention schedule maps each data category to a retention period and a lawful basis, and its default should be delete when no longer needed rather than keep indefinitely. For each category you state the purpose the data serves, the period it is retained, the basis for holding it (contract, legal obligation, legitimate interest, consent), and the trigger that starts the deletion clock. GDPR Article 30, records of processing activities, is where much of this is already documented for personal data, so the schedule should read as an extension of that record rather than a parallel artifact.
Data held past its usefulness is a liability, not an asset. Attackers exfiltrate what exists, not what you needed. Every retained record widens the breach blast radius, raises e-discovery cost, and enlarges the surface a deletion request must cover.
Soft delete marks a record as removed, usually with a tombstone flag or a deleted_at
timestamp, while the underlying data stays present and recoverable. It is the right choice when you need an undo
window, referential safety, or an audit trail of the removal itself. It is not erasure. The data is still there,
still queryable by anything that ignores the flag, and still within reach of a breach. Hard delete physically
removes the data so it cannot be recovered. It is what an erasure obligation actually contemplates, and it is the
correct end state once any soft delete grace period expires.
For backups, where hard deleting a single record inside an immutable set is not feasible, crypto-shredding is the standard answer. You encrypt the data, or a category of data, under a key, and when the retention or erasure trigger fires you destroy the key rather than the ciphertext. The encrypted bytes may still sit in the backup, but without the key they are unrecoverable, which renders the data effectively gone. Whichever mechanism you use, deletion has to propagate to every downstream copy, the derived tables, caches, search indexes, logs, and analytics stores, or the value simply survives where the propagation did not reach. Enumerating those destinations is a lineage problem: column-level lineage is what tells you where a value went so propagation can be made complete.
| Dimension | Soft delete | Hard delete |
|---|---|---|
| What it does | Flags the record as removed with a tombstone or deleted_at timestamp while leaving the data in place | Physically removes the data from the store |
| Recoverability | Recoverable, the record can be restored by clearing the flag | Irrecoverable by design once complete |
| Is the data actually gone | No, the data is still present and readable by anything that ignores the flag | Yes, the underlying bytes are removed |
| Typical use | Undo windows, referential safety, audit trail of the removal, staged deletion | End state after any grace period, erasure requests, expired retention |
| GDPR erasure sufficiency | Not sufficient on its own, the data still exists | Sufficient when propagated to all copies, treat as readiness not legal advice |
A legal hold is an override that suspends deletion for data relevant to actual or reasonably anticipated litigation, regulatory investigation, or audit. When a hold is placed, the affected records are exempted from the retention schedule and from any scheduled purge until the hold is lifted, regardless of what their normal retention period says. The hold has to take precedence over the schedule by design, because the schedule optimizes for minimizing retained data while the hold serves a different and higher obligation, preserving evidence.
The failure mode is spoliation, the destruction or loss of evidence that a party had a duty to preserve. If a routine purge deletes records that were under a duty to preserve, the consequences can include court sanctions, adverse inference instructions, and other penalties that far outweigh the storage the purge saved. This is why a legal hold cannot be an informal note. It has to be a machine enforceable state that the deletion pipeline checks before it removes anything, so that a hold reliably blocks a purge rather than depending on someone remembering it exists.
Deletion you cannot show is deletion you cannot rely on. A defensible posture produces a deletion record: for each deletion event, what was deleted, which systems it propagated to, when, under which policy or request, and the outcome in each destination. That log is the evidence that a retention schedule and a subject request were honored, and it is what turns a claim of erasure into something you can demonstrate on request.
Completeness is the hard part, and it is where lineage does the work. To show a deletion is complete you have to enumerate every place a given value traveled, then confirm removal or crypto-shredding in each. Column-level lineage is what makes that enumeration possible, because it traces a field to every downstream copy so no destination is silently missed. Framed as readiness rather than a certification claim, the deletion log plus lineage coverage is the evidence you assemble now, so that when a regulator, auditor, or data subject asks, the answer is a record and not an assertion.
Before the retention clock runs out, strip the PII you no longer need.
Try the redaction gateway