Tokenization vs Encryption: What Actually Takes Data Out of Scope

Two sentences you hear in almost every compliance conversation, both of them wrong:

“We encrypted everything, so this system is out of scope.”

“We tokenized the data, so GDPR doesn’t apply.”

Neither holds. Encryption rarely removes a system from PCI DSS scope, and tokenization never removes you from GDPR — because tokenization is pseudonymisation, and pseudonymised data is still personal data.

Both techniques are genuinely useful. They just buy different things than most teams assume. This guide covers what each one actually does, what it costs you operationally, and where the line really falls.

The Mechanical Difference

Encryption transforms a value with an algorithm and a key. The ciphertext still contains the information — that’s the point, it has to be recoverable. Whoever holds the key holds the data.

Tokenization replaces the value with an unrelated surrogate and stores the original somewhere else:

4111 1111 1111 1111   →   tok_9f2a8b3e4c7d

That token has no mathematical relationship to the card number. There is no key that reverses it, because there is nothing to reverse. Recovering the original means having access to the mapping.

The practical consequence shows up in a breach:

Both fail if the secret fails. The difference is what an attacker gets from the database alone, and how many separate systems they need to compromise.

What Tokenization Actually Does for PCI DSS

Here’s the part worth being precise about, because it’s where the money is.

The logic of scope reduction: a system that only ever holds tokens, and cannot use those tokens to obtain the original card number, may fall outside the cardholder data environment. Your analytics warehouse, your support tooling, your reporting pipeline — none of them ever touch a real PAN, so none of them need to be assessed as if they did.

For a company assessed annually, removing a dozen systems from scope is a real, measurable saving.

But scope doesn’t disappear — it concentrates.

The tokenization system itself moves firmly into scope: the vault, the mapping, the token generation, the de-tokenization path. You’ve taken sensitive data that was spread thinly across many systems and put it in one place. That place is now the most valuable target you own, and it will be assessed accordingly.

PCI SSC guidance is unambiguous that tokenization does not remove the obligation to maintain and validate compliance. Anyone selling tokenization as “PCI DSS goes away” is selling something.

What you’ve actually bought is a smaller, sharper problem: instead of defending twenty systems adequately, you defend one system extremely well.

The GDPR Trap

This is the misunderstanding that costs teams the most, and it’s worth quoting the regulation directly.

Article 4(5) defines pseudonymisation as:

«the processing of personal data in such a manner that the personal data can no longer be attributed to a specific data subject without the use of additional information, provided that such additional information is kept separately and is subject to technical and organisational measures…»

That describes tokenization precisely. Token in the database, mapping held separately under tighter controls.

Then Recital 26 closes the door:

«Personal data which have undergone pseudonymisation, which could be attributed to a natural person by the use of additional information should be considered to be information on an identifiable natural person.»

So tokenized data is still personal data. Which means all of it still applies:

• data subject access requests
• the right to erasure
• breach notification duties
• retention limits
• a lawful basis for processing

Only genuinely anonymous data falls outside GDPR — and Recital 26 sets that bar high, asking whether identification is possible using «all the means reasonably likely to be used», accounting for cost, time and available technology.

A token that can be reversed through a vault you operate does not come close to that bar. You hold the additional information by design.

What pseudonymisation does buy you is real, just narrower: it’s explicitly recognised as an appropriate technical measure under Article 32, it reduces the severity of a breach, and it strengthens your position on data minimisation. That’s worth having. It just isn’t an exemption.

Vaulted or Vaultless

Two architectures, and the choice has real consequences.

Vaulted keeps a lookup table: token ↔ original value. Simple to reason about, easy to revoke a token, easy to audit. The cost is that the vault is a single point of failure for both availability and security. Every tokenize and de-tokenize call depends on it, it grows forever, and it holds every secret in one place.

Vaultless derives tokens algorithmically rather than storing them. Nothing to replicate, nothing to grow, no lookup on the hot path. Better scaling, no availability bottleneck.

The honest caveat: vaultless tokenization is cryptography. The security rests entirely on key management, and the line between “vaultless tokenization” and “encryption with extra branding” is thinner than most vendor material admits. That’s fine — just don’t let the word “tokenization” convince you that you’ve escaped the key-management problem you were trying to avoid.

The Trade-off That Decides Your Design

This one catches teams late, usually after the analytics team files a ticket.

Deterministic tokens — the same input always produces the same token. You can join across tables, count distinct customers, deduplicate. Everything analytical keeps working.

Random tokens — every occurrence gets a different token. Nothing leaks. Nothing joins either.

Determinism leaks structure. If one token appears ten thousand times in your dataset, that’s a fact about the underlying data, and with an auxiliary dataset it can be enough to re-identify people even though no individual token is reversible. Under GDPR, this is squarely the «singling out» risk Recital 26 asks you to consider.

There’s no universally right answer. What’s wrong is picking without knowing you picked:

A common middle path: deterministic tokens scoped per context, so the same value tokenizes differently in the analytics domain than in the support domain, and the two datasets can’t be trivially correlated.

Format-Preserving Encryption

Sometimes the real constraint isn’t security, it’s a schema you can’t change.

A legacy system expects sixteen digits with a valid checksum. It validates on write, it has integer columns, and rewriting it is not on any roadmap. Format-preserving encryption solves exactly this: the ciphertext has the same shape as the plaintext. NIST specifies FF1 and FF3 for this in SP 800-38G.

Two things to be honest about, and both come straight from NIST’s own revision history.

FF3 needed fixing. After cryptanalysis by Durak and Vaudenay, NIST announced in 2017 that it would either shrink FF3’s tweak parameter or withdraw the method entirely. The tweak was reduced from 64 bits to 56, and the result was renamed FF3-1. That change lives in SP 800-38G Revision 1 — which, as of writing, is still a public draft, not a final publication.

Small domains are the real limit. Work by Hoang, Tessaro and Trieu pushed NIST to harden its guidance: what was a recommendation of at least one million possible inputs became a requirement in Revision 1. Encrypting a field with few possible values is inherently weak, no matter which method you pick.

So treat FPE as a compatibility tool, not a default. If you can change the schema, use standard authenticated encryption or proper tokenization. Reach for FPE when the downstream system genuinely cannot be changed.

Choosing Between Them

Work through it in this order:

1. Is the field small, well-defined and high-sensitivity? Card numbers, national IDs, account numbers. If yes, tokenization is a candidate. If it’s a free-text document, it isn’t.

2. Does the value flow through systems that have no business seeing it? This is the real question. Tokenization pays off when replacing the field early keeps most of your estate out of the blast radius. If almost every consumer needs the real value anyway, you’ve bought machinery and gained nothing.

3. Is this about audit scope or about protection at rest? Scope reduction is tokenization’s genuine superpower. Protection at rest is encryption’s job, and it does it with far less operational overhead.

4. Can you actually operate a vault? High availability, disaster recovery, key rotation, access control, latency budget. Tokenization is not a library — it’s a system, and it sits on the critical path of everything that needs the real value.

5. What breaks if it’s unavailable? With encryption, an outage of your KMS is bad. With a vault, an outage means no de-tokenization at all. Plan for it before you find out.

The short version

Use tokenization when a small sensitive field spreads across many systems and you want most of them out of scope.

Use encryption for everything else — data at rest, data in transit, large or unstructured content, and any case where nearly all consumers need the real value.

Use both in most real architectures. They solve different problems and compose well.

What Neither of Them Fixes

Worth stating plainly, because it’s where the actual incidents happen.

Neither technique helps if the application itself is authorised to de-tokenize and gets compromised. Neither helps against an insider with legitimate access to the vault. Neither replaces access control, audit logging, or knowing where your data actually lives.

If you don’t have an accurate picture of which systems hold which data, tokenization won’t give you one — it will just add a system. That inventory work comes first, and I’ve written about the data governance side of it on GCP, which is the natural companion to this.

And the same principle that runs through killing long-lived service account keys applies here: the strongest control is not having the secret in the place that gets compromised.

The Bottom Line

Encryption keeps the data inside the ciphertext and hands the risk to your key management. Tokenization removes the data entirely from the systems that don’t need it and concentrates the risk in one place you defend properly.

Tokenization reduces PCI DSS scope — it does not eliminate it. The vault moves into scope and becomes your crown jewel.

Tokenization is pseudonymisation, not anonymisation. Under Recital 26, the data stays personal, and every GDPR obligation stays with it. What you gain is a recognised technical measure and a smaller blast radius — not an exemption.

Pick deterministic or random deliberately, keep format-preserving encryption for the legacy systems that force your hand, and be honest about whether you can operate a vault before you build one.