Regex PII redactor for prompts, logs and datasets
A dependency-free Python script that replaces emails, phone numbers, US SSNs, Luhn-valid card numbers, IP addresses, IBANs and bearer tokens with placeholders before text goes to a model or into a dataset. Its limits are stated up front.
File in Security by AISkillHub Library
What it does
Support transcripts, logs and exported tickets are the most common source of eval sets and fine-tuning data, and they are full of personal data. This script removes the well-structured kinds of PII before the text is pasted into a prompt, sent to an API, or committed as a test fixture.
**What it detects**
| Type | Notes | |---|---| | `EMAIL` | Standard address format | | `PHONE` | Needs a separator or leading `+`; dates and bare digit runs are skipped | | `SSN` | US format `123-45-6789`, invalid ranges excluded | | `CARD` | 13 to 19 digits that pass the Luhn check, so order numbers are left alone | | `IPV4`, `IPV6` | Octets range-checked for IPv4 | | `IBAN` | Country code, check digits, grouped or ungrouped | | `BEARER` | `Bearer <token>` strings from copied headers |
**Example**
``` in: Contact [email protected] or +1 415-555-0132. Card 4111 1111 1111 1111. Order 4471. out: Contact [EMAIL_1] or [PHONE_1]. Card [CARD_1]. Order 4471. ```
`--numbered` maps the same value to the same placeholder within a run, so a conversation still makes sense after redaction ("[EMAIL_1] wrote back..."). `--report` prints counts per type, and `--fail-on-match` turns it into a check for CI or pre-commit hooks.
**Limits (read before relying on it)**
This is pattern matching, not entity recognition. It does **not** detect names, street addresses, dates of birth, account usernames, free-text health or financial details, or national IDs other than US SSNs. It misses values that are misspelled, obfuscated or split across lines. Treat its output as reduced-risk, not anonymized. For regulated data, use a dedicated PII detection service or human review in addition to this script.
Exit codes: 0 success, 1 PII found with `--fail-on-match`, 2 usage error.
How to use it
1. Copy `scripts/redact_pii.py` into your project. 2. Redact a stream or files: ``` cat transcript.txt | python3 scripts/redact_pii.py > clean.txt python3 scripts/redact_pii.py tickets/*.txt --in-place --numbered --report python3 scripts/redact_pii.py export.jsonl --only EMAIL --only PHONE > export.clean.jsonl ``` 3. Use it as a check without changing files: `python3 scripts/redact_pii.py fixtures/*.txt --fail-on-match` 4. Spot-check a sample of the output by hand, especially for names and addresses, which the script does not catch. 5. Keep the original files out of version control; commit only the redacted versions.
Python 3.8+, no packages, no environment variables.
Files
Save each file at the path shown, relative to your project root.