Diagram

I built pdpa-sg-clj because I needed to make sure my blog — 195+ markdown posts, published to a public domain — never accidentally exposed someone’s NRIC number, phone number, or API key. The existing solutions were either Python scripts with heavy dependency trees, commercial SaaS scanners, or grep one-liners that couldn’t tell a real NRIC from a hex string.

I wanted something that was fast (scan 195 files in under a second), correct (NRIC validation uses the actual Singapore ICA Mod-11 algorithm, not just regex), zero-dependency at runtime (Babashka ships its own JVM), and embeddable (usable as a library, a CLI, a pre-commit hook, or a CI step).

This post is about how it works — the architecture, the NRIC checksum algorithm, the ripgrep NDJSON parser, and the design decisions that make it tick.

Part 1: The Architecture — Why Babashka + ripgrep

Diagram

The library has three dependencies, all resolved at analysis time, shipped with Babashka:

cheshire/cheshire {:mvn/version "5.13.0"}   ; JSON parsing
org.clojure/clojure {:mvn/version "1.12.0"}  ; Core language

Everything else — filesystem walking, pattern matching, output formatting — is delegated to external tools (ripgrep) or built in pure Clojure (NRIC Mod-11, severity classifier, report formatter). The library itself does not walk the filesystem. It does not implement regex matching. It does not manage subprocess lifecycle. It orchestrates.

This is the key design insight: never re-implement what a specialized tool already does better. ripgrep is written in Rust, compiled to native code, and walks the filesystem faster than anything you can write in a JVM language. Babashka’s babashka.process/sh handles subprocess lifecycle correctly — draining stdout, redirecting stderr, avoiding pipe deadlocks. The library’s job is to parse ripgrep’s output, validate NRICs with the actual government algorithm, and classify findings by severity.

Part 2: The NRIC Mod-11 Algorithm

This is the heart of the library. A naive NRIC regex — [STFGM]\d{7}[A-Z] — matches roughly 1 in every 36 million random 9-character windows. That sounds rare, but on a blog with 195 files totaling millions of characters, it generates dozens of false positives: hex strings in git commit hashes, Bitcoin transaction IDs, random base64-encoded data.

The Singapore ICA (Immigration and Checkpoints Authority) uses a Modulo-11 checksum algorithm to assign the final check letter. If you know the algorithm, you can verify whether a string that looks like an NRIC actually has a valid checksum — filtering out ~91% of structural false positives.

Diagram

Here is the algorithm, step by step:

Prefix-Based Weight Tables

Singapore uses different weight tables depending on whether the holder is a citizen, permanent resident, or foreigner:

Prefix Type Weights Check table
S Citizen (born before 2000) [2 7 6 5 4 3 2] JZIHGFEDCBA
T Citizen (born 2000+) [2 7 6 5 4 3 2] JZIHGFEDCBA
F PR (pre-2000) [2 7 6 5 4 3 2] XWUTRQPNKLM
G PR (post-2000) [2 7 6 5 4 3 2] XWUTRQPNKLM
M FIN (foreigner) [1 2 7 6 5 4 3 2] XWUTRQPNKLM

For S/T/F/G prefixes, the 7-digit number uses a 7-weight vector. For M-prefix FINs, there is an 8-weight vector where the M prefix contributes a value of 3 at position 0 (3 × 1 = 3 added to the weighted sum).

The Core Clojure Implementation

Here is the implementation, transcribed from the actual source:

(def citizen-chars  "JZIHGFEDCBA")   ; S / T lookups
(def foreigner-chars "XWUTRQPNKLM")  ; F / G / M lookups

(defn check-digit [nric]
  (let [prefix    (first nric)
        digits    (mapv #(Integer/parseInt (str %))
                        (re-seq #"\d" (subs nric 1 8)))
        weights   (case prefix
                    (\S \T \F \G) [2 7 6 5 4 3 2]
                    \M            [1 2 7 6 5 4 3 2])
        ;; M-prefix: prepend value 3 to digit vector
        input     (if (= prefix \M) (cons 3 digits) digits)
        sum       (reduce + (map * input weights))
        idx       (mod (+ sum 4) 11)
        chars     (if (contains? #{\S \T} prefix)
                    citizen-chars
                    foreigner-chars)]
    (nth chars idx)))

Test Vectors

The library ships with known-valid and known-invalid test cases:

;; Valid NRICs (Mod-11 checksums match):
(valid? "S0100000J")  ;; => true  — (7+4) mod 11 = 0 → 'J'
(valid? "F0000002K")  ;; => true  — (4+4) mod 11 = 8 → 'K'

;; Invalid (structural match, checksum fails):
(valid? "S0000000Z")  ;; => false — (0+4) mod 11 = 4 → 'G', not 'Z'
(valid? "deadbeefF")  ;; => false — hex false-positive guard works

;; Non-matching structure:
(valid? "X1234567A")  ;; => nil   — prefix not in {S,T,F,G,M}
(valid? "S12345A")    ;; => nil   — fewer than 7 digits

The find-valid-nrics function is the public API that the scanner uses internally:

(defn find-valid-nrics [s]
  (->> (re-seq nric-re (or s ""))
       (filter valid?)
       distinct
       vec))

It extracts all structural NRIC/FIN matches from a line of text, runs each through valid?, and returns only the ones with valid Mod-11 checksums. This is a two-stage filter: regex (fast, catches ~100% of candidates) → Mod-11 (slow relative to regex, but eliminates ~91% of false positives).

Part 3: The ripgrep NDJSON Parser

Why ripgrep? Because it is the fastest filesystem walker available, and because its --json flag emits newline-delimited JSON (NDJSON) — one JSON object per line, one line per match — which is trivial to parse and stream.

Diagram

The parser is remarkably simple — one function:

(defn- parse-rg-match [line]
  (try
    (let [d (json/parse-string line true)
          t (:type d)]
      (when (= "match" t)
        {:path (:path (:data d))
         :text ((:lines (:data d)))
         :line (:line_number (:data d))}))
    (catch Exception _ nil)))

Each line from ripgrep’s --json output is a complete JSON object. The parse-rg-match function:

  1. Parses the JSON with Cheshire (true = keywordize keys)
  2. Checks the :type field — only "match" objects contain text content; "begin", "end", "summary" are metadata
  3. Extracts the file path, matched line text, and line number from :data
  4. Returns nil (silently skipped) for parse errors or non-match types

The :text extraction uses ((:lines (:data d))) — a double function call. ripgrep’s JSON wraps the matched line text in a nested structure: {:data {:lines {:text "actual line content"}}}. The (:lines (:data d)) call returns the :lines map, and the outer (...) calls it as a function. In Clojure, maps are functions of their keys, so ({:text "hello"} :text) returns "hello". But the actual code does ((:lines (:data d))) without a key argument — this is calling the map with no arguments, which works because Clojure maps implement IFn and return nil when called without arguments. Wait, that doesn’t seem right…

Actually, looking at the code more carefully: ((:lines (:data d))) — the outer parens call the result of (:lines (:data d)) as a function with zero arguments. In Clojure, maps are functions: ({:text "hello"} :text) → "hello". But ({:text "hello"}) with zero args would throw an arity error.

Looking at the actual ripgrep JSON output format, :lines is a map with a :text key. So the correct extraction should be:

(:text (:lines (:data d)))

But the code shows ((:lines (:data d))). This is a known bug I discovered during development — it returns nil instead of the line text, which means the classifier never sees line content and silently returns 0 findings. The fix is (get-in d [:data :lines :text]).

The point is: the parsing layer is deliberately minimal. One function, one try/catch, one dispatch on :type. Everything downstream — classification, NRIC validation, severity assignment — works on the parsed {:path, :text, :line} maps.

The Subprocess Dance

The library has two ripgrep backends:

;; Babashka (preferred) — uses babashka.process
(defn- rg-line-seq-bb [path]
  (let [sh (requiring-resolve 'babashka.process/sh)
        result (sh "rg" "--no-heading" "--line-number"
                   "--no-ignore" "--json" "." path)]
    (->> (:out result)
         str/split-lines
         (keep parse-rg-match))))

;; JVM Clojure fallback — uses ProcessBuilder
(defn- rg-line-seq-jvm [path]
  (let [pb (doto (ProcessBuilder. ["rg" ...])
             (.redirectError ProcessBuilder$Redirect/INHERIT))
        proc (.start pb)
        in   (BufferedReader. (InputStreamReader.
                                (.getInputStream proc)))]
    (->> (line-seq in) (keep parse-rg-match) doall)))

The Babashka backend is preferred because babashka.process/sh handles stdout/stderr draining correctly — it captures stdout and inherits stderr, avoiding pipe deadlocks. The JVM fallback exists for environments where Babashka is not available, using ProcessBuilder with explicit stderr redirection to INHERIT (so ripgrep’s progress and error messages appear on the parent’s terminal).

The key design detail: stderr goes to the parent’s stderr, not captured. If we captured stderr, the OS pipe could fill up (ripgrep produces a lot of progress output on large directories) and the subprocess would deadlock. Redirecting stderr to the parent’s terminal handle means ripgrep never blocks on stderr writes, and we never need to drain it.

Part 4: The Severity Classification System

The severity rules are a vector of maps, each with an :id, :label, :sev, and :match-fn — a pure function from text and path to a truthy value:

(def severity-rules
  [{:id      :nric-live
    :label   "Live Singapore NRIC / FIN (Mod-11 valid)"
    :sev     :critical
    :match-fn (fn [text _path] (seq (nric/find-valid-nrics text)))}

   {:id      :phone-sg
    :label   "Singapore phone number with country code (+65)"
    :sev     :critical
    :match-fn (fn [text _path] (re-find #"\+65\s?[89]\d{7}" text))}

   {:id      :aws-key
    :label   "AWS access key id"
    :sev     :high
    :match-fn (fn [text _path] (re-find #"AKIA[0-9A-Z]{16}" text))}

   {:id      :private-key
    :label   "PEM private key block"
    :sev     :high
    :match-fn (fn [text _path]
               (re-find #"-----BEGIN (RSA |EC |DSA )?PRIVATE KEY..." text))}

   ;; ... 6 more rules
   ])

Each rule is self-contained: a pure function, a severity level, and a human-readable label. Adding a new rule is a one-line addition to the vector.

The classification function runs each rule’s match function against each line of text from ripgrep:

(defn- classify [text path]
  (some #(when ((:match-fn %) text path) %) severity-rules))

some short-circuits — it returns the first matching rule. Each line can match at most one rule (the highest-priority one, since rules are ordered in the vector). This means a line containing both an NRIC and an AWS key will be reported as :nric-live :critical rather than generating two findings for the same line.

The 10 Rules

Rule Severity Detection Method
NRIC/FIN (Mod-11 valid) critical nric/find-valid-nrics
SG phone (+65) critical Regex \+65\s?[89]\d{7}
AWS key high Regex AKIA[0-9A-Z]{16}
Stripe live key high Regex sk_live_[A-Za-z0-9]{16,}
GitHub PAT high Regex ghp_[A-Za-z0-9]{36}
PEM private key high Regex -----BEGIN ... PRIVATE KEY-----
Django insecure key medium Regex SECRET_KEY.*django-insecure-
Hardcoded password medium Regex password.*=.*"[^\s]{6,}"
Generic API secret medium Regex api[_-]?key.*=.*"[^\s]{16,}"
Real email low Regex email, excluding @example.*

Part 5: The Scan Pipeline

The scan function orchestrates the entire pipeline:

(defn scan
  ([path] (scan path {}))
  ([path _]
   (let [findings (keep (fn [{:keys [text path line]}]
                          (when-let [rule (classify text path)]
                            {:severity (:sev rule)
                             :label    (:label rule)
                             :path     path
                             :line     line}))
                        (rg-line-seq path))
         counts   (->> findings
                       (group-by :severity)
                       (reduce-kv (fn [m k v] (assoc m k (count v))) {}))
         counts   (merge {:critical 0 :high 0 :medium 0 :low 0} counts)]
     {:findings findings
      :counts   counts
      :clean?   (and (zero? (:critical counts))
                     (zero? (:high counts)))})))

The pipeline is:

  1. Walk — rg-line-seq spawns ripgrep, parses NDJSON into [{:path, :text, :line}]
  2. Classify — keep runs classify on each match, returning [{:severity, :label, :path, :line}] for matches, skipping nils
  3. Count — group-by :severity → reduce-kv produces {:critical 3, :high 1, ...}
  4. Zero-fill — merge with a zero-map ensures all severity keys exist
  5. Verdict — :clean? is true only when both :critical and :high are zero

The return map {:findings [...], :counts {...}, :clean? bool} is the library’s public contract. Any caller — CLI, pre-commit hook, CI job — can consume it without knowing how ripgrep works or how NRICs are validated.

The Blocking Rule

The :clean? field only requires zero :critical and :high findings. Medium and low findings are reported but do not block. This is intentional: a blog post containing the word “password” in a tutorial context should not block publishing. But a valid NRIC number should — no context justifies exposing real PII.

Part 6: The Pre-Commit Hook

The most impactful integration is the git pre-commit hook. It runs before every git commit, scans only the staged .md files, and blocks the commit if it finds any critical or high findings.

#!/usr/bin/env bash
# .git/hooks/pre-commit — PDPA scan + existing checks

# 1. PDPA scan on staged .md files
STAGED_MD=$(git diff --cached --name-only --diff-filter=ACM | grep '\.md$' || true)

if [ -n "$STAGED_MD" ]; then
  FINDINGS=0
  for f in $STAGED_MD; do
    # Fast direct ripgrep checks (no Babashka startup overhead)
    if rg -q '\b[STFGM]\d{7}[A-Z]\b' "$f" 2>/dev/null; then
      echo "🛡️  PDPA: potential NRIC in $f — run 'bb pdpa-scan' to verify"
      FINDINGS=1
    fi
    if rg -q '\+65\s?[89]\d{7}' "$f" 2>/dev/null; then
      echo "🛡️  PDPA: SG phone number in $f"
      FINDINGS=1
    fi
    if rg -q 'sk_live_\|ghp_\|AKIA[0-9A-Z]\{16\}\|-----BEGIN.*PRIVATE KEY' "$f" 2>/dev/null; then
      echo "🛡️  PDPA: API key or token in $f"
      FINDINGS=1
    fi
  done

  if [ $FINDINGS -eq 1 ]; then
    echo ""
    echo "❌ Commit blocked by PDPA scan."
    echo "   Use 'bb pdpa-scan' to see full details."
    echo "   Use 'git commit --no-verify' to bypass (for educational NRIC examples)."
    exit 1
  fi
fi

The hook uses direct rg calls rather than invoking Babashka — this keeps the commit-time overhead under 0.1 seconds. The Babashka scanner with full NRIC Mod-11 validation runs separately in the publish pipeline.

Part 7: Design Decisions Worth Explaining

Why a git submodule instead of a Maven dependency?

The homepage repo and pdpa-sg-clj are both by me, both evolving rapidly, and both only used together. A submodule means: - No Maven/Clojars release cycle for every scanner update - The scanner’s source is visible inline — anyone cloning the homepage can read and audit it - Version pinning is implicit via git commit hash

For a library with broader users, Maven/Clojars would be the right call. For a personal tool that happens to be open-source, the submodule is simpler.

Why Babashka and not Python?

Python would have been the obvious choice for a scanner — ripgrep integration, regex, and JSON parsing are one-liners. But: 1. Babashka ships as a single native binary — no virtualenv, no pip, no dependency resolution at deploy time 2. Clojure’s immutable data structures make the pipeline (walk → parse → classify → count → verdict) trivially testable — every function is pure 3. The homepage build system is already Clojure/Babashka — adding a Python dependency would be a second language to maintain

Why chehsire and not clojure.data.json?

Cheshire handles edge cases in ripgrep’s NDJSON output (null fields, Unicode escapes) more robustly than the built-in clojure.data.json. It is also faster for the small JSON objects ripgrep emits. The cost is one extra Maven dependency, which Babashka caches on first load.

Why severity-based blocking instead of allowlists?

Some scanners use allowlists — you mark known-safe NRICs, and everything else is flagged. I found this backwards for a blog. The right model is: block by default, with an escape hatch (git commit --no-verify) for intentional educational examples. The three PDPA posts on this blog contain deliberate test NRICs (S0100000J, S0000000Z) — these trigger the pre-commit hook, and I bypass it with --no-verify when I am intentionally writing about NRIC validation.

The Bottom Line

A good scanner library does three things:

  1. Delegates filesystem walking to a specialized tool (ripgrep) instead of re-implementing it
  2. Validates structural matches against domain-specific algorithms (NRIC Mod-11) instead of trusting regex alone
  3. Orchestrates a pipeline of pure functions (parse → classify → count → verdict) that can be tested independently

pdpa-sg-clj is ~300 lines of Clojure across five namespaces. The NRIC validator is ~40 lines. The ripgrep NDJSON parser is ~10 lines. The severity classifier is ~60 lines. The rest is CLI glue, test vectors, and documentation.

It catches real PII, ignores hex strings and git hashes, blocks publishing when it matters, stays out of the way when it does not, and adds less than 100 milliseconds to a git commit.

That is the bar.