System design & data flow

PROOViD AML is a single ASP.NET Core (.NET 8) service with a clean four-layer split (API → Application → Domain → Infrastructure), PostgreSQL via EF Core 8, an own OpenIddict identity provider, and a small Python NER companion. Below is a detailed walk-through of how a request becomes a decision, with a dedicated diagram for every function.

1 · Full component map

Every runtime component and how a request flows through them — clients and the SDK/MCP, the edge, the dual-authority identity layer, the .NET service and its pure domain engine, the data & infrastructure tier, and the external sources.

flowchart TB
  subgraph clients[Clients]
    apic[API consumers - REST]
    sdk[TypeScript SDK]
    mcp[MCP server - AI agents]
    uis[Bundled UIs - onboarding / stats / cases / playground]
  end
  subgraph edge[Edge]
    traefik[Traefik ingress + cert-manager TLS]
  end
  subgraph idsec[Identity and access]
    idp[Own IdP - OpenIddict]
    kc[Keycloak]
    apikey[API-key handler]
  end
  subgraph appsvc[AMLService - .NET 8]
    ctrl[REST controllers + RFC7807]
    orch[ScreeningOrchestrator + full-text search]
    subgraph dengine[Domain engine - pure]
      seng[ScreeningEngine - TP/PM/FP/Unknown]
      match[Matching - trigram / Jaro-Winkler / phonetic / nickname]
      risk[RiskScoringEngine - 0-100 + band]
    end
    am[Adverse-media + relevance filter]
    mon[Monitoring + scheduled re-screen]
    thr[Turnover-threshold engine]
    casesvc[Cases + attachments + audit hash-chain]
    priv[GDPR privacy - DSR / consent / retention purge]
    quota[Quota + rate limit]
    bg[Batch BackgroundService]
    wh[Webhook dispatcher - HMAC]
  end
  subgraph datainfra[Data and infra]
    pg[(PostgreSQL 16 - pg_trgm / dmetaphone / tsvector)]
    ingest[Watchlist ingestion]
    ner[aml-ner - FastAPI + spaCy NER - optional, not the default relevance filter]
  end
  subgraph external[External]
    gdelt[GDELT news]
    byo[OpenSanctions / BYO provider]
    sources[UN / OFAC / EU / UK / Wikidata PEP]
    twh[Tenant webhook endpoints]
  end
  apic --> traefik
  sdk --> traefik
  mcp --> traefik
  uis --> traefik
  traefik --> ctrl
  ctrl --> apikey
  ctrl --> idp
  ctrl --> kc
  ctrl --> quota
  ctrl --> orch
  ctrl --> mon
  ctrl --> thr
  ctrl --> casesvc
  ctrl --> bg
  ctrl --> priv
  priv --> pg
  orch --> seng
  seng --> match
  match --> pg
  orch --> risk
  orch --> am
  am --> gdelt
  am --> ner
  orch --> byo
  thr --> orch
  mon --> orch
  bg --> orch
  ingest --> sources
  ingest --> pg
  orch --> pg
  casesvc --> pg
  orch --> wh
  mon --> wh
  thr --> wh
  wh --> twh
      

Data flows left-to-right/top-down along the animated edges: client → edge → auth → orchestrator → engine + providers → PostgreSQL, with webhooks fanning out to tenant endpoints.

2 · Screening: request → decision

The core path. The API gates the request on the tenant's per-minute rate limit and monthly quota, then the orchestrator matches the subject against the enabled watchlists (trigram + phonetic + nickname), classifies each candidate, scores risk, optionally checks adverse media, persists the result with explainability factors, and emits screening.completed. The response is one actionable verdict plus the matched entities and risk factors.

flowchart LR
  C[Client - POST /v1/screenings/check] --> A[API - authenticate, tenant from principal]
  A --> Q{Quota and rate OK}
  Q -- no --> R429[429 Too Many Requests]
  Q -- yes --> O[Orchestrator - build match options]
  O --> M[Match vs enabled watchlists - trigram / phonetic / nickname]
  M --> DB[(PostgreSQL - GIN indexes)]
  M --> E[Classify - TP / PM / FP / Unknown]
  E --> RS[Risk score + band]
  RS --> V[Adverse-media evidence - read from the prefetch cache, no network call]
  V --> P[Persist screening + factors]
  P --> WH[screening.completed webhook - HMAC]
  P --> RESP[Return 201 - Pass / Review / Fail]
      

Per-request knobs (lists, threshold, exactMatch, adverseMedia, includeDiagnostics) tune this path without changing the engine.

2.1 · Inside the match — fuzzy, phonetic & nickname

A name never has to be spelled exactly. Matching is two phases: fast candidate retrieval (high recall, index-driven) then precise scoring (high precision, which gates false positives). Typos, transliterations, sound-alikes and nicknames all still find the right entry, and the score decides whether it's kept.

• Trigram (fuzzy) — % on the GIN index catches misspellings & spelling variants (Mohammed / Muhammad).
• Phonetic — double-metaphone per token catches sound-alikes even with a wrong first name (Catherine / Kathryn).
• Nickname — hypocorism expansion (Jim → James, Bob → Robert) before retrieval.
• Scoring — Jaro-Winkler + token-set similarity (0–1), then a DOB / nationality boost or penalty; kept only if ≥ the tenant/request threshold.

flowchart TB
  IN[Input - full name + aliases + optional DOB / nationality] --> NORM[Normalize - lowercase, strip accents and punctuation]
  NORM --> NICK[Expand nicknames - Jim to James, Bob to Robert]
  NICK --> RET[Candidate retrieval - high recall]
  RET --> TRG[Trigram similarity - % on GIN index - typos and spelling]
  RET --> PHON[Phonetic - double-metaphone tokens - sound-alikes]
  TRG --> SC[Score each candidate - high precision]
  PHON --> SC
  SC --> JW[Jaro-Winkler + token-set similarity 0 to 1]
  JW --> DOB[DOB and nationality boost or penalty]
  DOB --> TH{Score at or above threshold}
  TH -- yes --> KEEP[Keep then classify - True / Potential / False]
  TH -- no --> DROP[Discard]
      

Worked example. Screening "Jose Kony" (misspelt): normalize → jose kony; the trigram index finds joseph kony and phonetic confirms the surname sounds the same; Jaro-Winkler + token-set score ≈ 0.9 — above the floor and the tenant threshold — so it is kept and classified against the enabled lists. A DOB mismatch lowers the score; exactMatch:true turns the fuzzy levers off.

3 · Watchlist ingestion

Sanctions and PEP sources are pulled directly (no per-lookup aggregator), normalized into one entity/name model, and indexed for fast fuzzy + phonetic retrieval. Ingestion runs on a schedule or via the admin API; the screening path only ever reads the local, indexed copy.

flowchart LR
  trig[Scheduled job or POST /v1/admin/watchlists/ingest] --> ing[WatchlistIngestionService]
  ing --> un[UN]
  ing --> ofac[OFAC]
  ing --> eu[EU]
  ing --> uk[UK]
  ing --> pep[Wikidata PEP]
  un --> norm[Normalize to one entity/name model]
  ofac --> norm
  eu --> norm
  uk --> norm
  pep --> norm
  norm --> idx[Build indexes - trigram GIN + per-token dmetaphone]
  idx --> pg[(PostgreSQL)]
      

Candidate retrieval uses the % trigram operator on the GIN index (≈7ms) and GIN array-overlap on the phonetic tokens — never a full scan.

The sources every screen checks against (the tenant chooses which are enabled; a per-request lists override can narrow them for one call):

Each feed is a built-in default and overridable per deployment (Watchlists:<Source>:FeedUrl); ingestion normalizes them all into one entity/name model before indexing.

4 · Adverse media — prefetched, never fetched in-screen

Adverse media does not query the news source during your request. A background sweep prefetches evidence into a cache; the screening path performs one indexed read and no network call at all. This is why a news-source outage or rate limit can neither slow down nor break a screen.

The sweep uses two GDELT surfaces: DOC retrieves candidate articles (metadata only — URL, headline, publisher, date), and the GKG slice each article was indexed in supplies the person entities GDELT itself extracted from it. That entity signal is what lets us recover on-topic articles whose headline never names the subject.

flowchart LR
  SW[Background sweep - every 15 min] --> DOC[GDELT DOC - candidate articles]
  DOC --> GKG[GDELT GKG slice - extracted person entities]
  GKG --> BIND[Bind subject - ordered adjacent span + rarity gate]
  BIND --> CACHE[(Prefetched evidence cache)]
  O[Orchestrator] --> Q{adverseMedia on}
  Q -- no --> SKIP[Status Skipped - we did not look]
  Q -- yes --> READ[Read cache - no network]
  CACHE --> READ
  READ --> HIT[Match with headline, publisher, date, category, matched person]
  READ --> UNAVAIL[No evidence yet - status Unavailable, subject enqueued]
      

A subject with no prefetched evidence returns Unavailable — “we did not check”, which is never served as a clean pass — and is enqueued for the next sweep. Every hit carries its evidence and a GDELT attribution. Full detail, including the measured recall/precision and the limits: Adverse media.

5 · Ongoing monitoring

Enrol a subject once (monitor: true) and the system re-screens it when watchlists change and on a scheduled cadence (daily delta + monthly full), alerting only when the outcome actually moves.

flowchart LR
  ENR[monitor:true on a screen] --> REG[(MonitoredSubject registry)]
  CHG[Watchlist change] --> SW[Re-screen monitored subjects]
  SCH[Scheduled cadence - daily / monthly] --> SW
  REG --> SW
  SW --> D{Outcome changed}
  D -- yes --> AL[monitoring.alert webhook]
  D -- no --> NO[No-op]
      

6 · Turnover-threshold monitoring

A consumer feeds turnover events for a subject; the engine keeps a rolling-window total and, when it crosses the tenant's configured threshold, auto-re-screens the subject and fires threshold.breached. Inert until a threshold is set.

flowchart LR
  EV[POST /v1/threshold/events] --> LED[(ThresholdLedger)]
  LED --> ROLL[Rolling-window turnover]
  ROLL --> D{At or over threshold}
  D -- yes --> RSC[Auto re-screen subject]
  RSC --> WH[threshold.breached webhook]
  D -- no --> ACC[Accumulate only]
      

7 · Async batch screening

For large files: upload a CSV/XLSX, the rows are parsed and a job is queued; a background worker claims it (multi-replica-safe via an xmin row version), screens each row through the same orchestrator, and the results are downloadable as CSV or JSON.

flowchart LR
  UP[POST /v1/screenings/batch - CSV or XLSX] --> PAR[Parse + map names]
  PAR --> JOB[(BatchJob - queued)]
  JOB --> BG[BackgroundService claims job - xmin]
  BG --> ORCH[Screen each row via the orchestrator]
  ORCH --> ROWS[(BatchJobRow results)]
  ROWS --> DL[GET /results - CSV or JSON]
      

8 · Case management, audit & regulator pack

Matches that need a human become cases: reviewed and dispositioned (cleared / confirmed / escalated) with a note thread and attachments, every change written to a tamper-evident, hash-chained audit log. A regulator pack exports the whole bundle as JSON, PDF or Parquet.

flowchart LR
  SCR[Screening match] --> CASE[(Case)]
  CASE --> REV[Review - clear / confirm / escalate + notes + attachments]
  REV --> AUD[Audit log - SHA-256 hash-chained]
  CASE --> PACK[Regulator pack]
  PACK --> J[JSON]
  PACK --> PDF[PDF]
  PACK --> PARQ[Parquet]
      

9 · GDPR data-subject flows

Data-subject rights are first-class: export a subject's full record, erase it (anonymize-in-place and de-link), restrict/object (which suppresses further processing), or rectify. Consent is captured with a versioned copy, and a retention job anonymizes expired PII automatically.

flowchart LR
  REQ[Data-subject request] --> EXP[Export - SAR bundle]
  REQ --> ER[Erase - anonymize in place + de-link]
  REQ --> RST[Restrict or object - set flag]
  REQ --> REC[Rectify - update snapshot]
  RST --> MON[Monitoring skips the subject]
  CON[Record consent] --> CV[(Versioned consent record)]
  RET[Retention purge - scheduled] --> ANON[Anonymize expired PII]
      

Technology