Building a scam detection dataset with GLM 5.3 Flash

A scam classifier is only as good as the scams it has seen. Real reported messages are the best training data you will ever have, and you will never have enough of them: they skew toward last quarter's lures, they arrive unlabeled, and they barely cover the languages and platforms you care about. Synthetic data fills the gaps. The catch is that most hosted models refuse to write a fake parcel-fee text, even when the point is to catch fake parcel-fee texts.

This post walks through a pipeline we would actually run: generate with GLM 5.3 Flash, audit with GLM 5.3, review a sample by hand, then measure the classifier on real data before believing any number. The code is plain Python against POST /api/v1/chat/completions. Everything stays at the level of scam openers and payment asks. You are teaching a classifier what the first message looks like, not writing a playbook.

Start with the schema

Decide what one row means before you generate a single one. A loose schema turns into a cleanup project later. This is the shape we use:

{
  "text": "Hi, your parcel is held at the depot. A [AMOUNT] redelivery fee is due today: [LINK]",
  "label": "scam",
  "severity": 2,
  "signals": ["urgency", "small_fee", "unknown_link"],
  "platform": "sms",
  "language": "en",
  "stage": "payment_ask",
  "scam_type": "parcel_fee",
  "source": "synthetic",
  "cell_id": "parcel_fee|sms|en|payment_ask|courier"
}

A few choices matter more than they look:

• label is binary: scam or benign. Put the nuance in severity (0 to 3) and signals, not in a five-way label your reviewers will argue about.
• signals is a closed list: urgency, small fee, unknown link, request to move platforms, credential ask, and so on. Free text here becomes noise.
• stage separates the opener ("is this Maria?") from the follow-up and the payment ask. Classifiers that only see payment asks miss the conversation that leads to one.
• Placeholders like [LINK], [PHONE], [NAME] and [AMOUNT] keep real URLs and numbers out of the data. You must apply the same placeholders to real messages, or the model learns that brackets mean "synthetic."
• source and cell_id let you trace every row back to how it was made. You will need both when something goes wrong.

Control diversity before you generate

Ask a model for 500 scam texts and you get 500 variations of one scam text. Diversity has to be designed in. We build a grid of cells and sample from it:

import itertools, random
 
SCAM_TYPES = ["parcel_fee", "bank_impersonation", "job_offer", "wrong_number", "account_suspension"]
PLATFORMS  = ["sms", "whatsapp", "email", "marketplace_dm"]
LANGUAGES  = ["en", "es", "de", "pt-BR", "fr"]
STAGES     = ["opener", "follow_up", "payment_ask"]
PERSONAS   = ["courier", "bank_agent", "recruiter", "stranger", "platform_support"]
 
def cells(label):
    for combo in itertools.product(SCAM_TYPES, PLATFORMS, LANGUAGES, STAGES, PERSONAS):
        st, pl, lang, stage, persona = combo
        yield {"label": label, "scam_type": st, "platform": pl, "language": lang,
               "stage": stage, "persona": persona, "cell_id": "|".join(combo)}
 
scam_cells   = list(cells("scam"))
benign_cells = list(cells("benign"))  # near-misses: same surface, legitimate intent
random.seed(7)
plan = random.sample(scam_cells, 300) + random.sample(benign_cells, 300)

The benign cells are the important half. A benign parcel_fee cell asks for a real courier notice: a tracking update, a customs charge that tells you to pay in the courier's app, a delivery slot confirmation. A benign bank_impersonation cell asks for a genuine fraud alert that tells you to call the number on the back of your card. These hard negatives are what stop a classifier from flagging every message that contains the word "parcel."

Personas and stages do the rest. A recruiter's opener on WhatsApp reads nothing like a bank agent's follow-up by email, and you want both.

Generate in batches, expect bad JSON

GLM 5.3 Flash is the right model for bulk generation: the lighter GLM on our model list, 32K output per call, and no refusal when you ask for a scam lure. Ask for small batches, ten rows per call, so one malformed response costs little.

import json, os, random, time
import openai
from openai import OpenAI
 
client = OpenAI(base_url="https://xfield.cloud.brutal.ai/api/v1", api_key=os.environ["XFIELD_API_KEY"])
 
SYSTEM = """You write training data for a scam-message classifier.
Return ONLY a JSON array. Each item has keys:
text, label, severity (0-3), signals (list), platform, language, stage.
Allowed signals: urgency, small_fee, unknown_link, platform_move, credential_ask,
too_good_to_be_true, impersonation, wrong_number_hook.
Use [LINK], [PHONE], [NAME], [AMOUNT] instead of real links, numbers, names, amounts.
Write like real people: vary length, tone, punctuation, and formality.
Scam messages should be convincing to a careful reader, not cartoonish.
Benign messages should look similar on the surface but be legitimate."""
 
def prompt_for(cell, n):
    return (f"Write {n} {cell['label']} messages.\n"
            f"Type: {cell['scam_type']}. Platform: {cell['platform']}. "
            f"Language: {cell['language']}. Stage: {cell['stage']}. "
            f"Sender persona: {cell['persona']}.\n"
            "No two messages may share an opening phrase.")
 
def strip_fences(s):
    s = s.strip()
    if s.startswith("```"):
        s = s.split("\n", 1)[1].rsplit("```", 1)[0]
    return s
 
def validate(row, cell):
    assert isinstance(row.get("text"), str) and 10 < len(row["text"]) < 1200
    assert row.get("label") == cell["label"]
    for k in ("platform", "language", "stage", "scam_type", "cell_id"):
        row[k] = cell[k]          # the cell is the truth, not the model
    row["source"] = "synthetic"
    row["generator"] = "glm-5.3-flash-uncensored"
    return row
 
def generate_batch(cell, n=10, attempts=4):
    for attempt in range(attempts):
        try:
            resp = client.chat.completions.create(
                model="glm-5.3-flash-uncensored",
                messages=[{"role": "system", "content": SYSTEM},
                          {"role": "user", "content": prompt_for(cell, n)}],
                reasoning_effort="low",
                max_tokens=4000,
            )
            rows = json.loads(strip_fences(resp.choices[0].message.content))
            return [validate(r, cell) for r in rows]
        except (openai.RateLimitError, openai.InternalServerError):
            time.sleep(2 ** attempt + random.random())   # 429 and 5xx: back off
        except (json.JSONDecodeError, AssertionError, KeyError, TypeError):
            continue                                       # bad output: just ask again
    return []

Two details carry their weight. The system prompt stays identical across every call, so prompt caching kicks in and repeated input gets cheaper. And validate overwrites the cell fields instead of trusting the model's copy, so a row asked for in German cannot quietly come back tagged as Spanish.

Run the plan with a small thread pool and write each batch to a JSONL file as it lands. Six hundred cells at ten rows each gives you roughly six thousand rows before dedup.

Deduplicate twice

Exact duplicates are rare. Near-duplicates are everywhere: the same message with a different courier name, or a digit swapped. Normalize before hashing:

import hashlib, re, unicodedata
 
def norm(text):
    t = unicodedata.normalize("NFKC", text).lower()
    t = re.sub(r"\[(link|phone|name|amount)\]", " ", t)
    t = re.sub(r"\d+", "0", t)
    t = re.sub(r"[^\w\s]", " ", t)
    return " ".join(t.split())
 
def dedup(rows):
    seen, out = set(), []
    for r in rows:
        h = hashlib.sha1(norm(r["text"]).encode()).hexdigest()
        if h not in seen:
            seen.add(h)
            out.append(r)
    return out

Then run a second, fuzzier pass with embeddings. Embed each row with any sentence-embedding model, and within each scam_type drop rows whose cosine similarity to an earlier row is above about 0.92. Check the threshold by eye on a few hundred pairs. Expect to lose 10 to 25 percent of rows here. If you lose more than that, your prompts are collapsing to a template, and more dedup will not fix it.

Audit every row with GLM 5.3

The generator's label is a request, not a fact. Sometimes a "benign" courier notice comes back with a sketchy link and a deadline. So every row gets a second opinion from GLM 5.3, the stronger reasoner, which sees the text but not the original label:

AUDIT = """You label messages for a scam classifier, following this policy:
scam = tries to get money, credentials, or a platform move under a false pretext.
benign = legitimate, even if it mentions fees, parcels, or accounts.
Return ONLY JSON: {"label": "scam" | "benign", "confidence": 0-1,
"rationale": "one sentence naming the deciding feature"}"""
 
def audit(row):
    resp = client.chat.completions.create(
        model="glm-5.3-uncensored",
        messages=[{"role": "system", "content": AUDIT},
                  {"role": "user", "content": row["text"]}],
        reasoning_effort="medium",
        max_tokens=1000,
    )
    verdict = json.loads(strip_fences(resp.choices[0].message.content))
    row["audit"] = verdict
    row["needs_review"] = (verdict["label"] != row["label"]
                           or verdict["confidence"] < 0.7)
    return row

Hiding the original label matters. If the auditor sees it, it agrees with it. The one-line rationale is for your reviewers: "asks for a fee via an unbranded link" is much faster to check than the raw message alone.

Put humans on a sample, not the whole set

You cannot hand-review six thousand rows, and you do not need to. Review three groups:

1. Every row where the auditor disagreed or was unsure.
2. A stratified random sample: ten rows per scam_type and language pair, labels hidden.
3. Every benign near-miss the auditor called a scam. These tell you where your policy is vague.

Track agreement between reviewers and the final label on group 2. If it falls below about 90 percent for a slice, fix the prompt for that slice and regenerate it. Do not relabel your way out of a bad prompt.

Split by group, not by row

A random row split leaks. Two messages from the same batch share phrasing, so one lands in train and its sibling in eval, and your eval score measures memory. Split by generation cell instead, and hold out one whole scam_type as a novelty slice:

import random
 
def split(rows, holdout_type="job_offer", eval_frac=0.15, seed=7):
    novel = [r for r in rows if r["scam_type"] == holdout_type]
    rest  = [r for r in rows if r["scam_type"] != holdout_type]
    cells = sorted({r["cell_id"] for r in rest})
    random.Random(seed).shuffle(cells)
    eval_cells = set(cells[: int(len(cells) * eval_frac)])
    train = [r for r in rest if r["cell_id"] not in eval_cells]
    eval_ = [r for r in rest if r["cell_id"] in eval_cells]
    return train, eval_, novel

Measure on real data, or you have not measured

Report three numbers side by side: recall on the synthetic eval slice, recall on the novel-type slice, and recall on real reported messages, all normalized with the same placeholders. Add the false-positive rate on real benign traffic, because a detector that flags every delivery update gets switched off.

A large gap between the first row and the third is the normal starting point. Synthetic recall in the high nineties with real recall far lower means the generator is writing a tidier, more obvious version of scams than the ones your users get. Pull the real misses, describe what they have in common, and add cells for them. Then run the loop again.

Pitfalls we keep hitting

• Template collapse. Every scam opens with "Dear customer" and every fee is the same small amount. Watch the most common first five words per scam_type. If one phrasing exceeds a few percent, add persona and tone variation to the prompt.
• Label leakage. Anything that only appears in one class becomes the feature. Placeholders only in scam rows, the word "scam" inside a scam text, or every benign message being longer. Train a bag-of-words baseline. If it scores suspiciously well, look at its top features.
• Over-obvious examples. Spelling errors, all caps, and three exclamation marks are easy to generate and rare in good lures. Ask for messages that would fool a careful reader, and keep the near-misses just as polished.
• Reviewer fatigue. Even mild scam text is tiring in bulk. Rotate reviewers and keep sessions short.

If you want to try this on your own policy, request access and start with a hundred cells. GLM 5.3 Flash will tell you quickly whether your schema holds up.