Skip to content

Web export (dashboard feed)

Build the dashboard's data feed: one compact JSON file per month plus a manifest.

The dashboard (see dashboard/) is a static site, so it cannot query a database. Instead the pipeline publishes small files it can download once and cache:

1
2
3
data/web/manifest.json     which months exist, row counts, "data through" date, cache-buster, and the
                           category table (NIBRS code, group, violent flag; see NIBRS_CATEGORIES)
data/web/YYYY-MM.json      every offense for one month, with repeated strings stored once

The feed is rebuilt in full from the master dataset every time (about a second), so late revisions to earlier months are always picked up. The build is all-or-nothing: it is validated first and written to a temporary folder, so a bad run can never leave a half-written or truncated feed behind.

File format, one row per offense::

1
2
3
4
5
{"m": "2026-09",
 "d": {"type": [...], "cat": [...], "off": [...], "nbhd": [...], "ward": ["1", ..., "14"]},
 "r": [[inc, "YYYY-MM-DDTHH:MM", lat, lon, address,
        typeIdx, catIdx, offenseIdx, nbhdIdx, district, firearm(0/1), estTime(0/1),
        wardIdx (-1 = no ward), masked(0/1)], ...]}

Masked rows (see MASKED_CATEGORIES and MASKED_OFFENSE_TERMS) have no street address and only a coarse location.

It also writes city.geojson, wards.geojson and neighborhoods.geojson (outlines with 2020 census population) for maps.

Run from the command line to rebuild from the master dataset::

1
python -m geocrime_stl.utils.web_export [--out DIR] [--source FILE] [--force]

WebExportError

Bases: RuntimeError

The feed could not be built safely. Nothing on disk was changed.

Source code in src/geocrime_stl/utils/web_export.py
183
184
class WebExportError(RuntimeError):
    """The feed could not be built safely. Nothing on disk was changed."""

build_web_feed(df=None, out_dir=None, *, force=False)

Rebuild the dashboard feed from the master dataset.

Parameters:

Name Type Description Default
df DataFrame | None

Cleaned crime rows for ALL months. If omitted, the master dataset is loaded from disk.

None
out_dir Path | str | None

Destination folder. Defaults to config.WEB_DIR (data/web).

None
force bool

Skip the plausibility checks (partial newest month, big drop in total rows).

False

Returns:

Type Description
dict

The manifest that was written.

Raises:

Type Description
WebExportError

If the input is unusable or the result fails validation. The existing feed is left untouched in that case.

Source code in src/geocrime_stl/utils/web_export.py
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
def build_web_feed(
    df: pd.DataFrame | None = None,
    out_dir: Path | str | None = None,
    *,
    force: bool = False,
) -> dict:
    """Rebuild the dashboard feed from the master dataset.

    Args:
        df: Cleaned crime rows for ALL months. If omitted, the master dataset is loaded from disk.
        out_dir: Destination folder. Defaults to ``config.WEB_DIR`` (``data/web``).
        force: Skip the plausibility checks (partial newest month, big drop in total rows).

    Returns:
        The manifest that was written.

    Raises:
        WebExportError: If the input is unusable or the result fails validation. The existing
            feed is left untouched in that case.
    """
    out_dir = Path(out_dir) if out_dir else WEB_DIR
    if df is None:
        df = load_master()
        if df.empty:
            raise WebExportError("Master dataset not found or empty. Run download_historical_data.py first.")
        try:
            df = ensure_wards(df)
        except Exception as err:  # e.g. GeoPandas not installed: build the feed without wards
            logger.warning("Wards could not be assigned (%s); the feed will have no ward data", err)

    d = _prepare(df)
    counts = d.groupby("month").size().to_dict()
    _validate(counts, _read_manifest(out_dir / "manifest.json"), force=force)

    # Write everything to a sibling temp folder, then swap it into place.
    tmp = out_dir.with_name(out_dir.name + ".tmp")
    if tmp.exists():
        shutil.rmtree(tmp)
    tmp.mkdir(parents=True)
    try:
        wards = _write_wards(tmp / "wards.geojson")
        _write_neighborhoods(tmp / "neighborhoods.geojson", d)
        _write_city(tmp / "city.geojson")
        for month, g in d.groupby("month", sort=True):
            with open(tmp / f"{month}.json", "w", encoding="utf-8") as f:
                json.dump(_month_file(month, g, wards), f, separators=(",", ":"))
        months = sorted(counts)
        manifest = {
            "schema": SCHEMA_VERSION,
            "built": datetime.now(timezone.utc).strftime("%Y%m%d%H%M%S"),  # cache-buster for the browser
            "through": d["t"].max().strftime("%Y-%m-%d"),
            # SLMPD posts each month whole, so the newest month counts as complete.
            "completeThrough": months[-1],
            "rows": int(len(d)),
            "incidents": int(d["inc"].nunique()),
            "unmapped": int(d["lat"].isna().sum()),
            "noWard": int(d["ward"].isna().sum()),
            "masked": int(d["masked"].sum()),
            "maskedCategories": list(MASKED_CATEGORIES),
            "categories": [{"cat": c, "code": k, "group": g, "violent": int(v)} for c, k, g, v in NIBRS_CATEGORIES],
            "maskedOffenses": [label for _, label in MASKED_OFFENSE_TERMS],
            "newCategories": d.attrs.get("new_categories", []),
            "months": [{"m": m, "n": int(counts[m])} for m in months],
        }
        (tmp / "manifest.json").write_text(json.dumps(manifest, separators=(",", ":")), encoding="utf-8")
        if out_dir.exists():
            shutil.rmtree(out_dir)
        out_dir.parent.mkdir(parents=True, exist_ok=True)
        tmp.rename(out_dir)
    except Exception:
        shutil.rmtree(tmp, ignore_errors=True)
        raise

    logger.info("Web feed written: %d months, %d rows, through %s -> %s",
                len(months), manifest["rows"], manifest["through"], out_dir)
    return manifest