Dataset utilities¶
Helpers for the master dataset: every month of cleaned crime data in one GeoParquet file.
The master file is the single source of truth. The monthly pipeline merges each new month into it, and everything downstream (GIS exports, the dashboard's web feed) is built from it.
ensure_wards(df)
¶
Add the ward column if it is missing or incomplete (older master files have none).
Needs GeoPandas. Rows that already have a ward are re-assigned too; the result is the same because the ward comes from the coordinates and the boundary file.
Source code in src/geocrime_stl/utils/dataset.py
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | |
load_master(path=None)
¶
Read the master dataset as a plain DataFrame (any geometry column is dropped).
Returns an empty DataFrame if the file does not exist yet.
Source code in src/geocrime_stl/utils/dataset.py
19 20 21 22 23 24 25 26 27 28 | |
merge_month(master, month_df, month, year)
¶
Replace one month of the master dataset with freshly processed rows.
Any existing rows for that month are removed first, so re-running a month (for example after SLMPD revises a file) never creates duplicates. Order within each month is preserved.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
master
|
DataFrame
|
The existing master DataFrame (may be empty). |
required |
month_df
|
DataFrame
|
The new month's cleaned rows, from |
required |
month
|
int
|
The month being merged (1-12). |
required |
year
|
int
|
The four-digit year being merged. |
required |
Source code in src/geocrime_stl/utils/dataset.py
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | |
save_master(df, parquet_path=None, geojson_path=None, *, write_geojson=None)
¶
Write the master dataset as GeoParquet and (optionally) GeoJSON.
The GeoJSON is a convenience for GIS users (it is large and is not committed to git). Pass write_geojson
to choose, or set the environment variable GEOCRIME_WRITE_GEOJSON=0 to skip it (the monthly GitHub Action does).
GeoPandas is imported here, not at module level, so the rest of this module works without it.
Source code in src/geocrime_stl/utils/dataset.py
76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 | |