hmwssb-data

HMWSSB open data mirror

Monthly datasets published by the Hyderabad Metropolitan Water Supply and Sewerage Board (HMWSSB) on Open Data Telangana, fetched on the last day of every month by a launchd job on Teja’s Mac (scripts/monthly-fetch.sh) and committed here when the portal adds a month. The portal only answers Indian connections (see Reachability below), which is why the fetch does not run on GitHub Actions. Part of the Hyderabad Urban Observatory.

Series	Portal dataset	Months	Files
`tankers/`	HMWSSB water tankers data	Jan 2022 →	`tankers/raw/*.csv` (as served), `tankers/tankers_monthly.csv` (stitched), `tankers/manifest.json`

Columns (tankers): year, month, division, section, noofbookings, delivered. noofbookings is the number of tanker bookings in that section that month, delivered the deliveries. Division and section codes match HMWSSB’s billing and water-connection datasets.

Running it yourself

python3 fetch.py            # downloads months not yet present, rewrites the stitched table
python3 fetch.py --refresh  # re-downloads everything
python3 fetch.py --offline  # no network: restitch from <series>/raw/ (what Actions runs)

The portal’s search does not index this dataset; the script reads it through the DKAN metastore API by dataset id (see fetch.py).

How the monthly update runs

launchd on Teja’s Mac, label in.urbanobservatory.hmwssb-fetch, last day of each month, 09:00 local (the portal posts the previous month around the 15th). launchd cannot express “last day”, so the plist fires on the 28th–31st and the script exits unless tomorrow is the 1st. Plist: ~/Library/LaunchAgents/in.urbanobservatory.hmwssb-fetch.plist (copy in scripts/). It runs scripts/monthly-fetch.sh: pull, fetch.py, commit and push as hulf-observatory. The push credential comes from gh auth token -u hulf-observatory at run time; nothing is stored in the repo or the plist. If the Mac is asleep at 09:00 the job runs at next wake; if it is off that day, the next run catches up (the script fetches every missing month). Log: ~/Library/Logs/hmwssb-fetch.log. Run it by hand with HMWSSB_FORCE=1 scripts/monthly-fetch.sh (a launchctl kickstart on any other day just logs “skipping”).
GitHub Actions stitch.yml runs on every push that touches */raw/** and restitches <series>_monthly.csv + manifest.json from the repo contents. It needs no network access to the portal.

To stop the job: launchctl bootout gui/$(id -u)/in.urbanobservatory.hmwssb-fetch and delete the plist.

Reinstalling on a new Mac

Needs: Python 3 on the PATH, gh logged in as hulf-observatory (gh auth login; it may stay the non-active account), and this repo cloned at ~/hulf-observatory/hmwssb-data (the plist has that path baked in; edit it if the clone lives elsewhere).

git clone https://github.com/hulf-observatory/hmwssb-data.git ~/hulf-observatory/hmwssb-data
cp ~/hulf-observatory/hmwssb-data/scripts/in.urbanobservatory.hmwssb-fetch.plist ~/Library/LaunchAgents/
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/in.urbanobservatory.hmwssb-fetch.plist
cd ~/hulf-observatory/hmwssb-data && HMWSSB_FORCE=1 scripts/monthly-fetch.sh   # test run

The test run should end with no new data or pushed <sha>. If it ends with a 403 from GitHub, gh auth token -u hulf-observatory is not returning a token: log in again with gh auth login.

Known data quirks

Dec 2022 is an empty file on the portal (header only), so the series has a gap.
Division renumbering, Mar 2026: divisions 1–34 are replaced by 71–94 over Mar–Sep 2026. Both schemes overlap for some months and section names carry (OLD) / (NEW) suffixes. Build a crosswalk before comparing sections across the change.
delivered equals noofbookings almost everywhere before 2026; treat the pre-2026 column as not independently measured.
Stray division codes appear in single months (31 Apr 2025, 6455 May 2025).

What exists where (for cleanup or handover)

Everything that exists for this mirror as of 19 Oct 2026, and what was tried and removed.

Thing	Where	Owner / account	Status
Repo `hulf-observatory/hmwssb-data`	github.com	GitHub account hulf-observatory	Public. GitHub Pages enabled from `main` / root (`https://hulf-observatory.github.io/hmwssb-data/`). No custom domain yet.
Workflow `stitch.yml`	`.github/workflows/`	runs with the repo’s own `GITHUB_TOKEN`	Active. Nothing to revoke.
launchd job `in.urbanobservatory.hmwssb-fetch`	Teja’s Mac, `~/Library/LaunchAgents/`	Teja’s macOS user	Active. Uses gh’s stored login for hulf-observatory; no separate token.
Fine-grained PAT “hmwssb-fetch worker”	github.com → Settings → Developer settings → Personal access tokens → Fine-grained	hulf-observatory	Revoked 19 Oct 2026. Was Contents read/write on this repo only, no expiry; its only copy was the deleted Worker’s secret.
Cloudflare Worker `hmwssb-fetch`	Cloudflare account hulf.observatory@gmail.com (id `d962c6372a92cc08b218d11c37440175`, workers.dev subdomain `hulf-observatory`)	—	Deleted 19 Oct 2026 with its `GITHUB_TOKEN` secret, after the reachability test below. The account still runs `hyd-tiles`; its API token stays at `~/.config/huo/cloudflare-token`.
Workflows `fetch.yml`, `diag.yml`	this repo’s history	—	Removed 19 Oct 2026 (Actions cannot reach the portal).

Reachability, tested 19 Oct 2026: data.telangana.gov.in (and data.gov.in) refuse TCP connections from GitHub Actions runners (US) and return 522 / timeouts to Cloudflare Workers, including from the colo nearest Hyderabad. They answer from an Indian residential connection in ~0.3 s. Any automation for Indian government open-data portals has to run from inside India.

Licence

Source: HMWSSB via Open Data Telangana, Municipal Administration & Urban Development department, Government of Telangana, under the Open Government Data Licence – India. Files under raw/ are unmodified copies; the stitched table and manifest are derived from them. The fetch script is MIT.