Roll out watcher releases

Publish watcher versions to PyPI, advertise them per environment, set a minimum version, force mandatory updates, roll the fleet back, and see what each version adds.

ForAdministrators

Lab PCs upgrade themselves toward the version you advertise per environment. You control that from Settings → Watchers. How an individual PC installs or runs self-update is in Install and update the watcher.

How fleet upgrades reach a PC

The in-process updater runs from the heartbeat tick on every platform (not only Windows services). Roughly once an hour the watcher:

Asks the server for the latest version and compares it to its own
Upgrades only if a newer version is available, no files have uploaded for several heartbeats, and no run was reported recently (the activity-window guard). Releases flagged mandatory skip this guard
Installs the new wheel and restarts into it (Windows services restart via failure-actions policy)
Emits update_started, then update_succeeded or update_failed, visible on the dashboard

Auto-update is disabled in the preview environment so pull-request preview deployments cannot push code to lab PCs.

Cut a new release

Releases are tag-driven and publish from a production commit.

Bump the version in watcher/pyproject.toml and merge through staging to production.

Tag and push from production:

git checkout production && git pull
git tag watcher-v1.3.0 && git push origin watcher-v1.3.0

Approve the pypi deployment under Actions → Publish watcher in GitHub.

In Settings → Watchers, pick the new version under Latest version. The list comes straight from PyPI, so a release shows up as soon as PyPI publishes it. If PyPI can’t be reached, the field accepts a typed version instead. Each environment has its own database, so set staging and production separately to roll the fleet gradually.

datahub.example.com/settings/watchers Settings Watchers page with Latest version and release controls

Always tag → publish → verify on PyPI → save. Saving Latest version before the wheel is live triggers a wave of update_failed events across the fleet.

Release settings

Field	Purpose
Latest version	The version watchers upgrade to. Leave it blank to pause self-updates.
Minimum supported version	Optional. A watcher older than this has its heartbeats rejected with `426 Upgrade Required`, so it shows as unresponsive until it updates. It still checks for updates every hour, so it upgrades itself as long as Latest version is newer. Leave it blank for no minimum.
Mandatory update	Skips the activity-window guard; see below. Has no effect while Latest version is blank.

Mandatory updates

Toggling Mandatory update skips the activity-window guard and fires the upgrade on the next hourly check. The server still compares the running version to Latest version, so a correctly pinned PC is not forced past its pin.

Use sparingly: a forced upgrade mid-acquisition can lose data. Reserve it for security fixes, wire-protocol breaks, or cases where leaving the bad version running is worse than restarting in flight.

Roll back

Rollback is another release: set Latest version back to the older tag for the affected environment(s), turn Mandatory update on if you need to bypass the activity-window guard, and wait for the next hourly tick. Turn Mandatory update off once the fleet converges. The older wheel remains on PyPI; it is no longer advertised.

If Minimum supported version is set above the version you roll back to, lower it first. Otherwise the rolled-back watchers can’t check in.

What each version adds

Some Data Hub features need a minimum watcher version. Set Latest version to at least the version below to turn a feature on across the fleet.

Version	What it adds
1.4.2	Once the watcher is running, a stop finishes within about 25 seconds, even when Data Hub isn’t answering. The watcher sends its events before its final heartbeat, and starts no new part or file once a stop is requested. A watcher that stops to install an automatic update tells Data Hub why, so files someone asked for with Upload stay requested while it comes back, and the Windows service hands an upgrade to its worker only once. The Windows service also records a config or env file it can’t load in `watcher.log`, and honors `DATA_HUB_WATCHER_LOG_LEVEL` from its env file. A file the instrument keeps open after its last write is uploaded once the instrument lets go of it: the watcher tries it again every minute until it can be read, and a request for it from the run page stays requested. Older watchers try such a file again only at the next start, and cancel a request for it after three tries.
1.4.1	Restarting a Windows instrument PC stops the watcher service cleanly. The watcher sends a final heartbeat, so it shows as Stopped, and files someone asked for with Upload stay requested and upload once the PC is back. Data Hub cancels those requests if the watcher isn’t back within about 15 minutes; see When the PC restarts or shuts down. Older watchers are ended by a restart without recording a stop. In auto mode, the files of a run reported just before a restart, a power cut or a crash upload after the next start; older watchers can leave them showing Pending, and they can be uploaded from the run page. A run whose ID contains `#`, `?` or another character with a special meaning in a web address gets its own files; older watchers send them to another run or don't upload them.
1.4.0	Admins can change the watcher’s config from Data Hub. The watcher picks up a saved change within about a minute and applies it, or rejects it and keeps its current settings when a check fails, such as a folder that doesn’t exist. After a change that matches more files, the watcher reports files already on the PC as detected but doesn’t upload them. A watcher whose folder is missing at startup keeps running and shows Waiting: folder not found, so the folder can be fixed from Data Hub. Older watchers can only be changed on the instrument PC, and exit at startup when their folder is missing.
1.3.0	A large file uploads several parts at once, and several files requested from the run page upload side by side. Together they stay within the `instrument.upload_parallelism` setting, 4 uploads by default. Older watchers send one part, or one requested file, at a time, so a large file goes no faster than a single connection allows. The watcher asks again for a part’s upload link after a connection error or a server error, instead of failing the file. It still fails the file when Data Hub rejects the request (a 4xx response). The `file_uploaded` event also records the file’s size and how long preparing and sending it took.
1.2.2	A run report that fails because the watcher can’t reach Data Hub, or because Data Hub returns a server error, is retried until it goes through. Older watchers leave the run unreported until another file in the run finishes or the watcher restarts.
1.2.1	Request upload works on instruments that upload automatically; older watchers in auto mode leave the file showing Uploading. The watcher also retries an automatic upload that didn’t finish, such as one cut off by a restart or an update, the next time it starts. It tries on up to three starts.
1.2.0	Files of 100 MB or more upload in 64 MB parts, and a failed part is retried on its own. Files larger than 5 GB can upload; older watchers are refused for files that large.
1.1.0	When a run gathers files from several folders, files with the same name are all kept instead of only the first. Later copies get a short tag in the name, such as `sample~3f9a1c2b.tif`. Adds the `data-hub-watcher state forget` command.

After upgrading to 1.1.0 or later, files that an older watcher skipped because of a duplicate name aren’t uploaded on their own. To upload them, stop the watcher, run data-hub-watcher state forget --prefix your_folder_name/ on the instrument PC, and start the watcher again. See Troubleshoot a watcher.

Operator-side failures

If update_started is followed by update_failed, or auto-update never fires, point operators at Troubleshoot a watcher and Install and update the watcher.