There have been a ton of new features and improvements since the last beta build!
Version format change
I’m adopting a simpler version format: $year.$month.$build, where $build starts at zero at the beginning of the month, and gets incremented for every prealpha, alpha, beta, or stable release. For non-stable releases, -$channel is appended to the version format.
This release, for example, is a re-release of the v2024.3.2-prealpha build. Discord users (almost) universally agreed that this build should supersede the prior (quite old) beta release.
People using prealpha and alpha builds: please consider changing to the beta release channel if you’d like a bit more stability.
New geo location tagger
PhotoStructure now adds Where/Country/Region/City tags for those photos and videos with Latitude and Longitude metadata.
Note that this feature is uses an embedded geo database, so no network access is required. This initial implementation only includes cities with a population of 1000 or greater. See the new tagGeo and tagGeoTemplate settings for more details. Edit: as with any prealpha, alpha, or beta release, you need to hop into that branch on github to see new settings: photostructure-for-servers/defaults.env at beta · photostructure/photostructure-for-servers · GitHub
Sync improvements
Previous builds of PhotoStructure had two work queues: one single-threaded work queue for videos, and one multithreaded work queue for images. This was a hack workaround to prevent concurrent ffmpeg invocations as ffmpeg attempts to use all cores by default, resulting in CPU overscheduling.
We’ve since found a fairly reliable way to single-thread ffmpeg, so sync now schedules both video and image work in a single queue, which greatly simplifies the code, and results in higher parallelism (!!). Anecdotally, prior builds would sync several hundred exemplar videos and photos in roughly 3 minutes. This build now completes that same task in under 90 seconds on the same hardware.
PhotoStructure’s task queuing system was also rewritten. Previous builds used a completely separate SQLite schema and database for work scheduling, in an attempt to keep that workload partitioned from the web service. With the new taskListCap setting, task schedulers receive backpressure if the Task table is “full.” This backpressure ensures the table doesn’t grow unbounded, so it felt safe to migrate it into the models database and delete the work queue database. This also allows web to schedule work for sync reliably without socket RPC or JSON watchfile overhead (again, allowing another good chunk of code to be deleted).
New stuck-task watchdog
For larger libraries with tens of thousands of ffmpeg transcodes, a sync could get “stuck” waiting for an ffmpeg transcode completion that exited abnormally. v2 builds had a hard timeout value based on video duration, but that proved problematic for slower computers and for more advanced codecs that require more computation to decode, so video transcode timeouts were dropped in v2023. More advanced video transcode timeouts were built that adjusted dynamically based on current system performance and processed pixel count, but this implementation was difficult to test rigorously, and the least-squares interpolation implementation was replaced with a new stuck-task watchdog.
When users reported their sync process was “stuck,” they’d always report that their system’s CPU was idle but that things weren’t done.
So, instead of fancy-pants pixels-processed-per-mimetype least-squares timeout interpolation complexity, why can’t PhotoStructure just do what the users are doing in these situations?
So now it does!
While sync is currently processing, every five minutes it will check if the system load is “idle” (by default, less than 50% of one busy core, but this is adjustable). If it is, any task that has run longer than the last check will be assumed to be stuck, and will be marked as failed. See the new stuckCheckIntervalMs and minBusyPct settings for more details.
To make this work, Tasks are now be abort-able externally, and know how to clean up gracefully, including killing child processes and notifying sync-report.
And several hundred (!!) bugfixes
See the release notes for more details!
