Version v2.1.0-alpha.7 is ready for testing

[mrm]

Alpha? What’s that?

Please note: alpha builds should be considered extremely experimental, and may not even launch on all platforms.

Read more about alpha and beta builds are here

Release notes

• There were some crash-bugs in alpha.4, alpha.5, and alpha.6.

  PhotoStructure for Desktops on macOS is still not working with this build, but all other editions of PhotoStructure seem to be functional. Please report here or on Discord if you see anything odd, unexpected, or possibly buggy.

• PhotoStructure for Docker users: If your docker or docker-compose scripts used $UID, please switch to using $PUID. If you used $GID, please switch to $PGID.

  Prior releases tried to be “nice” and support both $UID and $PUID (as well as both $GID and $PGID), but this turned out to be a bad idea. bash and other commands consider $UID and $GID to be reserved, read-only, and trustable environment variables, which could cause issues. We’ll just stick with the linuxserver.io standard. Details: https://photostructure.com/go/pgid

• Support for remote TCP GELF-compatible logging servers via the new PS_LOG_SERVER and PS_LOG_SERVER_LEVEL settings.

• / “Friendly” duration strings are now supported (after I typoed the fourth ISO duration string). See https://photostructure.com/getting-started/advanced-settings#duration for details.

• / Prior versions of PhotoStructure compiled the front-end javascript against an ES5 target, which caused older, unsupported iOS devices to not render the frontend. When we heard that Nighthawk’s Grandma’s iPad didn’t work, though, this had to be fixed. We know build against ES3, and should support ancient versions of Safari.

• // The prior build (alpha.2 and alpha.3) introduced globs, but having both scanPaths and globs resulted in confusion (and several bugs). File exclusion patterns were completely revisited in this build. Implementation details and usage are explained in this forum post.

• / Sidecar handling was improved: photo.JPEG now matches up with photo.JPG.xmp.

• When PhotoStructure copies files on macOS and Windows, it now retains file “birthtime” metadata. This isn’t a field that exists on standard Linux filesystems, so it’s not supported there. Set retainFileBirthtimes=false to disable this new behavior.

• Lazy loading is now configurable, via the new lazyLoadExtraVh setting. Use a smaller value if you’re serving your library over a constrained network.

• / Several sync report improvements:

  • The sync report directory can be opened via the nav menu (if accessed via localhost), or on PhotoStructure for Desktops, via the tray and system menus.

  • The README.txt now includes a comprehensive list of “states” for files and directories.

  • Added a new “at” column that’s ISO-date-time formatted, because most spreadsheet apps don’t know how to parse millis-from-common-epoch.

  • Sync reports no longer worryingly state that all sidecars were skipped–the sync report now states what file(s) the sidecar will be associated with, and only marked as excluded if they don’t match with any sibling photo or video file.

  • Sync reports now include a “started” state emitted after being dequeued from the work queue.

  • Prior sync report CSVs could contain a “details” cell that included newlines. Although Excel and LibreOffice parse these CSVs properly, Google Sheets don’t, and there was discussion asking if these newlines could be avoided. Good news, everyone, /\r?/n/g is replaced with ": " in the details column now!

  • If automatic organization is enabled (see the copyAssetsToLibrary setting), a new sync report row will be added when photos and videos are copied into your library.

• / When sync finishes for a given path, and retryEnqueued is true, sync will look at the last day of sync reports for paths that are “stuck”–paths that have a “enqueued” entry, but no subsequent “synced”, “timeout”, or “failed” entry, and retry them.

• A bug in URI root encoding caused alpha.2 through alpha.4 to have several sync reporting and progress panel-related errors, which should now be resolved.

• Depending on how PhotoStructure was shut down, the sync process could have been force-killed while still closing the database, which would result in SQLITE_CORRUPT. A new syncExitTimeoutMs setting has been added, which defaults to 1 minute–this should be enough to close SQLite even on the slowest remote HDDs and largest libraries, but now you can extend this if you must.`

• Work-in-progress files, (hidden files starting with .WIP-), used by metadata extraction and transcoding ops, now use a filesystem mutex to avoid race conditions (this caused random import failures on high-CPU-count servers).

• Filesystem watches for the same path are now shared within a given process (Node.js quietly fails when watch is invoked more than 3 times for the same path. Again, this would only impact users with high-CPU-count servers).

• Videos and images are no longer considered for aggregation (to avoid spurious live photo matches). (PhotoStructure will revert this when Live Photos are properly aggregated).

• PhotoStructure for Desktops has a “pre-flight check” of the library directory at startup. Prior versions could fall into an infinite loop if the directory permissions were wrong.

• Library directory suggestions now filter out any directories that are not read-write by the current user.

• The “ Not Connected” dialog no longer flashes epi(lepti)cally when the library server isn’t available.

• Backend state, liked current running version, isPaused, and current plan is now synchronized with the front-end after every XHR request.

• Fixed ./photostructure main --tail (prior versions would erroneously report the arg as being invalid).

• PhotoStructure for Desktop billing links now open in the current window

• Due to an unclosed http response, the webserver would hang if previews were missing

• Fixed includedPreviewTags setting’s capturedAt and exposureSettings support for non-standard tag locations. Preview images are tagged to let Apple Preview and Eye of Gnome properly extract exposure settings.

• / Reduced db mutex contention during backups by pausing work item dequeues–this could cause tasks to “time out” when they were just waiting for the backup to complete.

• The “your library is already open” and “your library is missing” preflight check dialog buttons for PhotoStructure for Desktops didn’t work properly. We use the new electron API properly now.

• File copies could erroneously timeout under heavy I/O, causing larger file imports to fail randomly. We now use a progress watchdog instead of a hard timeout.

• / Wrapped PhotoStructure for Desktops launch block in a try/catch to ensure errors got rendered to a user-visible dialog

• PhotoStructure for Docker: If $PUID or $PGID aren’t “effective” (either the current effective user id doesn’t match $PUID, or current effective group id doesn’t match $PGID), all commands (main, sync, web, …) will now emit a warning with a link to the forum post with the solution: https://forum.photostructure.com/t/1597/2.

• Process shutdown was refactored a bit:

  • To avoid premature shutdown (and dreaded SQLITE_CORRUPT errors), we now fully rely on “endable” component timeouts, rather than having a single top-level timeout. If, say, the db takes a while to shut down, the new code will be patient and wait for it now. Shutdown may take a bit longer, but in testing I haven’t measured slower shutdowns.

  • Child processes now listen on stdout for --exit, process signals, and the new exit shared-state event to initiate shutdown.

• Volume and mountpoint parsing now uses the validateMountpoints setting to only return user-rX directories.

• Expired subscription licenses are now auto-refreshed after upgrading to a new major/minor version.

• Multiprocess state sharing is now lockless to avoid multi-process deadlocks in alpha.3.

• Advisory locks use filesystem mutexes, rather than relying on SQLite unique constraints.

• Improved info --exclude-globs output

• File SHAs are cached and invalidated only if fs.Stats size or mtimeMs change. Prior versions would invalidate previously-cached SHAs too aggressively, which could result in the entire file being re-read several times unnecessarily.

• Pulled in latest versions of Electron, sharp, node, typescript, ExifTool, and other third-party libraries.

Installation instructions

Please take a backup of your system before continuing!

PhotoStructure for Desktop users

1. Shut down PhotoStructure
2. Download and install: https://photostructure.com/go/alpha

Running the alpha installer will replace the currently installed version. PhotoStructure for Desktops will detect that it’s an alpha build, and automatically upgrade you to newer alpha or beta builds, and then “stick” to stable builds (once v2.1-stable is released).

PhotoStructure for Docker users

1. Shut down PhotoStructure
2. Run docker pull photostructure/server:alpha in a terminal
3. Change your docker run command to use photostructure/server:alpha

PhotoStructure for Docker Compose users

1. docker-compose down
2. Edit your docker-compose.yml to use

image: photostructure/server:alpha

3. docker-compose pull ; docker-compose up -detach

PhotoStructure for Node users

1. Make sure you’ve got Node.js 16.15.0 or later installed. If you’re using nvm, run nvm install 16 ; nvm alias default node to get the latest.

2. Shut down PhotoStructure

3. cd into your photostructure-for-servers directory, and run git checkout alpha ; ./start.sh

I’m seeing “Cannot find module ‘stream/promises’”

Please upgrade to Node 16 or later. Instructions are above.

[mrm]

Known bugs in this build

Show-stoppers

• macOS Desktop builds are broken in alpha.7

• The docker :beta image was updated by pushing to :alpha. Only the reverse should be true (new :beta builds should be tagged with :alpha to make sure alpha testers get the latest and greatest).

• @avdp is still tormented by SQLITE_CORRUPT

• Windows 11 considered C:\ as “hidden” and didn’t walk into the directory when using “automatic” scan paths

• Sync doesn’t seem to quick-scan and no-op previously-imported directories (reported by @harrisc)

• Decoding HEIF with no embedded previews causes Windows to hang until timeout

• Invalid color profile applied to NEFs (as reported by @ssickle)

  • upgrade LibRaw to latest HEAD
  • add setting to always use embedded jpeg if resolution matches raw
  

  update: New Apple RAW DNGs use a white balance that LibRaw doesn’t handle correctly. “out.jpg” is from the embedded preview (which has correct white balance), and out-N.tiff are rendered using all of the different available whitebalance settings–all of which are incorrect:

  

  Note also that both Darktable and Rawtherapee don’t know how to handle this image format:
  

  

  rawtherapee-cli: /home/mrm/Downloads/IMG_0002.DNG: Corrupt data near 0x552c22

  So, the solution is to really lean on the preview image, if possible. The next build will defer to embedded images even if they’re smaller than the original RAW image.

Irritants

• progress panels don’t show at the beginning of sync runs (thanks for the report, @harrisc)

• PhotoStructure shutdown seems to be taking longer than expected

• When restarting, sync doesn’t restart the prior sync’s work queue (this is painful on very large directory scans)

• esc key doesn’t work on Edge

• Time zones in videos don’t seem to be respecting local time encoded filenames (see bug report)

• asset IDs are included in rebuild progress

• Upgrade ExifTool on Windows to use recent Strawberry Perl

• Prior scroll position isn’t restored in tag gallery views (thanks for the report, @Tigger on discord!)

(got any more? post here or tell me in Discord!)

[devon]

I can report that a manual scan works now, as per:

Files in /ps/tmp owned by root

PhotoStructure’s sync tries to be “clever” and not re-do directories it thinks has already been recently synchronized. Sometimes that cleverness is too clever by half. I suspect the first command “no-op”-ed (did nothing) because there was a record of that directory having been recently sync’ed. The second command ran because sync wasn’t sure it had recently completed those child directories. You can currently disable this cleverness by adding the --force flag, but that makes a bunch of other…