New "easy mode" for Docker coming in v2.1

[mrm]

Note: the following post is for the as-yet-unreleased next version of PhotoStructure, v2.1

As I’ve been testing the new v2.1 alpha build on docker, every time I spin up a new container, I think “gosh, can’t this be made simpler?” I think it can.

What we’ve got up to now

PhotoStructure’s docker setup currently requires at least 4 bind mounts:

1. The library directory, /ps/library
2. The scratch directory, /ps/tmp
3. The system configuration directory, /ps/config
4. The log directory, /ps/logs

There are a couple constraints on these directories:

1. The library directory needs to have a bunch of free disk space
2. The scratch disk must be on a local disk, and cannot be a disk using a “union” or mergefs filesystem.

More details about these volumes are in the docker setup page.

What we’ll have with v2.1

PhotoStructure v2.1 will require a /ps/library bind-mounted volume. Everything else is optional.

If you’ve got a big local disk that you want everything to live in, just bind-mount that one directory, and you’re done. Like this:

docker run \
  --stop-timeout 120 \
  -v "$HOME"/PhotoStructure:/ps/library \
  photostructure/server:stable

Implementation details

Here’s how v2.1 figures out directories when running in Docker; the first directory that exists, wins.

For your library directory, PhotoStructure still defaults to /ps/library (but can be changed from the default by setting PS_LIBRARY_DIR).

For your scratch directory, if /ps/tmp is a read/writable directory, return that. Otherwise, if /ps/cache is a read/writable directory, return that. Otherwise, return /ps/library/.photostructure/cache-$UID.

For your system settings: if /ps/config is a read/writable directory, return that. Otherwise, return /ps/library/.photostructure/docker-config.

For your log files: if /ps/logs is a read/writable directory, return that. Otherwise, return /ps/library/.photostructure/logs.

PhotoStructure still needs a local disk!

PhotoStructure still needs to have a cache dir that supports memory mapping, so if you want to store your library on mergefs or a union filesystem, you can bind-mount that volume to /ps/library, but you also need to provide a standard, local (fast!) filesystem bind-mounted to /ps/tmp.

Read more about why on this forum post.

Does this work for you?

So, docker users: Does this make sense? Will you use this?

Feedback is welcome: it’s easy to change stuff now, before I have to support backward compatibility…

Edits

20220129: prior version referenced the default directory as /ps/default, but to keep things simple, I’ve renamed it /ps/library. The thinking is that you have to have a library directory, but everything else is an implementation detail.

20220130: I’ve shoved the scratch, logs, and config directory to live under /ps/library/.photostructure to minimize the number of directories I’m writing into.

[avdp]

Yes, this makes sense. I’d probably use 2 mounts. library and then everything else in default. It’s essentially how I have it setup now but with the hassles of having to explicitly setup all these extra extra mounts.

[tkohhh]

I think this is a good change, and I’ll implement it. As for my recommended Unraid setup, I will modify it to use the default directory (stored on the appdata share), but still advise that an Originals Directory (stored on the array) and a source location be mapped as well.

Well done!!

[mrm]

Thanks for the feedback!

I was explaining this change to someone today and realized there wasn’t really a good reason to introduce a new /ps/default directory: /ps/library is always required, so that can just be the “default” to put everything else into. I just changed the top post (and the code) to reflect this change.

This change does means the “everything else goes into this directory” isn’t handled in only one line, but I suspect people will either use “full easy mode” (here’s a local directory, please just get to work), or “full explicit mode” (for users that want to lay everything out manually and don’t want extraneous directories in their library directory).

Am I missing anything?

[tkohhh]

This makes sense, definitely an improvement on the concept.

There is one thing that I would love to see with this…

Currently, if you want a “hybrid setup” where you split your full-size originals into a different folder from your “library” folder, you have to implement two settings: a mapped path for the storage location, and an environment variable (PS_ORIGINALS_DIR).

With this new concept, could you add a fifth mappable directory for the original files? In other words, maybe the mappable directory is /ps/originals… if that exists, then use that, otherwise, use /ps/library. This would eliminate the need for the PS_ORIGINALS_DIR environment variable without losing any of the configurability.

Just a thought!

[mrm]

@tkohhh

In other words, maybe the mappable directory is /ps/originals… if that exists, then use that, otherwise, use /ps/library.

  originalsDir: new StringSetting({
    category: SettingCategories.Paths,
    description: `This is the directory that PhotoStructure uses to store original images when "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative paths are evaluated from your libraryDir. This setting defaults to ".", which is the same as your PhotoStructure library directory, unless you are on docker, and a /ps/originals directory exists.\nIf you open your PhotoStructure library on a different computer, and that computer doesn't have access to your originals volume, full-screen zoom won't work, and non-transcoded videos will not play.\nThis system setting needs to be set appropriately on different computers (it won't be set automatically!)\nIf you have a large library and want to use an SSD, we recommend you set your libraryDir to your SSD, and use this setting to store your originals on a larger volume, rather than using the "previewsDir" setting.\nSee <https://forum.photostructure.com/t/hybrid-photostructure-libraries/775> and <https://forum.photostructure.com/t/new-easy-mode-for-docker-coming-in-v2-1/1278/5>.`,
    defaultValue: () => isDocker() && isReadWriteableDirectorySync("/ps/originals")
    ? "/ps/originals"
    : "."
  })

[tkohhh]

Beautiful! Out of curiosity: had you already implemented this, or did you do so at my suggestion? Either way, I’m glad it’s there!

[mrm]

I just added it this morning after your suggestion: thanks again!

[Red2StandingBy]

This would be perfect for my use case, at least!

[ericb]

So, about that v2.1… how’s it been going?

[mrm]

@ericb

v2.1… how’s it been going?

Apologies for the delay! I’m chasing down (one last?) threading error and I can release the alpha. Best case would be this afternoon.

[ericb]

Excellent. Can’t wait until the “last” threading error is fixed and I can try out the alpha and immediately hit a non-reproducible deadlock on startup on my own machine. Software!

[mrm]

LOL. I’ve recently added a bunch of tests that make child process simulate a litany of failure cases, like memory errors, volume parsing failures, broken pipes, … All to verify that recovery code behaves as expected on all platforms. It’s been a challenge!

[bdillahu]

You just need an @ericb test

[ericb]

The initial @ericb test might involve a quick run with fiu-run, randomly failing some low percent of reads or writes.

[Sunghei]

Just join the photostructure fun ! So is the 2.1 out? Running unraid. Also will there be a paid once pricing model ?

[mrm]

@Sunghei

Just join the photostructure fun !

Welcome!

@Sunghei

So is the 2.1 out?

Not yet, but it’s

@Sunghei

will there be a paid once pricing model ?

Not currently. I wrote about “perpetual” pricing here.

[Sunghei]

Yay can’t wait !

[adamf]

Agreed, I’m waiting to resume using the docker version and cleaning up my bind mounts, etc. with a fresh 2.1 database. Very excited!

[mrm]

Progress is happening: https://discord.com/channels/818905168107012097/818907922767544340/968950739407880314