Version 2.1.0-alpha is ready for testing!

Done per PM.

but they will still be visible in the library with a thumbnail or not at all?

Oh, I see, I need to check it out. Thank you!

I was running for a while (Thanks @mrm for help with the DB restore!), and PS crashed. Now trying to start it with --versbose flag seeing:

2022-05-09T11:36:38.148Z main-2626615 error Error onError() { event: 'fatal', message: "Can't restart sync, failure rate is too high.¹" }
2022-05-09T11:36:38.149Z main-2626615 error exit() exit() { reason: "Can't restart sync, failure rate is too high.¹",
  status: 12,
  ending: false }

Any way to quickly fix t?

Thanks!

If you want to send DM/email me the prior log lines, that’d be great.

Glob support is in the release I hope to get out tomorrow, BTW, not the current alpha. 1 release.

Emailed.

upgradeInsecureRequests=false

Where to set this up


It’s not working after the update to the alpha version.
Using the node variant

You need to place this into the settings.toml file of your library.

Just as a random mention, the ‘what’s new’ link points to version 1.0

That’s were I placed them. But doesn’t help and I get no connection to server after some time. Nothing interesting if I logcat

2 Likes

Welcome to PhotoStructure, @mexusbg!

The site.webmanifest is served as a static file: you should see it in your photostructure-for-servers/public directory.

It’s odd that the webservice is returning a 401: PhotoStructure’s static file server router returns 404s for unknown routes, so I believe that status is coming from your reverse proxy. What do you have in front of PhotoStructure?

1 Like

I guess I’m blind, but where are the sync reports stored?

They live in .photostructure/sync-reports in your PhotoStructure library.

I will add a proper link and web view for that directory hierarchy to the UI after user auth is done.

BMP support can work with PhotoStructure for Node, but only via GraphicsMagick and recompiling libVips. I’d suggest converting them (with GraphicsMagic) to PNG: that would be a lossless conversion, and should safe disk space. Native support would require pulling in a new library.

Can you email one of these .wmv files to support@photostructure.com so I can take a look? Those should be supported, but as WMV is a container format, there are a ton of different codec combinations, and ffmpeg might not be happy with whatever you’ve got (given the settings I currently feed it).

I don’t know if this is related to the new version or not, but I report this in case anyone else can correlate.

I was investigating a problem where the sync seemed to die, so I was logtailing inside the container. I was seeing lots of permissions errors with the app trying to delete items from both the imgcache and database copy in /ps/tmp.

I checked the permissions on the real filesystem and a bunch of directories were owned by root, instead of the container owner. I’ve been running the previous version for a while with no issue and the locations are the same (same docker-compose.yml with the UID set). I’ve changed the ownership of the whole mount for /ps/tmp back to my container user again and things seem happier.

There shouldn’t be any PhotoStructure process that doesn’t respect the provided UID/GID environment variables: the entry point ensures that:

So… If there are any files not owned by your configured UID, something else is amiss.

Ok, that’s interesting. I hear you, but it’s happened again. Nothing else should be interested in this directory structure, so all I can think of is either an unexpected behaviour or something in the way I’m launching, which is just docker-compose up -d.

Any suggestions for where to look or process owners to check?

Here’s a sample:

drwxr-xr-x  3 containeruser containeruser   16 May 15 01:53 2r/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:52 2v/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:53 2w/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 2z/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 35/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:52 36/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 37/
drwxr-xr-x  3 root       root         16 May 14 01:22 38/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 3c/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:50 3d/
drwxr-xr-x  3 root       root         16 May 14 01:21 3j/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 3q/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 41/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 46/
drwxr-xr-x  3 root       root         16 May 14 01:22 4d/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 4f/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 4x/
drwxr-xr-x  3 root       root         16 May 14 01:20 53/
drwxr-xr-x  3 root       root         16 May 14 01:22 55/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:50 5k/
drwxr-xr-x  3 root       root         16 May 14 01:20 5r/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:51 5x/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 62/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 68/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 6b/
drwxr-xr-x  4 root       root         26 May 14 01:22 6n/
drwxr-xr-x  3 root       root         16 May 14 01:22 6r/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:52 6y/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:51 7b/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 7c/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 7e/
drwxr-xr-x  3 root       root         16 May 14 01:22 7z/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:53 83/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 8k/
drwxr-xr-x  3 root       root         16 May 14 01:23 8n/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 8r/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 8u/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:50 8y/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 95/
drwxr-xr-x  3 root       root         16 May 14 01:23 96/
drwxr-xr-x  3 root       root         16 May 14 01:19 9d/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:53 9f/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:51 9m/
drwxr-xr-x  3 root       root         16 May 14 01:19 9n/
drwxr-xr-x  3 root       root         16 May 14 01:21 b0/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:53 b2/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:53 b6/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:52 b8/
drwxr-xr-x  3 root       root         16 May 14 01:20 bb/
drwxr-xr-x  3 root       root         16 May 14 01:20 bc/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:51 bm/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:51 c9/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 ce/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:51 cu/
drwxr-xr-x  3 root       root         16 May 14 01:22 d0/
drwxr-xr-x  3 root       root         16 May 14 01:19 de/
drwxr-xr-x  3 root       root         16 May 14 01:19 dj/
drwxr-xr-x  4 containeruser containeruser   26 May 15 01:53 du/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:52 dx/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 e3/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 e6/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:52 e9/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:52 eh/
drwxr-xr-x  3 root       root         16 May 14 01:22 em/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 ez/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:54 f1/
drwxr-xr-x  3 root       root         16 May 14 01:23 f2/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:50 fc/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:50 fd/
drwxr-xr-x  4 containeruser containeruser   26 May 15 01:54 fg/
drwxr-xr-x  3 root       root         16 May 14 01:23 fk/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:52 fq/
drwxr-xr-x  3 root       root         16 May 14 01:19 fw/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:51 fx/
drwxr-xr-x  3 root       root         16 May 14 01:22 fz/
drwxr-xr-x  3 root       root         16 May 14 01:22 g7/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:53 g9/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:51 gp/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:50 gw/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 h4/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:53 jn/
drwxr-xr-x  3 containeruser containeruser   16 May 15 01:55 kb/
containeruser@host:/media/bulkstorage/photostructure/cache/imgcache$

some relevant processes:

$ ps -ef|grep photostructure
root        1993    1663  0 May14 ?        00:00:00 su --preserve-environment node --command /usr/local/bin/node /ps/app/photostructure
containeruser    2044    1993  0 May14 ?        00:00:00 /usr/local/bin/node /ps/app/photostructure
containeruser    2119    2044  0 May14 ?        00:00:28 photostructure main
containeruser    4820    2119 82 May14 ?        01:53:34 photostructure sync
containeruser   34960    2119  0 00:40 ?        00:00:23 photostructure web
containeruser   72190    4820 15 01:54 ?        00:00:34 photostructure worker
containeruser   72865    4820 11 01:54 ?        00:00:19 photostructure worker

and

ps -ef|grep ps
root        1663    1565  0 May14 ?        00:00:00 /sbin/tini -- /ps/app/docker-entrypoint.sh
root        1993    1663  0 May14 ?        00:00:00 su --preserve-environment node --command /usr/local/bin/node /ps/app/photostructure
containeruser   2044    1993  0 May14 ?        00:00:00 /usr/local/bin/node /ps/app/photostructure
containeruser   74760   72865  7 01:58 ?        00:00:02 /usr/bin/perl -w /ps/app/node_modules/exiftool-vendored.pl/bin/exiftool -stay_open True -@ -
containeruser   74762    4820  6 01:58 ?        00:00:01 /usr/bin/perl -w /ps/app/node_modules/exiftool-vendored.pl/bin/exiftool -stay_open True -@ -

docker-compose.yml extract:

      # The userid to run PhotoStructure as:
      - "PUID=1000" # < CHANGE THIS LINE or delete this line to run as root. See below for details.

      # The groupid to run PhotoStructure as:
      - "PGID=1000" # < CHANGE THIS LINE or delete this line to run as root.

uid confirmation:

$ cat /etc/passwd | grep 1000
containeruser:x:1000:1000:Name:/home/containeruser:/bin/bash

I had a look a the the file you referenced above and I see the handling for /ps/tmp is different to the rest and you don’t chown the structure in there:

  if [ -z "$PS_NO_PUID_CHOWN" ]; then
    # Always make sure the settings, opened-by, and models directories are
    # read/writable by node:
    for dir in /ps/library/.photostructure/settings.toml \
      /ps/library/.photostructure/opened-by \
      /ps/library/.photostructure/models \
      /ps/config \
      /ps/logs \
      /ps/default; do
      maybe_chown "$dir"
    done

    # Special handling so we don't do something terrible if someone bind-mounts /tmp to /ps/tmp
    if [ -d /ps/tmp ]; then
      mkdir -p "/ps/tmp/.cache-$UID"
      maybe_chown "/ps/tmp/.cache-$UID"
    fi
  fi

It seems that all the files/directories owned by root are from yesterday: do you have more than one way to launch the docker container?

That’s what I’d expect: tini and su run as root, and everything else should be the UID. All the actual work is done by node and child processes.

Ah, nice sleuthing! I only added those chown commands so when you change PUID/UID (say, from root 0 to the default 1000 user id, that the new user would still have access to the files it needed to spin up the library).

So… didn’t get to play with 2.1.0 alpha a lot, but after an initial upgrade noticed that a lot of my pictures aren’t there… tried to do a restart sync, and then a full rebuild and it just ignored me…

Things were a bit “messy” on my install, and it’s gone through a bunch of testing updates and things, so just started fresh. Blew it all away (config and all) and redownloaded the docker (I’m on unRaid).

Configured as before…

I have 4 directories it’s supposed to scan. Three of them it pulls in thousands of pictures (I haven’t checked closely, but looks like most all) - but those are the three “small” directories. Then there is the main archive. It started on it, chewed just a few minutes, pulled in just a 1000 or so and stopped.

Log shows (for example):

1652886758215,/pictures/Archive/2009/2009-01_January/2009-01-04–2009-01-04_Disney – travel/2009-01-01T18.03.08_Disney-IMG_3849.jpg,enqueued,
1652886758215,/pictures/Archive/2009/2009-01_January/2009-01-04–2009-01-04_Disney – travel/2009-01-01T18.03.24_Disney-IMG_3850.jpg,enqueued,
1652886758226,/pictures/Archive/2009/2009-01_January,canceled,DirectoryWalker was ended,1131
1652886758226,/pictures/Archive/2009,canceled,DirectoryWalker was ended,1132
1652886758226,/pictures/Archive,canceled,DirectoryWalker was ended,111489
1652886760151,/pictures/Archive/1989/1989-10_October/19891007-Dillahunty_Wedding/020-29-Scan-101007-0001.jpg,failed,“BatchCluster has ended, cannot enqueue {”“id”":438,"“method”":"“buildAssetPreviews_”","“args”":[{"“assetId”":199,"“assetFiles”":[{""$ctor"":"“models.AssetFile”","“id”":219,"“assetId”":199,"“shown”":false,"“uri”":"“psfile://37Ajd4ybC/Archive/1989/1989-10_October/19891007-Dillahunty_Wedding/02.:false,”“recountAllTags”":false}]}",http://127.0.0.1:1787/asset/199,2643
~ ~

Where should I look from here? Permissions look right - for example, it pulled 4 pictures from one folder that has about 20. All set the same permissions.

I’m still seeing two problems. Scans don’t seem to populate new images into the database, even with small manually run single date folder scans.

Secondly, I’m still getting weird file ownership problems that I didn’t get with the previous version. This time, I carefully shut everything down, checked file ownership, started again with a stock standard sudo docker-compose up -d

This is the result (the top level of this is the bind mount point for /ps/tmp:

$ sudo docker logs photostructure
Please wait, setting up...
Your library is at <file:///ps/library>
PhotoStructure is ready:
  - <http://127.0.0.1:1787/>
  - <http://172.18.0.2:1787/>
  - <http://f4b1f198ce13:1787/>

Shutting down PhotoStructure...
Please wait, setting up...
Please wait, setting up...
Please wait, setting up...
Please wait, setting up...
Please wait, setting up...
Please wait, setting up...



$ ll
total 0
drwxr-xr-x 5 containeruser containeruser  48 Mar 17 09:38 ./
drwxr-xr-x 3 root       root        28 Mar 16 02:51 ../
drwxrwxr-x 2 containeruser containeruser  19 Mar 16 02:54 backup/
drwxrwxr-x 6 containeruser containeruser 135 May 17 08:47 cache/
drwxrwxr-x 4 containeruser containeruser  70 Mar 17 09:54 samples/


/cache$ ll
total 8
drwxrwxr-x  6 containeruser containeruser 135 May 17 08:47 ./
drwxr-xr-x  5 containeruser containeruser  48 Mar 17 09:38 ../
drwxr-xr-x  2 containeruser containeruser   6 May 12 09:20 .cache-1000/
drwxr-xr-x 15 containeruser containeruser 136 May 17 09:21 imgcache/
drwxr-xr-x  3 containeruser containeruser  20 May 18 21:21 local-db-0885-rqph-1u41-rutf/
-rw-r--r--  1 containeruser containeruser 126 Mar 16 02:57 .NoMedia
drwxr-xr-x  3 containeruser containeruser  37 May 12 09:21 sync-state-kt27ebt2j7/
-rw-r--r--  1 containeruser containeruser  37 Mar 16 03:01 .uuid


/cache/local-db-0885-rqph-1u41-rutf$ ll
total 0
drwxr-xr-x 3 containeruser containeruser  20 May 18 21:21 ./
drwxrwxr-x 6 containeruser containeruser 135 May 17 08:47 ../
drwxr-xr-x 4 root       root        64 May 18 03:57 models/


/cache/local-db-0885-rqph-1u41-rutf/models$ ll
total 225200
drwxr-xr-x 4 root       root              64 May 18 03:57 ./
drwxr-xr-x 3 containeruser containeruser        20 May 18 21:21 ../
drwxr-xr-x 2 root       root               6 May 18 03:57 backup/
-rw-r--r-- 1 root       root       230604800 May 18 03:57 db.sqlite3
drwxr-xr-x 2 root       root               6 May 18 03:57 .db.sqlite3.pslock/

/cache/imgcache$ ll
total 0
drwxr-xr-x 15 containeruser containeruser 136 May 17 09:21 ./
drwxrwxr-x  6 containeruser containeruser 135 May 17 08:47 ../
drwxr-xr-x  3 root       root        16 May 17 09:18 2z/
drwxr-xr-x  3 root       root        16 May 17 09:20 30/
drwxr-xr-x  3 root       root        16 May 17 09:19 38/
drwxr-xr-x  3 root       root        16 May 17 09:18 4n/
drwxr-xr-x  3 root       root        16 May 17 09:16 5d/
drwxr-xr-x  3 root       root        16 May 17 09:19 75/
drwxr-xr-x  3 root       root        16 May 17 09:20 7x/
drwxr-xr-x  3 root       root        16 May 17 09:18 82/
drwxr-xr-x  3 root       root        16 May 17 09:20 92/
drwxr-xr-x  3 root       root        16 May 17 09:19 9t/
drwxr-xr-x  3 root       root        16 May 17 09:18 cq/
drwxr-xr-x  3 root       root        16 May 17 09:18 g9/
drwxr-xr-x  3 root       root        16 May 17 09:18 gf/

If the DirectoryWalker ended, that means the error rate was too high and the “yikes, things are bad, I better not continue” circuit breaker blew.

If you set your logging to info and send me your logs I can take a look at what’s going on.