Done per PM.
but they will still be visible in the library with a thumbnail or not at all?
Oh, I see, I need to check it out. Thank you!
I was running for a while (Thanks @mrm for help with the DB restore!), and PS crashed. Now trying to start it with --versbose flag seeing:
2022-05-09T11:36:38.148Z main-2626615 error Error onError() { event: 'fatal', message: "Can't restart sync, failure rate is too high.¹" }
2022-05-09T11:36:38.149Z main-2626615 error exit() exit() { reason: "Can't restart sync, failure rate is too high.¹",
status: 12,
ending: false }
Any way to quickly fix t?
Thanks!
If you want to send DM/email me the prior log lines, that’d be great.
Glob support is in the release I hope to get out tomorrow, BTW, not the current alpha. 1 release.
Emailed.
upgradeInsecureRequests=false
Where to set this up
It’s not working after the update to the alpha version.
Using the node variant
You need to place this into the settings.toml file of your library.
Just as a random mention, the ‘what’s new’ link points to version 1.0
That’s were I placed them. But doesn’t help and I get no connection to server after some time. Nothing interesting if I logcat
2 Likes
Welcome to PhotoStructure, @mexusbg!
The site.webmanifest is served as a static file: you should see it in your photostructure-for-servers/public directory.
It’s odd that the webservice is returning a 401: PhotoStructure’s static file server router returns 404s for unknown routes, so I believe that status is coming from your reverse proxy. What do you have in front of PhotoStructure?
1 Like
I guess I’m blind, but where are the sync reports stored?
They live in .photostructure/sync-reports in your PhotoStructure library.
I will add a proper link and web view for that directory hierarchy to the UI after user auth is done.
BMP support can work with PhotoStructure for Node, but only via GraphicsMagick and recompiling libVips. I’d suggest converting them (with GraphicsMagic) to PNG: that would be a lossless conversion, and should safe disk space. Native support would require pulling in a new library.
Can you email one of these .wmv files to support@photostructure.com so I can take a look? Those should be supported, but as WMV is a container format, there are a ton of different codec combinations, and ffmpeg might not be happy with whatever you’ve got (given the settings I currently feed it).
I don’t know if this is related to the new version or not, but I report this in case anyone else can correlate.
I was investigating a problem where the sync seemed to die, so I was logtailing inside the container. I was seeing lots of permissions errors with the app trying to delete items from both the imgcache and database copy in /ps/tmp.
I checked the permissions on the real filesystem and a bunch of directories were owned by root, instead of the container owner. I’ve been running the previous version for a while with no issue and the locations are the same (same docker-compose.yml with the UID set). I’ve changed the ownership of the whole mount for /ps/tmp back to my container user again and things seem happier.
There shouldn’t be any PhotoStructure process that doesn’t respect the provided UID/GID environment variables: the entry point ensures that:
So… If there are any files not owned by your configured UID, something else is amiss.
Ok, that’s interesting. I hear you, but it’s happened again. Nothing else should be interested in this directory structure, so all I can think of is either an unexpected behaviour or something in the way I’m launching, which is just docker-compose up -d.
Any suggestions for where to look or process owners to check?
Here’s a sample:
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:53 2r/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:52 2v/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:53 2w/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 2z/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 35/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:52 36/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 37/
drwxr-xr-x 3 root root 16 May 14 01:22 38/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 3c/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:50 3d/
drwxr-xr-x 3 root root 16 May 14 01:21 3j/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 3q/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 41/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 46/
drwxr-xr-x 3 root root 16 May 14 01:22 4d/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 4f/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 4x/
drwxr-xr-x 3 root root 16 May 14 01:20 53/
drwxr-xr-x 3 root root 16 May 14 01:22 55/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:50 5k/
drwxr-xr-x 3 root root 16 May 14 01:20 5r/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:51 5x/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 62/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 68/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 6b/
drwxr-xr-x 4 root root 26 May 14 01:22 6n/
drwxr-xr-x 3 root root 16 May 14 01:22 6r/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:52 6y/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:51 7b/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 7c/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 7e/
drwxr-xr-x 3 root root 16 May 14 01:22 7z/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:53 83/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 8k/
drwxr-xr-x 3 root root 16 May 14 01:23 8n/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 8r/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 8u/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:50 8y/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 95/
drwxr-xr-x 3 root root 16 May 14 01:23 96/
drwxr-xr-x 3 root root 16 May 14 01:19 9d/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:53 9f/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:51 9m/
drwxr-xr-x 3 root root 16 May 14 01:19 9n/
drwxr-xr-x 3 root root 16 May 14 01:21 b0/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:53 b2/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:53 b6/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:52 b8/
drwxr-xr-x 3 root root 16 May 14 01:20 bb/
drwxr-xr-x 3 root root 16 May 14 01:20 bc/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:51 bm/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:51 c9/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 ce/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:51 cu/
drwxr-xr-x 3 root root 16 May 14 01:22 d0/
drwxr-xr-x 3 root root 16 May 14 01:19 de/
drwxr-xr-x 3 root root 16 May 14 01:19 dj/
drwxr-xr-x 4 containeruser containeruser 26 May 15 01:53 du/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:52 dx/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 e3/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 e6/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:52 e9/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:52 eh/
drwxr-xr-x 3 root root 16 May 14 01:22 em/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 ez/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:54 f1/
drwxr-xr-x 3 root root 16 May 14 01:23 f2/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:50 fc/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:50 fd/
drwxr-xr-x 4 containeruser containeruser 26 May 15 01:54 fg/
drwxr-xr-x 3 root root 16 May 14 01:23 fk/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:52 fq/
drwxr-xr-x 3 root root 16 May 14 01:19 fw/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:51 fx/
drwxr-xr-x 3 root root 16 May 14 01:22 fz/
drwxr-xr-x 3 root root 16 May 14 01:22 g7/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:53 g9/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:51 gp/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:50 gw/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 h4/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:53 jn/
drwxr-xr-x 3 containeruser containeruser 16 May 15 01:55 kb/
containeruser@host:/media/bulkstorage/photostructure/cache/imgcache$
some relevant processes:
$ ps -ef|grep photostructure
root 1993 1663 0 May14 ? 00:00:00 su --preserve-environment node --command /usr/local/bin/node /ps/app/photostructure
containeruser 2044 1993 0 May14 ? 00:00:00 /usr/local/bin/node /ps/app/photostructure
containeruser 2119 2044 0 May14 ? 00:00:28 photostructure main
containeruser 4820 2119 82 May14 ? 01:53:34 photostructure sync
containeruser 34960 2119 0 00:40 ? 00:00:23 photostructure web
containeruser 72190 4820 15 01:54 ? 00:00:34 photostructure worker
containeruser 72865 4820 11 01:54 ? 00:00:19 photostructure worker
and
ps -ef|grep ps
root 1663 1565 0 May14 ? 00:00:00 /sbin/tini -- /ps/app/docker-entrypoint.sh
root 1993 1663 0 May14 ? 00:00:00 su --preserve-environment node --command /usr/local/bin/node /ps/app/photostructure
containeruser 2044 1993 0 May14 ? 00:00:00 /usr/local/bin/node /ps/app/photostructure
containeruser 74760 72865 7 01:58 ? 00:00:02 /usr/bin/perl -w /ps/app/node_modules/exiftool-vendored.pl/bin/exiftool -stay_open True -@ -
containeruser 74762 4820 6 01:58 ? 00:00:01 /usr/bin/perl -w /ps/app/node_modules/exiftool-vendored.pl/bin/exiftool -stay_open True -@ -
docker-compose.yml extract:
# The userid to run PhotoStructure as:
- "PUID=1000" # < CHANGE THIS LINE or delete this line to run as root. See below for details.
# The groupid to run PhotoStructure as:
- "PGID=1000" # < CHANGE THIS LINE or delete this line to run as root.
uid confirmation:
$ cat /etc/passwd | grep 1000
containeruser:x:1000:1000:Name:/home/containeruser:/bin/bash
I had a look a the the file you referenced above and I see the handling for /ps/tmp is different to the rest and you don’t chown the structure in there:
if [ -z "$PS_NO_PUID_CHOWN" ]; then
# Always make sure the settings, opened-by, and models directories are
# read/writable by node:
for dir in /ps/library/.photostructure/settings.toml \
/ps/library/.photostructure/opened-by \
/ps/library/.photostructure/models \
/ps/config \
/ps/logs \
/ps/default; do
maybe_chown "$dir"
done
# Special handling so we don't do something terrible if someone bind-mounts /tmp to /ps/tmp
if [ -d /ps/tmp ]; then
mkdir -p "/ps/tmp/.cache-$UID"
maybe_chown "/ps/tmp/.cache-$UID"
fi
fi
It seems that all the files/directories owned by root are from yesterday: do you have more than one way to launch the docker container?
That’s what I’d expect: tini and su run as root, and everything else should be the UID. All the actual work is done by node and child processes.
Ah, nice sleuthing! I only added those chown commands so when you change PUID/UID (say, from root 0 to the default 1000 user id, that the new user would still have access to the files it needed to spin up the library).
So… didn’t get to play with 2.1.0 alpha a lot, but after an initial upgrade noticed that a lot of my pictures aren’t there… tried to do a restart sync, and then a full rebuild and it just ignored me…
Things were a bit “messy” on my install, and it’s gone through a bunch of testing updates and things, so just started fresh. Blew it all away (config and all) and redownloaded the docker (I’m on unRaid).
Configured as before…
I have 4 directories it’s supposed to scan. Three of them it pulls in thousands of pictures (I haven’t checked closely, but looks like most all) - but those are the three “small” directories. Then there is the main archive. It started on it, chewed just a few minutes, pulled in just a 1000 or so and stopped.
Log shows (for example):
1652886758215,/pictures/Archive/2009/2009-01_January/2009-01-04–2009-01-04_Disney – travel/2009-01-01T18.03.08_Disney-IMG_3849.jpg,enqueued,
1652886758215,/pictures/Archive/2009/2009-01_January/2009-01-04–2009-01-04_Disney – travel/2009-01-01T18.03.24_Disney-IMG_3850.jpg,enqueued,
1652886758226,/pictures/Archive/2009/2009-01_January,canceled,DirectoryWalker was ended,1131
1652886758226,/pictures/Archive/2009,canceled,DirectoryWalker was ended,1132
1652886758226,/pictures/Archive,canceled,DirectoryWalker was ended,111489
1652886760151,/pictures/Archive/1989/1989-10_October/19891007-Dillahunty_Wedding/020-29-Scan-101007-0001.jpg,failed,“BatchCluster has ended, cannot enqueue {”“id”":438,"“method”":"“buildAssetPreviews_”","“args”":[{"“assetId”":199,"“assetFiles”":[{""$ctor"":"“models.AssetFile”","“id”":219,"“assetId”":199,"“shown”":false,"“uri”":"“psfile://37Ajd4ybC/Archive/1989/1989-10_October/19891007-Dillahunty_Wedding/02.:false,”“recountAllTags”":false}]}",http://127.0.0.1:1787/asset/199,2643
~ ~
Where should I look from here? Permissions look right - for example, it pulled 4 pictures from one folder that has about 20. All set the same permissions.
I’m still seeing two problems. Scans don’t seem to populate new images into the database, even with small manually run single date folder scans.
Secondly, I’m still getting weird file ownership problems that I didn’t get with the previous version. This time, I carefully shut everything down, checked file ownership, started again with a stock standard sudo docker-compose up -d
This is the result (the top level of this is the bind mount point for /ps/tmp:
$ sudo docker logs photostructure
Please wait, setting up...
Your library is at <file:///ps/library>
PhotoStructure is ready:
- <http://127.0.0.1:1787/>
- <http://172.18.0.2:1787/>
- <http://f4b1f198ce13:1787/>
Shutting down PhotoStructure...
Please wait, setting up...
Please wait, setting up...
Please wait, setting up...
Please wait, setting up...
Please wait, setting up...
Please wait, setting up...
$ ll
total 0
drwxr-xr-x 5 containeruser containeruser 48 Mar 17 09:38 ./
drwxr-xr-x 3 root root 28 Mar 16 02:51 ../
drwxrwxr-x 2 containeruser containeruser 19 Mar 16 02:54 backup/
drwxrwxr-x 6 containeruser containeruser 135 May 17 08:47 cache/
drwxrwxr-x 4 containeruser containeruser 70 Mar 17 09:54 samples/
/cache$ ll
total 8
drwxrwxr-x 6 containeruser containeruser 135 May 17 08:47 ./
drwxr-xr-x 5 containeruser containeruser 48 Mar 17 09:38 ../
drwxr-xr-x 2 containeruser containeruser 6 May 12 09:20 .cache-1000/
drwxr-xr-x 15 containeruser containeruser 136 May 17 09:21 imgcache/
drwxr-xr-x 3 containeruser containeruser 20 May 18 21:21 local-db-0885-rqph-1u41-rutf/
-rw-r--r-- 1 containeruser containeruser 126 Mar 16 02:57 .NoMedia
drwxr-xr-x 3 containeruser containeruser 37 May 12 09:21 sync-state-kt27ebt2j7/
-rw-r--r-- 1 containeruser containeruser 37 Mar 16 03:01 .uuid
/cache/local-db-0885-rqph-1u41-rutf$ ll
total 0
drwxr-xr-x 3 containeruser containeruser 20 May 18 21:21 ./
drwxrwxr-x 6 containeruser containeruser 135 May 17 08:47 ../
drwxr-xr-x 4 root root 64 May 18 03:57 models/
/cache/local-db-0885-rqph-1u41-rutf/models$ ll
total 225200
drwxr-xr-x 4 root root 64 May 18 03:57 ./
drwxr-xr-x 3 containeruser containeruser 20 May 18 21:21 ../
drwxr-xr-x 2 root root 6 May 18 03:57 backup/
-rw-r--r-- 1 root root 230604800 May 18 03:57 db.sqlite3
drwxr-xr-x 2 root root 6 May 18 03:57 .db.sqlite3.pslock/
/cache/imgcache$ ll
total 0
drwxr-xr-x 15 containeruser containeruser 136 May 17 09:21 ./
drwxrwxr-x 6 containeruser containeruser 135 May 17 08:47 ../
drwxr-xr-x 3 root root 16 May 17 09:18 2z/
drwxr-xr-x 3 root root 16 May 17 09:20 30/
drwxr-xr-x 3 root root 16 May 17 09:19 38/
drwxr-xr-x 3 root root 16 May 17 09:18 4n/
drwxr-xr-x 3 root root 16 May 17 09:16 5d/
drwxr-xr-x 3 root root 16 May 17 09:19 75/
drwxr-xr-x 3 root root 16 May 17 09:20 7x/
drwxr-xr-x 3 root root 16 May 17 09:18 82/
drwxr-xr-x 3 root root 16 May 17 09:20 92/
drwxr-xr-x 3 root root 16 May 17 09:19 9t/
drwxr-xr-x 3 root root 16 May 17 09:18 cq/
drwxr-xr-x 3 root root 16 May 17 09:18 g9/
drwxr-xr-x 3 root root 16 May 17 09:18 gf/
If the DirectoryWalker ended, that means the error rate was too high and the “yikes, things are bad, I better not continue” circuit breaker blew.
If you set your logging to info and send me your logs I can take a look at what’s going on.