Thumbnails Missing - Server Install

adamf · June 9, 2021, 11:33pm

Hello!

I just set up PhotoStructure via docker-compose and noticed that I seem to be missing a significant amount of thumbnail photos. I at first assumed I did something wrong and so I deleted my entire library, cleared out all of the generated settings, the library, etc., and restarted the process.

This time, however, I disabled the automatic transcoding of videos to speed up the testing process. While I would expect to not have certain video thumbnails now available, I’m still missing a lot of PNG, HEIC, and RAW thumbnails as well as several (but not all) MPEG-4 and QuickTime video format thumbnails.

(Related to this, I attempted to force the disabling of auto start via a compose environment variable, but this did not appear to work - imports started anyway. Thus, I moved my preexisting content out of the library folder, manually shut down, edited the .toml file to disable video transcode, moved the preexisting content back in to the library folder structure, and then restarted the sync.)

If there is a good place to send a screenshot without publicly posting my photos, please let me know.

Here is my docker compose configuration:

# PhotoStructure - Media Consumption - Photos
  photostructure:
    image: photostructure/server:beta
    container_name: photostructure
    restart: unless-stopped
    networks:
      - $TRAEFIK_NETWORK
    security_opt:
      - no-new-privileges:true
    stop_grace_period: 2m
    volumes:
      - $MEDIADIR/photos:/ps/library
      - $DOCKERDIR/photostructure/tmp:/ps/tmp
      - $DOCKERDIR/photostructure/config:/ps/config
      - $DOCKERDIR/photostructure/logs:/ps/logs
    environment:
      - TZ
      - PUID
      - PGID
      - UMASK=002
    labels:
      - traefik.enable=true
      ## HTTP Routers
      - traefik.http.routers.photostructure-rtr.entrypoints=https
      - traefik.http.routers.photostructure-rtr.rule=Host(`photos.$DOMAINNAME`)
      ## Middlewares
      - traefik.http.routers.photostructure-rtr.middlewares=chain-authelia@file
      ## HTTP Services
      - traefik.http.routers.photostructure-rtr.service=photostructure-svc
      - traefik.http.services.photostructure-svc.loadbalancer.server.port=1787

Thanks very much!

Const · June 10, 2021, 12:25am

You can send email to Matthew support@photostructure.com, but keep in mind he is AFK till next week.

K

mrm · June 14, 2021, 10:01pm

Howdy @adamf, welcome to PhotoStructure!

PhotoStructure doesn’t import thumbnails (low-resolution images) by default. There’s a lot of detail (and instructions on how to tell PhotoStructure that you really do want to import everything) on this page:

In the future, know that you can either DM me images via the forum, or email me (at support@photostructure.com).

adamf · June 15, 2021, 12:50am

Hi Matthew, welcome back!

I think I should clarify, these are not small photos that weren’t imported - they were a variety of sizes of photos (and some videos) that were imported into the library, but thumbnails were not generated for them. I expected that to happen when I disabled video transcoding for the video previews, but not for the photos.

Please let me know if there’s additional info I could provide.

Thanks!

mrm · June 15, 2021, 1:49am

Ah, thanks for that clarification, and for sending me that screenshot.

Oof, that’s no good!

I haven’t seen previews not show up unless the previews directory (that’s in $path/to/your/library/.photostructure/previews) has been manually deleted (or mucked with). This could be due to the /ps/library path not being properly bind-mounted, which can lead to writes going to an ephemeral docker layer that gets deleted when the container was restarted (due to reboot or upgrade).

If you rebuild your library, does that fix the problem?

adamf · June 15, 2021, 4:35am

Hey @mrm,

So I’m finding what may be interesting in my folder permissions. First I should mention that my $DOCKERDIR is on my VM’s local disk, and $MEDIADIR is an NFS share on my Synology NAS.

Now, I have the NAS bind mounted, and it’s mounted on the VM via NFS. It’s mapping me to root on the NAS due to some limitations around uid, root_squash, etc. with NFS.

Since Photostructure is running via docker as my username and group name, many of the folders and files within the .photostructure folder are being generated as being chown’d by adam:adam via the container, whereas others are being generated without the UID/GID mapping as root/users, which may be causing part of this problem?

Assuming I created only the $DOCKERDIR/photostructure and its child subfolders of config, tmp, and logs manually, but the .photostructure folder structure within my media directory was not precreated, the permissions aren’t always matching. Please see the attached screenshot as an example, such as the licenses folder vs the models folder, or README.txt vs settings.toml.

folder permissions

mrm · June 15, 2021, 5:20am

Ugh, permission issues are always pesky!

If you’re using the beta docker image,:

don’t use the docker --user feature: just set the UID and GID environment variables in your docker config to the correct numbers (presumably whatever adam is).
sudo chown -r adam:adam $path/to/library to make sure everything’s starting off correctly
Restart PhotoStructure. You should find that the library and newly-copied-in assets are properly owned by adam:adam.

If anything goes awry, holler.

adamf · June 15, 2021, 5:37am

Thanks. I’m already setting the UID/GID environmental variables luckily, so currently running the chown.

However, I’m seeing that all of the previews created after resuming the container are still being created as root:users instead of adam:adam. Also, the newest folders created after that point in the previews folder (in my case, 153 and above) are also set to root:users.

Is it possible some of the processes in the container are not running as the UID/GID from the environmental variables while others are?

Thanks.

mrm · June 15, 2021, 2:32pm

Unfortunately, it’s software: surprising bugs are always possible.

PhotoStructure spawns a bunch of child processes, and those are supposed to inherit the UID and GID of the parent process. I’ll look into this today.

Could you email or dm me details about asset 69 versus asset 71? (Are they different kinds of file types?)

adamf · June 15, 2021, 3:45pm

Thanks again. I just emailed you some json files plus additional detail.

avdp · June 15, 2021, 5:03pm

Not sure this is the same issue, but I have folders that show 1,206 assets, but only ~140 thumbnails under them (I obscured the actual thumbnails for privacy reasons, but you might discern there are 20 across and 7 down)

Not seeing any permission issues. Permission are consistent, so if they were wrong, nothing should show up.

Version 1.0.0-beta.7
PhotoStructure for Docker on unraid
All disks are local (no usb, nfs, smb, etc)

I see a bunch of errors in the logs involving “previews”… They’re all informational, yet many of them show errors

{"ts":1623701244448,"l":"warn","ctx":"DirectoryEntry","msg":"children() failed to readdir(/ps/library/.photostructure/previews/000/023)","meta":{"errno":-2,"code":"ENOENT","syscall":"lstat","path":"/ps/library/.photostructure/previews/000/023/.61-fit-w1080.jpg","stack":["Error: ENOENT: no such file or directory, lstat '/ps/library/.photostructure/previews/000/023/.61-fit-w1080.jpg'"]}}
{"ts":1623701280249,"l":"warn","ctx":"DirectoryEntry","msg":"children() failed to readdir(/ps/library/.photostructure/previews/000/023)","meta":{"stack":["Error: readdir() timeout for /ps/library/.photostructure/previews/000/023","    at Object.t.readdir_ (/ps/app/bin/sync-file.js:9:243295)","    at async f.children (/ps/app/bin/sync-file.js:9:204775)","    at async W.childDirectoryEntries (/ps/app/bin/sync-file.js:9:183778)","    at async W.childFiles (/ps/app/bin/sync-file.js:9:184227)","    at async Object.t.thenOrElse (/ps/app/bin/sync-file.js:9:114386)"]}}
{"ts":1623701526831,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/027/.35-fit-w2160.jpg"]}}
{"ts":1623701527405,"l":"info","ctx":"UpdateAsset(2735)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":5627,"assetId":2735,"uri":"pslib:/2008/2008-11-22/IMG_1817.JPG","width":3888,"height":2592,"mimetype":"image/jpeg","sha":"LnXgNVJ6Km49niFjcrnf3S53E3ae1QuI","rotation":0,"path":"/ps/library/2008/2008-11-22/IMG_1817.JPG","mtime":1623371737698,"filesize":4080338,"fitSizes":"qhd,hd,qvga"}}}
{"ts":1623717107669,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/143/.85-fit-w1080.jpg"]}}
{"ts":1623717108179,"l":"info","ctx":"UpdateAsset(14385)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":31031,"assetId":14385,"uri":"pslib:/2020/2020-03-26/IMG_0765%20%281%29.JPG","width":3024,"height":4032,"mimetype":"image/jpeg","sha":"daUgbsToxNg5ezOROCOjY4FAJYIcLbYp","rotation":90,"path":"/ps/library/2020/2020-03-26/IMG_0765 (1).JPG","mtime":1623373114270,"filesize":3397968,"fitSizes":"uhd4k,fhd,wvga,qvga"}}}
{"ts":1623717113411,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/143/.88-fit-w1440.jpg"]}}
{"ts":1623717113862,"l":"info","ctx":"UpdateAsset(14388)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":31038,"assetId":14388,"uri":"pslib:/2020/2020-03-26/IMG_0766.JPG","width":4032,"height":3024,"mimetype":"image/jpeg","sha":"2ERMhkNgK2gHOxmS4mWWGL7D9HCe01gM","rotation":180,"path":"/ps/library/2020/2020-03-26/IMG_0766.JPG","mtime":1623373096483,"filesize":3113296,"fitSizes":"uhd4k,fhd,wvga,qvga"}}}
{"ts":1623722643347,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/026/.50-fit-w2160.jpg"]}}
{"ts":1623722643721,"l":"info","ctx":"UpdateAsset(2650)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":41799,"assetId":2650,"uri":"pslib:/2008/2008-03-22/IMG_1317-1.jpg","width":3888,"height":2592,"mimetype":"image/jpeg","sha":"MOYs9W/S69egLbzZiCn4nhcG6aKqDwGd","rotation":0,"path":"/ps/library/2008/2008-03-22/IMG_1317-1.jpg","mtime":1623371729181,"filesize":3056101,"fitSizes":"qhd,hd,qvga"}}}
{"ts":1623722645296,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/026/.53-fit-w2160.jpg"]}}
{"ts":1623722645625,"l":"info","ctx":"UpdateAsset(2653)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":41805,"assetId":2653,"uri":"pslib:/2008/2008-03-24/IMG_1321-1.JPG","width":3888,"height":2592,"mimetype":"image/jpeg","sha":"705fGR/cQsHqkJy91js+40BlpBr5iq/b","rotation":0,"path":"/ps/library/2008/2008-03-24/IMG_1321-1.JPG","mtime":1623371724545,"filesize":3049945,"fitSizes":"qhd,hd,qvga"}}}
{"ts":1623727244927,"l":"info","ctx":"UpdateAsset(6049)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":12479,"assetId":6049,"uri":"pslib:/2013/2013-08-20/SDC10586.JPG","width":3000,"height":4000,"mimetype":"image/jpeg","sha":"lLzyFgQNkq3shriKj4rGMTqpY37z9utF","path":"/ps/library/2013/2013-08-20/SDC10586.JPG","mtime":1623372003789,"filesize":5161849,"fitSizes":"qhd,hd,qvga"}}}
{"ts":1623727250128,"l":"info","ctx":"UpdateAsset(6050)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":12484,"assetId":6050,"uri":"pslib:/2013/2013-08-20/SDC10589.JPG","width":4000,"height":3000,"mimetype":"image/jpeg","sha":"HndCu4UC1GEcjwnqkAXOfhFUwZhF9k2n","rotation":0,"path":"/ps/library/2013/2013-08-20/SDC10589.JPG","mtime":1623372008447,"filesize":5223353,"fitSizes":"qhd,hd,qvga"}}}
{"ts":1623728256279,"l":"warn","ctx":"DirectoryEntry","msg":"children() failed to readdir(/ps/library/.photostructure/previews/000/191)","meta":{"errno":-2,"code":"ENOENT","syscall":"lstat","path":"/ps/library/.photostructure/previews/000/191/.72-sq-w480.jpg","stack":["Error: ENOENT: no such file or directory, lstat '/ps/library/.photostructure/previews/000/191/.72-sq-w480.jpg'"]}}
{"ts":1623728329983,"l":"info","ctx":"UpdateAsset(19192)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":51250,"assetId":19192,"uri":"pslib:/2015/2015-04-03/IMG_20150403_190705372_TOP.jpg","width":4160,"height":2340,"mimetype":"image/jpeg","sha":"9VPDhjv+XZ0Cjwk5+SK3KFOb3PGnC0Xb","path":"/ps/library/2015/2015-04-03/IMG_20150403_190705372_TOP.jpg","mtime":1586298280000,"filesize":4638564,"fitSizes":"fhd,wvga,qvga"}}}
{"ts":1623728332839,"l":"info","ctx":"UpdateAsset(7186)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":51255,"assetId":7186,"uri":"pslib:/2015/2015-04-03/IMG_20150403_190709866-1.jpg","width":4160,"height":2340,"mimetype":"image/jpeg","sha":"FwTOZrSUnsje48LNUe0OKTa0fXXSvmTk","path":"/ps/library/2015/2015-04-03/IMG_20150403_190709866-1.jpg","mtime":1623372171671,"filesize":2154809,"fitSizes":"fhd,wvga,qvga"}}}
{"ts":1623727770967,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/189/.98-fit-w1440.jpg"]}}
{"ts":1623727771267,"l":"info","ctx":"UpdateAsset(18998)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":50273,"assetId":18998,"uri":"pslib:/2013/2013-12-29/IMG_0929.JPG","width":3264,"height":2448,"mimetype":"image/jpeg","sha":"4AHIxpZ9NMj0rnYdqwD7SL0X1lk81yED","rotation":180,"path":"/ps/library/2013/2013-12-29/IMG_0929.JPG","mtime":1389386690000,"filesize":2107796,"fitSizes":"uhd4k,fhd,wvga,qvga"}}}
{"ts":1623700307391,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/015/.57-fit-w1080.jpg"]}}
{"ts":1623700307860,"l":"info","ctx":"UpdateAsset(1557)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":3229,"assetId":1557,"uri":"pslib:/2005/2005-09-10/IMG_0521-007.JPG","width":1944,"height":2592,"mimetype":"image/jpeg","sha":"DP04JwNQUcU1JqPcv1Hz9mmk3ynIPCgQ","path":"/ps/library/2005/2005-09-10/IMG_0521-007.JPG","mtime":1623371635302,"filesize":1582950,"fitSizes":"fhd,wvga,qvga"}}}
{"ts":1623700309211,"l":"info","ctx":"UpdateAsset(1557)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":3229,"assetId":1557,"uri":"pslib:/2005/2005-09-10/IMG_0521-007.JPG","width":1944,"height":2592,"mimetype":"image/jpeg","sha":"DP04JwNQUcU1JqPcv1Hz9mmk3ynIPCgQ","path":"/ps/library/2005/2005-09-10/IMG_0521-007.JPG","mtime":1623371635302,"filesize":1582950,"fitSizes":"fhd,wvga,qvga"}}}

mrm · June 15, 2021, 6:18pm

oh neat, TIL that it supported pass-through env values.

In the future, this might be easier: Want to share a screenshot, but blur the thumbnails?

I wonder if the lazy-loader isn’t fetching the next batch of thumbs? It fetches 128 thumbs per request. Does it fail to load on Safari or Firefox? I actually added a “Load More…” button that should have shown if the lazy-loader failed to run. I’d also check the browser’s developer tools: if the console has any errors, can you DM or email those to me?

adamf · June 15, 2021, 6:42pm

Yes indeed, though in this case I’m using a .env environmental variable file, which also behaves very differently than if you defined one via the env_file flag within the compose file.

I bring the container up with dcpu photostructure and my alias is alias dcpu='sudo docker-compose --env-file /home/adam/docker/.env -f /home/adam/docker/docker-compose.yml up -d'

avdp · June 15, 2021, 7:31pm

Thanks for the tip on blurring! Much easier indeed.

I checked the console under developer’s tool. It’s clean, no errors.

EDIT: same under Safari. I don’t have firefox

avdp · June 15, 2021, 7:40pm

I guess I am blind. Not an error, but the console does show &limit=128

mrm · June 15, 2021, 7:49pm

That’s the first batch loading, but there should be a series of those messages.

I suspect this issue is due to me being dumb/unimaginative: I didn’t think there were going to be > 128 files taken at exactly the same centisecond, but these look like they are all from 1974 with no hour/minute/second resolution: so they’re all “from the same moment”.

I’ll fix the pager in the next beta build to handle this situation. Thanks again for helping debug this issue!

avdp · June 15, 2021, 7:51pm

That makes sense! Those were all scanned photos from my dad’s childhood, no information on when they were taken. So in order for them to show up early in the timeline I tagged them all as 1/1 of my birth year. I am sure the time is the same as well.

EDIT: indeed, I just found another album with more than 128… real photos (not scans) with real timestamps and it shows all of them.

mrm · June 15, 2021, 8:29pm

@avdp I’m afraid we threadjacked @adamf here: I suspect his problem is different. I’ll see if I can split this topic.

mrm · June 16, 2021, 12:05am

OK, so, both 69 and 70 are JPEG assets: so we’re not fighting, say, heif or video conversion weirdness.

I’ve poked around my test library that’s running PhotoStructure for Docker v1.0.0-beta.X on an Ubuntu host with a UID and GID set to 1000, and all the file owner bits of preview files are correct.

It’s all stored on a local disk, though. I wonder if there’s funny uid mapping when a bindmount isn’t local?

What’s the container’s ps say? Mine looks like this:

$ docker exec -it psdc_photostructure_1 ps -ef | grep node
root        12     1  0 Jun14 ?        00:00:00 su --preserve-environment node -
node        23    12  0 Jun14 ?        00:00:00 /usr/local/bin/node /ps/app/phot
node        30    23  0 Jun14 ?        00:01:35 PhotoStructure
node        44    30  0 Jun14 ?        00:13:29 PhotoStructure web
node     21628    44  0 00:02 ?        00:00:00 /usr/bin/perl -w /ps/app/node_mo
node     22949    30 39 Jun15 ?        06:19:58 PhotoStructure sync

adamf · June 16, 2021, 12:41am

It’s very likely then that it’s an issue with the library generated files living on an NFS mount. Here’s my ps

node          19       8  0 Jun15 ?        00:00:00 /usr/local/bin/node /ps/app/
node          26      19  0 Jun15 ?        00:00:25 PhotoStructure
node          41      26  0 Jun15 ?        00:00:29 PhotoStructure web
node          48      26 36 Jun15 ?        00:56:54 PhotoStructure sync
node      107742      48  3 00:04 ?        00:00:50 PhotoStructure sync-file
node      108061      48  2 00:05 ?        00:00:46 PhotoStructure sync-file
node      108077      48  2 00:05 ?        00:00:41 PhotoStructure sync-file
node      115558  108077  0 00:30 ?        00:00:00 /usr/bin/perl -w /ps/app/nod
node      115791  108077 99 00:30 ?        00:12:18 ffmpeg -loglevel error -thre
node      116393  108061 99 00:31 ?        00:04:07 ffmpeg -loglevel error -thre

My assumption is that actions taken by the container that are explicitly inheriting the UID/GID are properly being generated, but anything else is falling back to the anon_uid/anon_gid of my all_squash NFS mount which is root.

I wonder if running the container as adam:users (the synology Users usergroup, 100) instead of adam:adam and leaving umask as 002 will help, because it’s root:users and not root:root on the NFS mapping. That would at least get the files in the same group.

Or, in another possible solution, is there a way to have all of the library transcodes/thumbnails/etc. created separately from where the actual photos live? Or would I instead have to take my existing photos out of /ps/library and then copy them in?

I already have Photostructure set to copy files in, but right now a majority of them are detected already in a subfolder of /ps/library. My use case there is that I use iCloud photo downloader to regularly download my iCloud photo library, and I’m currently putting it in a subfolder at (the bind mount equivalent of) /ps/library/icloud.

Thanks for any insight.

Edit: Here is a new ps after I recreated the entire library/container from scratch with the GID overlap on the NFS mount.

adam@laforge:~/docker/photostructure$ docker exec -it photostructure ps -ef | grep node
node          20       7  0 00:47 ?        00:00:00 /usr/local/bin/node /ps/app/
node          27      20  0 00:47 ?        00:00:02 PhotoStructure
node          40      27  2 00:47 ?        00:00:04 PhotoStructure web
node          74      27 47 00:48 ?        00:01:07 PhotoStructure sync
node          99      74 67 00:48 ?        00:01:32 PhotoStructure sync-file
node         116      74 61 00:48 ?        00:01:23 PhotoStructure sync-file
node         139      74 66 00:48 ?        00:01:29 PhotoStructure sync-file
node         339      74 65 00:48 ?        00:01:26 PhotoStructure sync-file
node         589      74 62 00:49 ?        00:01:21 PhotoStructure sync-file
node         904      74 68 00:49 ?        00:01:27 PhotoStructure sync-file
node        1378      74 71 00:49 ?        00:01:30 PhotoStructure sync-file
node        1922      74 70 00:49 ?        00:01:27 PhotoStructure sync-file
node        2851      74 65 00:49 ?        00:01:21 PhotoStructure sync-file
node        4001      74 66 00:49 ?        00:01:20 PhotoStructure sync-file
node        5257      74 64 00:49 ?        00:01:17 PhotoStructure sync-file
node        6698      74 66 00:49 ?        00:01:18 PhotoStructure sync-file
node       45878      99  6 00:49 ?        00:00:04 /usr/bin/perl -w /ps/app/nod
node       53901     139  5 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       55936     904  6 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       58886    1378  7 00:50 ?        00:00:04 /usr/bin/perl -w /ps/app/nod
node       60612    1922  7 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       60977     116  5 00:50 ?        00:00:02 /usr/bin/perl -w /ps/app/nod
node       60980     589  5 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       64332    2851  6 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       67444    4001  6 00:50 ?        00:00:02 /usr/bin/perl -w /ps/app/nod
node       67458    5257  6 00:50 ?        00:00:02 /usr/bin/perl -w /ps/app/nod
node       67518    6698  6 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       71469     339  6 00:50 ?        00:00:02 /usr/bin/perl -w /ps/app/nod

Edit 2: I noted that I can right click on an image with a broken thumbnail and open the image in a new tab, and it exists.

Here is what it looks like however - seems pretty random which ones do or do not show up.