Thumbnails Missing - Server Install

adamf · June 15, 2021, 5:37am

Thanks. I’m already setting the UID/GID environmental variables luckily, so currently running the chown.

However, I’m seeing that all of the previews created after resuming the container are still being created as root:users instead of adam:adam. Also, the newest folders created after that point in the previews folder (in my case, 153 and above) are also set to root:users.

Is it possible some of the processes in the container are not running as the UID/GID from the environmental variables while others are?

Thanks.

mrm · June 15, 2021, 2:32pm

Unfortunately, it’s software: surprising bugs are always possible.

PhotoStructure spawns a bunch of child processes, and those are supposed to inherit the UID and GID of the parent process. I’ll look into this today.

Could you email or dm me details about asset 69 versus asset 71? (Are they different kinds of file types?)

adamf · June 15, 2021, 3:45pm

Thanks again. I just emailed you some json files plus additional detail.

avdp · June 15, 2021, 5:03pm

Not sure this is the same issue, but I have folders that show 1,206 assets, but only ~140 thumbnails under them (I obscured the actual thumbnails for privacy reasons, but you might discern there are 20 across and 7 down)

Not seeing any permission issues. Permission are consistent, so if they were wrong, nothing should show up.

Version 1.0.0-beta.7
PhotoStructure for Docker on unraid
All disks are local (no usb, nfs, smb, etc)

I see a bunch of errors in the logs involving “previews”… They’re all informational, yet many of them show errors

{"ts":1623701244448,"l":"warn","ctx":"DirectoryEntry","msg":"children() failed to readdir(/ps/library/.photostructure/previews/000/023)","meta":{"errno":-2,"code":"ENOENT","syscall":"lstat","path":"/ps/library/.photostructure/previews/000/023/.61-fit-w1080.jpg","stack":["Error: ENOENT: no such file or directory, lstat '/ps/library/.photostructure/previews/000/023/.61-fit-w1080.jpg'"]}}
{"ts":1623701280249,"l":"warn","ctx":"DirectoryEntry","msg":"children() failed to readdir(/ps/library/.photostructure/previews/000/023)","meta":{"stack":["Error: readdir() timeout for /ps/library/.photostructure/previews/000/023","    at Object.t.readdir_ (/ps/app/bin/sync-file.js:9:243295)","    at async f.children (/ps/app/bin/sync-file.js:9:204775)","    at async W.childDirectoryEntries (/ps/app/bin/sync-file.js:9:183778)","    at async W.childFiles (/ps/app/bin/sync-file.js:9:184227)","    at async Object.t.thenOrElse (/ps/app/bin/sync-file.js:9:114386)"]}}
{"ts":1623701526831,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/027/.35-fit-w2160.jpg"]}}
{"ts":1623701527405,"l":"info","ctx":"UpdateAsset(2735)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":5627,"assetId":2735,"uri":"pslib:/2008/2008-11-22/IMG_1817.JPG","width":3888,"height":2592,"mimetype":"image/jpeg","sha":"LnXgNVJ6Km49niFjcrnf3S53E3ae1QuI","rotation":0,"path":"/ps/library/2008/2008-11-22/IMG_1817.JPG","mtime":1623371737698,"filesize":4080338,"fitSizes":"qhd,hd,qvga"}}}
{"ts":1623717107669,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/143/.85-fit-w1080.jpg"]}}
{"ts":1623717108179,"l":"info","ctx":"UpdateAsset(14385)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":31031,"assetId":14385,"uri":"pslib:/2020/2020-03-26/IMG_0765%20%281%29.JPG","width":3024,"height":4032,"mimetype":"image/jpeg","sha":"daUgbsToxNg5ezOROCOjY4FAJYIcLbYp","rotation":90,"path":"/ps/library/2020/2020-03-26/IMG_0765 (1).JPG","mtime":1623373114270,"filesize":3397968,"fitSizes":"uhd4k,fhd,wvga,qvga"}}}
{"ts":1623717113411,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/143/.88-fit-w1440.jpg"]}}
{"ts":1623717113862,"l":"info","ctx":"UpdateAsset(14388)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":31038,"assetId":14388,"uri":"pslib:/2020/2020-03-26/IMG_0766.JPG","width":4032,"height":3024,"mimetype":"image/jpeg","sha":"2ERMhkNgK2gHOxmS4mWWGL7D9HCe01gM","rotation":180,"path":"/ps/library/2020/2020-03-26/IMG_0766.JPG","mtime":1623373096483,"filesize":3113296,"fitSizes":"uhd4k,fhd,wvga,qvga"}}}
{"ts":1623722643347,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/026/.50-fit-w2160.jpg"]}}
{"ts":1623722643721,"l":"info","ctx":"UpdateAsset(2650)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":41799,"assetId":2650,"uri":"pslib:/2008/2008-03-22/IMG_1317-1.jpg","width":3888,"height":2592,"mimetype":"image/jpeg","sha":"MOYs9W/S69egLbzZiCn4nhcG6aKqDwGd","rotation":0,"path":"/ps/library/2008/2008-03-22/IMG_1317-1.jpg","mtime":1623371729181,"filesize":3056101,"fitSizes":"qhd,hd,qvga"}}}
{"ts":1623722645296,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/026/.53-fit-w2160.jpg"]}}
{"ts":1623722645625,"l":"info","ctx":"UpdateAsset(2653)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":41805,"assetId":2653,"uri":"pslib:/2008/2008-03-24/IMG_1321-1.JPG","width":3888,"height":2592,"mimetype":"image/jpeg","sha":"705fGR/cQsHqkJy91js+40BlpBr5iq/b","rotation":0,"path":"/ps/library/2008/2008-03-24/IMG_1321-1.JPG","mtime":1623371724545,"filesize":3049945,"fitSizes":"qhd,hd,qvga"}}}
{"ts":1623727244927,"l":"info","ctx":"UpdateAsset(6049)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":12479,"assetId":6049,"uri":"pslib:/2013/2013-08-20/SDC10586.JPG","width":3000,"height":4000,"mimetype":"image/jpeg","sha":"lLzyFgQNkq3shriKj4rGMTqpY37z9utF","path":"/ps/library/2013/2013-08-20/SDC10586.JPG","mtime":1623372003789,"filesize":5161849,"fitSizes":"qhd,hd,qvga"}}}
{"ts":1623727250128,"l":"info","ctx":"UpdateAsset(6050)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":12484,"assetId":6050,"uri":"pslib:/2013/2013-08-20/SDC10589.JPG","width":4000,"height":3000,"mimetype":"image/jpeg","sha":"HndCu4UC1GEcjwnqkAXOfhFUwZhF9k2n","rotation":0,"path":"/ps/library/2013/2013-08-20/SDC10589.JPG","mtime":1623372008447,"filesize":5223353,"fitSizes":"qhd,hd,qvga"}}}
{"ts":1623728256279,"l":"warn","ctx":"DirectoryEntry","msg":"children() failed to readdir(/ps/library/.photostructure/previews/000/191)","meta":{"errno":-2,"code":"ENOENT","syscall":"lstat","path":"/ps/library/.photostructure/previews/000/191/.72-sq-w480.jpg","stack":["Error: ENOENT: no such file or directory, lstat '/ps/library/.photostructure/previews/000/191/.72-sq-w480.jpg'"]}}
{"ts":1623728329983,"l":"info","ctx":"UpdateAsset(19192)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":51250,"assetId":19192,"uri":"pslib:/2015/2015-04-03/IMG_20150403_190705372_TOP.jpg","width":4160,"height":2340,"mimetype":"image/jpeg","sha":"9VPDhjv+XZ0Cjwk5+SK3KFOb3PGnC0Xb","path":"/ps/library/2015/2015-04-03/IMG_20150403_190705372_TOP.jpg","mtime":1586298280000,"filesize":4638564,"fitSizes":"fhd,wvga,qvga"}}}
{"ts":1623728332839,"l":"info","ctx":"UpdateAsset(7186)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":51255,"assetId":7186,"uri":"pslib:/2015/2015-04-03/IMG_20150403_190709866-1.jpg","width":4160,"height":2340,"mimetype":"image/jpeg","sha":"FwTOZrSUnsje48LNUe0OKTa0fXXSvmTk","path":"/ps/library/2015/2015-04-03/IMG_20150403_190709866-1.jpg","mtime":1623372171671,"filesize":2154809,"fitSizes":"fhd,wvga,qvga"}}}
{"ts":1623727770967,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/189/.98-fit-w1440.jpg"]}}
{"ts":1623727771267,"l":"info","ctx":"UpdateAsset(18998)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":50273,"assetId":18998,"uri":"pslib:/2013/2013-12-29/IMG_0929.JPG","width":3264,"height":2448,"mimetype":"image/jpeg","sha":"4AHIxpZ9NMj0rnYdqwD7SL0X1lk81yED","rotation":180,"path":"/ps/library/2013/2013-12-29/IMG_0929.JPG","mtime":1389386690000,"filesize":2107796,"fitSizes":"uhd4k,fhd,wvga,qvga"}}}
{"ts":1623700307391,"l":"info","ctx":"Sharp","msg":"Failed to buffer-clone sharp","meta":{"stack":["Error: Unsupported output format /ps/library/.photostructure/previews/000/015/.57-fit-w1080.jpg"]}}
{"ts":1623700307860,"l":"info","ctx":"UpdateAsset(1557)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":3229,"assetId":1557,"uri":"pslib:/2005/2005-09-10/IMG_0521-007.JPG","width":1944,"height":2592,"mimetype":"image/jpeg","sha":"DP04JwNQUcU1JqPcv1Hz9mmk3ynIPCgQ","path":"/ps/library/2005/2005-09-10/IMG_0521-007.JPG","mtime":1623371635302,"filesize":1582950,"fitSizes":"fhd,wvga,qvga"}}}
{"ts":1623700309211,"l":"info","ctx":"UpdateAsset(1557)","msg":"previews built","meta":{"assetPreviewInfo":{"assetFileId":3229,"assetId":1557,"uri":"pslib:/2005/2005-09-10/IMG_0521-007.JPG","width":1944,"height":2592,"mimetype":"image/jpeg","sha":"DP04JwNQUcU1JqPcv1Hz9mmk3ynIPCgQ","path":"/ps/library/2005/2005-09-10/IMG_0521-007.JPG","mtime":1623371635302,"filesize":1582950,"fitSizes":"fhd,wvga,qvga"}}}

mrm · June 15, 2021, 6:18pm

oh neat, TIL that it supported pass-through env values.

In the future, this might be easier: Want to share a screenshot, but blur the thumbnails?

I wonder if the lazy-loader isn’t fetching the next batch of thumbs? It fetches 128 thumbs per request. Does it fail to load on Safari or Firefox? I actually added a “Load More…” button that should have shown if the lazy-loader failed to run. I’d also check the browser’s developer tools: if the console has any errors, can you DM or email those to me?

adamf · June 15, 2021, 6:42pm

Yes indeed, though in this case I’m using a .env environmental variable file, which also behaves very differently than if you defined one via the env_file flag within the compose file.

I bring the container up with dcpu photostructure and my alias is alias dcpu='sudo docker-compose --env-file /home/adam/docker/.env -f /home/adam/docker/docker-compose.yml up -d'

avdp · June 15, 2021, 7:31pm

Thanks for the tip on blurring! Much easier indeed.

I checked the console under developer’s tool. It’s clean, no errors.

EDIT: same under Safari. I don’t have firefox

avdp · June 15, 2021, 7:40pm

I guess I am blind. Not an error, but the console does show &limit=128

mrm · June 15, 2021, 7:49pm

That’s the first batch loading, but there should be a series of those messages.

I suspect this issue is due to me being dumb/unimaginative: I didn’t think there were going to be > 128 files taken at exactly the same centisecond, but these look like they are all from 1974 with no hour/minute/second resolution: so they’re all “from the same moment”.

I’ll fix the pager in the next beta build to handle this situation. Thanks again for helping debug this issue!

avdp · June 15, 2021, 7:51pm

That makes sense! Those were all scanned photos from my dad’s childhood, no information on when they were taken. So in order for them to show up early in the timeline I tagged them all as 1/1 of my birth year. I am sure the time is the same as well.

EDIT: indeed, I just found another album with more than 128… real photos (not scans) with real timestamps and it shows all of them.

mrm · June 15, 2021, 8:29pm

@avdp I’m afraid we threadjacked @adamf here: I suspect his problem is different. I’ll see if I can split this topic.

mrm · June 16, 2021, 12:05am

OK, so, both 69 and 70 are JPEG assets: so we’re not fighting, say, heif or video conversion weirdness.

I’ve poked around my test library that’s running PhotoStructure for Docker v1.0.0-beta.X on an Ubuntu host with a UID and GID set to 1000, and all the file owner bits of preview files are correct.

It’s all stored on a local disk, though. I wonder if there’s funny uid mapping when a bindmount isn’t local?

What’s the container’s ps say? Mine looks like this:

$ docker exec -it psdc_photostructure_1 ps -ef | grep node
root        12     1  0 Jun14 ?        00:00:00 su --preserve-environment node -
node        23    12  0 Jun14 ?        00:00:00 /usr/local/bin/node /ps/app/phot
node        30    23  0 Jun14 ?        00:01:35 PhotoStructure
node        44    30  0 Jun14 ?        00:13:29 PhotoStructure web
node     21628    44  0 00:02 ?        00:00:00 /usr/bin/perl -w /ps/app/node_mo
node     22949    30 39 Jun15 ?        06:19:58 PhotoStructure sync

adamf · June 16, 2021, 12:41am

It’s very likely then that it’s an issue with the library generated files living on an NFS mount. Here’s my ps

node          19       8  0 Jun15 ?        00:00:00 /usr/local/bin/node /ps/app/
node          26      19  0 Jun15 ?        00:00:25 PhotoStructure
node          41      26  0 Jun15 ?        00:00:29 PhotoStructure web
node          48      26 36 Jun15 ?        00:56:54 PhotoStructure sync
node      107742      48  3 00:04 ?        00:00:50 PhotoStructure sync-file
node      108061      48  2 00:05 ?        00:00:46 PhotoStructure sync-file
node      108077      48  2 00:05 ?        00:00:41 PhotoStructure sync-file
node      115558  108077  0 00:30 ?        00:00:00 /usr/bin/perl -w /ps/app/nod
node      115791  108077 99 00:30 ?        00:12:18 ffmpeg -loglevel error -thre
node      116393  108061 99 00:31 ?        00:04:07 ffmpeg -loglevel error -thre

My assumption is that actions taken by the container that are explicitly inheriting the UID/GID are properly being generated, but anything else is falling back to the anon_uid/anon_gid of my all_squash NFS mount which is root.

I wonder if running the container as adam:users (the synology Users usergroup, 100) instead of adam:adam and leaving umask as 002 will help, because it’s root:users and not root:root on the NFS mapping. That would at least get the files in the same group.

Or, in another possible solution, is there a way to have all of the library transcodes/thumbnails/etc. created separately from where the actual photos live? Or would I instead have to take my existing photos out of /ps/library and then copy them in?

I already have Photostructure set to copy files in, but right now a majority of them are detected already in a subfolder of /ps/library. My use case there is that I use iCloud photo downloader to regularly download my iCloud photo library, and I’m currently putting it in a subfolder at (the bind mount equivalent of) /ps/library/icloud.

Thanks for any insight.

Edit: Here is a new ps after I recreated the entire library/container from scratch with the GID overlap on the NFS mount.

adam@laforge:~/docker/photostructure$ docker exec -it photostructure ps -ef | grep node
node          20       7  0 00:47 ?        00:00:00 /usr/local/bin/node /ps/app/
node          27      20  0 00:47 ?        00:00:02 PhotoStructure
node          40      27  2 00:47 ?        00:00:04 PhotoStructure web
node          74      27 47 00:48 ?        00:01:07 PhotoStructure sync
node          99      74 67 00:48 ?        00:01:32 PhotoStructure sync-file
node         116      74 61 00:48 ?        00:01:23 PhotoStructure sync-file
node         139      74 66 00:48 ?        00:01:29 PhotoStructure sync-file
node         339      74 65 00:48 ?        00:01:26 PhotoStructure sync-file
node         589      74 62 00:49 ?        00:01:21 PhotoStructure sync-file
node         904      74 68 00:49 ?        00:01:27 PhotoStructure sync-file
node        1378      74 71 00:49 ?        00:01:30 PhotoStructure sync-file
node        1922      74 70 00:49 ?        00:01:27 PhotoStructure sync-file
node        2851      74 65 00:49 ?        00:01:21 PhotoStructure sync-file
node        4001      74 66 00:49 ?        00:01:20 PhotoStructure sync-file
node        5257      74 64 00:49 ?        00:01:17 PhotoStructure sync-file
node        6698      74 66 00:49 ?        00:01:18 PhotoStructure sync-file
node       45878      99  6 00:49 ?        00:00:04 /usr/bin/perl -w /ps/app/nod
node       53901     139  5 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       55936     904  6 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       58886    1378  7 00:50 ?        00:00:04 /usr/bin/perl -w /ps/app/nod
node       60612    1922  7 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       60977     116  5 00:50 ?        00:00:02 /usr/bin/perl -w /ps/app/nod
node       60980     589  5 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       64332    2851  6 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       67444    4001  6 00:50 ?        00:00:02 /usr/bin/perl -w /ps/app/nod
node       67458    5257  6 00:50 ?        00:00:02 /usr/bin/perl -w /ps/app/nod
node       67518    6698  6 00:50 ?        00:00:03 /usr/bin/perl -w /ps/app/nod
node       71469     339  6 00:50 ?        00:00:02 /usr/bin/perl -w /ps/app/nod

Edit 2: I noted that I can right click on an image with a broken thumbnail and open the image in a new tab, and it exists.

Here is what it looks like however - seems pretty random which ones do or do not show up.

adamf · June 16, 2021, 3:24am

I’m adding a new reply as the prior, with multiple edits, was getting long. My updated compose is relocating the previews directory on the local VM of Ubuntu hard drive, and all preview folders/files are being owned by my adam user, but I’m still seeing all of the thumbnails not rendering on the page. I’m wondering if there are possibly other files/services that may be owned by root on the NFS share within the .photostructure folder that are causing this, or if there are other things to check.

I’m going to separately move a subset of my photos to the local VM disk and run a fresh instance again just to verify that works normally as expected. If it doesn’t, well at least we can cross NFS/permissions off the list.

For reference, my current compose is:

  photostructure:
    image: photostructure/server:beta
    container_name: photostructure
    restart: unless-stopped
    networks:
      - $TRAEFIK_NETWORK
    security_opt:
      - no-new-privileges:true
    stop_grace_period: 2m
    volumes:
      - $MEDIADIR/photos:/ps/library
      - $DOCKERDIR/photostructure/tmp:/ps/tmp
      - $DOCKERDIR/photostructure/config:/ps/config
      - $DOCKERDIR/photostructure/logs:/ps/logs
      - $DOCKERDIR/photostructure/previews:/ps/previews
    environment:
      - TZ
      - PUID
      - PGID=100
      - UMASK=002
      - PS_START_PAUSED=true
      - PS_TRANSCODE_VIDEOS=false
      - PS_PREVIEWS_DIR=/ps/previews
    labels:
      - traefik.enable=true
      ## HTTP Routers
      - traefik.http.routers.photostructure-rtr.entrypoints=https
      - traefik.http.routers.photostructure-rtr.rule=Host(`photos.$DOMAINNAME`)
      ## Middlewares
      - traefik.http.routers.photostructure-rtr.middlewares=chain-authelia@file
      ## HTTP Services
      - traefik.http.routers.photostructure-rtr.service=photostructure-svc
      - traefik.http.services.photostructure-svc.loadbalancer.server.port=1787

Stay tuned!

Edit 1: Moving the entire library folder structure to the VM local disk, and setting one specific subfolder of the NAS library as a bind mount, did not work - still some broken thumbnails out of my 170 test photos. Next up I’ll set the ‘copy’ option to yes instead of no. Here are the volumes and variables for these test iterations.

  photostructure:
    image: photostructure/server:beta
    container_name: photostructure
    restart: unless-stopped
    networks:
      - $TRAEFIK_NETWORK
    security_opt:
      - no-new-privileges:true
    stop_grace_period: 2m
    volumes:
      - $DOCKERDIR/photostructure/library:/ps/library
      - $DOCKERDIR/photostructure/tmp:/ps/tmp
      - $DOCKERDIR/photostructure/config:/ps/config
      - $DOCKERDIR/photostructure/logs:/ps/logs
      - $MEDIADIR/photos/icloud/adam/2021/06:/myphotos
    environment:
      - TZ
      - PUID
      - PGID=100
      - UMASK=002
      - PS_START_PAUSED=true
      - PS_TRANSCODE_VIDEOS=false

Edit 2: Testing the same as above with everything local, but now having the /myphotos path be copied in instead of not copied. Same behavior.

Edit 3: Testing the same as above, but removing the /myphotos mount and manually copying in those 170 files to /ps/library is the same result. I’ve got to conclude at this point it isn’t a permissions issue but something going on with docker, my OS, compose, etc.

Thanks again.

adamf · June 16, 2021, 5:42am

@mrm Okay! Major breakthrough here. I was looking at one page of thumbnails that wasn’t loading and I kept hitting refresh. Suddenly, I’d see new thumbnails work and others fail to load. Then I decided to check the browser console: tons of 429 errors - rate limit.

On a hunch, I went to the IP address of Photostructure instead of my reverse proxy. Everything loads without issue. As it turns out, all of the thumbnails, however they are loading, are triggering the rate limit middleware of Traefik (my reverse proxy) whose configuration is as such:

http:
  middlewares:
    rate-limit:
      rateLimit:
        average: 100
        burst: 50

I don’t think I’ve experienced this type of issue before with lots of content loading on a page. I’m wondering if it is how the thumbnails are embedded and called that’s causing the rate limit issue, and if those calls could theoretically be changed to not all be separate GET calls?

Thanks!

mrm · June 16, 2021, 3:17pm

You can actually configure both: the originals directory, library directory, and previews directory are all configurable. Check the system and library settings.

Switching to an http/2 server engine would conceivably help, by I’ve not looked into that.

Good job finding the issue!

Update: http/2 support for Express has been waiting for 2.5 years, and http/2 support in fastify (which would require a non-trivial code migration) is experimental. .

adamf · June 16, 2021, 6:29pm

I’m trying to understand how I could keep rate limiting enabled but also use Traefik. Is that something I’d want to use PS_TRUST_PROXY for?

Also, if I use PS_ORIGINALS_DIR, can I somehow leverage that and copy assets to library, but not copy my existing iCloud photo downloader library? (Trying to avoid duplicating this as it is 450GB)

IE: I have my library config/db/etc. in the docker folder on SSD, but I want the originals on the NAS. My iCloud downloaded photos are already on the NAS. If I set the originals to live at /mnt/storage/media/photos and my iCloud repo is at /mnt/storage/media/photos/icloud, will they be copied or left in place? I wasn’t sure how that functionality worked since files in the library aren’t moved, but technically now my library is on the SSD, and only the originals are on the NAS, if that makes sense.

Thank you.

mrm · June 16, 2021, 9:34pm

No, that only manages how Express handles headers. See Express behind proxies

You can set average to 0 to disable ratelimiting, but it should be higher than burst. I’d set burst to something like 256.

Batches of tiny thumbnails come in 128, so that the hit rate should be expected to be at least that.

As long as files are already found in the PS_ORIGINALS_DIR, they won’t be copied/duplicated.

They’ll be left in place.

adamf · June 16, 2021, 9:42pm

Perfect. I think I’ve landed here for the moment, pending my testing of hw acceleration. Thanks for your help.

  photostructure:
    image: photostructure/server:beta
    container_name: photostructure
    restart: unless-stopped
    networks:
      - $TRAEFIK_NETWORK
    security_opt:
      - no-new-privileges:true
    stop_grace_period: 2m
    volumes:
      - $MEDIADIR/photos:/photos
      - $DOCKERDIR/photostructure/library:/ps/library
      - $DOCKERDIR/photostructure/tmp:/ps/tmp
      - $DOCKERDIR/photostructure/config:/ps/config
      - $DOCKERDIR/photostructure/logs:/ps/logs
    environment:
      - TZ
      - PUID
      - PGID
      - UMASK=002
      - PS_ORIGINALS_DIR=/photos
      - PS_ENABLE_VIPS_CACHE=true
      - PS_MAX_ASSET_FILE_SIZE_BYTES=5000000000

My iCloud downloaded photos are in a subfolder within $MEDIADIR/photos.

Edit: I’ve also simply disabled rate limiting on this one container by using a different combination of Traefik middlewares. I’m putting it behind Authelia SSO so likely no problems here.

mrm · June 16, 2021, 10:02pm

That config looks reasonable to me.

(I’d benchmark the VIPS cache: it didn’t make my machine resize faster)