Thumbnails Missing - Server Install

Support
21

I’m adding a new reply as the prior, with multiple edits, was getting long. My updated compose is relocating the previews directory on the local VM of Ubuntu hard drive, and all preview folders/files are being owned by my adam user, but I’m still seeing all of the thumbnails not rendering on the page. I’m wondering if there are possibly other files/services that may be owned by root on the NFS share within the .photostructure folder that are causing this, or if there are other things to check.

I’m going to separately move a subset of my photos to the local VM disk and run a fresh instance again just to verify that works normally as expected. If it doesn’t, well at least we can cross NFS/permissions off the list.

For reference, my current compose is:

  photostructure:
    image: photostructure/server:beta
    container_name: photostructure
    restart: unless-stopped
    networks:
      - $TRAEFIK_NETWORK
    security_opt:
      - no-new-privileges:true
    stop_grace_period: 2m
    volumes:
      - $MEDIADIR/photos:/ps/library
      - $DOCKERDIR/photostructure/tmp:/ps/tmp
      - $DOCKERDIR/photostructure/config:/ps/config
      - $DOCKERDIR/photostructure/logs:/ps/logs
      - $DOCKERDIR/photostructure/previews:/ps/previews
    environment:
      - TZ
      - PUID
      - PGID=100
      - UMASK=002
      - PS_START_PAUSED=true
      - PS_TRANSCODE_VIDEOS=false
      - PS_PREVIEWS_DIR=/ps/previews
    labels:
      - traefik.enable=true
      ## HTTP Routers
      - traefik.http.routers.photostructure-rtr.entrypoints=https
      - traefik.http.routers.photostructure-rtr.rule=Host(`photos.$DOMAINNAME`)
      ## Middlewares
      - traefik.http.routers.photostructure-rtr.middlewares=chain-authelia@file
      ## HTTP Services
      - traefik.http.routers.photostructure-rtr.service=photostructure-svc
      - traefik.http.services.photostructure-svc.loadbalancer.server.port=1787

Stay tuned!

Edit 1: Moving the entire library folder structure to the VM local disk, and setting one specific subfolder of the NAS library as a bind mount, did not work - still some broken thumbnails out of my 170 test photos. Next up I’ll set the ‘copy’ option to yes instead of no. Here are the volumes and variables for these test iterations.

  photostructure:
    image: photostructure/server:beta
    container_name: photostructure
    restart: unless-stopped
    networks:
      - $TRAEFIK_NETWORK
    security_opt:
      - no-new-privileges:true
    stop_grace_period: 2m
    volumes:
      - $DOCKERDIR/photostructure/library:/ps/library
      - $DOCKERDIR/photostructure/tmp:/ps/tmp
      - $DOCKERDIR/photostructure/config:/ps/config
      - $DOCKERDIR/photostructure/logs:/ps/logs
      - $MEDIADIR/photos/icloud/adam/2021/06:/myphotos
    environment:
      - TZ
      - PUID
      - PGID=100
      - UMASK=002
      - PS_START_PAUSED=true
      - PS_TRANSCODE_VIDEOS=false

Edit 2: Testing the same as above with everything local, but now having the /myphotos path be copied in instead of not copied. Same behavior.

Edit 3: Testing the same as above, but removing the /myphotos mount and manually copying in those 170 files to /ps/library is the same result. I’ve got to conclude at this point it isn’t a permissions issue but something going on with docker, my OS, compose, etc.

Thanks again.

22

@mrm Okay! Major breakthrough here. I was looking at one page of thumbnails that wasn’t loading and I kept hitting refresh. Suddenly, I’d see new thumbnails work and others fail to load. Then I decided to check the browser console: tons of 429 errors - rate limit.

On a hunch, I went to the IP address of Photostructure instead of my reverse proxy. Everything loads without issue. As it turns out, all of the thumbnails, however they are loading, are triggering the rate limit middleware of Traefik (my reverse proxy) whose configuration is as such:

http:
  middlewares:
    rate-limit:
      rateLimit:
        average: 100
        burst: 50

I don’t think I’ve experienced this type of issue before with lots of content loading on a page. I’m wondering if it is how the thumbnails are embedded and called that’s causing the rate limit issue, and if those calls could theoretically be changed to not all be separate GET calls?

Thanks!

23

You can actually configure both: the originals directory, library directory, and previews directory are all configurable. Check the system and library settings.

Switching to an http/2 server engine would conceivably help, by I’ve not looked into that.

Good job finding the issue!

Update: http/2 support for Express has been waiting for 2.5 years, and http/2 support in fastify (which would require a non-trivial code migration) is experimental. :poop: .

24

I’m trying to understand how I could keep rate limiting enabled but also use Traefik. Is that something I’d want to use PS_TRUST_PROXY for?

Also, if I use PS_ORIGINALS_DIR, can I somehow leverage that and copy assets to library, but not copy my existing iCloud photo downloader library? (Trying to avoid duplicating this as it is 450GB)

IE: I have my library config/db/etc. in the docker folder on SSD, but I want the originals on the NAS. My iCloud downloaded photos are already on the NAS. If I set the originals to live at /mnt/storage/media/photos and my iCloud repo is at /mnt/storage/media/photos/icloud, will they be copied or left in place? I wasn’t sure how that functionality worked since files in the library aren’t moved, but technically now my library is on the SSD, and only the originals are on the NAS, if that makes sense.

Thank you.

25

No, that only manages how Express handles headers. See Express behind proxies

You can set average to 0 to disable ratelimiting, but it should be higher than burst. I’d set burst to something like 256.

Batches of tiny thumbnails come in 128, so that the hit rate should be expected to be at least that.

As long as files are already found in the PS_ORIGINALS_DIR, they won’t be copied/duplicated.

They’ll be left in place.

1 Like
26

Perfect. I think I’ve landed here for the moment, pending my testing of hw acceleration. Thanks for your help.

  photostructure:
    image: photostructure/server:beta
    container_name: photostructure
    restart: unless-stopped
    networks:
      - $TRAEFIK_NETWORK
    security_opt:
      - no-new-privileges:true
    stop_grace_period: 2m
    volumes:
      - $MEDIADIR/photos:/photos
      - $DOCKERDIR/photostructure/library:/ps/library
      - $DOCKERDIR/photostructure/tmp:/ps/tmp
      - $DOCKERDIR/photostructure/config:/ps/config
      - $DOCKERDIR/photostructure/logs:/ps/logs
    environment:
      - TZ
      - PUID
      - PGID
      - UMASK=002
      - PS_ORIGINALS_DIR=/photos
      - PS_ENABLE_VIPS_CACHE=true
      - PS_MAX_ASSET_FILE_SIZE_BYTES=5000000000

My iCloud downloaded photos are in a subfolder within $MEDIADIR/photos.

Edit: I’ve also simply disabled rate limiting on this one container by using a different combination of Traefik middlewares. I’m putting it behind Authelia SSO so likely no problems here.

27

That config looks reasonable to me.

(I’d benchmark the VIPS cache: it didn’t make my machine resize faster)

1 Like