Docker container exiting without logs

tyderian · August 23, 2021, 11:37am

Hi,

I spent a fair bit of time last night trying to install PhotoStructure under Docker on my Synology NAS and ran into some issues.
I initially used the suggested Docker shell script wrapper, but found that within about 6 seconds of starting the docker container exited without generating any logs.
Both docker logs -f <ID> and the photostructure ~/.cache/PhotoStructure/logs were blank.

I then moved to using docker-compose:

version: "3.7"
services:
  photostructure:
    image: photostructure/server
#    command: node /ps/app/photostructure main --verbose
    restart: on-failure
    stop_grace_period: 2m

    volumes:
      - type: bind
        source: /volume1/PhotoStructure # < CHANGE THIS LINE
        target: /ps/library

      - type: bind
        source: "${HOME}/.cache/photostructure"
        target: /ps/tmp

      - type: bind
        source: "${HOME}/.config/PhotoStructure-docker"
        target: /ps/config

      - type: bind
        source: "${HOME}/.config/PhotoStructure/logs"
        target: /ps/logs

      - type: bind
        source: "/volume1/Document/Photos" # < CHANGE THIS LINE
        target: /var/synology-photos

    ports:
      - 1787:1787/tcp

    environment:
      - "PS_LOG_LEVEL=error"
#      - "PUID=0" # < CHANGE THIS LINE to match the userid you want PhotoStructure to run under
#      - "PGID=0" # < CHANGE THIS LINE to match the groupid you want PhotoStructure to run under

As you can see I tried to use the command option at the top (and then commented it out later) to try and get some useful output when starting in non-detached mode. This didn’t help.
I stripped out Watchtower, I tried switching from running as a user account to running as root, I tried upgrading docker-compose… Nothing helped.

I’m running the latest version as of last night, on a fully patched DS1812+.
I have Synology Docker installed and running a couple of containers via the UI, but they’re in different locations and mapped to different ports… I have ample disk space and system resources. My user account has full permissions to docker (via the UI) and has also been added to the docker user group.

Does anyone have any ideas as to what I can do? The lack of error output makes it really hard for me to diagnose.

avdp · August 23, 2021, 1:21pm

I don’t have a synology, so for what it is worth, this is what gets run for my container on Unraid.

I see a few differences. No idea which (if any) is significant. Do all those source directories exists on your NAS? Personally, I’d try replacing ${HOME} with the full path.

docker run -d --name='Photostructure' --net='bridge' -e TZ="America/New_York" -e 'PS_LOG_LEVEL'='error' -e 'PS_FFMPEG_HWACCEL'='false' -e 'PS_KEYWORD_REPARENTING'='move' -e 'PS_WHO_TAGS'='["People"]' -e 'PS_TRANSCODE_VIDEOS'='false' -p '1787:1787/tcp' -v '/mnt/cache/photostructure/':'/ps/library':'rw' -v '/mnt/cache/appdata/photostructure/tmp/':'/ps/tmp':'rw' -v '/mnt/cache/appdata/photostructure/config/':'/ps/config':'rw' -v '/mnt/cache/appdata/photostructure/logs/':'/ps/logs':'rw' -v '/mnt/cache/photostructure/Photosbackup/':'/photos-backup':'rw' -v '/mnt/user/PhotoArchive/':'/photos':'ro' --restart unless-stopped 'photostructure/server:1.1.0'

mrm · August 23, 2021, 1:43pm

Howdy: welcome to PhotoStructure, @tyderian , and apologies that your first impression wasn’t smooth!

I’d make a couple changes:

Like @avdp suggested, try replacing the bind mount paths to be absolute paths to existing directories. ${HOME} should work, but…
change your environment section to look like

    environment:
      - "PS_LOG_LEVEL=debug"
      - "PS_LOG_STDOUT=1"
      - "PS_TAIL_LOGS=1"

PS_LOG_STDOUT=1 tees logs to both logfiles and the console. You’ll want to remove this later, as docker doesn’t take kindly to excessively chatty containers.

Please report back on how it goes!

tyderian · August 24, 2021, 9:55am

Hi guys,

So as my regular user account I hard coded all the paths, and added the extra logging variables…
Photostructure starts… then exits a few seconds later:

CONTAINER ID   IMAGE                                 COMMAND                  CREATED              STATUS                          PORTS                                            NAMES
403b5c303d26   photostructure/server                 "/sbin/tini -- /ps/a…"   About a minute ago   Exited (0) About a minute ago                                                    USER_photostructure_1

The contents of my .config folder tree (after container startup) is as follows:

~/.config$ ls -alR
.:
total 0
drwx------ 1 USER users  78 Aug 22 20:20 .
drwxr-xr-x 1 USER users 120 Aug 24 19:46 ..
drwxr-xr-x 1 USER users   8 Aug 22 20:20 PhotoStructure
drwxr-xr-x 1 USER users   0 Aug 22 20:20 PhotoStructure-docker

./PhotoStructure:
total 0
drwxr-xr-x 1 USER users  8 Aug 22 20:20 .
drwx------ 1 USER users 78 Aug 22 20:20 ..
drwxr-xr-x 1 USER users  0 Aug 22 20:20 logs

./PhotoStructure/logs:
total 0
drwxr-xr-x 1 USER users 0 Aug 22 20:20 .
drwxr-xr-x 1 USER users 8 Aug 22 20:20 ..

./PhotoStructure-docker:
total 0
drwxr-xr-x 1 USER users  0 Aug 22 20:20 .
drwx------ 1 USER users 78 Aug 22 20:20 ..

Terminating the dead container and running it again as root I also get no logs, but now I see this in /var/log/messages:

2021-08-24T19:52:52+10:00 DiskStation kernel: [3883053.200945] aufs au_opts_verify:1571:dockerd[31624]: dirperm1 breaks the protection by the permission bits on the lower branch
2021-08-24T19:52:52+10:00 DiskStation kernel: [3883053.238295] aufs au_opts_verify:1571:dockerd[31624]: dirperm1 breaks the protection by the permission bits on the lower branch
2021-08-24T19:52:52+10:00 DiskStation kernel: [3883053.389384] aufs au_opts_verify:1571:dockerd[24813]: dirperm1 breaks the protection by the permission bits on the lower branch
2021-08-24T19:52:57+10:00 DiskStation coredump: Process node[18] dumped core on signal [4].

This sounds almost like a kernel bug?

Linux DiskStation 3.10.105 #25556 SMP Thu Mar 18 12:51:35 CST 2021 x86_64 GNU/Linux synology_cedarview_1812+

avdp · August 26, 2021, 1:10am

That definitely look like a kernel bug… from 2017. Which is how old the kernel version the Synology runs 3.10.105 was released on 2/10/2017. Not sure what you can do about this though. Are you running the container in privileged mode? I see some random google posts indicating this may cause this error.

tyderian · August 26, 2021, 7:08am

I checked this, and it does not appear to be in privileged mode:

unpriv_user@Diskstation:~$ docker inspect --format='{{.HostConfig.Privileged}}' 434a7ca7206b
false

I’ve done a lot of googling and was unable to find posts from people who were successfully able to update their old Synology kernels by compiling newer ones.

mrm · August 28, 2021, 6:12pm

Totally spitballing here, but you can crank up logging and write to stdout and then use docker logs: it may at least emit something…

Add this to your docker command:

-e PS_LOG_LEVEL=debug -e PS_LOG_STDOUT=1 -e PS_TAIL_LOGS=1

More logging config is here:

github.com

photostructure/photostructure-for-servers/blob/1f6d907769fd2b01900eb88ce64bf0d732998aa0/defaults.env#L231

    
      
          #
          # Should PhotoStructure scan your library after all other paths are
          # synchronized?
          #
          # PS_SCAN_LIBRARY_LAST="true"
          
          

          

          
##############################################################################
          #
          # Settings for System.logging:
          #
          
          

          
# +------------+
          # |  logColor  |
          # +------------+
          #
          # Output all logs with terminal escape codes to colorize output. If NO_COLOR
          # is set, this defaults to false. See <https://no-color.org/>.
          #

tyderian · September 3, 2021, 4:27am

Sorry, unfortunately I seem to be having no luck with this.
I think my only option to get this working would be to get a newer NAS, or a microserver.

I’ll definitely revisit this in the next few months as we need a better solution than just folders on the NAS.

mrm · September 3, 2021, 3:54pm

If node is dumping a core file, either it or a native library is not happy. It could be due to it being compiled against a different set of dynamically linked libraries or different headers. Either way, I suspect that you’re correct: the ancient OS is what’s causing you grief.

I have an ancient NAS as well: make sure it’s isolated from the internet to shield it from all the security issues that have been found (and mostly fixed) in more recent kernels.

You’ll want to run PhotoStructure on any other computer you have: it’ll run on basically everything (just make sure the OS is up to date).