I’m trying to move my assets to S3 using s3fs, but it may be too slow or otherwise problematic for PS to be usable. PS sees the asset directory and can scan it, but it seems to freeze often. So far it only sees 9 assets out of 40K, and it is “done” scanning.
I’m posting this in the Support category, but I realize that S3 is not supported. If anyone can help me figure out how to make this work, I’d appreciate it. If not, I’ll just remain on block storage.
One thing I’m seeing is that PS thinks drives are full when they aren’t. You can see that /var/ps_assets has 280T free: this is the S3 space. But check out /ps/logs:
It says there are 4.9G free on a 5.2G drive, but that it’s full. /ps/logs isn’t even S3 storage - it’s a 5G block-storage volume mounted to a Digital Ocean droplet. Might this be causing any performance issues, like slowness or shutdowns?
/ps/library is odd, too: PS can’t determine its size or usage, but it’s the same volume as /ps/logs. From docker-compose.yaml:
volumes:
- /opt/containers/photostructure/config:/ps/config:rw
- /mnt/ps_config:/ps/library:rw
- /mnt/ps_config/.photostructure/logs:/ps/logs:rw
- /tmp/photostructure-docker:/ps/tmp:rw
- /mnt/ps_assets:/var/ps_assets:rw
/dev, /proc/acpi, and /proc/scsi are also “full” when they’re nearly completely empty. I don’t know what those directories are, but is it a problem that PS thinks they’re full?
Here’s a sync-file log with lots of “slow” and “deferred” warnings. Does this reflect PS struggling to scan via s3fs?
{"ts":1625710044836,"l":"warn","ctx":"Sentry","msg":"Failed to set up sentry: TypeError: Cannot read property 'init' of undefined"}
{"ts":1625710044932,"l":"warn","ctx":"Deferred({\"cmd\":\"ps\",\"args\":[\"-p\",\"1027,1042,25,38\",\"-wwwo\",\"pid,lstart,command\"]})",
"msg":"settled multiple times (already resolved)","meta":{"value":{"result":" PID STARTED COMMAND\n 25 Thu Jul 8
01:59:06 2021 PhotoStructure\n 38 Thu Jul 8 01:59:08 2021 PhotoStructure web\n 1027 Thu Jul 8 02:06:53 2021 PhotoStructure sync\
n 1042 Thu Jul 8 02:07:22 2021 PhotoStructure sync-file\n","code":0,"pid":1057}}}
{"ts":1625710045380,"l":"warn","ctx":"LibraryHealthChecks","msg":"cleanupTmp(): deleting /var/ps_assets/.tmp-gqfner"}
{"ts":1625710055076,"l":"warn","ctx":"time(fs.mkdirp)","msg":"slow","meta":{"elapsed":9044}}
{"ts":1625710078980,"l":"warn","ctx":"time(fs.copyFile)","msg":"slow","meta":{"elapsed":33529}}
{"ts":1625710081262,"l":"warn","ctx":"time(fs.sha)","msg":"slow","meta":{"elapsed":2277}}
{"ts":1625710091441,"l":"warn","ctx":"time(fs.rmrf)","msg":"slow","meta":{"elapsed":10178}}
{"ts":1625710091593,"l":"warn","ctx":"Deferred({\"cmd\":\"ps\",\"args\":[\"-p\",\"1027,1042,25,38\",\"-wwwo\",\"pid,lstart,command\"]})",
"msg":"settled multiple times (already resolved)","meta":{"value":{"result":" PID STARTED COMMAND\n 25 Thu Jul 8
01:59:06 2021 PhotoStructure\n 38 Thu Jul 8 01:59:08 2021 PhotoStructure web\n 1027 Thu Jul 8 02:06:53 2021 PhotoStructure sync\
n 1042 Thu Jul 8 02:07:22 2021 PhotoStructure sync-file\n","code":0,"pid":1071}}}
{"ts":1625710103052,"l":"warn","ctx":"time(fs.stat)","msg":"slow","meta":{"elapsed":11444}}
{"ts":1625710103052,"l":"warn","ctx":"LibraryHealthChecks","msg":"cleanupTmp(): deleting /var/ps_assets/.tmp-gqfner"}
{"ts":1625710109565,"l":"warn","ctx":"time(fs.stat)","msg":"slow","meta":{"elapsed":6120}}
{"ts":1625710115209,"l":"warn","ctx":"time(fs.copyFile)","msg":"slow","meta":{"elapsed":11764}}
{"ts":1625710127763,"l":"warn","ctx":"time(fs.rmrf)","msg":"slow","meta":{"elapsed":12076}}
{"ts":1625710128784,"l":"warn","ctx":"ChildProcess","msg":"stdoutResult(): on(exit)","meta":{"pid":1083,"cmd":"renice","exitCode":1,"args
":["-1","-p","1082"],"elapsedMs":6,"result":""}}
{"ts":1625710131813,"l":"warn","ctx":"time(fs.stat)","msg":"slow","meta":{"elapsed":2542}}
{"ts":1625710136090,"l":"warn","ctx":"time(fs.stat)","msg":"slow","meta":{"elapsed":4276}}
{"ts":1625710136091,"l":"warn","ctx":"time(tags.readRawTags)","msg":"slow","meta":{"elapsed":6820}}
{"ts":1625710136282,"l":"warn","ctx":"AssetFileImporter(/var/ps_assets/2004/01/12/flying_lynx-.jpg)","msg":"rejected: minDimensionsFilter
"}
{"ts":1625710137862,"l":"warn","ctx":"Deferred({\"cmd\":\"ps\",\"args\":[\"-p\",\"1027,1042,25,38\",\"-wwwo\",\"pid,lstart,command\"]})",
"msg":"settled multiple times (already resolved)","meta":{"value":{"result":" PID STARTED COMMAND\n 25 Thu Jul 8
01:59:06 2021 PhotoStructure\n 38 Thu Jul 8 01:59:08 2021 PhotoStructure web\n 1027 Thu Jul 8 02:06:53 2021 PhotoStructure sync\
n 1042 Thu Jul 8 02:07:22 2021 PhotoStructure sync-file\n","code":0,"pid":1092}}}
{"ts":1625710137936,"l":"warn","ctx":"LibraryHealthChecks","msg":"cleanupTmp(): deleting /var/ps_assets/.tmp-gqfner"}
{"ts":1625710145848,"l":"warn","ctx":"ChildProcess","msg":"endProcess(1082)","meta":{"killed":false,"connected":false}}
{"ts":1625710146135,"l":"warn","ctx":"Deferred({\"cmd\":\"ps\",\"args\":[\"-p\",\"1082,1042\",\"-wwwo\",\"pid,lstart,command\"]})","msg":
"settled multiple times (already resolved)","meta":{"value":{"result":" PID STARTED COMMAND\n 1042 Thu Jul 8 02:07
:22 2021 PhotoStructure sync-file\n","code":0,"pid":1095}}}
{"ts":1625710146788,"l":"warn","ctx":"ChildProcess","msg":"stdoutResult(): on(exit)","meta":{"pid":1104,"cmd":"renice","exitCode":1,"args
":["-1","-p","1103"],"elapsedMs":5,"result":""}}
{"ts":1625710146788,"l":"warn","ctx":"ChildProcess","msg":"stdoutResult(): on(close)","meta":{"pid":1104,"cmd":"renice","exitCode":1,"arg
s":["-1","-p","1103"],"elapsedMs":5,"result":""}}
{"ts":1625710146788,"l":"warn","ctx":"Deferred({\"cmd\":\"renice\",\"args\":[\"-1\",\"-p\",\"1103\"]})","msg":"settled multiple times (al
ready resolved)","meta":{"value":{"result":"","code":1,"pid":1104}}}