Syncing issue with nested folders

Bug
1

I found a number of missing files and I’m stumped as to what causes it. So I had mapped to a folder in the docker image at /pictures and that contained 2023/Turks and Caicos/Eliza. 129 images in the turks folder and only 16 jpgs in the eliza folder. I might only get 9 of the 16 from the eliza folder.

If I map /pictures directly to the Eliza folder and only scan that one mapped folder on a fresh start it finds all 16.

If I map /pictures/Turks to the turks folder, move Eliza out of turks and map that to /pictures/Eliza. Then again it fails to find all 16.

I got looking at this because my file count of all pictures is much higher than what I’ve got imported.

For the 16 files here is one error from the sync log:

{“ts”:1697512361349,“l”:“warn”,“ctx”:“Error”,“msg”:“onError()”,“meta”:{“event”:“nonFatal”,“error”:“unhandledRejection: end() called before task completed ({"gracefully":false,"lastTask":{"command":"{\"id\":1025,\"method\":\"_readRawTags\",\"args\":[\"/pictures/2023/Turks and Caicos/Eliza/2…ctures/2023/Turks and Caicos/Eliza/2023-09-24_17.10.39.jpg"]}}}) at BatchProcess._BatchProcess_end (/opt/photostructure/node_modules/batch-cluster/src/BatchProcess.ts:418:11); observe (/opt/photostructure/node_modules/batch-cluster/src/Deferred.ts:116:15)”}}

I’m running prealpha docker image.

2

Thanks for reporting, and apologies for the glitch!

Can you set debug logging, retry, and send me the resulting “sync” and “worker“ logs? Hopefully that will include the context that I need to determine what’s going awry.

Oh—also, be sure to check the sync reports, there might be something in there.

3

So is Javascript heap out of memory. I tweaked these settings but it didn’t help. I got the exact same result. Do any of these affect the node jobs?

  - PS_MAX_MEMORY_MB="4096"
  - PS_MAX_RSS_MEMORY_MB="4096"
  - PS_SHARP_THREADS_PER_JOB=2
  - PS_MAX_SYNC_FILE_JOBS=2
  - PS_MAX_TASKS_PER_PROCESS=50

Oddly I don’t think it took any longer, but the result was the same. FATAL ERRORs due to javascript heap memory.

logs here: Dropbox - ph-missing.zip - Simplify your life

One file which didn’t work is: /pictures/Eliza/2023-09-24_17.10.39.jpg

4

Oddly I can repro this now with just the folder of 16 jpgs. Last night when I tried only that folder i got all the files. Is it possible there is something different about some of these jpgs? I did take a lightroom update yesterday and these files were exported from lightroom.

5

I haven’t seen a heap overflow for a while, so there’s got to be something (either the file or the system setup) that’s causing things to go awry. I’m away from my desk this week, but I’ll check when I get back.

6

Can I invoke the command to extract the info and generate the thumbnails directly to see if I can repro it? These are 33MP images it’s barfing on.

7

You can run sync directly on any given file or folder. You can add the --force argument to make sure sync fully reimports the file into the library database, rebuilds the image previews, and, if necessary, rebuilds the video transcode (if the asset needs it): PhotoStructure | PhotoStructure Tools

8

Ok, thx, maybe I can find a simpler repro. This got me looking at my total file counts and they are way off so hopefully this one small folder represents the main problem.

9

Always seems to fail on _readrawtags with certain files. I’ve looked at the tags using exiftool and cant see anything in common with the failed files. I can provide one somehow when you get back.

A couple things weird about the tools. Most of them would not work for me until I set PS_LIBRARY_DIR in my docker env. I was see files and what not, but the cli tools would not work. Your instructions say to use “-u node”, but in my docker it is the user photostructure who got the PUID/PGID. I have to run the commands with “-u photostructure”. When I use “-u node” i get permission errors about not have access to the library folder.