Slow / stalled rebuild

I last used PhotoStructure a few months ago. I upgraded to 1.0.0-beta.3 and it wanted to rebuild my library. It said “around 12 hours remain” so I left my PC on overnight. I came back the next morning and it had barely progressed. There were ~43,000 items remaining last night, and 12 hours later there’s still 42,400 items remaining.

The system I’m running it on isn’t the fastest or newest thing in the world (Core i5 6500), but older versions were definitely faster to rebuild. I only have 16GB RAM though. I’ve noticed that PhotoStructure’s RAM usage will grow until it reaches around 4.6GB:
image

then it will rapidly drop down to ~1GB:
image

so my guess would be that some process is hitting an out-of-memory error. There’s several instances of something that looks like a Node.js OOM error in the sync logs:

{“ts”:1622309186526,“l”:“warn”,“ctx”:“Error”,“msg”:“onError(): Error: onStderr(\r\n<— Last few GCs —>\r\n\r\n[21592:0000494300000000] 131018 ms: Scavenge 3829.6 (3967.3) → 3829.6 (3967.3) MB, 115.9 / 0.0 ms (average mu = 0.598, current mu = 0.430) allocation failure \r\n[21592:0000494300000000] 131139 ms: Scavenge…\nError: onStderr(\r\n<— Last few GCs —>\r\n\r\n[21592:0000494300000000] 131018 ms: Scavenge 3829.6 (3967.3) → 3829.6 (3967.3) MB, 115.9 / 0.0 ms (average mu = 0.598, current mu = 0.430) allocation failure \r\n[21592:0000494300000000] 131139 ms: Scavenge 3829.6 (3967.3) → 3829.6 (3967.3) MB, 120.2 / 0.0 ms (average mu = 0.598, current mu = 0.430) allocation failure \r\n[21592:0000494300000000] 131256 ms: Scavenge 3829.6 (3967.3) → 3829.6 (3967.3) MB, 117.8 / 0.0 ms (average mu = 0.598, current mu = 0.430) allocation failure \r”,“meta”:{“event”:“nonFatal”,“message”:“sync-file: internal error”}}

Sync log: http://d.ls/photostructure/sync-5540-002.log

When I get a chance, I might try it on my work PC instead, which is a beast of a machine (32-core Threadripper 3970X and 128 GB RAM) and see how it goes there instead.

System information

Version 1.0.0-beta.3
Edition PhotoStructure for Desktops (beta channel)
Plan plus
OS Windows 10 (10.0.19043) on x64
PowerShell 5.1.19041
Electron 12.0.8
Video support FFmpeg 4.2.3
HEIF support heif-convert: not installed
Free memory 6.4 GB / 17 GB
CPUs 4 × Intel(R) Core™ i5-6500 CPU @ 3.20GHz
Concurrency Target system use: 75% (3 syncs, 2 threads/sync)
Web uptime 12 hours, 30 minutes
Library metrics 26,966 assets


79,718 image files
778 video files
1,372 tags
Library path D:\PhotoStructure
Health checks

  • Library and support directories are OK
  • Used memory used by web (26 MB) is OK
  • RSS memory used by web (78 MB) is OK

Volume information

mount size free volume id label
C:\ 1000 GB 48 GB 2BNf9oGmo
D:\ 1 TB 61 GB 2qFb6q6JU Data

I let it continue since I posted earlier today (which was around 7 hours ago) and it still says 42,003 items remaining, so it only processed around 400 items in 7 hours. I’m going to exit it for now since it seems like it’s not actually progressing. Should I just delete the entire library and let it reindex everything from scratch?

Sorry that PhotoStructure’s misbehaving for you!

I suspect the same behavior will happen again.

Can you turn on debug level logging and send me your logs?

@mrm Here’s the logs after running it for around 10 minutes with debug logging enabled:

…(deleted)…

Please let me know if there’s anything else you need :slight_smile:

I’ve seen something similar, i think. Photostructure 1.0.0-beta3 in a docker container, ~1Tb library with 150K assets, mostly photos.

After upgrade to beta3, Photostructure spent about a week in “rebuild”, and based on “photostructure logtail” it was mostly updating “When” tags. On 4-threaded box, load average stayed below 2 most of the time, and logtail looked like sync processes restarted every now and then and library closed-reopened about every couple of minutes.

I added these settings at some point, which subjectively made process at least twice as was as it was before (I have 32GB of physical ram):
- PS_DB_CACHE_SIZE_MB=512
- PS_MAX_MEMORY_MB=2000
- PS_MAX_RSS_MEMORY_MB=4000
- PS_MAX_TASKS_PER_PROCESS=2000
- PS_CPU_LOAD_PERCENT=150

After rebuild was done, library rescan started and went for another 4-5 days. All in all, it feels that this is the longest rebuild/rescan I encountered after upgrade, so far.

1 Like

Ah! Thanks for the clue, @adept! I just timed the tag count update module on my larger library, and found that it was taking about 40 seconds. I’ll see if I can make that process faster.

Update: I’m making it faster now, and it’ll be in beta.4

1 Like

It looks like only the sync-file process keeling over from memory consumption. Can you email me your db and a sample file so I can try to reproduce the issue with JavaScript memory profiling?

Sure, I’ll do that now. Sorry for the delay!

@Daniel and @adept can you two verify that beta.9 solves this issue? It does for me, at least for my large test library.

Thanks again for reporting and assisting in debugging!

I am having this issue right now with beta.9
Progress is stuck - it hasn’t processed anything. CPU usage is low, essentially idle. I have shutdown the container and restarted a couple times, it processes a bunch then stalls again.

  PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND                                              
    1 root      20   0    0.8m   0.3m   0.0   0.0   0:00.12 S /sbin/tini -- /ps/app/docker-entrypoint.sh           
    7 root      20   0    1.1m   0.6m   0.0   0.0   0:00.00 S  `- su --preserve-environment node --command /usr/lo+
   18 node      20   0  214.1m  29.0m   0.0   0.1   0:00.04 S      `- /usr/local/bin/node /ps/app/photostructure   
   25 node      20   0  290.3m  76.4m   0.7   0.2   0:06.39 S          `- PhotoStructure                           
   40 node      20   0  309.5m 104.4m   0.0   0.3   0:15.34 S              `- PhotoStructure web                   
   46 node      29   9  322.6m 121.3m   2.6   0.4   8:15.57 S              `- PhotoStructure sync                  
   74 node      29   9  361.4m 133.4m   0.0   0.4   7:06.37 S                  `- PhotoStructure sync-file         
  622 node      29   9  360.8m 132.1m   0.0   0.4  10:56.33 S                  `- PhotoStructure sync-file         
17706 node      29   9  353.7m 127.0m   0.0   0.4   1:26.92 S                  `- PhotoStructure sync-file         
18560 node      29   9  354.2m 132.4m   0.0   0.4   2:09.60 S                  `- PhotoStructure sync-file         
30285 node      29   9  360.9m 134.5m   0.0   0.4   2:35.55 S                  `- PhotoStructure sync-file         
32177 node      29   9  357.2m 169.2m   0.0   0.5  18:51.30 S                  `- PhotoStructure sync-file         
 4718 node      29   9  354.4m 168.7m   0.0   0.5  21:16.04 S                  `- PhotoStructure sync-file         
 5420 node      29   9  354.2m 168.9m   0.0   0.5   3:22.91 S                  `- PhotoStructure sync-file         
 5455 node      29   9  360.2m 174.8m   0.0   0.5   3:30.64 S                  `- PhotoStructure sync-file         
13601 root      20   0    1.6m   1.1m   0.0   0.0   0:00.01 S sh                                                   
13607 root      20   0    2.0m   1.3m   0.0   0.0   0:00.00 R  `- top

Sorry for the delay in getting back to you. Life is busy these days :slight_smile:

I just updated from beta3 to beta9 and it’s definitely faster! In 10 minutes or so it got through more photos than the old version did in ~9 hours (it seems like the old one just got completely stuck after a while).

I can only echo what others have been saying : beta.9 is blazingly fast for me! Thanks a lot!

I’m not sure if this is the right thread but my rebuild seems to stall at around 98% of completion.
The last few 1000-2000 pictures do not get processed:

image

Even after several “restart sync” button clicks it does not completely finish.

Dang! Can you send me debug logs, please?

https://photostructure.com/faq/error-reports/#how-to-manually-send-your-logs

done, sent you a link to the log files asked. If you need more infos just ask, thx for taking the time to look into this problem.