Slow / stalled rebuild

I last used PhotoStructure a few months ago. I upgraded to 1.0.0-beta.3 and it wanted to rebuild my library. It said “around 12 hours remain” so I left my PC on overnight. I came back the next morning and it had barely progressed. There were ~43,000 items remaining last night, and 12 hours later there’s still 42,400 items remaining.

The system I’m running it on isn’t the fastest or newest thing in the world (Core i5 6500), but older versions were definitely faster to rebuild. I only have 16GB RAM though. I’ve noticed that PhotoStructure’s RAM usage will grow until it reaches around 4.6GB:
image

then it will rapidly drop down to ~1GB:
image

so my guess would be that some process is hitting an out-of-memory error. There’s several instances of something that looks like a Node.js OOM error in the sync logs:

{“ts”:1622309186526,“l”:“warn”,“ctx”:“Error”,“msg”:“onError(): Error: onStderr(\r\n<— Last few GCs —>\r\n\r\n[21592:0000494300000000] 131018 ms: Scavenge 3829.6 (3967.3) → 3829.6 (3967.3) MB, 115.9 / 0.0 ms (average mu = 0.598, current mu = 0.430) allocation failure \r\n[21592:0000494300000000] 131139 ms: Scavenge…\nError: onStderr(\r\n<— Last few GCs —>\r\n\r\n[21592:0000494300000000] 131018 ms: Scavenge 3829.6 (3967.3) → 3829.6 (3967.3) MB, 115.9 / 0.0 ms (average mu = 0.598, current mu = 0.430) allocation failure \r\n[21592:0000494300000000] 131139 ms: Scavenge 3829.6 (3967.3) → 3829.6 (3967.3) MB, 120.2 / 0.0 ms (average mu = 0.598, current mu = 0.430) allocation failure \r\n[21592:0000494300000000] 131256 ms: Scavenge 3829.6 (3967.3) → 3829.6 (3967.3) MB, 117.8 / 0.0 ms (average mu = 0.598, current mu = 0.430) allocation failure \r”,“meta”:{“event”:“nonFatal”,“message”:“sync-file: internal error”}}

Sync log: http://d.ls/photostructure/sync-5540-002.log

When I get a chance, I might try it on my work PC instead, which is a beast of a machine (32-core Threadripper 3970X and 128 GB RAM) and see how it goes there instead.

System information

Version 1.0.0-beta.3
Edition PhotoStructure for Desktops (beta channel)
Plan plus
OS Windows 10 (10.0.19043) on x64
PowerShell 5.1.19041
Electron 12.0.8
Video support FFmpeg 4.2.3
HEIF support heif-convert: not installed
Free memory 6.4 GB / 17 GB
CPUs 4 × Intel(R) Core™ i5-6500 CPU @ 3.20GHz
Concurrency Target system use: 75% (3 syncs, 2 threads/sync)
Web uptime 12 hours, 30 minutes
Library metrics 26,966 assets


79,718 image files
778 video files
1,372 tags
Library path D:\PhotoStructure
Health checks

  • Library and support directories are OK
  • Used memory used by web (26 MB) is OK
  • RSS memory used by web (78 MB) is OK

Volume information

mount size free volume id label
C:\ 1000 GB 48 GB 2BNf9oGmo
D:\ 1 TB 61 GB 2qFb6q6JU Data

I let it continue since I posted earlier today (which was around 7 hours ago) and it still says 42,003 items remaining, so it only processed around 400 items in 7 hours. I’m going to exit it for now since it seems like it’s not actually progressing. Should I just delete the entire library and let it reindex everything from scratch?

Sorry that PhotoStructure’s misbehaving for you!

I suspect the same behavior will happen again.

Can you turn on debug level logging and send me your logs?

@mrm Here’s the logs after running it for around 10 minutes with debug logging enabled:

…(deleted)…

Please let me know if there’s anything else you need :slight_smile:

I’ve seen something similar, i think. Photostructure 1.0.0-beta3 in a docker container, ~1Tb library with 150K assets, mostly photos.

After upgrade to beta3, Photostructure spent about a week in “rebuild”, and based on “photostructure logtail” it was mostly updating “When” tags. On 4-threaded box, load average stayed below 2 most of the time, and logtail looked like sync processes restarted every now and then and library closed-reopened about every couple of minutes.

I added these settings at some point, which subjectively made process at least twice as was as it was before (I have 32GB of physical ram):
- PS_DB_CACHE_SIZE_MB=512
- PS_MAX_MEMORY_MB=2000
- PS_MAX_RSS_MEMORY_MB=4000
- PS_MAX_TASKS_PER_PROCESS=2000
- PS_CPU_LOAD_PERCENT=150

After rebuild was done, library rescan started and went for another 4-5 days. All in all, it feels that this is the longest rebuild/rescan I encountered after upgrade, so far.

1 Like

Ah! Thanks for the clue, @adept! I just timed the tag count update module on my larger library, and found that it was taking about 40 seconds. I’ll see if I can make that process faster.

Update: I’m making it faster now, and it’ll be in beta.4

1 Like

It looks like only the sync-file process keeling over from memory consumption. Can you email me your db and a sample file so I can try to reproduce the issue with JavaScript memory profiling?

Sure, I’ll do that now. Sorry for the delay!

@Daniel and @adept can you two verify that beta.9 solves this issue? It does for me, at least for my large test library.

Thanks again for reporting and assisting in debugging!