Any tips on getting my import/resync to use more disk/CPU/RAM, or figure out what’s actually limiting it? It’s using only ~20% of CPU and ~1 MB/s of the hard drives.
I’m trying out alpha.7 on a Synology DS920+ with the Docker GUI. So far I have:
configured the Docker container with “High” CPU priority
configured the Docker container to be able to use all 4GB of my RAM
set environment variable PS_PROCESS_PRIORITY=Normal
Any other ideas? Perhaps it’s a bug that PhotoStructure is only doing 1 concurrent import and 1 gfx/process (whatever that is), even though it’s detecting 4 cores and targeting 200% usage.
The code that determines concurrency looks at memory as a hard limiting factor, and won’t exceed that. I guess I could apply that clamp before I apply the cpuPercent setting, though, because if someone sets it to higher than 100%, it’s certainly signal that they are ok with higher load.
I’ll take a look at this. Thanks for the heads up!
PhotoStructure is the single most important application on my NAS, so when it’s rebuilding I want it done ASAP
(I got the NAS to prevent data rot and centralize my storage, and while theoretically I want to do Plex/Jellyfin/other stuff sometime, PhotoStructure is the only application I actually care about at this time).
I forgot I’d already added a manual override–try setting maxConcurrentImports=3 in your system settings.toml, or the environment variable in your docker run command. (Note that PhotoStructure looks for an environment variable override using both the PS_MAX_CONCURRENT_IMPORTS and maxConcurrentImports keys. I made the SHOUTY_CASE variant to make env pedants happy).
I think I’d try 2 or 3 first–using 4 will be almost certainly slower than 3, and may make your NAS unresponsive.
I’d also set your cpuLoadPercent to only 100. Although some “CPU percent utilization” (like Windows Task Manager) count one busy CPU as 100%, and 4 busy CPUs as 400%, PhotoStructure adds up all busy “cpu” percents and divides by the number of "cpu"s (where a “cpu” is the number of reported CPUs, which may be 2 ⨉ core_count, like on your hardware).
# +------------------+
# | cpuLoadPercent |
# +------------------+
#
# This setting is a rough goal for PhotoStructure to load the system during
# library synchronization. A higher value here will allow PhotoStructure to
# run more tasks in parallel, but may impact your system's responsiveness.
# Setting this value to 0 will still allow 1 import to run.
#
# This setting is ignored if "maxConcurrentImports" and
# "sharpThreadsPerProcess" are set.
#
# environment keys: "PS_CPU_LOAD_PERCENT"
# minValue: 0
# maxValue: 200
#
# cpuLoadPercent = 75
# +------------------------+
# | maxConcurrentImports |
# +------------------------+
#
# How many imports can PhotoStructure schedule concurrently? This will be
# clamped between 1 and 32.
#
# If not set, a sensible value will be computed based on "cpuLoadPercent".
#
# If set explicitly, this and "sharpThreadsPerProcess" will override
# "cpuLoadPercent" and "maxConcurrentImportsWhenRemote" settings.
#
# aliases: "maxSyncFileJobs"
# environment keys: "PS_MAX_CONCURRENT_IMPORTS" or "PS_MAX_SYNC_FILE_JOBS"
#
# maxConcurrentImports = undefined
# +----------------------------------+
# | maxConcurrentImportsWhenRemote |
# +----------------------------------+
#
# How many concurrent files can be imported if the library is on a remote
# volume? This defaults to 2 to try to avoid overwhelming HDD I/O on the
# remote NAS. If this is larger than (cpus.length * cpuLoadPercent) or max
# child processes given available memory, this value will be ignored.
#
# aliases: "maxSyncFileJobsWhenRemote"
# environment keys: "PS_MAX_CONCURRENT_IMPORTS_WHEN_REMOTE" or
# "PS_MAX_SYNC_FILE_JOBS_WHEN_REMOTE"
#
# maxConcurrentImportsWhenRemote = 2
(Given this, I won’t change the current code).
If you ever see any settings documentation that’s confusing or wrong, or think of ways to improve the verbiage, please share!
Make sure you enable snapshots and periodic data scrubs! Those weren’t enabled by default (at least on my NAS).
Also know that btrfs, which Synology uses, doesn’t do automatic data scrub recovery unless your data redundancy via RAID is sufficient. Unfortunately, I don’t believe RAID on btrfs is considered stable.
So–as far as I understand, btrfs will detect bitrot, but can’t automatically repair bitrot.
(I’d also rsync everything important to an external drive every quarter or year or so, and trade drives with a relative when you see them for holidays, just to have an offsite backup)
Thank you for the concern and writing PhotoStructure | How do I safely store my files?. I think I’m actually okay with my Synology using btrfs with SHR Synology Hybrid Raid with two hard drives, with snapshots and periodic data scrubs enabled, and with data healing enabled per How do I enable File self-healing on DSM? - Synology Knowledge Center. It can’t be enabled for the whole NAS, only on “shared folders”, but that’s the root folder you create for files during setup anyway so no big deal. It’s even enabled on my docker photostructure and Plex folders, too, though idk whether that’s necessary. Looks like it’s new since DSM 6.1, so data healing hasn’t always been a thing. And it’s goofy to have to specifically enable it, and the UI doesn’t call it “data healing”.
ECC RAM does concern me, though. It would be a bummer if a cosmic ray flipped some bits while writing some photo edits to disc. But there’s still a hefty price premium for ECC and I expect to do very little editing, so the risk is low enough for now to be acceptable.
Am I imagining things, or does PhotoStructure system loading depends on the order in which videos/photos are encountered during sync?
I have not verified this scientifically, but it seems to be that:
Processing a folder full of 1MB jpg images is be limited by disk I/O or PS_MAX_CONCURRENT_IMPORTS. E.g. I can set ...CONCURRENT_IMPORTS=6 and still see my NAS CPU usage below 50% on average
Processing a folder full of videos is limited by CPU ffmpeg processing. I have to drop to ...CONCURRENT_IMPORTS=3 which causes just 2 ffmpeg processes to spawn. Otherwise it spawns 4+ processes, sync seems to stall, and I have to restart to see progress again.
Here is some speculation and suggestion:
PS_FFMPEG_THREADS docs say it’s “threads … per transcode” and “multiple transcodes will be invoked concurrently”. It defaults to 1, which tells me it helps PhotoStructure target PS_CPU_LOAD_PERCENT by running more/fewer transcodes simultaneously.
The stated reason for avoiding 100% CPU utilization is to avoid affecting system responsiveness. While I can totally see that being an issue in general, I have not observed Handbrake or Jellyfin bring the system to a standstill. My understanding is that both use ffmpeg, and I would assume they let ffmpeg pick the appropriate number of threads since it has some smarts about choosing threads based on the codecs. So maybe if PhotoStructure sync is just waiting for video transcodes anyway, it would be more efficient and obvious to run a single ffmpeg process instead of multiple single-threaded ffmpeg processes.
Maybe moving ffmpeg processing from concurrent with imports to after imports would be better? PhotoStructure could do all regular syncs first, then all ffmpeg transcodes serially at the very end with only one ffmpeg process at a time.
I think you could let users test this new workflow with 0 new settings by adding a “-1” or “0” accepted value to PS_FFMPEG_THREADS. Currently it only accepts 1 to 6. Interpreting “-1” or “0” as “Let ffmpeg decide how many threads to use” is tantamount to telling PhotoStructure not to process anything else concurrently, so it makes sense to let that bump ffmpeg processing to the very end of the sync process.
Or maybe don’t even call it part of sync at all. Sync is for everyone, ffmpeg encoding is PLUS-only, and the PLUS-only step happens after regular sync.
Thank you for listening, and I totally understand if this is not worth addressing at this time. Import (ideally) only happens once. While it has an outsize impact on users’ initial impressions, it has a limited long-term impact.
# ----------------------------------
# PS_FFMPEG_THREADS or ffmpegThreads
# ----------------------------------
#
# How many threads should ffmpeg use per transcode?
#
# Note that multiple transcodes will be invoked concurrently. Setting to 0
# will use all available CPU threads, which will grind your poor server into
# dust.
#
# Minimum value: 1
# Maximum value: 6
#
# PS_FFMPEG_THREADS="1"
So: if the user is advanced enough to set PS_MAX_CONCURRENT_IMPORTS – maybe I should just trust them?
I’ll do that.
function _maxConcurrentImports() {
return toGt0(Settings.maxConcurrentImports.valueOrDefault) ?? maxCpus()
}
PS_FFMPEG_THREADS
When set to 1, each in-flight transcode is one ffmpeg thread, which matches what the scheduler already budgets for. So at defaults there shouldn’t be ffmpeg-driven CPU oversubscription.
On the stall itself
If you can reproduce it under default settings, the most useful thing would be a temporal log capture from a window when sync went idle, as well as the sync-report from that time.
Also worth noting: deferring transcoding to a separate post-sync phase has issues – namely, how we would handle presenting those video assets to the browser before ffmpeg does its transcode magicks. Do we wait to mark those video assets as not “shown” until the transcode runs? That could be hours or days!
Also: I really don’t want to make anything I’ve already shipped as LITE be “carpet-pulled” to PLUS. I have a ton of new features in the hopper that I can use to incentivize people to subscribe to PLUS (hopefully they will be compelling!)
how we would handle presenting those video assets to the browser before ffmpeg does its transcode magicks. Do we wait to mark those video assets as not “shown” until the transcode runs? That could be hours or days!
I was assuming that was already figured out for LITE users – but it isn’t, so yes I agree that’s a thorny issue!
I doubt a stall will happen on default settings, as it only runs 1-2 ffmpeg threads which is very close to 75% of CPU utilization. But for sure I will let you know if I am able to capture that.
It occurs to me this could be desirable after all.
If it’s going to be hours or days to process, then it’ll be hours or days regardless of whether it’s done in the middle or end of import. In the extreme case, if PhotoStructure happens to process a folder full of videos first, then you’re going to show barely any assets to users for several days.
For the long processing, I also think a time estimate would be helpful. It’s hard to tell whether it’s stalled not making progress, or it’s just a really long import.
I reverted to default settings about 36hr ago. I’m still at 55,601 assets / 65,436 image files / 1,552 video files. Now it’s one FFmpeg process going instead of several, so 25% CPU usage instead of ~70%.
If it’s going to take days for each video file to import that’s OK, but it would be nice to see some kind of minimal process indicator for each video. A 3-day Handbrake encode showed me progress with total elapsed time so far, and estimate of time remaining. The estimate of time remaining changed a lot, but knowing roughly 3-4days is better than having no idea if it should be 2hr or 1 month.
Thanks for pushing on this @nuk — shipped on main for the next release:
Per-file progress on /sync. Any task running more than ~30 seconds now shows up in a “Currently working on” list under the status banner with filename, percent complete, and an ETA. For long transcodes that’s vacation_4k.mov — 47% — ~8h remain instead of just “55,000 remain”.
Bucketed sync ETA. The top-line estimate used to disappear on mixed image+video runs because the variance guard tripped on the combined pool. It’s now computed per-kind, so it shows up far more often.
Stuck-ffmpeg watchdog. If ffmpeg emits progress ticks but speed stays under 0.01× — what a genuine hang looks like — the transcode is aborted after 5 minutes instead of wedging the queue. Tunable via a new ffmpegStallTimeoutMs setting if your hardware needs a longer window; set to 0 to disable.
Together these answer the “stalled or just slow?” question directly: if you see pct climbing and a real speed number, it’s working. If not, the watchdog will end it and move on.
Allowing PS_FFMPEG_THREADS=0: yes. Small change, low risk. 0 passes straight through to ffmpeg’s -threads 0 (“pick the best count for this codec” heuristic). When set, PhotoStructure automatically serializes video transcodes so you get the single-big-ffmpeg pattern without having to touch PS_MAX_CONCURRENT_IMPORTS — image imports and other non-transcoding work stay concurrent. Shipping in the next release.
Automatically serializing transcodes when ffmpegThreads=0: ok, we’ll try it (but that section of the code is critical to sync not deadlocking – if anything goes south, the new auto-serialize-transcode codepath can be skipped by not setting ffmpegThreads=0.
Post-sync transcoding phase: still a hard no from me, for the reasons I gave above — “asset exists, thumbnail shows, clicking does nothing” is a confusing state that’s worse than “asset hasn’t appeared yet.”
The “sync seems to stall” symptom: partly addressed by the stall-watchdog I mentioned in the reply above — if any specific ffmpeg is actually wedged (progress ticks with speed < 0.01× for 5 min), it’s now aborted rather than blocking a queue slot until the total-duration timeout hits. That’s “hung”, not “slow” — the slow-but-working case isn’t affected.
Observations like yours are genuinely useful — keep them coming.
