Initial import of library fails

Support
, ,
1

I’d hoped the new betas would improve my import woes, but I still can’t get my photos imported.

Import fails about 1/4 of the way into a 31,000 image library.
On viewing the logs, everything gets enqueued and these two types of errors pop up on processing:

Timeout img.ValidFile cache after 28522ms

BatchCluster has ended, cannot enqueue {“id”:510,“method”:“buildAssetPreviews_”,“args”:[{“assetId”:8792,“assetFiles”:[{“$ctor”:“models.AssetFile”,“id”:11411,“assetId”:8792,“shown”:false,“uri”:“psfile://2tPi1zFoW/photos/2012/Botswana/E-M5/Botswana-E-M5-08544.ORF”,"mou…:false,“recountAllTags”:false}]}

I tried to start from scratch again with a full reinstall and using just ONE of the same folders that was successfully indexed before in its entirety. MY intention was then to add folder-by-folder to isolate any issues related to the images. Only 376 of 2000 photos imported despite the same setup completely indexing the same folder before!

I really want this app to succeed because it has the best UI out there and the tag extraction is amazing. However, none of that matters if the app can’t import my photos. PhotoStructure has ever successfully imported my photos despite trying stable/beta/alpha, desktop and docker, windows and Ubuntu.

Version|2.1.0-alpha.7|
Edition|PhotoStructure for Desktops (alpha channel)|
plan|plus|
OS|Pop!_OS 22.04 LTS on x64|
Electron|19.0.10|
Video support|FFmpeg 4.4.2-0ubuntu0.22.04.1|
HEIF support|/usr/bin/heif-convert|
Free memory|26 GB / 34 GB|
CPUs|12 × AMD Ryzen 5 3600X 6-Core Processor|

2

Sorry to hear that sync is misbehaving for you! I understand how frustrating this is.

For almost every operation, PhotoStructure has either a hard timeout or a watchdog to prevent sync from getting “stuck” on problematic files, directories, or volumes.

The error you mentioned is from the image validation code failing to complete in 28 seconds for one of your .ORF files. If you’ve got a slow/remote HDD or busy CPU, it’s possible it wasn’t be able to raster the ORF to a TIFF in 28 seconds. (My 15 year old windows laptop which I use for benchmarking timeouts renders large ORFs in ~8 seconds, when otherwise idle, fwiw).

That 28 seconds is from the setting commandTimeoutMs, which defaults to 25 seconds. You may want to bump that up to 1 minute (set commandTimeoutMs=1m). You can also disable jpeg, raw, and video file validations if you’d like: see validateJpegImages, validateRawImages, and validateVideos settings explained here.

I’ll bump the isValidImage timeout to be a minute as the default in the next build.

If this doesn’t fix stuff, try either or both of these options:

  1. Host your library database on a local SSD–this should help SQLite db concurrency. Your originals can live elsewhere.

  2. Try disabling concurrency by setting PS_CPU_LOAD_PERCENT=0. This is obviously just a stopgap solution, but if things suddenly work, that’s valuable information (for me).

If you still have issues, please send me recent sync reports and debug logs, if possible.