skipLibraryOpenLocks

[ericb]

# +------------------------+
# |  skipLibraryOpenLocks  |
# +------------------------+
#
# DANGEROUS: if set, PhotoStructure will not do any library opened-by locking,
# which could result in concurrent library writes, which can cause database
# corruption. If only one system will open this library concurrently, it's OK
# to set this to true.
#
# environment: "PS_SKIP_LIBRARY_OPEN_LOCKS"
#
# skipLibraryOpenLocks = false

I’m using PS for Servers 1.1.0 on Ubuntu 20.04… and slooowly doing an initial import of ~700,000 images on a many-core CPU with a relatively slow spinny-disk. How DANGEROUS is DANGEROUS in this case? Is it worth trying out this option to see if it speeds up the import, or is this really the classic shootOffOwnFoot = true config option in disguise?

[mrm]

Welcome to PhotoStructure, @ericb!

@ericb

How DANGEROUS is DANGEROUS in this case?

Hmm, where 1 is a stubbed toe, and 10 is cosmic calamity, it’s probably a 3.

This setting only comes into play if you are fighting a bug in PhotoStructure’s “opened-by” locking.

Taking a step back: PhotoStructure uses SQLite to persist your library database.

SQLite only supports concurrent read/write on a local filesystem.

If a library is somehow mounted on another computer (either by exporting the local library via SMB/NFS/whatever), or the library was mounted from an external NAS, bad things will happen if PhotoStructure runs sync concurrently on multiple machines against a single library database (basically it will be a case of random-one-in-wins, and data loss is almost guaranteed).

To guard against this, PhotoStructure writes “opened-by” lockfiles into the /path/to/library/.photostructure/opened-by directory. They’re just JSON files: you can open them up to see what’s in them:

mrm@nuc:~/psdc$ cd library/.photostructure/opened-by/
mrm@nuc:~/psdc/library/.photostructure/opened-by$ jq . *.json

{
  "serviceName": "main",
  "createdAt": 1629087713840,
  "updatedAt": 1629228833860,
  "hostname": "c8bed863b72c",
  "systemUID": "p12j-92ar-wbn3-jn07",
  "v": "1.1.0",
  "pid": 25
}

Getting back to your actual issue:

@ericb

and slooowly doing an initial import of ~700,000 images on a many-core CPU with a relatively slow spinny-disk

You should be able to run 5-10 sync-file processes concurrently against an HDD (but it totally depends on the contents of your library, the speed of your disk, and the speed of each core). Keep an eye on iowait (via top or any top-like tool) to see if you’re overwhelming your HDD.

Check the about page (http://localhost:1787/about) to see what PhotoStructure has scheduled by default. If you want to try higher parallelism, you can explicitly set the number of sync-file jobs and image operation threads per job with the maxSyncFileJobs and sharpThreadsPerJob settings: either edit your ~/.config/PhotoStructure/settings.toml or use environment variables: both work.

(like, maxSyncFileJobs=8 ./start.sh)

Lots more details about settings are here.

You may be afflicted by some as-yet-undiscovered bug in PhotoStructure, too: if you crank up parallelism and don’t see thumbnails whizzing by, set LOG_LEVEL=debug and send me your logfiles.

[ericb]

Thanks - in my case, I think I’m a victim of too many CPU cores (20, which shows up as 40 in Linux due to hyperthreading). So I had to decrease my max sync-file procs down to 8 down from the defaults. I was also trying to go all the way to 24 at first, since I had plenty of cores to burn… but after experimenting a bit, 6 sync-files works fine with no problems, 8 sync-files is the hairy edge where I get an occasional SQLite locked/busy error, and anything more than that and basically all I get are a constant stream of SQLite locked errors until things give up and exit.

Other than moving to, say, postgres… at least for initial imports, any thoughts on an approach like:

• Divide up the assets sync found to import into maxSyncFileJobs number of groups
• Use a dedicated sqlite db per sync-file instance that that instance exclusively writes to
• Combine all the sqlite dbs together into one big final one at the very end of the initial import (by one process, so probably very CPU intensive for a bit, but shouldn’t have any write conflicts).

? Or any sort of variation on that theme to shard up the work and combine it later.

By the time my import finishes with maxSyncFileJobs set to 8 in another… six days or so?? It’ll probably be a moot point, but could help others in the future. Or again… maybe allow postgres as a config option for PS for servers? No need for any of the “opened-by” locks at all, then, as a bonus.

[mrm]

Is you library database sitting on an HDD, or on SSD?

My test machine can run 18 concurrent sync-file, but only on fast SSD.

Hybrid PhotoStructure libraries

PhotoStructure offers a lot of flexibility with how you lay out your photos and videos. Initial library setup When you’re initially setting up PhotoStructure, you can set up the following: You can put your library directory on either a local or remote filesystem (if you’re on Windows, though, map your network drive to a letter first: heavy disk I/O against UNC paths can be flaky) You can choose to leave your existing photos and videos where they are already, or automatically organize t…

That said, it’d be nice to support Postgres or MariaDB as an option for very large libraries or users (like you) that need heavy concurrency support.

Another possible solution which I haven’t tested yet would be switching from process-threading (which is what PhotoStructure currently uses) to web worker threading. Sharing the same may avoid some amount of lock contention.

@ericb

any sort of variation on that theme to shard up the work and combine it later

This is an interesting idea! Imports would look “bursty” to users, though, as batches of work were merged back into the final library.

Shuffling around what the work is to “import a file” to avoid concurrent db access might be a (much) simpler approach (and avoid map/reduce batches).

Currently importing a file looks roughly like

1. Walk though directories and look for files to import.

Then, for each file that passes import filters:

2. Is the file URI already in AssetFile? If so, is the size and largest-mtime-of-file-or-sidecar match the current value in the db? If so, we’re done with that file.

3. If not, run a series of db queries and file operations (including SHA and image hashing) to try to find any prior row in the Asset table. If no prior asset matches, add a new Asset and insert a new AssetFile

4. For the referenced Asset, find the “best” variation, and ensure the previews and transcoded video is in order.

The time-consuming operations are

1. File SHA (10ms-10s, depends on the size of the file and speed of the disk)
2. Image hash (100ms-1.5s, depending on image resolution and CPU speed)
3. Preview generation (100ms-5s, depending on image resolution and CPU speed)
4. Transcoding (can be .2x-4x the duration of the video, depending on speed of the disk and CPU speed and count)

DB operations typically complete in between a millisecond to 10ms (SQLite is fast!), as long as there isn’t any lock contention.

I’d like to fix this. I’ll think about how I can adjust the sync workloads.

[ericb]

My database is on a ZFS zpool with some ye olde spinning disks backing it, but I did an experiment where I just put the database on a RAM disk instead and there wasn’t really much difference - the problem, for me at least, really just seems to be massive SQLite write contention. I’ve tried fiddling around with the db timeout options, too. I can definitely make things worse with those… but not too much better.

Imports would indeed look very bursty for users with something like I described, but for someone like me with hundreds of thousands of photos doing an initial import… I’d rather it be done in half the time than look good while it’s working.

In terms of some of the dupe-detection work, at least for exact duplicates, there are a few general-purpose de-dupe tools out there (rmlint, jdupes, etc.) that each have tried out different techniques around hashing to speed things up for the cases that most files aren’t exact dupes (probably the typical case). E.g. use a much faster hash than SHA1 (xxHash, clhash, etc.) as even with hardware support the SHA family isn’t blazing fast - and you don’t need a cryptographic hash family for simple de-dupe work. If hashes differ, then clearly the files are different. If they match, then go ahead and do a byte-by-byte compare of the files. There’s also been experimentation with just reading in the first, say, page-size worth of bytes of both files and hashing or doing a byte-comparison rather than reading in the whole file from disk for faster detection of files that are definitely different, with mixed results IIRC (since Linux at least is pretty aggressive at doing read-aheads and filling up the page cache anyway, even if you never use what it read-aheads for you).

I know checking the db for a matching hash should just be a read operation & not conflict too much, but would it maybe be worth sticking a bloom filter in front of those reads, either in sqlite itself or just in-memory in each sync-file app, to avoid looking for hashes that definitely aren’t going to be found? (Either avoid all such lookups if you’re still using a single central sqlite db instance, or avoid at least some of them if using an in-memory per-sync-file job bloom filter).

I’m not going to be much help with ideas on speeding up preview generation or transcoding - I think those have already been pretty reasonably optimized!

I’m probably in a small minority of users with this sort of issue in the first place, so while performance issues are fun to work on… other asked-for features might be worth more development time, honestly. I’m just spitballing while waiting for my massive import!

[mrm]

I appreciate the spitballing!

After I bang out fave/hide/exclude/delete, I think moving code around to only require sync and web to load db instances may really help on large core, slow I/O, and larger (200k+) libraries.

[bdillahu]

Happened to be looking at an unrelated system involving hashes and discovered they were moving to blake3 for performance reasons… not really my world, but thought I would toss it out.

The chart sure looks fast

GitHub - BLAKE3-team/BLAKE3: the official Rust and C implementations of the...