As mentioned I am rebuilding my database on windows 10 desktop with 2 b1. There is no indication of the progress on the main page and the About dialogue box has no close button; I have to close it via the task bar.
When I access via web access on an Android device I see progress and the About box has a close cross
Hello again,
sorry for the late reply. Since I had my complete NAS on a system HDD of unknown ages (very old), I took the opportunity and got a new SSD for my system. Took me a while to get everything up and running again.
We can actually find out: I keep backups of the settings files in that directory. if you
cd ...path/to/your/library/.photostructure ; grep copyAssetsToLibrary *settings*that would be enlightening (please DM or email me that if you want me to see whatās going on!)
settings.toml:# | copyAssetsToLibrary |
settings.toml:copyAssetsToLibrary = false
settings.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211026165913.toml:# | copyAssetsToLibrary |
old/settings-20211026165913.toml:copyAssetsToLibrary = false
old/settings-20211026165913.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211026115705.toml:# | copyAssetsToLibrary |
old/settings-20211026115705.toml:copyAssetsToLibrary = false
old/settings-20211026115705.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211025190007.toml:# | copyAssetsToLibrary |
old/settings-20211025190007.toml:copyAssetsToLibrary = false
old/settings-20211025190007.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211026112841.toml:# | copyAssetsToLibrary |
old/settings-20211026112841.toml:copyAssetsToLibrary = true
old/settings-20211026112841.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211025185837.toml:# | copyAssetsToLibrary |
old/settings-20211025185837.toml:# copyAssetsToLibrary = true
old/settings-20211025185837.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
Not sure if that helps 
Regarding the speed of the delete / favourite operations:
Sadly the SSD did not make any difference in the speed of the operations, they were still very slow with the favourite one sometimes even just throwing an timeout.
My library was scattered around 2 HDDs and mounted using mergerfs. I though that might be a problem and moved everything and got rid of the mergerfs but it did not make any difference.
Finally I tried starting PhotoStructure not as a docker container but on the system directly (the āfor nodeā installation method) and that was it: The operations are now super fast.
So, it was somehow related to the docker method.
I noticed that sometimes other things were slow in docker, too, so that makes kind of sense.
Iām fine with using PhotoStructure that way, so no further support needed for me, but if I can help to find the issue Iāll be happy to do that.
If it helps:
Iām on OpenMediaVault stable 5.6.13 (Debian 10 buster)
Docker version 20.10.10, build b485636
Not encrypted system disk
Encrypted data disks
library + cache + config lives on the non-encrypted system disk
The actual media files are on an encrypted data disk
Thanks a lot!
Thanks for those details! Iāll review them when I get a chance.
FWIW Iām working on a āreduced db contentionā mode for sync to run under, where all database writes go through one process. This means only sync and web will be writing to the db, rather than sync, web, and all the sync-file processes. The prior approach is fine on smaller libraries and computers with only a few CPU cores, or with libraries stored on very fast SSDs, but high db write contention has proven problematic on computers with high core counts and larger libraries (and caused large latencies for the webservice).
ā¦but high db write contention has proven problematic on computers with high core counts and larger libraries (and caused large latencies for the webservice).
This is probably the cause of issues Iāve been having lately. Though the āreduced contention modeā sounds like it could come with performance issues of its own.
Hereās a question - must all operations take place on a single database? Or could you have one or more āstagingā databases used during sync operations which occasionally get merged into the main database with a single large write operation? If so the main database could be left alone to serve content for the webservice most of the time.
This is an interesting idea, and might be great feature in the future to support merging libraries!
Merging is actually pretty expensive due to deduplication heuristics and wasted work creating previews that result in just getting deleted. Itād also be potentially super confusing to the user as several copies of a given photo show up, and then disappear as PhotoStructure re-aggregates.
This merging code is actually already implemented, and is what runs the ārebuildā process: rebuild assumes the state of the library is not properly aggregated, re-aggregates all assets, and then verifies that all preview image sets are correct.
My current hypothesis is that because database operations are actually fairly cheap (say, 1-10ms), if I single-thread the ops, and send all non-trivial work to threads, I can eke out the best performance for a sync job. This is what Iāve been working on for the past 2 weeks (itās the first time PhotoStructure has used worker_threads, so getting that to work everywhere seamlessly and reliably has taken a bit of work), but all core tests are passing again, and Iām banging out the last couple issues in the library test suite now.
I had imagined it in a way that would be transparent to the user, at the database level rather than library level. Under the assumption that frequent writing to the main database is the problematic part, the deduplication would take place in the secondary database using the main database only as a source to read from. The UI would not need to expose that a smaller temporary database is being used for writing instead of the main database directly.
My current hypothesis is that because database operations are actually fairly cheap (say, 1-10ms), if I single-thread the ops, and send all non-trivial work to threads, I can eke out the best performance for a sync job.
I see, I was under the impression that you were starting to reach the limits of what a single SQLite db can handle at a reasonable speed. If the operations can actually be much cheaper under those conditions then thereās indeed a lot less need for a solution like the above. Good luck wrangling with the worker_threads - I look forward to seeing how much it might help.