PhotoStructure will try to fix the orientation of all asset variations, and only if it succeeds for all of them, does it report âsuccessâ.
Unfortunately, Iâm finding that the rotation operation timeout on the frontend is too impatient for some of my assets on my really slow NAS (especially if it has a couple copies), so I need to rethink what constitutes a âsuccessâ versus a âfailureâ for this operation.
Thanks for the update!
Thanks for the insights. Is there a writeup somewhere that describes a method to migrate away from my original mess of photos? What I mean is original - photostructure-A â photostructureB⌠Once I am happy with A and B, I would sever the references/mention of the original. The original is a rough, messy collection of nearly 20 years of photos.
Another related question would be to migrating the photostructure to somewhere like digitalocean. thank you
I installed beta 1 of Windows desktop. It auto started but then went to error "ChildService(sync). on Stdout():::::::::.
Database starts repairing to 100% and then program stops. I restarted with database repair again to 100% and again fail. This has occurred several times with no successful start.
Anything I can do or do I reconfigure again?
Chris
Hello from a new user,
I was trying PhotoStructure and figured with the new features I could just go with the beta version.
Three things I noticed. First:
Second:
Deleting an image is very unresponsive. When I click on the trash can icon seemingly nothing happens. The request then takes 7 seconds to complete (according to the network tab of the developer tools) and only then I get any response (but deleting works). IMO it would be better to have at least some progress indicator or even an asynchronous action which starts deletion in the background and immediately gives feedback.
Same goes for marking an image as an favourite btw.
I would assume this is because my pictures are on a rather slow drive?
Please let me know if there is anything I can provide you with to help debugging.
Third:
The icon for archiving images somehow looks like a download icon to me. Initially I wanted to download a file through that button. Not sure where this resemblence comes from, but most likely the downwards arrow is the thing that reminds me of a download button.
Thanks for the work btw, Iâm really liking this so far!
Oof, apologies! Can you email me any logs you have in %APPDATA%\PhotoStructure\logs? They might explain what went awry.
If you want to completely reset your library (imports will need to start from scratch!), I just wrote up these instructions.
Hello, and welcome to PhotoStructure! Thanks for taking the time to share feedback!
This is surprising: settings shouldnât just ârevertâ to defaults, but the settings infrastructure is, uh, tactifully speaking, involved.
For example, if you set the environment variable PS_COPY_ASSETS_TO_LIBRARY in your docker configuration to true, for example, the settings page will show the current environment setting value, and if you change it on the settings page, it will persist that value to your libraryâs settings.toml, but because environment variable values always win, the value will stay true.
We can actually find out: I keep backups of the settings files in that directory. if you cd ...path/to/your/library/.photostructure ; grep copyAssetsToLibrary *settings* that would be enlightening (please DM or email me that if you want me to see whatâs going on!)
Harumph! Thatâs no good! This operation should be instantaneous: weâre just writing a timestamp into a row in your database (if you speak SQL, weâre just doing an UPDATE AssetFile SET deletedAt= :now WHERE id = :assetFileId). If this is taking a long time, somethingâs up with the disk I/O on your library volume. Is a backup happening? Is it going through a RAID rebuild? Is PhotoStructure sync doing an import? (Sync runs reniced to low priority, but on slow remote disks, it still can overwhelm the disk controller, as renice only impacts the CPU scheduling, not I/O scheduling).
If you click the main menu, click âpause syncâ, wait a couple seconds, and then delete is suddenly fast, it means weâre running too many concurrent sync-file processes to keep the db read-performant. You can control this by setting PS_CPU_LOAD_PERCENT=25 (it defaults to 75) or PS_MAX_SYNC_FILE_JOBS=2 (more detailed instructions are here).
This is actually a bit more involved: when you 
an asset, we not only write to the db (UPDATE Asset SET rating = :likeRating WHERE id = :assetId), but we write that metadata to each asset file variation (either in a sidecar or to the file itself, depending on your configuration, see this section of settings).
My library is super messy: I have some assets with 20 variations/duplicates. The metadata writing happens in parallel, but even then, it can take a couple seconds for 20 files on several NASes to get written to.
Ah, I see that! Iâm hoping that when I add an actual download link to the right of the rotate icon, itâll be distinct enough from archive to not confuse:
(the archive icon is actually the material standard, but if you have a better icon, please share: good icons are hard!)
I donât know if this issue was from the latest beta or earlier, however none of my video files are importing. I tried to play around with the command line tools a bit to see if I could see what is going on, but this file was recently shot but canât seem to get imported. The photo files seem to be okay.
/ps/app # ./photostructure sync-file /root/pictures/Ryan_S21/20211025_190709.mp4
{"path":"/root/pictures/Ryan_S21/20211025_190709.mp4","rejected":["File was deleted"],"elapsedMs":183}
{"ready":true}
/ps/app # ./photostructure info ~/pictures/Ryan_S21/20211025_190709.mp4
[
{
nativePath: '/root/pictures/Ryan_S21/20211025_190709.mp4',
ignoredBecause: [ 'notExists' ]
}
]Ubuntu 21.10 PhotoStructure for Desktops. Fresh install of v 2.0.0-beta.1
Where was the problem?
Import sync incomplete and errors and corruption follow attempt to finish it.
What did you expect PhotoStructure to do?
Import all of a de-duped test folder of 2055 images
What did it actually do?
Imported 1316 of 2055. Tried to restart sync, nothing. Restarted application and was met with an sqlite error (see attached screenshot). Restarted again and was met with this error:
Restarted again and was greeted with corruption:
Please attach screenshots/screen recording/example images if applicable. Feel free to email support@photostructure.com directly if anything is privacy-related.
I think that path may be incorrect?
When youâre shelled into your docker container, remember that paths are from the bind root destination, not the path thatâs valid on your host machine.
As an example, if your docker command had -v ~/pictures:/pics, youâd want to do ./photostructure info /pics/Ryan_S21/20211025_190709.mp4
Ugh, apologies for the most unpleasant greeting. What disk is you library on? HDD, SSD, or a remote volume?
O ok thanks, that helped. I was able to run info on the mp4 file, and it was able to list out all of the data for the file. However I could not locate it in Photostructure by searching for any of the relavent tags or types or anything. I then did the sync-file command, it ran and then now I am able to see the video file in photostructure. Meanwhile, I can view all of the jpg images that were taken and synced at the same time, only the mp4 videos are missing.
Yeah: I need to add filename and a bunch of other metadata to the search index.
You can force PhotoStructure to import a specific file and the logs may explain whatâs going on:
./photostructure sync-file --debug /pics/Ryan_S21/20211025_190709.mp4
(feel free to send me the output if you want me to decipher whatâs going on)
PhotoStructure files are on a local SSD, original photo and video directories are on a local HDD (Ironwolf 4tb 5900rpm)
I have now reset my library following your instructions; it will take days to import all the files. I can see progress from the About page but there is no progress indicator on the main page. That would be useful to know if the program is actually running and how far it has progressed.
BTW I did email logs to you for your information about the database problem
As mentioned I am rebuilding my database on windows 10 desktop with 2 b1. There is no indication of the progress on the main page and the About dialogue box has no close button; I have to close it via the task bar.
When I access via web access on an Android device I see progress and the About box has a close cross
Hello again,
sorry for the late reply. Since I had my complete NAS on a system HDD of unknown ages (very old), I took the opportunity and got a new SSD for my system. Took me a while to get everything up and running again.
We can actually find out: I keep backups of the settings files in that directory. if you
cd ...path/to/your/library/.photostructure ; grep copyAssetsToLibrary *settings*that would be enlightening (please DM or email me that if you want me to see whatâs going on!)
settings.toml:# | copyAssetsToLibrary |
settings.toml:copyAssetsToLibrary = false
settings.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211026165913.toml:# | copyAssetsToLibrary |
old/settings-20211026165913.toml:copyAssetsToLibrary = false
old/settings-20211026165913.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211026115705.toml:# | copyAssetsToLibrary |
old/settings-20211026115705.toml:copyAssetsToLibrary = false
old/settings-20211026115705.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211025190007.toml:# | copyAssetsToLibrary |
old/settings-20211025190007.toml:copyAssetsToLibrary = false
old/settings-20211025190007.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211026112841.toml:# | copyAssetsToLibrary |
old/settings-20211026112841.toml:copyAssetsToLibrary = true
old/settings-20211026112841.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211025185837.toml:# | copyAssetsToLibrary |
old/settings-20211025185837.toml:# copyAssetsToLibrary = true
old/settings-20211025185837.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
Not sure if that helps 
Regarding the speed of the delete / favourite operations:
Sadly the SSD did not make any difference in the speed of the operations, they were still very slow with the favourite one sometimes even just throwing an timeout.
My library was scattered around 2 HDDs and mounted using mergerfs. I though that might be a problem and moved everything and got rid of the mergerfs but it did not make any difference.
Finally I tried starting PhotoStructure not as a docker container but on the system directly (the âfor nodeâ installation method) and that was it: The operations are now super fast.
So, it was somehow related to the docker method.
I noticed that sometimes other things were slow in docker, too, so that makes kind of sense.
Iâm fine with using PhotoStructure that way, so no further support needed for me, but if I can help to find the issue Iâll be happy to do that.
If it helps:
Iâm on OpenMediaVault stable 5.6.13 (Debian 10 buster)
Docker version 20.10.10, build b485636
Not encrypted system disk
Encrypted data disks
library + cache + config lives on the non-encrypted system disk
The actual media files are on an encrypted data disk
Thanks a lot!
Thanks for those details! Iâll review them when I get a chance.
FWIW Iâm working on a âreduced db contentionâ mode for sync to run under, where all database writes go through one process. This means only sync and web will be writing to the db, rather than sync, web, and all the sync-file processes. The prior approach is fine on smaller libraries and computers with only a few CPU cores, or with libraries stored on very fast SSDs, but high db write contention has proven problematic on computers with high core counts and larger libraries (and caused large latencies for the webservice).
âŚbut high db write contention has proven problematic on computers with high core counts and larger libraries (and caused large latencies for the webservice).
This is probably the cause of issues Iâve been having lately. Though the âreduced contention modeâ sounds like it could come with performance issues of its own.
Hereâs a question - must all operations take place on a single database? Or could you have one or more âstagingâ databases used during sync operations which occasionally get merged into the main database with a single large write operation? If so the main database could be left alone to serve content for the webservice most of the time.
This is an interesting idea, and might be great feature in the future to support merging libraries!
Merging is actually pretty expensive due to deduplication heuristics and wasted work creating previews that result in just getting deleted. Itâd also be potentially super confusing to the user as several copies of a given photo show up, and then disappear as PhotoStructure re-aggregates.
This merging code is actually already implemented, and is what runs the ârebuildâ process: rebuild assumes the state of the library is not properly aggregated, re-aggregates all assets, and then verifies that all preview image sets are correct.
My current hypothesis is that because database operations are actually fairly cheap (say, 1-10ms), if I single-thread the ops, and send all non-trivial work to threads, I can eke out the best performance for a sync job. This is what Iâve been working on for the past 2 weeks (itâs the first time PhotoStructure has used worker_threads, so getting that to work everywhere seamlessly and reliably has taken a bit of work), but all core tests are passing again, and Iâm banging out the last couple issues in the library test suite now.
I had imagined it in a way that would be transparent to the user, at the database level rather than library level. Under the assumption that frequent writing to the main database is the problematic part, the deduplication would take place in the secondary database using the main database only as a source to read from. The UI would not need to expose that a smaller temporary database is being used for writing instead of the main database directly.
My current hypothesis is that because database operations are actually fairly cheap (say, 1-10ms), if I single-thread the ops, and send all non-trivial work to threads, I can eke out the best performance for a sync job.
I see, I was under the impression that you were starting to reach the limits of what a single SQLite db can handle at a reasonable speed. If the operations can actually be much cheaper under those conditions then thereâs indeed a lot less need for a solution like the above. Good luck wrangling with the worker_threads - I look forward to seeing how much it might help.