I have lots to learn ā clicking the trash can works fine, and rotating looks like it works but it complains that ādarn that didnt workā ā it may be that some of my original photos are read-only.
In general I should figure out how to make a master copy and quit referencing the original photos ā entirely ā indeed i hope to phase out the old computer and its old hard drives completely.
PhotoStructure will try to fix the orientation of all asset variations, and only if it succeeds for all of them, does it report āsuccessā.
Unfortunately, Iām finding that the rotation operation timeout on the frontend is too impatient for some of my assets on my really slow NAS (especially if it has a couple copies), so I need to rethink what constitutes a āsuccessā versus a āfailureā for this operation.
Thanks for the insights. Is there a writeup somewhere that describes a method to migrate away from my original mess of photos? What I mean is original - photostructure-A ā photostructureBā¦ Once I am happy with A and B, I would sever the references/mention of the original. The original is a rough, messy collection of nearly 20 years of photos.
Another related question would be to migrating the photostructure to somewhere like digitalocean. thank you
I installed beta 1 of Windows desktop. It auto started but then went to error "ChildService(sync). on Stdout():::::::::.
Database starts repairing to 100% and then program stops. I restarted with database repair again to 100% and again fail. This has occurred several times with no successful start.
I was trying PhotoStructure and figured with the new features I could just go with the beta version.
Three things I noticed. First:
I had a docker-compose based installation of the stable channel up and running, configured for a specific import path with āNo thanks, I like my photos and videos where they already are.ā selected
I upgraded to the beta channel and restarted the application and it was running correctly
I then started a yearly subscription and returned to my instance
The import path disappeared and āYes, please copy my photos and videos into my PhotoStructure Library.ā was selected
I do not know if this configuration change came through the upgrade or through the plus subscription - but it was unexpected
Second:
Deleting an image is very unresponsive. When I click on the trash can icon seemingly nothing happens. The request then takes 7 seconds to complete (according to the network tab of the developer tools) and only then I get any response (but deleting works). IMO it would be better to have at least some progress indicator or even an asynchronous action which starts deletion in the background and immediately gives feedback.
Same goes for marking an image as an favourite btw.
I would assume this is because my pictures are on a rather slow drive?
Please let me know if there is anything I can provide you with to help debugging.
Third:
The icon for archiving images somehow looks like a download icon to me. Initially I wanted to download a file through that button. Not sure where this resemblence comes from, but most likely the downwards arrow is the thing that reminds me of a download button.
Thanks for the work btw, Iām really liking this so far!
Hello, and welcome to PhotoStructure! Thanks for taking the time to share feedback!
This is surprising: settings shouldnāt just ārevertā to defaults, but the settings infrastructure is, uh, tactifully speaking, involved.
For example, if you set the environment variable PS_COPY_ASSETS_TO_LIBRARY in your docker configuration to true, for example, the settings page will show the current environment setting value, and if you change it on the settings page, it will persist that value to your libraryās settings.toml, but because environment variable values always win, the value will stay true.
We can actually find out: I keep backups of the settings files in that directory. if you cd ...path/to/your/library/.photostructure ; grep copyAssetsToLibrary *settings* that would be enlightening (please DM or email me that if you want me to see whatās going on!)
Harumph! Thatās no good! This operation should be instantaneous: weāre just writing a timestamp into a row in your database (if you speak SQL, weāre just doing an UPDATE AssetFile SET deletedAt= :now WHERE id = :assetFileId). If this is taking a long time, somethingās up with the disk I/O on your library volume. Is a backup happening? Is it going through a RAID rebuild? Is PhotoStructure sync doing an import? (Sync runs reniced to low priority, but on slow remote disks, it still can overwhelm the disk controller, as renice only impacts the CPU scheduling, not I/O scheduling).
If you click the main menu, click āpause syncā, wait a couple seconds, and then delete is suddenly fast, it means weāre running too many concurrent sync-file processes to keep the db read-performant. You can control this by setting PS_CPU_LOAD_PERCENT=25 (it defaults to 75) or PS_MAX_SYNC_FILE_JOBS=2 (more detailed instructions are here).
This is actually a bit more involved: when you an asset, we not only write to the db (UPDATE Asset SET rating = :likeRating WHERE id = :assetId), but we write that metadata to each asset file variation (either in a sidecar or to the file itself, depending on your configuration, see this section of settings).
My library is super messy: I have some assets with 20 variations/duplicates. The metadata writing happens in parallel, but even then, it can take a couple seconds for 20 files on several NASes to get written to.
Ah, I see that! Iām hoping that when I add an actual download link to the right of the rotate icon, itāll be distinct enough from archive to not confuse:
(the archive icon is actually the material standard, but if you have a better icon, please share: good icons are hard!)
I donāt know if this issue was from the latest beta or earlier, however none of my video files are importing. I tried to play around with the command line tools a bit to see if I could see what is going on, but this file was recently shot but canāt seem to get imported. The photo files seem to be okay.
Ubuntu 21.10 PhotoStructure for Desktops. Fresh install of v 2.0.0-beta.1
Where was the problem?
Import sync incomplete and errors and corruption follow attempt to finish it.
What did you expect PhotoStructure to do?
Import all of a de-duped test folder of 2055 images
What did it actually do?
Imported 1316 of 2055. Tried to restart sync, nothing. Restarted application and was met with an sqlite error (see attached screenshot). Restarted again and was met with this error:
Restarted again and was greeted with corruption:
Please attach screenshots/screen recording/example images if applicable. Feel free to email support@photostructure.com directly if anything is privacy-related.
When youāre shelled into your docker container, remember that paths are from the bind root destination, not the path thatās valid on your host machine.
As an example, if your docker command had -v ~/pictures:/pics, youād want to do ./photostructure info /pics/Ryan_S21/20211025_190709.mp4
O ok thanks, that helped. I was able to run info on the mp4 file, and it was able to list out all of the data for the file. However I could not locate it in Photostructure by searching for any of the relavent tags or types or anything. I then did the sync-file command, it ran and then now I am able to see the video file in photostructure. Meanwhile, I can view all of the jpg images that were taken and synced at the same time, only the mp4 videos are missing.
I have now reset my library following your instructions; it will take days to import all the files. I can see progress from the About page but there is no progress indicator on the main page. That would be useful to know if the program is actually running and how far it has progressed.
BTW I did email logs to you for your information about the database problem
As mentioned I am rebuilding my database on windows 10 desktop with 2 b1. There is no indication of the progress on the main page and the About dialogue box has no close button; I have to close it via the task bar.
When I access via web access on an Android device I see progress and the About box has a close cross
sorry for the late reply. Since I had my complete NAS on a system HDD of unknown ages (very old), I took the opportunity and got a new SSD for my system. Took me a while to get everything up and running again.
We can actually find out: I keep backups of the settings files in that directory. if you cd ...path/to/your/library/.photostructure ; grep copyAssetsToLibrary *settings* that would be enlightening (please DM or email me that if you want me to see whatās going on!)
settings.toml:# | copyAssetsToLibrary |
settings.toml:copyAssetsToLibrary = false
settings.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211026165913.toml:# | copyAssetsToLibrary |
old/settings-20211026165913.toml:copyAssetsToLibrary = false
old/settings-20211026165913.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211026115705.toml:# | copyAssetsToLibrary |
old/settings-20211026115705.toml:copyAssetsToLibrary = false
old/settings-20211026115705.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211025190007.toml:# | copyAssetsToLibrary |
old/settings-20211025190007.toml:copyAssetsToLibrary = false
old/settings-20211025190007.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211026112841.toml:# | copyAssetsToLibrary |
old/settings-20211026112841.toml:copyAssetsToLibrary = true
old/settings-20211026112841.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
old/settings-20211025185837.toml:# | copyAssetsToLibrary |
old/settings-20211025185837.toml:# copyAssetsToLibrary = true
old/settings-20211025185837.toml:# "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
Not sure if that helps
Regarding the speed of the delete / favourite operations:
Sadly the SSD did not make any difference in the speed of the operations, they were still very slow with the favourite one sometimes even just throwing an timeout.
My library was scattered around 2 HDDs and mounted using mergerfs. I though that might be a problem and moved everything and got rid of the mergerfs but it did not make any difference.
Finally I tried starting PhotoStructure not as a docker container but on the system directly (the āfor nodeā installation method) and that was it: The operations are now super fast.
So, it was somehow related to the docker method.
I noticed that sometimes other things were slow in docker, too, so that makes kind of sense.
Iām fine with using PhotoStructure that way, so no further support needed for me, but if I can help to find the issue Iāll be happy to do that.
If it helps:
Iām on OpenMediaVault stable 5.6.13 (Debian 10 buster)
Docker version 20.10.10, build b485636
Not encrypted system disk
Encrypted data disks
library + cache + config lives on the non-encrypted system disk
The actual media files are on an encrypted data disk
Thanks for those details! Iāll review them when I get a chance.
FWIW Iām working on a āreduced db contentionā mode for sync to run under, where all database writes go through one process. This means only sync and web will be writing to the db, rather than sync, web, and all the sync-file processes. The prior approach is fine on smaller libraries and computers with only a few CPU cores, or with libraries stored on very fast SSDs, but high db write contention has proven problematic on computers with high core counts and larger libraries (and caused large latencies for the webservice).
ā¦but high db write contention has proven problematic on computers with high core counts and larger libraries (and caused large latencies for the webservice).
This is probably the cause of issues Iāve been having lately. Though the āreduced contention modeā sounds like it could come with performance issues of its own.
Hereās a question - must all operations take place on a single database? Or could you have one or more āstagingā databases used during sync operations which occasionally get merged into the main database with a single large write operation? If so the main database could be left alone to serve content for the webservice most of the time.
This is an interesting idea, and might be great feature in the future to support merging libraries!
Merging is actually pretty expensive due to deduplication heuristics and wasted work creating previews that result in just getting deleted. Itād also be potentially super confusing to the user as several copies of a given photo show up, and then disappear as PhotoStructure re-aggregates.
This merging code is actually already implemented, and is what runs the ārebuildā process: rebuild assumes the state of the library is not properly aggregated, re-aggregates all assets, and then verifies that all preview image sets are correct.
My current hypothesis is that because database operations are actually fairly cheap (say, 1-10ms), if I single-thread the ops, and send all non-trivial work to threads, I can eke out the best performance for a sync job. This is what Iāve been working on for the past 2 weeks (itās the first time PhotoStructure has used worker_threads, so getting that to work everywhere seamlessly and reliably has taken a bit of work), but all core tests are passing again, and Iām banging out the last couple issues in the library test suite now.