Released v2.0.0-beta.1

[mrm]

Salutations, brave beta testers! Version 2.0 is now feature-complete.

This is a beta build

Standard beta disclaimers apply: read more here.

Remember that library upgrades are a one-way street.

You may want to run this on a test library first.

Where can I get the beta installer?

• For PhotoStructure for Desktops: https://photostructure.com/go/beta

• For PhotoStructure for Node: git checkout beta.
  To return to the stable release, git stash -u ; git checkout main

• For PhotoStructure for Docker: change the image to photostructure/server:beta. Remember to docker-compose pull or docker pull before restarting the container!

New features

If you’re coming from v1.1, here are some of the more obvious new features:

• Initial support for liking, archiving, removing, and deleting assets!

How to archive, remove, and delete photos and videos in your PhotoStructure...

• Added new “copy path” option in the asset info panel.

• Full screen mode! Tap the toggle button in the header or the f key to toggle fullscreen mode.

• Client and server code can now show informational and error messages (or “toasts” to let you undo some actions, notify you of network problems, tell you when you need to upgrade, and more.

• Rebuilt UX view persistence to be dramatically faster thanks to proper component keying and smarter fetch heuristics: clicking the “back” button is instantaneous, and prior scroll positions are correctly retained.

• Asset rendering at high resolutions is now dramatically faster. Prior builds didn’t trust the database for rotation metadata, which led to a (potentially expensive) exif parsing step before streaming the original asset to the browser.

• On computers with at least 6 CPUs, preview images are now optimized with mozjpeg defaults, which reduces images by up to 30% with minimal reduction in visible quality. See the new jpegMinimized setting for details.

Bug fixes since v2.0.0-alpha.1

• Sync should now automatically pick up new files. There were a couple issues:

  • v1.0 changed how PhotoStructure keeps track of sync progress, and prior progress metadata would “trick” sync into thinking there was nothing to do for subsequent runs. New sync jobs started after a completed job will start with new metadata, rather than adopting the prior run metadata.

  • v1.0 added readdir caching, and those cached directory contents could prevent new files in existing directories from being picked up. sync now deletes the readdir cache properly after an import completes.

• Fixed unexpected missing asset file id error, which could prevent file variations from being imported properly

• Stale opened-by lockfiles are now properly deleted on startup, and we reduced the default value for the openLockStaleMinutes setting from 1 hour to 5 minutes. This should resolve issues like this.

• For docker users or libraries stored on remote volumes, lock contention could result in the local replica being overwritten with an empty library database. Lock directories are retained much longer now, and replica mtime and db sizes are now used to determine if replicas are “stale.” Local db replicas are now stored in library-uuid-specific subdirectories to prevent overwrites when multiple library are open concurrently.

• Restored title bars that got lost in v2.0.0-alpha.1

• PhotoStructure for Desktop macOS icons don’t fight with the traffic lights any more.

• Fixed undraggable windows

• Monthly vs annual pricing is now handled in the Stripe Checkout page to make initial subscription setup simpler.

More details are in the release notes:

2021 PhotoStructure release notes

How does this version compare with other versions?

Here’s a chart comparing recent stable versions:

Why aren’t you releasing this as v2.0 stable yet?

There are a couple issues I still need to poke at resolve:

• macOS about page is not draggable
• apple photos library originals are not imported for some people
• progress panel seems to not render on v2?
• rotation is broken for HEIF?

What’s next?

Running v2.0 on my own library, I’ve realized that multi-user authorization needs to happen sooner than later: even with password protection, it’s not great to allow visitors to mess with the settings page or shut down the server, and letting them click “empty trash” is really not OK (which is why I hid those buttons behind a setting for this release: it’s still useful for local-only libraries).

Once that’s in, then metadata editing and face management makes more sense (as being only available to “library owners”).

[bdillahu]

Unraid docker setup… did the update and tried to start:

{“fatal”:true,“exit”:true,“status”:12,“pid”:27,“ppid”:20,“error”:“Library setup failed: ¹: ModelDbJanitor(/ps/library/.photostructure) ModelDbJanitor.setup(): Failed to set up /ps/tmp/local-db-h7n4-7r78-4nqf-k5ve/models/db.sqlite3: Quick integrity check failed: *** in database main ***\nPage 1008 is never used\nPage 1009 is never used\nPage 1010 …”}
ModelDbJanitor(/ps/library/.photostructure) ModelDbJanitor.setup(): Failed to set up /ps/tmp/local-db-h7n4-7r78-4nqf-k5ve/models/db.sqlite3: Quick integrity check failed: *** in database main ***
Page 1008 is never used
Page 1009 is never used
Page 1010 …
{“fatal”:true,“exit”:true,“status”:14,“pid”:27,“ppid”:20,“error”:“main setup failed: ModelDbJanitor(/ps/library/.photostructure) ModelDbJanitor.setup(): Failed to set up /ps/tmp/local-db-h7n4-7r78-4nqf-k5ve/models/db.sqlite3: Quick integrity check failed: *** in database main ***\nPage 1008 is never used\nPage 1009 is never used\nPage 1010 …¹”}

I’ll fiddle with things, but thought I would pass it over in case it was obvious to you.

[mrm]

Oof, apologies!

Can you email me your library database? It’ll be in /ps/library/.photostructure/models/db.sqlite3.

[tkohhh]

Up and running! Can’t wait to see if the Sync issue is well and truly fixed!

[bvwelch]

Hi! I signed in to say we love the delete feature in the apha version but noticed the new beta so we are trying it out now. At first I had trouble until I remembered to do a pull – apparently I already had a prior beta. Anyway the new beta is running and merrily scanning our photos. thanks!

[bvwelch]

I have lots to learn – clicking the trash can works fine, and rotating looks like it works but it complains that ‘darn that didnt work’ – it may be that some of my original photos are read-only.

In general I should figure out how to make a master copy and quit referencing the original photos – entirely – indeed i hope to phase out the old computer and its old hard drives completely.

[mrm]

@bvwelch

rotating looks like it works but it complains that ‘darn that didnt work’ –

PhotoStructure will try to fix the orientation of all asset variations, and only if it succeeds for all of them, does it report “success”.

Unfortunately, I’m finding that the rotation operation timeout on the frontend is too impatient for some of my assets on my really slow NAS (especially if it has a couple copies), so I need to rethink what constitutes a “success” versus a “failure” for this operation.

Thanks for the update!

[bvwelch]

Thanks for the insights. Is there a writeup somewhere that describes a method to migrate away from my original mess of photos? What I mean is original - photostructure-A – photostructureB… Once I am happy with A and B, I would sever the references/mention of the original. The original is a rough, messy collection of nearly 20 years of photos.

Another related question would be to migrating the photostructure to somewhere like digitalocean. thank you

[harrisc]

I installed beta 1 of Windows desktop. It auto started but then went to error "ChildService(sync). on Stdout():::::::::.
Database starts repairing to 100% and then program stops. I restarted with database repair again to 100% and again fail. This has occurred several times with no successful start.

Anything I can do or do I reconfigure again?

Chris

[nphesse]

Hello from a new user,

I was trying PhotoStructure and figured with the new features I could just go with the beta version.

Three things I noticed. First:

• I had a docker-compose based installation of the stable channel up and running, configured for a specific import path with “No thanks, I like my photos and videos where they already are.” selected
• I upgraded to the beta channel and restarted the application and it was running correctly
• I then started a yearly subscription and returned to my instance
• The import path disappeared and “Yes, please copy my photos and videos into my PhotoStructure Library.” was selected
  I do not know if this configuration change came through the upgrade or through the plus subscription - but it was unexpected

Second:
Deleting an image is very unresponsive. When I click on the trash can icon seemingly nothing happens. The request then takes 7 seconds to complete (according to the network tab of the developer tools) and only then I get any response (but deleting works). IMO it would be better to have at least some progress indicator or even an asynchronous action which starts deletion in the background and immediately gives feedback.
Same goes for marking an image as an favourite btw.
I would assume this is because my pictures are on a rather slow drive?

Please let me know if there is anything I can provide you with to help debugging.

Third:
The icon for archiving images somehow looks like a download icon to me. Initially I wanted to download a file through that button. Not sure where this resemblence comes from, but most likely the downwards arrow is the thing that reminds me of a download button.

Thanks for the work btw, I’m really liking this so far!

[mrm]

Oof, apologies! Can you email me any logs you have in %APPDATA%\PhotoStructure\logs? They might explain what went awry.

If you want to completely reset your library (imports will need to start from scratch!), I just wrote up these instructions.

[mrm]

@nphesse

Hello from a new user,

Hello, and welcome to PhotoStructure! Thanks for taking the time to share feedback!

@nphesse

The import path disappeared and “Yes, please copy my photos and videos into my PhotoStructure Library.” was selected

This is surprising: settings shouldn’t just “revert” to defaults, but the settings infrastructure is, uh, tactifully speaking, involved.

For example, if you set the environment variable PS_COPY_ASSETS_TO_LIBRARY in your docker configuration to true, for example, the settings page will show the current environment setting value, and if you change it on the settings page, it will persist that value to your library’s settings.toml, but because environment variable values always win, the value will stay true.

@nphesse

I do not know if this configuration change came through the upgrade or through the plus subscription - but it was unexpected

We can actually find out: I keep backups of the settings files in that directory. if you cd ...path/to/your/library/.photostructure ; grep copyAssetsToLibrary *settings* that would be enlightening (please DM or email me that if you want me to see what’s going on!)

@nphesse

Deleting an image is very unresponsive

Harumph! That’s no good! This operation should be instantaneous: we’re just writing a timestamp into a row in your database (if you speak SQL, we’re just doing an UPDATE AssetFile SET deletedAt= :now WHERE id = :assetFileId). If this is taking a long time, something’s up with the disk I/O on your library volume. Is a backup happening? Is it going through a RAID rebuild? Is PhotoStructure sync doing an import? (Sync runs reniced to low priority, but on slow remote disks, it still can overwhelm the disk controller, as renice only impacts the CPU scheduling, not I/O scheduling).

If you click the main menu, click “pause sync”, wait a couple seconds, and then delete is suddenly fast, it means we’re running too many concurrent sync-file processes to keep the db read-performant. You can control this by setting PS_CPU_LOAD_PERCENT=25 (it defaults to 75) or PS_MAX_SYNC_FILE_JOBS=2 (more detailed instructions are here).

@nphesse

Same goes for marking an image as an favourite btw

This is actually a bit more involved: when you an asset, we not only write to the db (UPDATE Asset SET rating = :likeRating WHERE id = :assetId), but we write that metadata to each asset file variation (either in a sidecar or to the file itself, depending on your configuration, see this section of settings).

My library is super messy: I have some assets with 20 variations/duplicates. The metadata writing happens in parallel, but even then, it can take a couple seconds for 20 files on several NASes to get written to.

@nphesse

The icon for archiving images somehow looks like a download icon to me

Ah, I see that! I’m hoping that when I add an actual download link to the right of the rotate icon, it’ll be distinct enough from archive to not confuse:

(the archive icon is actually the material standard, but if you have a better icon, please share: good icons are hard!)

[rgrandy]

I don’t know if this issue was from the latest beta or earlier, however none of my video files are importing. I tried to play around with the command line tools a bit to see if I could see what is going on, but this file was recently shot but can’t seem to get imported. The photo files seem to be okay.

/ps/app # ./photostructure sync-file /root/pictures/Ryan_S21/20211025_190709.mp4

{"path":"/root/pictures/Ryan_S21/20211025_190709.mp4","rejected":["File was deleted"],"elapsedMs":183}
{"ready":true}

/ps/app # ./photostructure info  ~/pictures/Ryan_S21/20211025_190709.mp4

[
  {
    nativePath: '/root/pictures/Ryan_S21/20211025_190709.mp4',
    ignoredBecause: [ 'notExists' ]
  }
]

[kyle]

Ubuntu 21.10 PhotoStructure for Desktops. Fresh install of v 2.0.0-beta.1

• Where was the problem?
  Import sync incomplete and errors and corruption follow attempt to finish it.

• What did you expect PhotoStructure to do?
  Import all of a de-duped test folder of 2055 images

• What did it actually do?
  Imported 1316 of 2055. Tried to restart sync, nothing. Restarted application and was met with an sqlite error (see attached screenshot). Restarted again and was met with this error:

Restarted again and was greeted with corruption:

Please attach screenshots/screen recording/example images if applicable. Feel free to email support@photostructure.com directly if anything is privacy-related.

[mrm]

@rgrandy

# ./photostructure info ~/pictures/Ryan_S21/20211025_190709.mp4

I think that path may be incorrect?

When you’re shelled into your docker container, remember that paths are from the bind root destination, not the path that’s valid on your host machine.

As an example, if your docker command had -v ~/pictures:/pics, you’d want to do ./photostructure info /pics/Ryan_S21/20211025_190709.mp4

[mrm]

@kyle

greeted with corruption

Ugh, apologies for the most unpleasant greeting. What disk is you library on? HDD, SSD, or a remote volume?

[rgrandy]

O ok thanks, that helped. I was able to run info on the mp4 file, and it was able to list out all of the data for the file. However I could not locate it in Photostructure by searching for any of the relavent tags or types or anything. I then did the sync-file command, it ran and then now I am able to see the video file in photostructure. Meanwhile, I can view all of the jpg images that were taken and synced at the same time, only the mp4 videos are missing.

[mrm]

@rgrandy

However I could not locate it in PhotoStructure by searching for any of the relevant tags or types or anything

Yeah: I need to add filename and a bunch of other metadata to the search index.

You can force PhotoStructure to import a specific file and the logs may explain what’s going on:

./photostructure sync-file --debug /pics/Ryan_S21/20211025_190709.mp4

(feel free to send me the output if you want me to decipher what’s going on)

[kyle]

PhotoStructure files are on a local SSD, original photo and video directories are on a local HDD (Ironwolf 4tb 5900rpm)

[harrisc]

@mrm

Oof, apologies! Can you email me any logs you have in %APPDATA%\PhotoStructure\logs? They might explain what went awry.

If you want to completely reset your library (imports will need to start from scratch!), I just wrote up these instructions.

I have now reset my library following your instructions; it will take days to import all the files. I can see progress from the About page but there is no progress indicator on the main page. That would be useful to know if the program is actually running and how far it has progressed.

BTW I did email logs to you for your information about the database problem