Show more import and sync details

Feature Requests
,
1

Sync status is still pretty opaque in PhotoStructure, and is frequently a cause of frustration for users.

Adding the info tool, library metrics and tag counts were tiny steps to help with this, and I wrote this article to give a bit more context, but the situation is still not OK.

The issue is that getting a browser to render a list with hundreds of thousands filenames is a bit of a trick: but maybe that’s not what’s needed?

A beta user suggested adding, after an import is run, a summary that is displayed showing: # of files imported, # of de-dupes, and # of files not imported. I think that would help, but it still doesn’t really answer the question, “why didn’t my file(s) get imported.”

I think a new “recently imported assets” and “recently updated assets” view (via the search view?) would help to show what did get imported.

I wonder if a “recently skipped” view would help (where it’s not images, but just paths to directories and files along with one or more reasons why they were skipped? This would be the first table-of-text view in PhotoStructure. It’d basically be the last N pathnames that sync visited and decided not to import (along with the filter(s) that caused the file to be passed over).

(I’ve even considered writing to a new “sync-status” SQLite db, that could then just be opened by something like the DB Browser for SQLite, and then no new UI would be needed, but this would really only help advanced users).

So, dear users of PhotoStructure: have you seen a UI that handled this sort of thing gracefully that I should look at? Do you have other ideas I can entertain?

Update, July 2022: this isn’t a “solved” issue, as sync reports aren’t very accessible, but it’s a start.

2

How about a “recently imported assets” and/or “recently skipped assets” page that categories the hundreds of thousands of assets by sync attempt (datetime). So you’d see a list of sync attemps and can click to expand any given one. If there get to be some huge number of sync attempts, maybe they can be further categorized by week or month. And if there are still too many assets within sync attempts, maybe they can be categorized by some commonly-problematic attribute (like assets between 1MB and 5MB, assets between 5MB and 25MB; or photo assets and video assets)?

1 Like
3

I of course like text reports and things… those are always good, even if just summaries. It gives you a warm and fuzzy to see “100 files processed, 100 files successful” or something.

As a couple of ideas:

Mylio, when it has a long list to present, gives a link in the interface/dialog box and if you click it spawns a separate browser window with a pretty plain (but potentially very, very long) list of paths that had an issue (or whatever). If you are delving into a problem, that is often enough to give a clue, or at least confirm which files didn’t work.

My addition here is that maybe you could use your existing interface. Have another selection criterial (like when, camera, lens, etc.) which is “errors” - I know you couldn’t show the image, but a little “box” with a filename and ability to hover or click on it to get more details. Maybe broken into sections based on directory (so each directory with files which had issues would have a separate section on the screen). I say this with the thought that you already have the infrastructure and code to handle very large displays in this manner.

4

A simple csv export with the following fields would be sufficient for now:

Source/Pictures/(messy library) file and path
Reason for failure
Size of file
Permissions of file (777/755/500…) and owner
Date of last sync attempt

While it would be super slick to have it in the GUI, maybe just start with a csv report from the GUI and move on from there?

For the current solution on the cli:

I cannot get the find output and the Photostructure list command outputs to work together, none of the files line up full path…it would be good if there were also a wildcard capacity for Photostructure info

./photostructure info */familyphotos/somephoto.jpg
5

And I’ll add… even when it makes it to the GUI, please allow for exporting the list in some parseable format (csv, Json, txt, something). It’s potentially pretty long and having the ability to (for instance) use it in a script with exiftool or something to fix issues would be handy.

6

@mrm as you know I am struggling currently with what appears to me a somewhat inconsistent behavior of PhotoStructure across different platforms and different version (Windows vs. Ubuntu, 1.1 vs. 2.0.0b) when it comes to how many pictures and videos are getting imported. Without further information why something didn’t get imported/processed it leaves a very uneasy feeling to me as a user. How many pictures am I missing? Can I trust the tool?

Hence, I would really appreciate an improvement here. I can’t point to a very slick (from an UI perspective) example, but if you check www.allwaysync.com they provide the needed information for the synchronization purposes.

For PhotoStructure I would like to know how many files did it find and read, how many where imported and how many and which (!) where not imported and why. Maybe a simple table with the columns/rows and numbers where you can, if you want to know, click on the number to get to the details?

In my case I use a fixed share on my NAS as the picture-import location. I would also appreciate a comparison feature. A button that will check the PhotoStructure library vs my original photo location to check what’s missing (because you can bet that users will not always pay attention to the import report) and explicitly trying to import those again incl. the same report from above to hunt down corrupt files or maybe even a bug in PhotoStructure. I guess that would help to make the application even more robust resp. assure the user that everything is fine. :slight_smile:

image
image1047Ă—777 76.9 KB

4 Likes
7

New eyeballs here.

Initial sync showed status of the sync for each target and estimate of time remaining. But re-syncing after adding nomedia to some folders and … crickets.

Sync seems to be running because it can be paused. It would be nice to see the status, or at least an indication it’s in process.

8

Adding another voice here!

I’d also love to have some more insight into the syncing progress. I’m using several tools on different devices to get photos onto my PS drives (Syncthing, foldersync, etc). Occasionally something goes wrong. If PS showed me a list of recent syncs (by folder) this would help me catch these hiccups.

9

@waltwooton @Ramblurr thanks for the feedback!

Would you be OK with PhotoStructure spitting out a CSV with verbose sync results that you could open in Excel/OpenOffice? (I then don’t have to find a high-quality scales-to-million-row html table widget!)

The .CSV would look something like this:

ts path status details url
1650930793117 /opt/photos/ scanned found 132 files in 3 child directories http://localhost:1787/tags/fs/CJKLSD
1650930793827 /opt/photos/.hidden skipped hiddenFilter
1650930793828 /opt/photos/path-with-nomedia skipped noMediaFilter
1650930794767 /opt/photos/invalid.jpeg skipped invalid JPEG
1650930795721 /opt/photos/prior.CR2 no-op already in sync http://localhost:1787/asset/1234
1650930795721 /opt/photos/tiny.jpg skipped minDimensionsFilter
1650930797646 /opt/photos/new.CR2 imported http://localhost:1787/asset/2345

Update for what’s implemented in v2.1:

The “ts” column is the millis from 1970-01-01 timestamp when that row was written.

The “path” column is the native path of the directory or file.

The “state” column for directories will be

  • “scanning” when initially found
  • “skipped” if the directory has .NoMedia or considered “ignorable”
  • “timeout” if readdir() failed
  • “scanned” if all the contents of the directory were scanned or enqueued.

The “state” column for files will be

  • “enqueued” if the file is not already in sync with the library, and why
  • “rejected” if the file doesn’t pass library filters
  • “new” if the file was newly imported into your library
  • “noop” if the file had been previously imported and not changed since
  • “synced” if the file had been previously imported, changed, and updated in your library
  • “deleted” if the file was determined to be deleted

These reports live in $libraryDir/. photostructure/sync-reports

4 Likes
10

From my perspective this is ok for the exhaustive report, but I would appreciate an error report within the tool itself, I kindly refer to my post above. I don’t care that much about the skipped ones because of a “hidden filter”, “noMediaFilter”, “no-op” or even the “imported”, but I do care about proper errors. Here, I would also suggest to not label “invalid JPEG” as “skipped”, but really as “error” as otherwise it’s very cumbersome to filter, group and hunt-down the errors (I am talking here from a user perspective, I understand that per se for the tool an invalid JPEG is not an error).

I really would like the program to be smart enough to store the skipped and erroneous ones in the DB (or in a file) so that those pictures could be imported again after fixing (either the picture or the bug in the program). This would avoid a complete re-import.

In my tests with my simple family-photo library, I have between 10-15% pictures missing and at the moment I wouldn’t know how to handle that as a normal user.

11

That would certainly be helpful for diagnosing issues, yes.

But in conjunction some sort of high-level sync status page that is easily viewable from the browser (especially on mobile) that lists a summary of recent sync: Sync timestamp, # imported, # errors, etc. That would be my first port of call several times a week to check on the health of my PS instance.

12

My concern is somewhat simpler, and perhaps misplaced.

If I start a sync manually in v.1.1 desktop, i see no indication it’s started. I did for the initial sync, and I continue to see such in the scheduled syncs.

This is somewhat of a moot point because I plan to rebuild and switch to PS for nodes once I get a new NAS built. But it’s disconcerting not to see a response in the UI. Not sure if this is a bug or a feature.:wink:

A csv file is perfectly serviceable. I’m not far enough along to have an opinion about the contents.

13

For people who are already familiar with PhotoStructure the csv format above is fine. Moreover they can fire up exiftool, investigate the failures and fix what needs to be fixed. For a new user this has very little meaning and needs to be hidden or at least do not throw this at them. As @Ramblurr suggested, maybe produce some sort of summary with nicer UI to let less experience users know how many assets were imported, how many dups were de-duped, how many skipped etc? I know, we are asking a lot again from you, but if this was requested, gotta be implemented.

K

14

Yes that makes sense to me to show directly in UI info about what wasn’t imported. And have a link for people to directly open the new CSV sync reports.

I haven’t reviewed any sync reports yet, but ideally it starts with super easy to understand stuff like a link to PhotoStructure | Why didn't my file get imported into my Library?, and a summary saying 9398 files too small, 7 files too large, 45 files in hidden directories, 78 files in NoMedia directories, 3993 files unsupported format, and so on.

1 Like