Import is broken on the new beta

[kyle]

Windows 10 PhotoStructure for Desktops. Fresh install of v 2.0.0-beta.1. Library and referenced photos on SSD.

• Where was the problem?
  Import sync misses a huge number of de-duped, valid images that are all imported by the other applications I use (Capture One, and have tested with PhotoPrism)

• What did you expect PhotoStructure to do?
  Import all of a de-duped test folder of 2055 images (mix of .raf, .DNG, .nef, .rw2, .jpg). It didn’t work on Ubuntu desktop (imported 1316 of 2055) and I was asked on forum if they were on HDD so I went. All SSD this time AND set photo checks to their most permissive value.

• What did it actually do?
  Detected 936 image files and imported 890.

No version of PhotoStructure to date has successfully imported all of my images and it is frustrating given that the UI is so perfect.

[mrm]

Thanks for your report: and sincere apologies that things aren’t working for you (especially on this small test set!)

If PhotoStructure isn’t importing any given file properly, it could be due to:

1. one of the filters rejecting the file (you can see if a file will be filtered, even with the most permissive filter settings, using the info tool):

photostructure info /path/to/image.jpg --filter filters

[
  {
    "nativePath": "/path/to/image.jpg",
    "filters": {
      "accepted": [
        "exifExtFilter",
        "fileSizeFilter",
        "notHiddenCheap",
        "noMediaFilter",
        "hasMimeType",
        "supportedMimeTypeFilter",
        "notRejected",
        "minDimensionsFilter",
        "minVideoDurationFilter"
      ],
      "rejected": []
    }
  }
]

2. a file may not be considered “valid”, which may actually be due to corruption, or may be a bug in the file integrity tester (which should determine if there’s been file corruption due to bad copies or bitrot). This can be verified with the info tool:

photostructure info /path/to/image.jpg --filter validFile

[
  {
    "nativePath": "/path/to/image.jpg",
    "validFile": "OK"
  }
]

Here’s an example failure:

[
  {
    "nativePath": "/path/to/truncated.jpg",
    "validFile": "/path/to/truncated.jpg is invalid: Premature end of JPEG file⁶³⁷"
  }
]

3. a bug in the job scheduler (which queues up what files to process as it scans a directory hierarchy, and then pops off work from that queue). I’ve got a bunch of tests for this component, so I’m fairly confident this isn’t the issue, but job queues are tricky to test under different parallel loads.

4. a bug in sync-file, which is preventing a given file from being imported. The defect (or defects) could arrise due to file contens or some specific state of the library database. You can test this too, just by running sync-file directly on any missing file. Note that info is always read-only, and never updates your library database. sync-file actually imports the given file(s) into your library, if it can.

photostructure sync-file --info /path/to/file.jpg

[kyle]

Thanks for this. Do I need PhotoStructure for servers to run these command line checks?
Also, I reinstalled and tried importing the same photos into my PhotoStructure library vs leaving them where they were on the other ssd. 1,868 assets of 2050 successfully imported this time.

[mrm]

@kyle

Do I need PhotoStructure for servers to run these command line checks

Yeah: if you’re comfortable with the command line and have git installed, it should only take a minute to get going. PhotoStructure | PhotoStructure for Node

@kyle

I reinstalled and tried importing the same photos into my PhotoStructure library vs leaving them where they were on the other ssd

Interesting, thanks for sharing.

There (obviously) shouldn’t be a difference, though: it should work properly regardless of whether you’ve opted into auto-organization. I’ll think about this.

[kyle]

Auto-Organization isn’t working for me either. I tired importing my whole library and it stopped halfway through. Restarted sync and got
“Timeout currentLibraryLockOwner after 25008ms”

[mrm]

Crap, sorry you’re seeing this.

A library lock is acquired to prevent libraries on network shares from being written to concurrently (which results in a borked library database). If your library is on a local-only drive, you can turn off the lock acquisition code by setting skipLibraryOpenLocks to true, either by editing the settings.toml in your library, or setting this environment variable.

Also: this shouldn’t happen (obviously!): if you want to share your logs, I can check to see if there’s something else amiss going on.