PhotoStructure isn't finding my stuff!

Been running the import for 10 hours and it’s only found a very small subset of my library. This is using the automatic setting.

PhotoStructure can get “stuck” during imports in several ways:

Wasting time scanning irrelevant folders

PhotoStructure imports are 2 separate jobs:

  1. scanning your system for photos and videos to import, and then

  2. actually importing each file.

The progress panel should tell you

  • what folder it is scanning (if 1. isn’t done)
  • how many files it’s found to import
  • what files it’s currently importing
  • how many it’s finished.

There are many directory patterns PhotoStructure uses to avoid system and application directories so it doesn’t waste time scanning hierarchies that don’t have assets, but this list isn’t comprehensive: a scan can still get “stuck” looking in places it doesn’t need to.

If you’ve seen this happen, please reply with the path to the directory hierarchy, if you think it’s generally ignorable. If it’s not a common system or application path, you can tell PhotoStructure to ignore a folder hierarchy by placing a NoMedia file or folder in the root or the directory you want to be skipped over.

Very large directories

Version 0.9.1 can get stuck trying to infer sibling metadata with directories that have many thousand (5k+) entries (especially an issue if the volume with these larger directories is mounted on a slow network share.)

Version 1.0 should address this bug, but until then, either skip that directory with nomedia or move files into new subdirectories so no folder has so many direct children.

If you already have a backup (please do that first!), you can try this perl script to break up enormous directories:

First, make sure you have a backup. Then open a terminal, cd into your gigantic directory, and then paste that script as one big copy-paste. Let it chunk through everything, and then PhotoStructure can handle it from there.

Problematic video files

PhotoStructure automatically transcodes videos so they are viewable with your browser.

Transcoding videos can take >5x the video duration to transcode. PhotoStructure has timeouts for importing files (so it doesn’t get stuck!) but video transcodes are throttled to only one at a time, so those may make your import seem stuck while PhotoStructure is waiting on ffmpeg/VLC. You may want to change the maxVideoTranscodeDurationSecs to limit this issue.

Corrupt files

PhotoStructure knows how to validate JPEG and videos, but other files may have corruption that causes the importer process, sync-file, to crash.

We actually expect these crashes, and will then try to skip over these especially problematic files, but it may be a “wedge” and not a “crash,” in that instead of the process dying quickly, the process instead becomes unresponsive and PhotoStructure has to wait for the (pessimistic) timeout before restarting the sync-file process.

Your files are being excluded from import

If your images are very small, or inadvertantly excluded due to them being in a system directory or NoMedia directory, you’ll need to remedy that. See these pages for details:

Is there a way to find which files were not imported and why?

I keep photos/videos in folders grouped by 1000 files to avoid large folders issues
I have

Name Count
.JPG 19376
.mp4   551
.psd    89
.tif     1

But in PhotoStructure I see imported only 18265 Images and 517 video files.

I have the following settings

transcodeVideos = false
maxAssetFileSizeBytes = 50000000000 # added extra 2 zeros to ensure even large video files would be included

I don’t see any logs that would say which files are missing and why.

I can open sqlite database and extracted the list of files imported

WITH
    x AS (
             SELECT
                 SUBSTR(uri, LENGTH(uri) - charindex('/', reverse(uri)) + 2) AS filename
             FROM
                 AssetFile
         )
SELECT
    filename
FROM
    x
ORDER BY
    filename;

Then comparing to the actual list of files I can find what files are missing.
So there are 838 files missing. All 89 psd files were not imported, which is probably fine, as maybe PhotoStructure doesn’t support this format. But it also didn’t import 26 mp4 files and 723 jpg files for unknown to me reason.

Also it seems that PhotoStructure is not even showing something that is supposedly imported

SELECT
    COUNT(*)
FROM
    AssetFile
WHERE
    uri LIKE '%mp4%'

shows that database has 525 files while as I mentioned above PhotoStructure shows only 517 video files imported. Which is confusing because now I don’t even know how to identify which of those 525-517=8 are missing

A added an example of the photo and video files that are missing from the import

As you found, the photoshop images won’t import, as I don’t have a reliable way to render them for display.

You can get an exact list of all files included or excluded via the list tool:

https://photostructure.com/server/tools/#which-files-in-a-directory-didnt-get-imported

The info tool can tell you why a specific file didn’t get imported.

Manually running sync --verbose will give tons of logs, including file filter accept/reject rules per path.

Both the 0098.jpg and 0002.mp4 aren’t imported due to their small dimensions:

$ photostructure info ~/Desktop/0098.jpg

...
  {
    filters: {
      accepted: [
        'exifExtFilter',
        'notHiddenCheap',
        'hasMimeType',
        'supportedMimeTypeFilter',
        'notRejected',
        'minVideoDurationFilter'
      ],
      rejected: [ 'fileSizeFilter', 'minDimensionsFilter' ]
    },
...
]

You can override these filters with the minVideoDimension, minImageDimension, and minAssetFileSizeBytes settings.

You didn’t mention your OS, but if you decide to enable transcoding later, I’d recommend FFmpeg (as it’s much more reliable for transcodes and frame extractions, which can also result in erroneously skipped files).

If you find other things amiss, please share!

1 Like