Troubleshooting missing files

Support
1

Running v2.0.0-beta.1 in docker. Saw on another post that you’re releasing the 2.1 alpha soon, so happy to retest with that if it makes more sense.

Finally got everything imported and transcoded (~4000 videos), but I noticed a lot of files missing. I ran through the steps from the server tools page to find what files were missing, and found about ~800 didn’t make it into the library.

I pulled the info on several of the files that didn’t import, but based on everything I could see the file should have imported. The videos are in the same directory and recorded with the same device, so aside from length there’s nothing different.

I did increase my maxasset size because I have larger videos I wanted transcoded. I increased it to 5GB, and the file that did not import is around 1GB. Not sure if maybe size is the factor here?

DID NOT IMPORT

[
  {
    _PhotoStructureVersion: '2.0.0-beta.1',
    capturedAt: {
      date: ExifDateTime {
        year: 2021,
        month: 11,
        day: 18,
        hour: 20,
        minute: 50,
        second: 11,
        millisecond: 0,
        tzoffsetMinutes: -300,
        rawValue: '2021:11:18 20:50:11',
        zoneName: 'America/New_York'
      },
      localCentiseconds: 2021111820501100,
      precisionMs: 1000,
      src: 'tags:CreateDate'
    },
    dimensions: { height: 3840, width: 2160 },
    duration: 167.435,
    errors: [],
    filters: {
      accepted: [
        'exifExtFilter',
        'fileSizeFilter',
        'notHiddenCheap',
        'noMediaFilter',
        'hasMimeType',
        'supportedMimeTypeFilter',
        'notRejected',
        'minDimensionsFilter',
        'minVideoDurationFilter'
      ],
      rejected: []
    },
    geohash: 'dr441y',
    geohashNumeric: 426905662,
    ignoredBecause: [],
    imageHash: {
      dominantColors: 'White (#FFFFFF), Charcoal (#36454F), Dark slate grey (#2F4F4F), Jet (#343434), Black olive (#3B3C36), Ebony (#555D50), Beige (#F6F1DE)',
      isGreyscale: false,
      meanHash: 'z/9/AQMAAIBYQQ0PAzOywNnx3+9jwEDj',
      modes: [
        2559, 475, 479,
         255, 914, 918,
        2998
      ]
    },
    Make: 'Apple',
    mimetype: 'video/quicktime',
    Model: 'iPhone 11 Pro Max',
    nativePath: '/ps/library/video-main/2021/2021-11/2021-11-iPhone 11 Pro Max/IMG_3428.MOV',
    needsTranscoding: true,
    rotation: 90,
    sha: 'iaLK6u/Fbb2ooOG9ctCsTlws+IQtBW2u',
    tags: [
      [ 'Camera', 'Apple', 'iPhone 11 Pro Max' ],
      [
        'When',
        { name: '2021', ordinal: 7979 },
        { displayName: 'Nov', name: '11', ordinal: 2 }
      ],
      [ 'Type', 'Video', 'QuickTime' ],
      [
        'fs',
        'Library',
        'video-main',
        '2021',
        '2021-11',
        '2021-11-iPhone 11 Pro Max'
      ]
    ],
    tz: 'America/New_York',
    tzSource: 'from Lat/Lon',
    uri: 'pslib:/video-main/2021/2021-11/2021-11-iPhone%2011%20Pro%20Max/IMG_3428.MOV',
    validFile: 'OK',
    variantSortCriteria: {
      count: 0,
      fileSize: 22,
      isBrowserSupported: false,
      isCover: false,
      mtime: 13643905,
      resolution: 11,
      schemeIdx: 3,
      uri: 'pslib:/video-main/2021/2021-11/2021-11-iPhone%2011%20Pro%20Max/IMG_3428.MOV'
    }
  }
]

DID IMPORT

[
  {
    _PhotoStructureVersion: '2.0.0-beta.1',
    capturedAt: {
      date: ExifDateTime {
        year: 2021,
        month: 11,
        day: 30,
        hour: 23,
        minute: 31,
        second: 37,
        millisecond: 0,
        tzoffsetMinutes: -300,
        rawValue: '2021:11:30 23:31:37',
        zoneName: 'America/New_York'
      },
      localCentiseconds: 2021113023313700,
      precisionMs: 1000,
      src: 'tags:CreateDate'
    },
    dimensions: { height: 1080, width: 1920 },
    duration: 12.0083333333333,
    errors: [],
    filters: {
      accepted: [
        'exifExtFilter',
        'fileSizeFilter',
        'notHiddenCheap',
        'noMediaFilter',
        'hasMimeType',
        'supportedMimeTypeFilter',
        'notRejected',
        'minDimensionsFilter',
        'minVideoDurationFilter'
      ],
      rejected: []
    },
    geohash: 'dr443z',
    geohashNumeric: 426905727,
    ignoredBecause: [],
    imageHash: {
      dominantColors: 'Tan (#D2B48C), Khaki (#C3B091), Beaver (#9F8170), Apricot (#FBCEB1), Blood red (#660000)',
      isGreyscale: false,
      meanHash: 'y8vBwcvLS8cwNDAwMDCwAPP/8dHRlw8B',
      modes: [
        3848, 3617,
        3841, 3621,
        3589, 3845,
        1594
      ]
    },
    Make: 'Apple',
    mimetype: 'video/quicktime',
    Model: 'iPhone 11 Pro Max',
    nativePath: '/ps/library/video-main/2021/2021-11/2021-11-iPhone 11 Pro Max/IMG_3481.MOV',
    needsTranscoding: false,
    rotation: 180,
    sha: 'oT2jeTRBe7dhpvBsHnwPHoAW42EH48kk',
    tags: [
      [ 'Camera', 'Apple', 'iPhone 11 Pro Max' ],
      [
        'When',
        { name: '2021', ordinal: 7979 },
        { displayName: 'Nov', name: '11', ordinal: 2 }
      ],
      [ 'Type', 'Video', 'QuickTime' ],
      [
        'fs',
        'Library',
        'video-main',
        '2021',
        '2021-11',
        '2021-11-iPhone 11 Pro Max'
      ]
    ],
    tz: 'America/New_York',
    tzSource: 'from Lat/Lon',
    uri: 'pslib:/video-main/2021/2021-11/2021-11-iPhone%2011%20Pro%20Max/IMG_3481.MOV',
    validFile: 'OK',
    variantSortCriteria: {
      count: 0,
      fileSize: 14,
      isBrowserSupported: false,
      isCover: false,
      mtime: 13652773,
      resolution: 9,
      schemeIdx: 3,
      uri: 'pslib:/video-main/2021/2021-11/2021-11-iPhone%2011%20Pro%20Max/IMG_3481.MOV'
    }
  }
]
2

If filters.rejected is empty, the file should have been imported, unless it fails file validation, which I didn’t run in the info filter chain… I just realized how confusing that might be, so the next build will include file validation in that filter chain…

In the next build, a new sync report will be generated that should explain these sorts of issues. See this for more details: Show more import and sync details - #9 by mrm

3

Gotcha. If I turn off file validation and re-sync, would they potentially import? Or are they ignored forever?

4

If there isn’t an issue with the file, just running sync should pick up the file.

It depends: if the file is found to be invalid, PhotoStructure will add it to your library database’s ShaBlocklist table. You can open your library database (using something like SQLite Browser) and clear that table (by running delete * from ShaBlocklist). I found that my sync runs were running slowly when a bunch of files in a given folder all suffered from bitrot, and PhotoStructure was having to re-validate each file.

If the file isn’t invalid, (see the validFile field from info’s output), then the file should be examined every time sync runs for that folder.

5

Gotcha… I’ll definitely take a look at that.

Both files had a value of OK for validFile. Does it make more sense to just wait for the next build to see the new reports?

6

Yeah. I’ll try to get it out ASAP.

7

i noticed that the one importing was transcode false and the one not imported was transcode true. I would look at the debug logs and the result of the ffmpeg transcoding operation and if it failed for some reason?

1 Like
8

I saw that too. I haven’t checked the other files, so I’m going to wait until the new alpha comes out and see if the import reports help at all.