BUG? - Too few pictures imported

Bug
1

I imported tens of thousands of photos. The progress bar was seeing tens of thousands of photos. They processed suspiciously fast (less than 1 hour). Sure enough, there are only about 2,500 assets in the library. Way too few even if I had a 50% rate of duplication…

So I looked in the logs. I see A LOT of errors.

This is 1.0.0-beta.4 on Docker (specifically a NAS running UNRAID)

I tried to send the logs through the UI, that fails also.

{"ts":1623249885307,"l":"debug","ctx":"ChildProcess","msg":"execFile()","meta":{"command":"/ps/app/tools/linux-x64/jpegtran/jpegtran","args":["-outfile","/dev/null","/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg"],"maxAgeMs":30000}}
{"ts":1623249885312,"l":"warn","ctx":"ChildProcess","msg":"stdoutResult(): on(error)","meta":{"cmd":"/ps/app/tools/linux-x64/jpegtran/jpegtran","args":["-outfile","/dev/null","/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg"],"opts":{"timeout":30000},"error":{"errno":-2,"code":"ENOENT","syscall":"spawn /ps/app/tools/linux-x64/jpegtran/jpegtran","path":"/ps/app/tools/linux-x64/jpegtran/jpegtran","spawnargs":["-outfile","/dev/null","/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg"],"stack":["Error: spawn /ps/app/tools/linux-x64/jpegtran/jpegtran ENOENT","    at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)","    at onErrorNT (node:internal/child_process:480:16)","    at processTicksAndRejections (node:internal/process/task_queues:83:21)"]}}}
{"ts":1623249885312,"l":"warn","ctx":"Deferred({\"cmd\":\"/ps/app/tools/linux-x64/jpegtran/jpegtran\",\"args\":[\"-outfile\",\"/dev/null\",\"/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg\"]})","msg":".reject()","meta":{"errno":-2,"code":"ENOENT","syscall":"spawn /ps/app/tools/linux-x64/jpegtran/jpegtran","path":"/ps/app/tools/linux-x64/jpegtran/jpegtran","spawnargs":["-outfile","/dev/null","/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg"],"stack":["Error: spawn /ps/app/tools/linux-x64/jpegtran/jpegtran ENOENT","    at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)","    at onErrorNT (node:internal/child_process:480:16)","    at processTicksAndRejections (node:internal/process/task_queues:83:21)"]}}
{"ts":1623249885312,"l":"info","ctx":"Deferred(AssetFileImporter(/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg).apply)","msg":".reject()","meta":{"cause":{"cause":{"errno":-2,"code":"ENOENT","syscall":"spawn /ps/app/tools/linux-x64/jpegtran/jpegtran","path":"/ps/app/tools/linux-x64/jpegtran/jpegtran","spawnargs":["-outfile","/dev/null","/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg"]},"retriable":false,"fatal":false},"retriable":false,"fatal":false,"stack":["Error: spawn /ps/app/tools/linux-x64/jpegtran/jpegtran ENOENT","    at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)","    at onErrorNT (node:internal/child_process:480:16)","    at processTicksAndRejections (node:internal/process/task_queues:83:21)"]}}
{"ts":1623249885312,"l":"debug","ctx":"ValidFile","msg":"isIgnorableValidationError","meta":{"result":false}}
{"ts":1623249885312,"l":"debug","ctx":"ChildProcess","msg":"stdoutResult.errorIsRetriable()","meta":{"result":false,"error":{"errno":-2,"code":"ENOENT","syscall":"spawn /ps/app/tools/linux-x64/jpegtran/jpegtran","path":"/ps/app/tools/linux-x64/jpegtran/jpegtran","spawnargs":["-outfile","/dev/null","/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg"],"stack":["Error: spawn /ps/app/tools/linux-x64/jpegtran/jpegtran ENOENT","    at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)","    at onErrorNT (node:internal/child_process:480:16)","    at processTicksAndRejections (node:internal/process/task_queues:83:21)"]},"cmd":"/ps/app/tools/linux-x64/jpegtran/jpegtran","args":["-outfile","/dev/null","/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg"]}}
{"ts":1623249885312,"l":"debug","ctx":"AssetFileFinder(/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg)","msg":"apply(): validFile rejected: Error: invalid file /pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg: code ENOENT: spawn /ps/app/tools/linux-x64/jpegtran/jpegtran ENOENT⁶³²"}
2

More info. ignored because notExists? Can’t photostructure not handle spaces in paths?

/ps/app # ./photostructure info /pictures/201511 November 2016/00016IMG_00016_BU
RST20161114172221_COVER.jpg
[
{ nativePath: ‘/pictures/201511’, ignoredBecause: [ ‘notExists’ ] },
{ nativePath: ‘/ps/app/November’, ignoredBecause: [ ‘notExists’ ] },
{
nativePath: ‘/ps/app/2016/00016IMG_00016_BURST20161114172221_COVER.jpg’,
ignoredBecause: [ ‘notExists’ ]
}
]
/ps/app # jpegtran -verbose -outfile /dev/null “/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_CO
VER.jpg”
libjpeg-turbo version 2.1.0 (build 20210424)
Copyright (C) 2009-2021 D. R. Commander
Copyright (C) 2015, 2020 Google, Inc.
Copyright (C) 2019-2020 Arm Limited
Copyright (C) 2015-2016, 2018 Matthieu Darbois
Copyright (C) 2011-2016 Siarhei Siamashka
Copyright (C) 2015 Intel Corporation
Copyright (C) 2013-2014 Linaro Limited
Copyright (C) 2013-2014 MIPS Technologies, Inc.
Copyright (C) 2009, 2012 Pierre Ossman for Cendio AB
Copyright (C) 2009-2011 Nokia Corporation and/or its subsidiary(-ies)
Copyright (C) 1999-2006 MIYASAKA Masaru
Copyright (C) 1991-2020 Thomas G. Lane, Guido Vollbeding

Emulating The Independent JPEG Group’s software, version 8d 15-Jan-2012

Start of Image
Miscellaneous marker 0xe1, length 14855
Miscellaneous marker 0xe1, length 733
Define Quantization Table 0 precision 0
Define Quantization Table 1 precision 0
Miscellaneous marker 0xe2, length 3062
Start Of Frame 0xc0: width=4048, height=3036, components=3
Component 1: 2hx2v q=0
Component 2: 1hx1v q=1
Component 3: 1hx1v q=1
Define Huffman Table 0x00
Define Huffman Table 0x10
Define Huffman Table 0x01
Define Huffman Table 0x11
Start Of Scan: 3 components
Component 1: dc=0 ac=0
Component 2: dc=1 ac=1
Component 3: dc=1 ac=1
Ss=0, Se=63, Ah=0, Al=0
End Of Image

3

Crap, it looks like there’s something wrong with the docker image: is it missing the jpegtran tool?

The error with the info tool is due to your shell splitting the filename on whitespace. Try wrapping your filename with double quotes.

4

well, I did figure out that indeed, none of my thousands of jpg imported. It imported all my gif, png, tiff, heic

I do see jpegtran

/ # find .|grep jpegtran
./usr/bin/jpegtran
./ps/app/tools/linux-x64/jpegtran
./ps/app/tools/linux-x64/jpegtran/jpegtran
./ps/app/tools/mac-x64/jpegtran
./ps/app/tools/mac-x64/jpegtran/jpegtran
./ps/app/tools/win-ia32/jpegtran
./ps/app/tools/win-ia32/jpegtran/jpegtran.exe
./ps/app/tools/win-ia32/jpegtran/libjpeg-62.dll
5

Here the info with the quotes.

/ps/app # ./photostructure info "/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg"
[
  {
    _PhotoStructureVersion: '1.0.0-beta.4',
    capturedAt: {
      date: ExifDateTime {
        year: 2016,
        month: 11,
        day: 14,
        hour: 17,
        minute: 22,
        second: 25,
        millisecond: 411,
        tzoffsetMinutes: -300,
        rawValue: '2016:11:14 17:22:25.411922',
        zoneName: 'America/New_York'
      },
      localCentiseconds: 2016111417222541,
      offset: -300,
      src: 'tags:SubSecDateTimeOriginal'
    },
    dimensions: { width: 4048, height: 3036 },
    errors: [],
    exposureSettings: {
      focalLength: '4.7 mm',
      iso: 127,
      aperture: 2,
      shutterSpeed: '1/120'
    },
    filters: {
      accepted: [
        'exifExtFilter',
        'fileSizeFilter',
        'notHiddenCheap',
        'noMediaFilter',
        'hasMimeType',
        'supportedMimeTypeFilter',
        'notRejected',
        'minDimensionsFilter',
        'minVideoDurationFilter'
      ],
      rejected: []
    },    geohash: 'dppj37',
    geohashNumeric: 425378919,
    ignoredBecause: [],
    imageHash: {
      meanHash: '8BIA9v////+DgPnBAIDBHgAAAP////8=',
      isGreyscale: false,
      modes: [
        2743, 2746,
        3624, 2750,
        1829, 1825,
        2742
      ],
      dominantColors: 'Laurel green (#A9BA9D), Artichoke (#8F9779), Desert (#C19A6B), Ecru (#C2B281), Raw umber (#826644), Umber (#635147), Ash grey (#B2BEB5)'
    },
    Make: 'Google',
    mimetype: 'image/jpeg',
    Model: 'Pixel XL',
    nativePath: '/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg',
    needsTranscoding: false,
    rotation: 0,
    sha: 'XCEIGSuNCsk3wdn/9JZTMT5pMwNl6oP1',
    tags: [
      [ 'Camera', 'Google', 'Pixel XL' ],
      [
        'When',
        { name: '2016', ordinal: 7984 },
        { name: '11', displayName: 'Nov', ordinal: 2 }
      ],
      [ 'Type', 'Image', 'JPEG' ],
      [ 'fs', '37PF4nuuM', '201511 November 2016' ]
    ],
    tz: 'America/New_York',
    tzSource: 'from Lat/Lon',
    uri: 'psfile://37PF4nuuM/201511%20November%202016/00016IMG_00016_BURST20161114172221_COVER.jpg',
    validFile: 'invalid file /pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg: code no such file or directory: spawn /ps/app/tools/linux-x64/jpegtran/jpegtran no such file or directory:⁶³²',
    variantSortCriteria: {
      resolution: 12,
      mtime: 12556362,
      schemeIdx: 2,
      isCover: true,
      count: 0,
      isBrowserSupported: true,
      fileSize: 9,
      uri: 'psfile://37PF4nuuM/201511%20November%202016/00016IMG_00016_BURST20161114172221_COVER.jpg'
    }
  }
]
6

jpegtran seems to do something…

/ps/app/tools/linux-x64/jpegtran # jpegtran -verbose -outfile /dev/null "/pictures/201511 November 2016/00016IMG_00016_BURST20161114172221_COVER.jpg"
libjpeg-turbo version 2.1.0 (build 20210424)
Copyright (C) 2009-2021 D. R. Commander
Copyright (C) 2015, 2020 Google, Inc.
Copyright (C) 2019-2020 Arm Limited
Copyright (C) 2015-2016, 2018 Matthieu Darbois
Copyright (C) 2011-2016 Siarhei Siamashka
Copyright (C) 2015 Intel Corporation
Copyright (C) 2013-2014 Linaro Limited
Copyright (C) 2013-2014 MIPS Technologies, Inc.
Copyright (C) 2009, 2012 Pierre Ossman for Cendio AB
Copyright (C) 2009-2011 Nokia Corporation and/or its subsidiary(-ies)
Copyright (C) 1999-2006 MIYASAKA Masaru
Copyright (C) 1991-2020 Thomas G. Lane, Guido Vollbeding

Emulating The Independent JPEG Group's software, version 8d  15-Jan-2012

Start of Image
Miscellaneous marker 0xe1, length 14855
Miscellaneous marker 0xe1, length 733
Define Quantization Table 0  precision 0
Define Quantization Table 1  precision 0
Miscellaneous marker 0xe2, length 3062
Start Of Frame 0xc0: width=4048, height=3036, components=3
    Component 1: 2hx2v q=0
    Component 2: 1hx1v q=1
    Component 3: 1hx1v q=1
Define Huffman Table 0x00
Define Huffman Table 0x10
Define Huffman Table 0x01
Define Huffman Table 0x11
Start Of Scan: 3 components
    Component 1: dc=0 ac=0
    Component 2: dc=1 ac=1
    Component 3: dc=1 ac=1
  Ss=0, Se=63, Ah=0, Al=0
End Of Image
7

I didn’t think it would help, but just in case, I updated to beta.5, no change/improvement.

Version 1.0.0-beta.5
Edition PhotoStructure for Docker
Plan plus
Licensed to 
OS Alpine Linux v3.13 on x64 (docker)
Node.js 16.3.0
ExifTool 12.26
[Video support](https://photostructure.com/getting-started/video-support/) FFmpeg 4.3.1
[HEIF support](https://photostructure.com/getting-started/heif-support/) /usr/bin/heif-convert
Free memory 1 GB / 34 GB
CPUs 4 × Intel(R) Xeon(R) E-2224 CPU @ 3.40GHz
Concurrency Target system use: 75% (3 syncs, 2 threads/sync)
Web uptime 4 minutes, 31 seconds
[Library metrics](https://photostructure.com/faq/metrics/) 528 assets
--- ---
1,078 image files
507 video files
691 tags
Library path /ps/library
Health checks * Library and support directories are OK
* Used memory used by web (27 MB) is OK
* RSS memory used by web (110 MB) is OK
8

Ugh, I think the alpine docker image may be grumpy with the libc compiled jpegtran: we want the musl jpegtran in /usr/bin.

I’m away from my desk until next week: beta. 3 or beta.2 may be the best build to run in docker, at least until I fix this next week.

There’s also a couple settings that let you tweak the tools path, so that could be a workaround as well (but require a bit more work than just pulling the prior image: they should be tagged with the full version)

1 Like
9

beta.2 works! beta.3,4,5 don’t.

1 Like
10

Thanks for that detective work: I’ll get on this first thing when I return to my desk tomorrow morning.

11

Hmmm… for me the first image with this issue is beta.4, and beta.3 works fine

12

I could swear it wasn’t working on beta.3… Maybe I screwed something up in my testing…

No matter, looking forward to test beta.6 on Monday!

13

Actually, I retried on beta.3 and it does indeed work fine. I probably was confused with what appears to be another bug, where a library sync (or initial scan) doesn’t appear to show any pictures/assets until the container is restarted.

14

I’ve just released beta.6 with the one-line-fix for this issue. There are a couple other bugs I need to address (like the search results rendering incorrectly) that will come in beta.7 later today.

1 Like
15

I can confirm this bug is definitely fixed on beta.6