Version 1.0.0-alpha.2 is ready for testing!

[mrm]

Updates from v1.0.0-alpha.1:

• PhotoStructure for Desktops builds available on macOS, Windows, and Ubuntu (alpha.1’s desktop builds were broken)

• I’ve reverted the UserData directory downcasing that was changed in
  -alpha.1: I believe I’ve fixed the problem with Electron startup that caused
  this issue. Sorry for the changes!

• Say hello to .cr3 support! LibRaw v0.20.1 is now included in all editions
  of PhotoStructure, as Ubuntu and Fedora distros are tracking older versions
  that don’t support .cr3.

• The main PhotoStructure for Desktops binary now supports command-line
  options, so running PhotoStructure-1.0.0-x86_64.AppImage --verbose on a
  terminal is now a thing.

• PhotoStructure for Desktops has a new “Open in browser…” in the View menu and the system tray menu.

Installation instructions are here:

V1.0.0-alpha.1 is out! Alpha testers: assemble

Alpha? What’s that? Remember: alpha builds should be considered extremely experimental, and may not even launch on all platforms. Read more about alpha, beta, and stable releases here. Questions? Comments? Feel free to reply here or to the new discord server: PhotoStructure What’s in 1.0.0-alpha.1? A lot: New .heif support New expanded RAW image support New curators: person tag extraction and parsing Browse-by-filesystem 40+ minor features and bugfixes PhotoStructure’s infrastructure …

Note: there seems to be an issue with the bounce service. To work around this for alpha.2, please set

PS_BOUNCE_MINUTES=0

[John]

Upgraded fine here…

Rebuilding your library, about 6 days remain

I’ll need to have a dig around as I expected these alphas to pick up my HEIC/HEIF files but that doesn’t seem to have happened. The health check in the about screen shows its installed fine.

[Senpai]

Just updated docker image on my Synology tp alpha(2), first small thing is that the whats-new link (https://photostructure.com/about/v-0-9/) opens the about of v0.9

[GavinCampbell]

Docker updates have been smooth. No issues to report yet.

[mrm]

@Senpai

the whats-new link opens the about of v0.9

Ah, good catch, thanks: I haven’t written up the v1.0.0 post yet (it goes to https://photostructure.com/whats-new/ and the redirect will change as soon as the v1.0.0 post is live).

[mrm]

@john you can use the info tool to make sure it’s working on your system correctly, and even run sync-file to import any missing file “by hand”. Both tools have --help and --verbose modes to let you see what’s going on (and feel free to share those logs with me if you think something’s amiss).

PhotoStructure Tools

You can also DM me an example file (nothing private, please!) and I can verify that it works as expected on my test machines.

[psu16502]

I’m having a problem where alpha.2 is not picking up any of my fuji raw files. I use docker-compose to run a fresh photostructure on my photos and it never picks up the .RAF files.

Here’s an example of photostructure info on a RAF file

/ps/app $ photostructure info /var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE\ 2021/2021-03-01\ salinas-monterey/Capture/DSCF1628.RAF 
[
  {
    _PhotoStructureVersion: '1.0.0-alpha.2',
    cameraId: 'SerialNumber:7CA04272',
    capturedAt: {
      date: ExifDateTime {
        year: 2021,
        month: 2,
        day: 26,
        hour: 15,
        minute: 20,
        second: 41,
        millisecond: 0,
        tzoffsetMinutes: -480,
        rawValue: '2021:02:26 15:20:41',
        zoneName: 'America/Los_Angeles'
      },
      localCentiseconds: 2021022615204100,
      offset: -480,
      src: 'tags:DateTimeOriginal'
    },
    dimensions: { width: 6032, height: 4032 },
    errors: [],
    exposureSettings: {
      focalLength: '35.0 mm',
      iso: 400,
      aperture: 2,
      shutterSpeed: '1/4000'
    },
    filters: {
      accepted: [
        'exifExtFilter',
        'fileSizeFilter',
        'notHiddenCheap',
        'noMediaFilter',
        'hasMimeType',
        'supportedMimeTypeFilter',
        'notRejected',
        'minDimensionsFilter',
        'minVideoDurationFilter'
      ],
      rejected: []
    },
    geohash: '9q923f',
    geohashNumeric: 325355630,
    ignoredBecause: [],
    imageHash: { dominantColors: '' },
    lensId: 'LensSerialNumber:58A22215',
    lensInfo: '35mm f/2',
    lensMake: 'Fujifilm',
    lensModel: 'XF35mm F2 R WR',
    Make: 'Fujifilm',
    mimetype: 'image/x-fujifilm-raf',
    Model: 'X-E3',
    nativePath: '/var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF',
    needsTranscoding: false,
    rotation: 0,
    sha: '7jzGpSIEWt/sHLCtgwKdlrgIj3TI9gOk',
    tags: [
      [ 'Camera', 'Fujifilm', 'X-E3' ],
      [ 'Lens', 'Fujifilm', 'XF35mm F2 R WR' ],
      [
        'When',
        { name: '2021', ordinal: 7979 },
        { name: '2', displayName: 'Feb', ordinal: 11 }
      ],
      [ 'Type', 'Image', 'Raw', 'Fujifilm' ],
      [
        'fs',
        'var',
        'heartofgold',
        'photos',
        'fujixe3',
        '_C1-SESSIONS',
        '_ARCHIVE 2021',
        '2021-03-01 salinas-monterey',
        'Capture'
      ]
    ],
    tz: 'America/Los_Angeles',
    tzSource: 'from Lat/Lon',
    uri: 'file:///var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE%202021/2021-03-01%20salinas-monterey/Capture/DSCF1628.RAF',
    validFile: 'invalid file /var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF: readableToFile() failed: Problem from /var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/…',
    variantSortCriteria: {
      resolution: 13,
      mtime: 13453180,
      schemeIdx: 1,
      isCover: false,
      count: 0,
      isBrowserSupported: false,
      fileSize: 14,
      uri: 'file:///var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE%202021/2021-03-01%20salinas-monterey/Capture/DSCF1628.RAF'
    }
  }
]
/ps/app $ 

something with “validFile” not sure how to interpret it

And here is an example of photostructure sync-file --verbose on the same RAF file

/ps/app $ photostructure sync-file --verbose /var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE\ 2021/2021-03-01\ salinas-monterey/Capture/DSCF1628.RAF
LogTail(): Tailing /ps/logs/**/*.log...
2021-03-24T18:31:44.366Z sync-file-313 info  Service(sync-file) setup()
2021-03-24T18:31:44.392Z sync-file-313 info  SettingsIO.importFileSettings(/ps/config/settings.toml) loaded { tomlMap: { copyAssetsToLibrary: false, libraryPath: '/ps/library', scanAllDrives: true },
  imported:
   [ { name: 'copyAssetsToLibrary', value: false, persist: false },
     { name: 'libraryPath', value: '/ps/library', persist: false },
     { name: 'scanAllDrives', value: true, persist: false } ] }
2021-03-24T18:31:44.399Z sync-file-313 info  SettingsIO.importFileSettings(/ps/config/settings.toml) loaded { tomlMap: { copyAssetsToLibrary: false, libraryPath: '/ps/library', scanAllDrives: true },
  imported:
   [ { name: 'copyAssetsToLibrary', value: false, persist: false },
     { name: 'libraryPath', value: '/ps/library', persist: false },
     { name: 'scanAllDrives', value: true, persist: false } ] }
2021-03-24T18:31:44.400Z sync-file-313 info  SettingsIO _libraryHasSettings { result: true,
  settings: '/ps/library',
  librarySettingsFile: '/ps/library/.photostructure/settings.toml' }
2021-03-24T18:31:44.409Z sync-file-313 info  SettingsIO.importFileSettings(/ps/library/.photostructure/settings.toml) loaded { tomlMap: { reportErrors: true },
  imported: [ { name: 'reportErrors', value: true, persist: false } ] }
2021-03-24T18:31:44.409Z sync-file-313 warn  Sentry Failed to set up sentry: TypeError: Cannot read property 'init' of undefined
2021-03-24T18:31:44.410Z sync-file-313 info  Service(sync-file) setup() { version: '1.0.0-alpha.2',
  start: 1616610704054,
  argv:
   [ '/usr/local/bin/node',
     '/ps/app/bin/sync-file.js',
     '--verbose',
     '/var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF' ],
  arch: 'x64',
  platform: 'linux',
  isDocker: true,
  isPacked: true,
  isElectron: false,
  versions:
   { node: '14.16.0',
     v8: '8.4.371.19-node.18',
     uv: '1.40.0',
     zlib: '1.2.11',
     brotli: '1.0.9',
     ares: '1.16.1',
     modules: '83',
     nghttp2: '1.41.0',
     napi: '7',
     llhttp: '2.1.3',
     openssl: '1.1.1j',
     cldr: '37.0',
     icu: '67.1',
     tz: '2020a',
     unicode: '13.0' },
  settings: { logLevel: 'info', httpPort: 1787, rpcPort: 1807, libraryPath: '/ps/library' },
  NODE_ENV: 'production',
  PS_CACHE_DIR: '/ps/tmp',
  PS_EXPOSE_NETWORK_WITHOUT_AUTH: 'true',
  PS_LIBRARY_PATH: '/ps/library',
  PS_LOG_COLOR: 'true',
  PS_LOG_DIR: '/ps/logs',
  PS_LOG_LEVEL: 'info',
  PS_LOG_STDOUT: 'true',
  PS_TAIL_LOGS: 'true' }
2021-03-24T18:31:44.411Z sync-file-313 info  Library(/ps/library) new()
2021-03-24T18:31:44.441Z sync-file-313 info  currentLibraryLockOwner(/ps/library/.photostructure) currentLibraryLockOwner { result:
   { file: { nativePath: '/ps/library/.photostructure/opened-by/867kc4e-14.json' },
     serviceName: 'main',
     createdAt: 1616609940901,
     updatedAt: 1616609940901,
     hostname: 'bbb451a26a98',
     pid: 14 } }
2021-03-24T18:31:44.447Z sync-file-313 info  SettingsIO.importFileSettings(/ps/library/.photostructure/settings.toml) loaded { tomlMap: { reportErrors: true },
  imported: [ { name: 'reportErrors', value: true, persist: false } ] }
2021-03-24T18:31:44.451Z sync-file-313 info  StatsDbDir Set up statsDbDir dir /ps/tmp/sync-state-8ww1q95n2u
2021-03-24T18:31:44.457Z sync-file-313 info  Vacuum checkRemoteVacuuming() received RPC value { value: false }
2021-03-24T18:31:44.461Z sync-file-313 info  ModelDbJanitor(/ps/library/.photostructure) Finished setup { primaryDbDir: { nativePath: '/ps/library/.photostructure' },
  backupDir: { nativePath: '/ps/library/.photostructure/models/backup' } }
2021-03-24T18:31:44.461Z sync-file-313 info  ModelDbJanitor(/ps/library/.photostructure) ModelDb reading from /ps/library/.photostructure/models/db.sqlite3
2021-03-24T18:31:44.461Z sync-file-313 info  Library(/ps/library) Library.setup() finished.
2021-03-24T18:31:44.483Z sync-file-313 info  Db(models) setting up new db connection to /ps/library/.photostructure/models/db.sqlite3 { priorWasNull: true }
2021-03-24T18:31:44.485Z sync-file-313 info  mkdb(/ps/library/.photostructure/models/db.sqlite3) Dynamically setting dbCacheSize to 192 { dbFileSize: 11550720 }
2021-03-24T18:31:44.551Z sync-file-313 info  SyncFileService isUnhealthy() { result: false,
  ok:
   [ 'Library and support directories are OK',
     'Used memory used by sync-file (25 MB) is OK',
     'RSS memory used by sync-file (92 MB) is OK' ],
  warn: [],
  bad: [],
  fail: [] }
2021-03-24T18:31:44.574Z sync-file-313 info  ExifTool Started child process ExifTool:328
2021-03-24T18:31:44.581Z sync-file-313 info  Renice Renice pid 328 to BelowNormal
{"path":"/var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF","error":"Error: invalid file /var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF: readableToFile() failed: applyWip(): empty after apply for /ps/tmp/imgcache/e4/77/gg22kuewu7t9puquwutx/dcraw.tiff⁶³⁷","memMb":33,"rssMb":150}
{"ready":true}
2021-03-24T18:31:45.036Z sync-file-313 info  AssetFileImporter(/var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF) Setting timeout to 2 minutes, 35 seconds
2021-03-24T18:31:45.304Z sync-file-313 info  SharpReadable sharpReadable() -> dcraw() { src:
   '/var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF' }
2021-03-24T18:31:46.365Z sync-file-313 warn  SharpReadable strategy failed for /var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF: Error: readableToFile() failed: applyWip(): empty after apply for /ps/tmp/imgcache/e4/77/gg22kuewu7t9puquwutx/dcraw.tiff⁶⁷
2021-03-24T18:31:46.366Z sync-file-313 info  Deferred(AssetFileImporter(/var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF).apply) .reject() { cause:
   { cause:
      Error: applyWip(): empty after apply for /ps/tmp/imgcache/e4/77/gg22kuewu7t9puquwutx/dcraw.tiff
          at V.applyWip (/ps/app/bin/sync-file.js:9:198802)
          at async V.applyIfEmpty_ (/ps/app/bin/sync-file.js:9:199083)
          at async Object.t.readableToFile (/ps/app/bin/sync-file.js:9:289473)
          at async Object.t.firstResolvedDefinedPromise (/ps/app/bin/sync-file.js:9:113684)
          at async /ps/app/bin/sync-file.js:9:298215
          at async g.time (/ps/app/bin/sync-file.js:9:114471)
          at async x (/ps/app/bin/sync-file.js:9:301953)
          at async t.AssetFileFinder.byExistingAsset_ (/ps/app/bin/sync-file.js:9:741288)
          at async a.thunk (/ps/app/bin/sync-file.js:9:740112)
          at async a.thunk (/ps/app/bin/sync-file.js:9:743556),
     retriable: true,
     fatal: false,
     stack:
      [ 'Error: applyWip(): empty after apply for /ps/tmp/imgcache/e4/77/gg22kuewu7t9puquwutx/dcraw.tiff',
        '    at V.applyWip (/ps/app/bin/sync-file.js:9:198802)',
        '    at async V.applyIfEmpty_ (/ps/app/bin/sync-file.js:9:199083)',
        '    at async Object.t.readableToFile (/ps/app/bin/sync-file.js:9:289473)',
        '    at async Object.t.firstResolvedDefinedPromise (/ps/app/bin/sync-file.js:9:113684)',
        '    at async /ps/app/bin/sync-file.js:9:298215' ] },
  retriable: false,
  fatal: false,
  stack:
   [ 'Error: applyWip(): empty after apply for /ps/tmp/imgcache/e4/77/gg22kuewu7t9puquwutx/dcraw.tiff',
     '    at V.applyWip (/ps/app/bin/sync-file.js:9:198802)',
     '    at async V.applyIfEmpty_ (/ps/app/bin/sync-file.js:9:199083)',
     '    at async Object.t.readableToFile (/ps/app/bin/sync-file.js:9:289473)',
     '    at async Object.t.firstResolvedDefinedPromise (/ps/app/bin/sync-file.js:9:113684)',
     '    at async /ps/app/bin/sync-file.js:9:298215' ] }
2021-03-24T18:31:46.368Z sync-file-313 warn  Error onError(): Error: invalid file /var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF: readableToFile() failed: applyWip(): empty after apply for /ps/tmp/imgcache/e4/77/gg22kuewu7t9puquwutx/dcraw.tiff⁶³⁷
Error: applyWip(): empty after apply for /ps/tmp/imgcache/e4/77/gg22kuewu7t9puquwutx/dcraw.tiff
    at V.applyWip (/ps/app/bin/sync-file.js:9:198802)
    at async V.applyIfEmpty_ (/ps/app/bin/sync-file.js:9:199083)
    at async Object.t.readableToFile (/ps/app/bin/sync-file.js:9:289473)
    at async Object.t.firstResolvedDefinedPromise (/ps/app/bin/sync-file.js:9:113684)
    at async /ps/app/bin/sync-file.js:9:298215 { event: 'nonFatal',
  message:
   'Failed to import /var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF⁶' }
2021-03-24T18:31:46.368Z sync-file-313 info  SyncFileService importFile() finished in 1907ms { result:
   { path:
      '/var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF',
     error:
      'Error: invalid file /var/heartofgold/photos/fujixe3/_C1-SESSIONS/_ARCHIVE 2021/2021-03-01 salinas-monterey/Capture/DSCF1628.RAF: readableToFile() failed: applyWip(): empty after apply for /ps/tmp/imgcache/e4/77/gg22kuewu7t9puquwutx/dcraw.tiff⁶³⁷' } }
2021-03-24T18:31:46.369Z sync-file-313 info  Service(sync-file) exit() { status: 0, reason: 'Processed files from argv', waitForJobs: true, ending: false }
2021-03-24T18:31:46.620Z sync-file-313 info  Service(sync-file) exit(): waiting for enqueued jobs to complete... { pendingNames: [] }
2021-03-24T18:31:46.620Z sync-file-313 info  Service(sync-file) exit(): enqueued jobs finished.
2021-03-24T18:31:46.620Z sync-file-313 info  Endable endEndables() { isTest: false, isSingleSpecTests: false }
2021-03-24T18:31:46.621Z sync-file-313 info  rpc.Client(db@localhost:1807#0) end() { ending: true, timings: {}, pendingSize: 0 }
2021-03-24T18:31:46.621Z sync-file-313 info  rpc.Client(db@localhost:1807#0) close() { ended: true }
2021-03-24T18:31:46.674Z sync-file-313 info  PromiseTimer timings: { 'tags.readRawTags': { sumSec: 0.49, mean: 120, sd: 120, k: 4 },
  'exiftool.read()': { sumSec: 0.216, mean: 220, k: 1 },
  'AssetFileFinder.byEqlUri': { sumSec: 0.197, mean: 200, k: 1 },
  'fs.sha': { sumSec: 0.195, mean: 200, k: 1 },
  'fs.readFile': { sumSec: 0.045, mean: 23, sd: 4.9, k: 2 },
  'tags.readTags': { sumSec: 0.035, mean: 35, k: 1 },
  'AssetFileFinder.tags': { sumSec: 0.035, mean: 35, k: 1 } }
2021-03-24T18:31:46.674Z sync-file-313 warn  PromiseTimer error counts: [ [ 'fs.unlink', 1 ], [ 'img.sharpReadable.RAF', 1 ] ]
2021-03-24T18:31:46.674Z sync-file-313 info  PromiseTimer memory use on exit: { ok:
   [ 'Used memory used by sync-file (33 MB) is OK',
     'RSS memory used by sync-file (150 MB) is OK' ],
  warn: [],
  bad: [] }
2021-03-24T18:31:46.679Z sync-file-313 info  Library(/ps/library) onEnd(): finished.
2021-03-24T18:31:46.679Z sync-file-313 info  Db(models) closing db Database {
  inTransaction: false,
  open: false,
  memory: false,
  readonly: false,
  name: '/ps/library/.photostructure/models/db.sqlite3' }
/ps/app $ 

[mrm]

@psu16502 can you DM me an example .RAF file? LibRaw seems to not be handling it correctly.

(I’ve got example .RAF files in my test suite from an X-T3, X-T4, and X-S10, but the X-E3 might produce slightly differently encoded files).

[narensankar]

MacOS image download error -

AccessDenied Access Denied4BHTXCCQYTK82D5WEOI+FvgxjvJYoIsyQHtMwqjgrA10HUUZCiq2sDrv7chyszdv7jetutcyogSZI0e+7zTmILpfa5k=

[mrm]

@narensankar

MacOS image download error

Thanks for reporting! I’ll get that fixed as soon as I can.

[mrm]

There’s an issue a couple users have found with the periodic bounce service. If you’re finding that PhotoStructure shuts down unexpectedly after 5+ hours, set PS_BOUNCE_MINUTES=0 to disable the bounce service.

I’ll have this fixed in the next release, which will be -beta.1.