Sync missing files with same format as the rest

Support
,
1

|Version|1.1.0|
|Edition|PhotoStructure for Servers|
|Plan|lite|
|OS|Ubuntu 21.04 on x64|
|Node.js|16.14.0|
|Video support|FFmpeg 4.3.2-0+deb11u1ubuntu1|
|HEIF support|/usr/bin/heif-convert|
|Free memory|14 GB / 17 GB|
|CPUs|4 × Intel(R) Core™ i5-7400T CPU @ 2.40GHz|

Sync is missing some of the files. From around 15000 photos, 500 are not imported.
All the photos are in ARW format taken from two cameras and are 100% NOT corrupted and the same format as the imported.

Can’t see anything different with photostructure info.
I’m sure that photostructure sync /path/filename.ARW DID import a missing file. (EDIT: maybe not)

Please help.

Tried the beta but it worked worse for me, showing server slowdown toasts and not fixing the issue.

2

Info of imported file:

[
  {
    _PhotoStructureVersion: '1.1.0',
    cameraId: 'InternalSerialNumber:00000000',
    capturedAt: {
      date: ExifDateTime {
        year: 2022,
        month: 2,
        day: 13,
        hour: 15,
        minute: 43,
        second: 24,
        millisecond: 176,
        tzoffsetMinutes: 60,
        rawValue: '2022:02:13 15:43:24.176+01:00',
        zoneName: 'UTC+1'
      },
      localCentiseconds: 2022021315432417,
      precisionMs: 10,
      src: 'tags:SubSecDateTimeOriginal'
    },
    dimensions: { height: 5120, width: 7168 },
    errors: [],
    exposureSettings: {
      aperture: 3.5,
      focalLength: '50.0 mm',
      iso: 6400,
      shutterSpeed: '1/500'
    },
    filters: {
      accepted: [
        'exifExtFilter',
        'fileSizeFilter',
        'notHiddenCheap',
        'noMediaFilter',
        'hasMimeType',
        'supportedMimeTypeFilter',
        'notRejected',
        'minDimensionsFilter',
        'minVideoDurationFilter'
      ],
      rejected: []
    },
    ignoredBecause: [],
    imageHash: {
      dominantColors: 'Charcoal (#36454F), Black (#000000), Apricot (#FBCEB1), Licorice (#1A1110), Cerulean (#007BA7), Blue sapphire (#126180)',
      isGreyscale: false,
      meanHash: 'YP+DJzw8DwA+Pj8+fHgAAD8/Pz9/fAAA',
      modes: [
         478, 1097, 3845,
        1536,  503,  219,
         499
      ]
    },
    imageId: 'ShutterCount:0',
    lensId: 'LensID:Sigma 50mm F1.4 DG HSM | A',
    lensInfo: '50mm f/1.4',
    lensMake: 'Sigma',
    lensModel: '50mm F1.4 DG HSM | A',
    Make: 'Sony',
    mimetype: 'image/x-sony-arw',
    Model: 'α7 IV',
    nativePath: '/storage/A7C/A7402915.ARW',
    needsTranscoding: false,
    rotation: 0,
    sha: 'pR1PD/9EX4VI/j/X0pw4tORPFayc3HRX',
    tags: [
      [ 'Camera', 'Sony', 'α7 IV' ],
      [ 'Lens', 'Sigma', '50mm F1.4 DG HSM | A' ],
      [
        'When',
        { name: '2022', ordinal: 7978 },
        { displayName: 'Feb', name: '2', ordinal: 11 }
      ],
      [ 'Type', 'Image', 'Raw', 'Sony' ],
      [ 'fs', '3MDC1yTZn', 'A7C' ]
    ],
    tz: 'UTC+01',
    tzSource: 'offsetMinutesToZoneName from OffsetTime',
    uri: 'psfile://3MDC1yTZn/A7C/A7402915.ARW',
    validFile: 'OK',
    variantSortCriteria: {
      count: 0,
      fileSize: 14,
      isBrowserSupported: false,
      isCover: false,
      mtime: 13706456,
      resolution: 14,
      schemeIdx: 2,
      uri: 'psfile://3MDC1yTZn/A7C/A7402915.ARW'
    }
  }
]

Info of not imported file:

[
  {
    _PhotoStructureVersion: '1.1.0',
    cameraId: 'InternalSerialNumber:00000000',
    capturedAt: {
      date: ExifDateTime {
        year: 2022,
        month: 2,
        day: 13,
        hour: 15,
        minute: 43,
        second: 24,
        millisecond: 348,
        tzoffsetMinutes: 60,
        rawValue: '2022:02:13 15:43:24.348+01:00',
        zoneName: 'UTC+1'
      },
      localCentiseconds: 2022021315432434,
      precisionMs: 10,
      src: 'tags:SubSecDateTimeOriginal'
    },
    dimensions: { height: 5120, width: 7168 },
    errors: [],
    exposureSettings: {
      aperture: 3.5,
      focalLength: '50.0 mm',
      iso: 6400,
      shutterSpeed: '1/500'
    },
    filters: {
      accepted: [
        'exifExtFilter',
        'fileSizeFilter',
        'notHiddenCheap',
        'noMediaFilter',
        'hasMimeType',
        'supportedMimeTypeFilter',
        'notRejected',
        'minDimensionsFilter',
        'minVideoDurationFilter'
      ],
      rejected: []
    },
    ignoredBecause: [],
    imageHash: {
      dominantColors: 'Charcoal (#36454F), Black (#000000), Apricot (#FBCEB1), Cerulean (#007BA7), Licorice (#1A1110)',
      isGreyscale: false,
      meanHash: 'YP+DJzw8DwA+Pj8+fHgAAD8/Pz9/fAAA',
      modes: [
        478, 1097, 3845,
        503, 1536, 1540,
        219
      ]
    },
    imageId: 'ShutterCount:0',
    lensId: 'LensID:Sigma 50mm F1.4 DG HSM | A',
    lensInfo: '50mm f/1.4',
    lensMake: 'Sigma',
    lensModel: '50mm F1.4 DG HSM | A',
    Make: 'Sony',
    mimetype: 'image/x-sony-arw',
    Model: 'α7 IV',
    nativePath: '/storage/A7C/A7402916.ARW',
    needsTranscoding: false,
    rotation: 0,
    sha: 'p3b/SgtB0F4psE0cGds3towtTcWCE9oN',
    tags: [
      [ 'Camera', 'Sony', 'α7 IV' ],
      [ 'Lens', 'Sigma', '50mm F1.4 DG HSM | A' ],
      [
        'When',
        { name: '2022', ordinal: 7978 },
        { displayName: 'Feb', name: '2', ordinal: 11 }
      ],
      [ 'Type', 'Image', 'Raw', 'Sony' ],
      [ 'fs', '3MDC1yTZn', 'A7C' ]
    ],
    tz: 'UTC+01',
    tzSource: 'offsetMinutesToZoneName from OffsetTime',
    uri: 'psfile://3MDC1yTZn/A7C/A7402916.ARW',
    validFile: 'OK',
    variantSortCriteria: {
      count: 0,
      fileSize: 14,
      isBrowserSupported: false,
      isCover: false,
      mtime: 13706456,
      resolution: 14,
      schemeIdx: 2,
      uri: 'psfile://3MDC1yTZn/A7C/A7402916.ARW'
    }
  }
]
3

Manual sync of not imported file (with --debug)

./photostructure sync --debug /storage/A7C/A7402916.ARW
2022-02-15T09:56:16.223Z sync-2523899 info  SettingsIO.importFileSettings(/home/sony/.config/PhotoStructure/settings.toml) loaded { imported:
   { copyAssetsToLibrary: false,
     libraryDir: '/storage/photostructure/library',
     scanAllDrives: false,
     scanPaths: [ '/storage/A7C' ] } }
2022-02-15T09:56:16.229Z sync-2523899 info  SettingsIO _libraryHasSettings { result: true,
  settings: '/storage/photostructure/library',
  librarySettingsFile: '/storage/photostructure/library/.photostructure/settings.toml' }
2022-02-15T09:56:16.233Z sync-2523899 info  SettingsIO.importFileSettings(/storage/photostructure/library/.photostructure/settings.toml) loaded { imported: { reportErrors: false } }
2022-02-15T09:56:16.236Z sync-2523899 info  Service(sync) setup() { version: '1.1.0',
  start: 1644918975856,
  argv:
   [ '/usr/bin/node',
     '/home/sony/photostructure/bin/sync.js',
     '--debug',
     '/storage/A7C/A7402916.ARW' ],
  arch: 'x64',
  platform: 'linux',
  isDocker: false,
  os: 'Ubuntu 21.04 on x64',
  isPacked: true,
  isElectron: false,
  versions:
   { node: '16.14.0',
     v8: '9.4.146.24-node.20',
     uv: '1.43.0',
     zlib: '1.2.11',
     brotli: '1.0.9',
     ares: '1.18.1',
     modules: '93',
     nghttp2: '1.45.1',
     napi: '8',
     llhttp: '6.0.4',
     openssl: '1.1.1m+quic',
     cldr: '40.0',
     icu: '70.1',
     tz: '2021a3',
     unicode: '14.0',
     ngtcp2: '0.1.0-DEV',
     nghttp3: '0.1.0-DEV' },
  settings:
   { logLevel: 'debug',
     httpPort: 1787,
     rpcPort: 1807,
     libraryDir: '/storage/photostructure/library' },
  NODE_ENV: 'production',
  PS_CACHE_DIR: '/storage/photostructure/cache',
  PS_HTTP_PORT: '1787',
  PS_LIBRARY_DIR: '/storage/photostructure/library',
  PS_LOCALHOST: '127.0.0.1',
  PS_PREVIEWS_DIR: '/storage/photostructure/previews',
  PS_RPC_PORT: '1807',
  PS_SCAN_ALL_DRIVES: 'false',
  PS_SCAN_MY_PICTURES: 'false',
  PS_SCAN_PATHS: '/storage/A7C' }
2022-02-15T09:56:16.238Z sync-2523899 info  Library(/storage/photostructure/library) new()
2022-02-15T09:56:16.239Z sync-2523899 debug Library(/storage/photostructure/library) setup() started
2022-02-15T09:56:16.239Z sync-2523899 debug OpenedByIO() throwIfAvailable() { from: '(library setup)' }
2022-02-15T09:56:16.242Z sync-2523899 debug JsonFileStore prior /home/sony/.config/PhotoStructure/.system-uid.json { uid: '1412-42bk-b0d3-040u',
  version: '2.0.0-beta.1',
  type: 'system',
  createdAt: 1644774158801 }
2022-02-15T09:56:16.250Z sync-2523899 debug OpenedBy.maybeRemovePriorLock(/storage/photostructure/library/.photostructure/opened-by/34eju35-2476395.json) Retaining prior lockfile
2022-02-15T09:56:16.250Z sync-2523899 debug OpenedBy.maybeRemovePriorLock(/storage/photostructure/library/.photostructure/opened-by/34eju35-2523735.json) Retaining prior lockfile
2022-02-15T09:56:16.251Z sync-2523899 debug OpenedBy.maybeRemovePriorLock(/storage/photostructure/library/.photostructure/opened-by/34eju35-2523899.json) Retaining prior lockfile
2022-02-15T09:56:16.252Z sync-2523899 debug OpenedBy.maybeRemovePriorLock(/storage/photostructure/library/.photostructure/opened-by/34eju35-319059.json) Retaining prior lockfile
2022-02-15T09:56:16.253Z sync-2523899 debug OpenedBy.maybeRemovePriorLock(/storage/photostructure/library/.photostructure/opened-by/34eju35-319071.json) Retaining prior lockfile
2022-02-15T09:56:16.255Z sync-2523899 debug currentLibraryLockOwner(/storage/photostructure/library/.photostructure) read { jsons:
   [ { file:
        { nativePath: '/storage/photostructure/library/.photostructure/opened-by/34eju35-2476395.json' },
       serviceName: 'sync',
       createdAt: 1644911460877,
       updatedAt: 1644915788205,
       hostname: 'somedomain.com',
       systemUID: '1412-42bk-b0d3-040u',
       v: '1.1.0',
       pid: 2476395 },
     { file:
        { nativePath: '/storage/photostructure/library/.photostructure/opened-by/34eju35-2523735.json' },
       serviceName: 'info',
       createdAt: 1644918889930,
       updatedAt: 1644918889931,
       hostname: 'somedomain.com',
       systemUID: '1412-42bk-b0d3-040u',
       v: '1.1.0',
       pid: 2523735 },
     { file:
        { nativePath: '/storage/photostructure/library/.photostructure/opened-by/34eju35-2523899.json' },
       serviceName: 'sync',
       createdAt: 1644918976239,
       updatedAt: 1644918976240,
       hostname: 'somedomain.com',
       systemUID: '1412-42bk-b0d3-040u',
       v: '1.1.0',
       pid: 2523899 },
     { file:
        { nativePath: '/storage/photostructure/library/.photostructure/opened-by/34eju35-319059.json' },
       serviceName: 'main',
       createdAt: 1644831144095,
       updatedAt: 1644917544115,
       hostname: 'somedomain.com',
       systemUID: '1412-42bk-b0d3-040u',
       v: '1.1.0',
       pid: 319059 },
     { file:
        { nativePath: '/storage/photostructure/library/.photostructure/opened-by/34eju35-319071.json' },
       serviceName: 'web',
       createdAt: 1644831142914,
       updatedAt: 1644916453776,
       hostname: 'somedomain.com',
       systemUID: '1412-42bk-b0d3-040u',
       v: '1.1.0',
       pid: 319071 } ] }
2022-02-15T09:56:16.256Z sync-2523899 debug ChildProcess stdoutResult(): execFile { cmd: 'ps',
  args: [ '-p', '2476395,2523735,2523899,319059,319071', '-wwwo', 'pid,lstart,command' ] }
2022-02-15T09:56:16.259Z sync-2523899 debug ChildEnv env { SHELL: '/bin/bash',
  SUDO_GID: '1000',
  PS_SCAN_ALL_DRIVES: 'false',
  SUDO_COMMAND: '/usr/bin/su sony',
  PWD: '/home/sony/photostructure',
  LOGNAME: 'sony',
  PS_CACHE_DIR: '/storage/photostructure/cache',
  HOME: '/home/sony',
  LANG: 'C',
  LS_COLORS:
   'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:',
  PS_HTTP_PORT: '1787',
  PS_LIBRARY_DIR: '/storage/photostructure/library',
  LESSCLOSE: '/usr/bin/lesspipe %s %s',
  TERM: 'xterm',
  LESSOPEN: '| /usr/bin/lesspipe %s',
  PS_RPC_PORT: '1807',
  USER: 'sony',
  SHLVL: '1',
  PS_PREVIEWS_DIR: '/storage/photostructure/previews',
  PS_LOCALHOST: '127.0.0.1',
  PATH:
   '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/libraw:/opt/local/bin:/opt/local/sbin',
  PS_SCAN_MY_PICTURES: 'false',
  SUDO_UID: '1000',
  MAIL: '/var/mail/sony',
  PS_SCAN_PATHS: '["/storage/A7C"]',
  OLDPWD: '/home/mexus',
  _: './photostructure',
  NODE_ENV: 'production',
  VIPS_MIN_STACK_SIZE: '2m',
  VIPSHOME: '/target',
  LC_ALL: 'C',
  PS_NEVER_IGNORED: '["/storage/A7C/A7402916.ARW"]' }
2022-02-15T09:56:16.259Z sync-2523899 debug ChildProcess execFile() { command: 'ps',
  args: [ '-p', '2476395,2523735,2523899,319059,319071', '-wwwo', 'pid,lstart,command' ],
  maxAgeMs: 15000 }
2022-02-15T09:56:16.276Z sync-2523899 debug Ps pidInfo() { pids: [ 2476395, 2523735, 2523899, 319059, 319071 ],
  result:
   [ { pid: 319059, start: 2022-02-14T09:32:03.000Z, cmd: 'PhotoStructure' },
     { pid: 319071, start: 2022-02-14T09:32:04.000Z, cmd: 'PhotoStructure web' },
     { pid: 2476395, start: 2022-02-15T07:51:00.000Z, cmd: 'PhotoStructure sync' },
     { pid: 2523899, start: 2022-02-15T09:56:15.000Z, cmd: 'PhotoStructure sync' } ] }
2022-02-15T09:56:16.277Z sync-2523899 debug currentLibraryLockOwner(/storage/photostructure/library/.photostructure) Unlinking zombie opened-by files: [ { file:
     { nativePath: '/storage/photostructure/library/.photostructure/opened-by/34eju35-2523735.json' },
    serviceName: 'info',
    createdAt: 1644918889930,
    updatedAt: 1644918889931,
    hostname: 'somedomain.com',
    systemUID: '1412-42bk-b0d3-040u',
    v: '1.1.0',
    pid: 2523735 } ]
2022-02-15T09:56:16.279Z sync-2523899 info  currentLibraryLockOwner(/storage/photostructure/library/.photostructure) currentLibraryLockOwner { result:
   { file:
      { nativePath: '/storage/photostructure/library/.photostructure/opened-by/34eju35-319059.json' },
     serviceName: 'main',
     createdAt: 1644831144095,
     updatedAt: 1644917544115,
     hostname: 'somedomain.com',
     systemUID: '1412-42bk-b0d3-040u',
     v: '1.1.0',
     pid: 319059 } }
2022-02-15T09:56:16.279Z sync-2523899 debug OpenedByIO() throwIfAvailable() { from: '(library setup)',
  lockOwner:
   { file:
      { nativePath: '/storage/photostructure/library/.photostructure/opened-by/34eju35-319059.json' },
     serviceName: 'main',
     createdAt: 1644831144095,
     updatedAt: 1644917544115,
     hostname: 'somedomain.com',
     systemUID: '1412-42bk-b0d3-040u',
     v: '1.1.0',
     pid: 319059 } }
2022-02-15T09:56:16.280Z sync-2523899 debug OpenedByIO() throwIfAvailable() { from: '(library setup)',
  json:
   { serviceName: 'sync',
     createdAt: 1644918976239,
     updatedAt: 1644918976240,
     hostname: 'somedomain.com',
     systemUID: '1412-42bk-b0d3-040u',
     v: '1.1.0',
     pid: 2523899 } }
2022-02-15T09:56:16.289Z sync-2523899 info  SettingsIO.importFileSettings(/storage/photostructure/library/.photostructure/settings.toml) loaded { imported: { reportErrors: false } }
2022-02-15T09:56:16.293Z sync-2523899 debug JsonFileStore prior /storage/photostructure/library/.photostructure/.library-uid.json { uid: 'nhkb-nutw-w5mp-7v4x',
  version: '2.0.0-beta.1',
  type: 'library',
  createdAt: 1644774158800 }
2022-02-15T09:56:16.294Z sync-2523899 debug JsonFileStore prior /home/sony/.config/PhotoStructure/.system-uid.json { uid: '1412-42bk-b0d3-040u',
  version: '2.0.0-beta.1',
  type: 'system',
  createdAt: 1644774158801 }
2022-02-15T09:56:16.295Z sync-2523899 debug BaseFile(/storage/photostructure/cache/sync-state-5kt2qphu1a/README.txt) applyIfEmpty(): non-empty
2022-02-15T09:56:16.296Z sync-2523899 info  StatsDbDir Set up statsDbDir dir /storage/photostructure/cache/sync-state-5kt2qphu1a
2022-02-15T09:56:16.297Z sync-2523899 debug RpcClient Creating new RPC client to 1807
2022-02-15T09:56:16.301Z sync-2523899 debug RpcClient Connected to 1807
2022-02-15T09:56:16.302Z sync-2523899 debug rpc.Client(db@localhost:1807#0) Resolved 'pong to sync@2523899 from main@319059 at Tue Feb 15 2022 11:56:16 GMT+0200 (Eastern European Standard Time)'
2022-02-15T09:56:16.303Z sync-2523899 debug rpc.Client(db@localhost:1807#0) Resolved false
2022-02-15T09:56:16.303Z sync-2523899 info  Vacuum checkRemoteVacuuming() received RPC value { value: false }
2022-02-15T09:56:16.303Z sync-2523899 debug Vacuum setVacuuming() { value: false }
2022-02-15T09:56:16.305Z sync-2523899 debug rpc.Client(db@localhost:1807#0) Resolved [ { filesystem: '/dev/mapper/ubuntu--vg-ubuntu--lv',
    mountpoint: '/',
    size: 124687908864,
    used: 34835992576,
    available: 83474059264,
    ignorable: false,
    remote: false,
    uuid: 'ee4cc9e6-2fb2-4f59-8594-73eb73982ccd' },
  { filesystem: '/dev/mapper/ubuntu--vg-lv--0',
    mountpoint: '/home',
    size: 124687908864,
    used: 104692084736,
    available: 13617967104,
    ignorable: false,
    remote: false,
    uuid: 'f5efccf9-e358-48fa-bcec-4e843da027a3' },
  { filesystem: '/dev/sda1',
    mountpoint: '/storage',
    size: 1967934717952,
    used: 876896493568,
    available: 991001559040,
    ignorable: false,
    remote: false,
    label: 'Storage',
    uuid: 'd52c5696-194c-4298-a599-92f2677ce8f3' } ]
2022-02-15T09:56:16.307Z sync-2523899 debug Library(/storage/photostructure/library) isOnRemoteVolume() { result: false }
2022-02-15T09:56:16.308Z sync-2523899 info  ModelDbJanitor(/storage/photostructure/library/.photostructure) Finished setup { primaryDbDir: { nativePath: '/storage/photostructure/library/.photostructure' },
  backupDir: { nativePath: '/storage/photostructure/library/.photostructure/models/backup' } }
2022-02-15T09:56:16.308Z sync-2523899 info  ModelDbJanitor(/storage/photostructure/library/.photostructure) ModelDb reading from /storage/photostructure/library/.photostructure/models/db.sqlite3
2022-02-15T09:56:16.309Z sync-2523899 info  Db(stats) setting up new db connection to /storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3 { priorWasNull: true }
2022-02-15T09:56:16.310Z sync-2523899 info  mkdb() Dynamically setting dbCacheSize to 192 { db: '/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3',
  dbFileSize: 147456 }
2022-02-15T09:56:16.312Z sync-2523899 info  SQLite verifyDb(/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3): OK
2022-02-15T09:56:16.315Z sync-2523899 debug Migration({"schema":"stats","db":"/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3"}) migration filter { migrationName: '20181030_stats',
  migrationDate: 2018-10-29T22:00:00.000Z,
  gitDate: 2021-08-16T01:54:24.000Z }
2022-02-15T09:56:16.315Z sync-2523899 debug Migration({"schema":"stats","db":"/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3"}) migration filter { migrationName: '20190312_files_not_bytes',
  migrationDate: 2019-03-11T22:00:00.000Z,
  gitDate: 2021-08-16T01:54:24.000Z }
2022-02-15T09:56:16.315Z sync-2523899 debug Migration({"schema":"stats","db":"/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3"}) migration filter { migrationName: '20190515_rebuild_queue',
  migrationDate: 2019-05-14T21:00:00.000Z,
  gitDate: 2021-08-16T01:54:24.000Z }
2022-02-15T09:56:16.315Z sync-2523899 debug Migration({"schema":"stats","db":"/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3"}) migration filter { migrationName: '20200106_no_dir_stats',
  migrationDate: 2020-01-05T22:00:00.000Z,
  gitDate: 2021-08-16T01:54:24.000Z }
2022-02-15T09:56:16.317Z sync-2523899 debug Migration({"schema":"stats","db":"/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3"}) latest(): already applied 20181030_stats.sql
2022-02-15T09:56:16.317Z sync-2523899 debug Migration({"schema":"stats","db":"/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3"}) latest(): already applied 20190312_files_not_bytes.sql
2022-02-15T09:56:16.317Z sync-2523899 debug Migration({"schema":"stats","db":"/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3"}) latest(): already applied 20190515_rebuild_queue.sql
2022-02-15T09:56:16.317Z sync-2523899 debug Migration({"schema":"stats","db":"/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3"}) latest(): already applied 20200106_no_dir_stats.sql
2022-02-15T09:56:16.318Z sync-2523899 info  Migration({"schema":"stats","db":"/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3"}) up to latest! { appliedMigrations: [] }
2022-02-15T09:56:16.318Z sync-2523899 info  Library(/storage/photostructure/library) Library.setup() finished.
2022-02-15T09:56:16.318Z sync-2523899 debug rpc.Client(db@localhost:1807#0) Resolved false
2022-02-15T09:56:16.321Z sync-2523899 debug ChildProcess stdoutResult(): execFile { cmd: 'cpuid', args: [ '-1' ] }
2022-02-15T09:56:16.322Z sync-2523899 debug ChildProcess execFile() { command: 'cpuid', args: [ '-1' ], maxAgeMs: 5000 }
2022-02-15T09:56:16.327Z sync-2523899 debug JsonFileStore prior /storage/photostructure/library/.photostructure/.library-uid.json { uid: 'nhkb-nutw-w5mp-7v4x',
  version: '2.0.0-beta.1',
  type: 'library',
  createdAt: 1644774158800 }
2022-02-15T09:56:16.328Z sync-2523899 debug ChildProcess stdoutResult.errorIsRetriable() { result: false,
  error:
   Error: cpuid exec'ed without a pid
       at /home/sony/photostructure/bin/sync.js:9:124685
       at processTicksAndRejections (node:internal/process/task_queues:96:5),
  cmd: 'cpuid',
  args: [ '-1' ] }
2022-02-15T09:56:16.330Z sync-2523899 info  SystemIds cpuid failed Error: cpuid exec'ed without a pid
    at /home/sony/photostructure/bin/sync.js:9:124685
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-02-15T09:56:16.333Z sync-2523899 debug SystemIds sids() { result:
   [ 'lm:27PKfUjtkCZJWPF',
     'cm:2QDwijBFy7iwE1U',
     'bm:2C9D6FQudu89RWj',
     'ma:26FxrJJCb1QBBFD',
     'ma:4vG5bGSwFK35ZfM',
     'ma:kSkvUg4F6W74Ee5',
     'ma:v2Tw1eaXU1HCjiB',
     'li:36EaLdADrZAah4C',
     'vl:3MDC1yTZnhNAu6Z',
     'vl:r3VdktTo6fMHPRM',
     'vl:Z1d184amJoMHea7' ] }
2022-02-15T09:56:16.335Z sync-2523899 debug Licensing licenses() { result: [] }
2022-02-15T09:56:16.336Z sync-2523899 info  SyncService setup(): library is at /storage/photostructure/library
2022-02-15T09:56:16.337Z sync-2523899 info  ChildService pathToService() { result: Promise { <pending> },
  cmd: 'sync-file.js',
  isPacked: true,
  dirs:
   [ '/home/sony/photostructure/bin/sync-file.js',
     '/home/sony/photostructure/app.asar/sync-file.js',
     '/home/sony/photostructure/dist/app/sync-file.js' ] }
2022-02-15T09:56:16.337Z sync-2523899 info  SyncService syncFileCluster(): sync-file found at /home/sony/photostructure/bin/sync-file.js
2022-02-15T09:56:16.361Z sync-2523899 info  Db(models) setting up new db connection to /storage/photostructure/library/.photostructure/models/db.sqlite3 { priorWasNull: true }
2022-02-15T09:56:16.361Z sync-2523899 info  mkdb() Dynamically setting dbCacheSize to 192 { db: '/storage/photostructure/library/.photostructure/models/db.sqlite3',
  dbFileSize: 12746752 }
2022-02-15T09:56:16.401Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144193,
     createdAt: 1644831144190,
     id: 1,
     name: 'enqueueAssetUpdates',
     updatedAt: 1644831144193,
     value: 'psfile://Z1d184amJ/',
     version: 3 } }
2022-02-15T09:56:16.404Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144198,
     createdAt: 1644831144197,
     id: 2,
     name: 'enqueueAssetUpdates',
     updatedAt: 1644831144198,
     value: 'psfile://r3VdktTo6/',
     version: 3 } }
2022-02-15T09:56:16.406Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144201,
     createdAt: 1644831144200,
     id: 3,
     name: 'enqueueAssetUpdates',
     updatedAt: 1644831144201,
     value: 'psfile://3MDC1yTZn/',
     version: 3 } }
2022-02-15T09:56:16.409Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144204,
     createdAt: 1644831144203,
     id: 4,
     name: 'enqueueAssetUpdates',
     updatedAt: 1644831144204,
     value: 'pslib:',
     version: 3 } }
2022-02-15T09:56:16.411Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144208,
     createdAt: 1644831144207,
     id: 5,
     name: 'enqueueAssetFileUpdates',
     updatedAt: 1644831144208,
     value: 'psfile://Z1d184amJ/',
     version: 11 } }
2022-02-15T09:56:16.412Z sync-2523899 debug enqueueAssetFileUpdatesForUri(psfile://Z1d184amJ/) Added outdated assetFiles: { queue: 'psfile://Z1d184amJ/', assetFileCount: 0 }
2022-02-15T09:56:16.413Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144210,
     createdAt: 1644831144210,
     id: 6,
     name: 'enqueueAssetFileUpdates',
     updatedAt: 1644831144210,
     value: 'psfile://r3VdktTo6/',
     version: 11 } }
2022-02-15T09:56:16.413Z sync-2523899 debug enqueueAssetFileUpdatesForUri(psfile://r3VdktTo6/) Added outdated assetFiles: { queue: 'psfile://r3VdktTo6/', assetFileCount: 0 }
2022-02-15T09:56:16.415Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144213,
     createdAt: 1644831144212,
     id: 7,
     name: 'enqueueAssetFileUpdates',
     updatedAt: 1644831144213,
     value: 'psfile://3MDC1yTZn/',
     version: 11 } }
2022-02-15T09:56:16.415Z sync-2523899 debug enqueueAssetFileUpdatesForUri(psfile://3MDC1yTZn/) Added outdated assetFiles: { queue: 'psfile://3MDC1yTZn/', assetFileCount: 0 }
2022-02-15T09:56:16.418Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144215,
     createdAt: 1644831144214,
     id: 8,
     name: 'enqueueAssetFileUpdates',
     updatedAt: 1644831144215,
     value: 'pslib:',
     version: 11 } }
2022-02-15T09:56:16.419Z sync-2523899 debug enqueueAssetFileUpdatesForUri(pslib:) Added outdated assetFiles: { queue: 'pslib:', assetFileCount: 0 }
2022-02-15T09:56:16.420Z sync-2523899 debug OpenedByIO() throwIfAvailable() { from: '(library health check)' }
2022-02-15T09:56:16.420Z sync-2523899 debug OpenedByIO() throwIfAvailable() { from: '(library health check)',
  lockOwner:
   { file:
      { nativePath: '/storage/photostructure/library/.photostructure/opened-by/34eju35-319059.json' },
     serviceName: 'main',
     createdAt: 1644831144095,
     updatedAt: 1644917544115,
     hostname: 'somedomain.com',
     systemUID: '1412-42bk-b0d3-040u',
     v: '1.1.0',
     pid: 319059 } }
2022-02-15T09:56:16.420Z sync-2523899 debug OpenedByIO() throwIfAvailable() { from: '(library health check)',
  json:
   { serviceName: 'sync',
     createdAt: 1644918976239,
     updatedAt: 1644918976240,
     hostname: 'somedomain.com',
     systemUID: '1412-42bk-b0d3-040u',
     v: '1.1.0',
     pid: 2523899 } }
2022-02-15T09:56:16.423Z sync-2523899 debug LibraryHealthChecks checkDir() { dir: '/storage/photostructure/cache',
  desc: 'Cache directory',
  bestVolume: '/storage' }
2022-02-15T09:56:16.424Z sync-2523899 debug LibraryHealthChecks checkDir() { dir: '/storage/photostructure/library',
  desc: 'Library directory',
  bestVolume: '/storage' }
2022-02-15T09:56:16.424Z sync-2523899 debug LibraryHealthChecks checkDir() { dir: '/storage/photostructure/library',
  desc: 'Library originals directory',
  bestVolume: '/storage' }
2022-02-15T09:56:16.424Z sync-2523899 debug LibraryHealthChecks checkDir() { dir: '/storage/photostructure/library/.photostructure/models/db.sqlite3',
  desc: 'Library model DB',
  bestVolume: '/storage' }
2022-02-15T09:56:16.426Z sync-2523899 debug Heartbeat assertPing() { heartbeat:
   { '$ctor': 'models.Heartbeat',
     createdAt: 1644831143730,
     id: 1,
     name: 'ping',
     updatedAt: 1644918976425 } }
2022-02-15T09:56:16.427Z sync-2523899 info  ModelDbUpdater Starting asset file updates { afqName: 'WorkQueue()',
  afqEnded: false,
  afqRunnable: true,
  idleStats:
   { doNotRun: { isPaused: false, ending: false, doNotRunImpl: false },
     idleListeners: [],
     runnableIdleListeners: [],
     pendingWorkItems: [] } }
2022-02-15T09:56:16.428Z sync-2523899 info  ModelDbUpdater Completed asset file updates { afqName: 'WorkQueue()',
  afqEnded: true,
  afqRunnable: false,
  idleStats:
   { doNotRun: { isPaused: false, ending: false, doNotRunImpl: false },
     idleListeners: [],
     runnableIdleListeners: [],
     pendingWorkItems: [] } }
2022-02-15T09:56:16.430Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144193,
     createdAt: 1644831144190,
     id: 1,
     name: 'enqueueAssetUpdates',
     updatedAt: 1644831144193,
     value: 'psfile://Z1d184amJ/',
     version: 3 } }
2022-02-15T09:56:16.432Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144198,
     createdAt: 1644831144197,
     id: 2,
     name: 'enqueueAssetUpdates',
     updatedAt: 1644831144198,
     value: 'psfile://r3VdktTo6/',
     version: 3 } }
2022-02-15T09:56:16.433Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144201,
     createdAt: 1644831144200,
     id: 3,
     name: 'enqueueAssetUpdates',
     updatedAt: 1644831144201,
     value: 'psfile://3MDC1yTZn/',
     version: 3 } }
2022-02-15T09:56:16.435Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144204,
     createdAt: 1644831144203,
     id: 4,
     name: 'enqueueAssetUpdates',
     updatedAt: 1644831144204,
     value: 'pslib:',
     version: 3 } }
2022-02-15T09:56:16.436Z sync-2523899 info  ModelDbUpdater Starting asset updates { aqName: 'WorkQueue()',
  aqEnded: false,
  aqRunnable: true,
  idleStats:
   { doNotRun: { isPaused: false, ending: false, doNotRunImpl: false },
     idleListeners: [ 'WorkQueue(AssetPreview)', 'WorkQueue()' ],
     runnableIdleListeners: [ 'WorkQueue(AssetPreview)', 'WorkQueue()' ],
     pendingWorkItems: [] } }
2022-02-15T09:56:16.437Z sync-2523899 info  ModelDbUpdater run(): finished
2022-02-15T09:56:16.438Z sync-2523899 info  TagSql forceRunTagMaintenance()
2022-02-15T09:56:16.438Z sync-2523899 info  Tag findByPath { tagPath: [ 'fs' ], dbIsNull: false }
2022-02-15T09:56:16.440Z sync-2523899 info  displayNameForVolsha(3MDC1yTZn) find() { vol:
   { filesystem: '/dev/sda1',
     mountpoint: '/storage',
     size: 1967934717952,
     used: 876896493568,
     available: 991001559040,
     ignorable: false,
     remote: false,
     label: 'Storage',
     uuid: 'd52c5696-194c-4298-a599-92f2677ce8f3' } }
2022-02-15T09:56:16.440Z sync-2523899 info  fixFileRootTag(Tag(18)) updated tag { id: 18, path: [ 'fs', '3MDC1yTZn' ], displayName: 'Storage' }
2022-02-15T09:56:16.457Z sync-2523899 debug TagSql vacuumLeafTags() { candidates: [] }
2022-02-15T09:56:16.458Z sync-2523899 info  TagSql vacuumLeafTags() complete. { candidates: [] }
2022-02-15T09:56:16.458Z sync-2523899 info  SyncService modelDbUpdater completed.
2022-02-15T09:56:16.459Z sync-2523899 debug Operation applyOnce(): already done { prior:
   { '$ctor': 'models.Operation',
     completedAt: 1644831144235,
     createdAt: 1644831144234,
     id: 9,
     name: 'applyNewTagger',
     updatedAt: 1644831144235,
     value: 'fs',
     version: 1 } }
4

rest of manual sync debug (reached max symbols of forum)

2022-02-15T09:56:16.459Z sync-2523899 info  SyncService directoryTaggerOp completed. Starting normal sync.
2022-02-15T09:56:16.462Z sync-2523899 info  SyncService Spawning new sync-file { execPath: '/usr/bin/node',
  syncFile: '/home/sony/photostructure/bin/sync-file.js',
  maxProcs: 3 }
2022-02-15T09:56:16.462Z sync-2523899 debug ChildProcess spawn() { command: '/usr/bin/node',
  args: [ '/home/sony/photostructure/bin/sync-file.js' ],
  maxAgeMs: 1800000 }
2022-02-15T09:56:16.467Z sync-2523899 info  sync-file Started child process sync-file:2523913
2022-02-15T09:56:16.468Z sync-2523899 debug ChildProcess stdoutResult(): execFile { cmd: 'renice', args: [ '9', '-p', '2523913' ] }
2022-02-15T09:56:16.468Z sync-2523899 debug ChildProcess execFile() { command: 'renice', args: [ '9', '-p', '2523913' ], maxAgeMs: 10000 }
2022-02-15T09:56:16.474Z sync-2523899 info  Renice Renice pid 2523913 to BelowNormal
2022-02-15T09:56:16.475Z sync-2523899 debug Pids addPid() wrote /home/sony/.config/PhotoStructure/pids/2523913.json { pid: 2523913,
  ppid: 2523899,
  cmd: 'sync-file',
  maxAgeMs: 1860000,
  startTime: 1644918976472 }
2022-02-15T09:56:16.973Z sync-2523899 info  SyncService Spawning new sync-file { execPath: '/usr/bin/node',
  syncFile: '/home/sony/photostructure/bin/sync-file.js',
  maxProcs: 3 }
2022-02-15T09:56:16.974Z sync-2523899 debug ChildProcess spawn() { command: '/usr/bin/node',
  args: [ '/home/sony/photostructure/bin/sync-file.js' ],
  maxAgeMs: 1800000 }
2022-02-15T09:56:16.982Z sync-2523899 info  sync-file Started child process sync-file:2523927
2022-02-15T09:56:16.983Z sync-2523899 debug ChildProcess stdoutResult(): execFile { cmd: 'renice', args: [ '9', '-p', '2523927' ] }
2022-02-15T09:56:16.984Z sync-2523899 debug ChildProcess execFile() { command: 'renice', args: [ '9', '-p', '2523927' ], maxAgeMs: 10000 }
2022-02-15T09:56:16.994Z sync-2523899 info  Renice Renice pid 2523927 to BelowNormal
2022-02-15T09:56:16.996Z sync-2523899 debug Pids addPid() wrote /home/sony/.config/PhotoStructure/pids/2523927.json { pid: 2523927,
  ppid: 2523899,
  cmd: 'sync-file',
  maxAgeMs: 1860000,
  startTime: 1644918976991 }
2022-02-15T09:56:18.033Z sync-2523899 info  TagSql forceRunTagMaintenance()
2022-02-15T09:56:18.034Z sync-2523899 info  Tag findByPath { tagPath: [ 'fs' ], dbIsNull: false }
2022-02-15T09:56:18.038Z sync-2523899 info  fixFileRootTag(Tag(18)) updated tag { id: 18, path: [ 'fs', '3MDC1yTZn' ], displayName: 'Storage' }
2022-02-15T09:56:18.071Z sync-2523899 debug TagSql vacuumLeafTags() { candidates: [] }
2022-02-15T09:56:18.071Z sync-2523899 info  TagSql vacuumLeafTags() complete. { candidates: [] }
2022-02-15T09:56:18.071Z sync-2523899 info  Service(sync) exit() { status: 0, reason: 'Finished', waitForJobs: true, ending: false }
2022-02-15T09:56:18.324Z sync-2523899 info  Service(sync) exit(): waiting for enqueued jobs to complete... { pendingNames: [] }
2022-02-15T09:56:18.324Z sync-2523899 info  Service(sync) exit(): enqueued jobs finished.
2022-02-15T09:56:18.325Z sync-2523899 info  Endable endEndables() { isTest: false, isSingleSpecTests: false }
2022-02-15T09:56:18.326Z sync-2523899 debug Endable vacuumEndables() [ [ 'first',
    [ 'tagsCache stats',
      'rawTagsCache stats',
      'Idle',
      'BaseFileCache stats',
      'imageCacheCleanup',
      'readdirCacheCleanup',
      'PosixFileCache stats',
      'readdirCache stats',
      'rpc.Client(db@localhost:1807#0)',
      'sync-file',
      'ModelDbUpdater',
      'ProgressUpdater',
      'SyncService.maybeRestartSync()',
      'TagAssetsWithFiles',
      'Syncs(import paths fs sync)' ] ],
  [ 'logger', [ 'LogWriter(/home/sony/.config/PhotoStructure/logs)' ] ],
  [ 'postdb', [ 'stats logger', 'OpenedByIO()' ] ],
  [ 'predb', [ 'Library(/storage/photostructure/library)' ] ],
  [ 'db',
    [ 'ModelDbJanitor(/storage/photostructure/library/.photostructure)',
      'Db(models)',
      'Db(stats)' ] ],
  [ 'service', [ 'statsDbJanitor vacuum' ] ] ]
2022-02-15T09:56:18.327Z sync-2523899 debug Endable endEndables(): ending first
2022-02-15T09:56:18.329Z sync-2523899 info  BaseFileCache stats { size: 20, cacheHitsSync: 5, cacheMisses: 20 }
2022-02-15T09:56:18.329Z sync-2523899 info  PosixFileCache stats { size: 32, cacheHitsSync: 61, cacheMisses: 32 }
2022-02-15T09:56:18.329Z sync-2523899 info  readdirCache stats { size: 2, cacheHitsSync: 1, cacheMisses: 3 }
2022-02-15T09:56:18.330Z sync-2523899 info  rpc.Client(db@localhost:1807#0) end() { ending: true, timings: {}, pendingSize: 0 }
2022-02-15T09:56:18.330Z sync-2523899 info  rpc.Client(db@localhost:1807#0) close() { ended: true }
2022-02-15T09:56:18.331Z sync-2523899 info  sync-file childEndCounts {}
2022-02-15T09:56:18.671Z sync-2523899 info  sync-file on(childExit) 2523913
2022-02-15T09:56:18.675Z sync-2523899 debug Pids addPid() wrote /home/sony/.config/PhotoStructure/pids/2523913.json { pid: 2523913,
  ppid: 2523899,
  cmd: 'sync-file',
  maxAgeMs: 1,
  startTime: 1644918918673 }
2022-02-15T09:56:18.949Z sync-2523899 info  sync-file on(childExit) 2523927
2022-02-15T09:56:18.950Z sync-2523899 debug Endable endEndables(): ending service
2022-02-15T09:56:18.950Z sync-2523899 debug Endable endEndables(): ending predb
2022-02-15T09:56:18.951Z sync-2523899 info  Db(stats) closing db Database {
  name: '/storage/photostructure/cache/sync-state-5kt2qphu1a/stats/db.sqlite3',
  open: true,
  inTransaction: false,
  readonly: false,
  memory: false }
2022-02-15T09:56:18.953Z sync-2523899 info  OpenedByIO() debug 'onEnd(): unlinking /storage/photostructure/library/.photostructure/opened-by/34eju35-2523899.json'
2022-02-15T09:56:18.955Z sync-2523899 info  Library(/storage/photostructure/library) onEnd(): finished.
2022-02-15T09:56:18.955Z sync-2523899 debug Endable endEndables(): ending db
2022-02-15T09:56:18.955Z sync-2523899 info  Db(models) closing db Database {
  name: '/storage/photostructure/library/.photostructure/models/db.sqlite3',
  open: true,
  inTransaction: false,
  readonly: false,
  memory: false }
2022-02-15T09:56:18.955Z sync-2523899 debug Endable endEndables(): ending postdb
2022-02-15T09:56:18.956Z sync-2523899 info  Service(sync) timings: { 'Service.startup()': { sumSec: 0.131, mean: 130, k: 1 },
  'db.rebuildTagFts': { sumSec: 0.031, mean: 31, k: 1 } }
2022-02-15T09:56:18.957Z sync-2523899 info  Service(sync) memory use on exit { ok: [ 'Memory by sync (28 MB) is OK' ], warn: [], bad: [], fail: [] }
2022-02-15T09:56:18.957Z sync-2523899 debug Endable endEndables(): ending logger
2022-02-15T09:56:18.959Z sync-2523899 debug Pids addPid() wrote /home/sony/.config/PhotoStructure/pids/2523927.json { pid: 2523927,
  ppid: 2523899,
  cmd: 'sync-file',
  maxAgeMs: 1,
  startTime: 1644918918958 }

PS the file didn’t get imported by manual sync. Maybe I was wrong with the first post

5

Howdy @mexus, welcome to PhotoStructure! Sorry that you’re banging up against this parsing bug.

There are several issues you’re hitting here: I’ve certainly fixed several in the v2.1 branch (like the invalid cameraId and imageId: see “deduplication improvements” in the release notes), but there may be more issues I’ve yet to encounter.

Can you email one of these .ARW files to support@photostructure.com so I can see what’s going on?

Cheers,

Matthew

6

Hello @mrm, thank you for looking into it.

I don’t believe I can send you the files over email since they are quite big. I’m uploading them to S3.

https://dev-fidweb.s3.eu-central-1.amazonaws.com/arw/A7402915-imported.ARW
https://dev-fidweb.s3.eu-central-1.amazonaws.com/arw/A7402916-notImported.ARW
https://dev-fidweb.s3.eu-central-1.amazonaws.com/arw/A7C01757-imported.ARW
https://dev-fidweb.s3.eu-central-1.amazonaws.com/arw/A7C01758-notImported.ARW

When do you expect to release v2.1 version or is it safe to switch to this branch now? Can I upgrade without having to rescan the whole library?

Thank you.

7

Thanks! I replied over email, but in case you missed it, I’ve fetched them now, and you can remove them from your s3 bucket.

It looks like the library I use to convert raw image formats, LibRAW, isn’t able to handle a couple of those files.

Darktable also couldn’t load the A7402916-notImported.ARW file, for example.

Are those files straight out of the camera?

As soon as I fix the last couple test failures in my continuous integration stack. Sometimes these are easy to fix, and sometimes they are evidence of a bigger issue that takes more time to remedy. :\

I’d stay on stable for now: the v2.0 alpha and beta builds had some show-stopping library database corruption errors for some people.

PhotoStructure will automatically upgrade your database to the new v2.x schema, but realize this migration is one-way.

I’ve made a number of deduplication improvements recently, however, so if you have the spare cycles, and you notice assets aren’t deduplicated correctly, you can run a rebuild to re-aggregate your library.

8

I did a bit more looking into how to add proper support for newer RAW image formats, and just found this tidbit in the library I use for rendering RAW images:

we do not consider production-quality rendering to be in the scope of LibRaw’s functionality

That’s … frustrating.

What I can do immediately is use the embedded full-size JPEG or TIFF in these raw image formats, and only fall back to dcraw/libRAW if that metadata has been stripped.

Thanks for the example images!

9

OK, I did a bit more research. LibRAW’s latest snapshot includes several more camera formats, so I’m upgrading PhotoStructure to use that version.

10

I’ve deleted the files from the bucket.

OK, I did a bit more research. LibRAW’s latest snapshot includes several more camera formats, so I’m upgrading PhotoStructure to use that version.

From the changelog it seems that is supports the Sony raw lossless compressed format, even though Sony A7 IV is not listed.

@mrm Can I update just this libRAW library or I have to wait for the 2.1 release? Also will it fix the issue or I need the re-duplication changes as well?

Also I can generate a JPEG for each ARW, but the grouping wasn’t working for me and was seeing both the JPEG and ARW files, so I’ve deleted the JPEGs and rebuilt the library (before this thread). Also if they are grouped in one photo in the library is there a way to download the ARW, since this functionality wasn’t working for me and I don’t really know.

Thanks

11

Yes, if you’re running PhotoStructure for Node: set NOGIT=1 to disable automatic updates in start.sh, and then replace the dcraw_emu binary in your tools/ subdirectory. (If you want to build from git, I wrote up instructions here: you’ll want to adjust the URL you download, of course.)

All variations of an asset should be visible in the “info” panel (click the “i” in the upper right corner), and then click the ellipses by the filename you want to download.

1 Like
12

@mrm thanks for the reply!

I’ve added the NOGIT option, compiled, prepared and replaced the executables from the latest LibRAW snapshot (the one with compressed lossless support).

Is there a way to get all my photos to import with version 1.1.0?
Like if I set PS_VALIDATE_RAW_IMAGES=“false” or another option?

13

PhotoStructure’s preview generation first sorts all the variants of a given asset, picks the “best”, and tries to build a preview of that image. If the best doesn’t work, it tries the next best, and so on, until it runs out of variants. So, assuming there are no bugs, having a RAW and JPG pair with a RAW that can’t be rendered will just mean the “primary” variant that’s chosen with be the JPG.

That may help, but if there’s a bug in preview generation, it’ll still bite us, because PhotoStructure will still try to render the raw as a broken TIFF.

Let me get v2.1 out and then we don’t have to do janky workarounds that we have to unwind later.

14

@mrm I think disabling the auto update and replacing libRaw executables with manually compiled from last snapshot fixes the problem for me (as far as I tested).

But there is another bug for me - wrote a separate topic for it. Filters not being kept between views.

Any news or ETA on the 2.1? I’m not rushing you, just curious.

15

I’m still banging on it. I found several issues with PhotoStructure’s concurrency library and those seem to be resolved now, and am trying to get CI all green before I ship the first alpha build.

image

It was over 400 failing this morning, but don’t do any linear interpolation on that progress…

No problem, sorry I can’t give a tighter timeframe.