Photo (not) corrupted

Expected Behavior

NEF files are imported without error.

Current Behavior

PhotoStructure refuses to import some NEF files saying that they are corrupted, but they aren’t.
Metadata are read correctly, then the call to dcraw fails.

2021-06-15T18:55:26.084Z sync-file-29458 warn  SharpReadable strategy failed for /pictures/Altre/2017/02/20170218 Tanneron 4282.NEF: Error: readableToFile() failed: Problem from /pictures/Altre/2017/02/20170218 Tanneron 4282.NEF: data corrupted at 879074⁶⁷

If I open the same file on my PC with dcraw (v. 9.27) and the same parameters I have no error and I get a good TIFF file in output.

Steps to Reproduce

Mailed a link to the NEF file.


PhotoStructure docker beta.7 running on Linux. Library mounted on NFS on the host and shared as a volume in the container.

Thanks for mailing the example .NEF.

I believe the issue should be addressed in beta.8 (or later):

:sparkles:/:bug: Fixed a race condition in child process handling affecting videos and raw images that could cause these files to not be imported.

I’ll keep this open for a couple days to give you time to upgrade and verify things seem to be working…

On beta.9 the issue seems to be still there, should I do a full rebuild, or delete the library folder and restart from scratch ?

If you run sync (via the main menu) it should pick up the .NEF: if it doesn’t, can you send me the output of running

docker exec -it $name_of_your_photostructure_container sh
./photostructure  sync-file /path/to/your.nef --verbose

Ok, with sync it picks up the NEF files and produces the same “data corrupted” error as before.

2021-06-23T08:37:13.312Z sync-file-3052 info  SharpReadable sharpReadable() -> dcraw() { src: '/pictures/Altre/2017/02/20170218 Tanneron 4274.NEF' }
2021-06-23T08:37:13.476Z sync-file-3052 info  ChildProcess stdout.on(data): /usr/bin/dcraw_emu { pid: 3082 }
2021-06-23T08:37:13.477Z sync-file-3052 warn  Deferred({"cmd":"which","args":["dcraw_emu"]}) settled multiple times (already resolved) { value: { result: '/u    sr/bin/dcraw_emu\n', code: 0, pid: 3082 } }
2021-06-23T08:37:13.498Z sync-file-3052 warn  SharpReadable strategy failed for /pictures/Altre/2017/02/20170218 Tanneron 4274.NEF: Error: readableToFile() f    ailed: Problem from /pictures/Altre/2017/02/20170218 Tanneron 4274.NEF: data corrupted at 1606274⁶⁷
2021-06-23T08:37:13.500Z sync-file-3052 info  Deferred(AssetFileImporter(/pictures/Altre/2017/02/20170218 Tanneron 4274.NEF).apply) .reject() { cause:
   { cause:
      'Problem from /pictures/Altre/2017/02/20170218 Tanneron 4274.NEF: data corrupted at 1606274',
     retriable: true,
     fatal: false,
     stack: undefined },
  retriable: false,
  fatal: false }
2021-06-23T08:37:13.507Z sync-file-3052 warn  Error onError(): Error: invalid file /pictures/Altre/2017/02/20170218 Tanneron 4274.NEF: readableToFile() failed: Problem from /pictures/Altre/2017/02/20170218 Tanneron 4274.NEF: data corrupted at 1606274⁶³⁷ { event: 'nonFatal', message: 'Failed to import /pictures/Altre/2017/02/20170218 Tanneron 4274.NEF⁶' }
2021-06-23T08:37:13.507Z sync-file-3052 info  SyncFileService importFile() finished in 2800ms { result:
   { path: '/pictures/Altre/2017/02/20170218 Tanneron 4274.NEF',
      'Error: invalid file /pictures/Altre/2017/02/20170218 Tanneron 4274.NEF: readableToFile() failed: Problem from /pictures/Altre/2017/02/20170218 Tanneron 4274.NEF: data corrupted at 1606274⁶³⁷' } }
2021-06-23T08:37:13.509Z sync-file-3052 info  Service(sync-file) exit() { status: 0, reason: 'Processed files from argv', waitForJobs: true, ending: false }

That’s frustrating, sorry. There’s got to be something different between how I tried to reproduce your issue and how you’re reproducing it: what’s the NFS server you’ve got? What Linux is serving your docker container?

I can reproduce the problem with the files on a local directory, it is not dependent of NFS.
The host is running Ubuntu 18.04.5.
I will send you a zip of the directory with the troublesome files by mail (there will be also some MOV files that ffmpeg doesn’t like).

I use this command line:

  --name photostructure \
  --detach \
  --restart on-failure `# < optional: starts PhotoStructure on boot.` \
  --stop-timeout 120 `# < gives PhotoStructure 2 minutes to shut down cleanly.` \
  --publish 1787:1787 \
  -e TZ="$(cat /etc/timezone)" \
  -v "$PSLIBRARY":/ps/library \
  -v "$PSTMP":/ps/tmp \
  -v "$PSCONFIG":/ps/config \
  -v "$PSLOGS":/ps/logs \
  -e PS_LOG_LEVEL=info \
  -e PS_TAIL_LOGS=1 \
  -v "/home/user/tmphoto:/pictures"             `# < CHANGE THIS LINE` \

I will send you a zip of the tmphoto directory.

I may have found the issue: there was a memory leak in steam handling, and incorrect error recovery code in the parent process. I’m fixing this now.

Thanks again for the report!