Missing file - no visible errors or rejections

Bug
1

Some of my files are still not synced without any visible errors

Please see the example

The full log is available in the gist as well

I’ve also included a problematic video file to the gist as well with the same symptoms

$ ./photostructure sync-file F:/SynologyDrive/Photos/0006/5242.jpg
...
{"ready":true}

$ ./photostructure info F:/SynologyDrive/Photos/0006/5242.jpg
...
errors: [],
...
rejected: []
...
ignoredBecause: []
...
validFile: 'OK',
...

$ ./photostructure list | grep 5242
# nothing
2

Thanks for the report: I’ll take a look tomorrow morning.

1 Like
3

New ETA? I have 15 files with the same symptoms

4

I investigated this a bit by myself.

I found that actually all my files were added to the library but the “missing” files from the ./photostructure list command are visible with ./photostructure list --where "Asset.shown = 0"

Then with ./photostructure list --where "Asset.shown = 0" | ./photostructure sync-file --force --info I can find why exactly my files are not shown

Some of my jpg files have

2021-03-27T12:11:07.906Z sync-file-25956 warn ImageHash(F:\SynologyDrive\Photos\0006\5242.jpg) Failed to imageHash(F:\SynologyDrive\Photos\0006\5242.jpg) { stack:
   [ 'Error: VipsJpeg: Invalid SOS parameters for sequential JPEG',
     'VipsJpeg: out of order read at line 0',
     '' ] }

While it is awesome to know that some of your files are corrupted and I think you should definitely keep such information and show in the asset info UI. But those file are perfectly viewable and you can still show them. Moreover, I think it is unacceptable that running sync-file without --info flag doesn’t show that those images are somehow broken.

 ./photostructure sync-file f:/SynologyDrive/Photos/0006/5245.jpg
{"path":"psfile://32hPdZdkz/SynologyDrive/Photos/0006/5245.jpg","assetFileId":19851,"assetId":19674,"memMb":38,"rssMb":110}
{"ready":true}

It seems, that you could fix the missing file via adding failOnError: false to your sharp calls. There is also a way to subscribe to the warning event Access Libvips failure/warnings with failOnError flagged false · Issue #2032 · lovell/sharp · GitHub not to lose them.

I am willing to be more helpful and make a Pull Request but I fail to find unminimized sources of your project.

Some files have a bit different issue

   [ 'Error: VipsJpeg: Corrupt JPEG data: premature end of data segment',
     'VipsJpeg: out of order read at line 1760',
     '' ] }

And, yes, those images are really corrupted at least partially. But all my comments above - apply.

Regards, problematic mp4 files, they have the following error

2021-03-27T12:11:31.263Z sync-file-25956 info ExifTool onStderr(20692):Error: Format error in file - C:/Users/mnaou/AppData/Local/Temp/PhotoStructure/imgcache/5w/wn/9j6z8p5hz5kfw7pbpkx1/wip-frame.jpg
2021-03-27T12:11:31.272Z sync-file-25956 warn ExifTool onError() (ending or ignorable): failed to run -charset
filename=utf8
-codedcharacterset=utf8
-sep

-E
-all=
C:\Users\mnaou\AppData\Local\Temp\PhotoStructure\imgcache\5w\wn\9j6z8p5hz5kfw7pbpkx1\wip-frame.jpg
-ignoreMinorErrors
-execute { stack:
   [ 'Error: Format error in file - C:/Users/mnaou/AppData/Local/Temp/PhotoStructure/imgcache/5w/wn/9j6z8p5hz5kfw7pbpkx1/wip-frame.jpg',
     '    at WriteTask.parser (C:\\Users\\mnaou\\photostructure-for-servers\\node_modules\\exiftool-vendored\\dist\\ExifToolTask.js:69:23)',
     '    at WriteTask.<anonymous> (C:\\Users\\mnaou\\photostructure-for-servers\\node_modules\\batch-cluster\\dist\\Task.js:122:51)',
     '    at step (C:\\Users\\mnaou\\photostructure-for-servers\\node_modules\\batch-cluster\\dist\\Task.js:33:23)',
     '    at Object.next (C:\\Users\\mnaou\\photostructure-for-servers\\node_modules\\batch-cluster\\dist\\Task.js:14:53)',
     '    at C:\\Users\\mnaou\\photostructure-for-servers\\node_modules\\batch-cluster\\dist\\Task.js:8:71' ] }

While when I run exiftool.exe on my mp4 files manually, I see no problem

1 Like
5

Thanks for looking into this! I’ll be able to dig into this in a day or two.

1 Like
6

I’ve done a bit of reverse engineering with the problematic mp4 files. Using procmon I found the call to the ffmpeg that extracts the frame from the video file. Apparently some of my files are broken as ffmpeg frame extraction produces empty jpg file. So, somehow I have files that are perfectly playable but invalid by their structure. I am going to recode them in order to be able to add them to the library

7

I’ve added another file to the gist above.

I found that PhotoStructure internally calls ffmpeg

ffmpeg -loglevel error -i F:\SynologyDrive\Photos\0017\16908.mp4 -ss 1.2 -vframes 1 -f singlejpeg -y C:\Users\mnaou\AppData\Local\Temp\PhotoStructure\imgcache\73\rg\nk8r8ggm2m79rzue8vgs\wip-frame.jpg

But for whatever reason it uses timestamp 1.2 while video is actually shorter

ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 F:\SynologyDrive\Photos\0017\16908.mp4
1.156000
8

Overall PhotoStructure doesn’t work fine with too short videos even if I set minVideoDurationSec = 0 in settings.toml

9

The 1.2 seconds is a workaround for many videos (especially AVCHD) whose initial frames are not decodable via ffmpeg.

PhotoStructure is supposed to look at the duration of the video and set the frame to be the min(duration, 1.2s), but perhaps that isn’t working correctly?

10

Actually all the problematic videos with that issue had duration less than 1.2. I am going to remove them from my collection anyway, but I think you should add to the configuration comment not to set it below 1.2. As I said I just had it set to 0

11

Oh, sorry, apparently I don’t know how to read. Yes, your min logic doesn’t work

12

Can you send me one of these short videos?

13

I’ve added the problematic video into the gist in the first message of the topic

14

I believe the new stream handling code in beta.10 may have addressed this issue: after you upgrade and sync completes, are those videos now in your library?