Why AssetFile.shown = 0

Bug
1

I have 405 files that for whatever reason have AssetFile.shown=0 but I can’t find any reason for that and they are not becoming AssetFile.shown=1 and corresponding Asset row is not being created even after ./photostructure sync-file --force --info and no errors shown

example

$ ./photostructure sync-file --info --force F:/SynologyDrive/Photos/0001/0157.jpg --no-color
...
... lots of info without any errors
...

$ ./photostructure list --where="AssetFile.shown = 0 and AssetFile.uri like '%/0157.jpg'"
F:\SynologyDrive\Photos\0001\0157.jpg
2

These column names are bad.

Asset.shown is set to 1 for assets that are ready to be presented in the UI.

AssetFile.shown is set to 1 for the file variants that are “primary” for their asset.

There should only be one primary asset file for any shown asset.

All of the asset files with shown=0 are “duplicates.”

1 Like
3

Is there a way how to determine what is this duplicate to?

4

You can look at the other asset files associated to the same asset.

If it’s the only asset file for the asset, I suspect it failed to create a preview image set, or ffmpeg failed to transcode, and the asset’s shown column should be 0 as well.

1 Like
5

Awesome. Then that’s a bug in the deduplication logic :slight_smile:

select uri from AssetFile where assetId = 98

psfile://32hPdZdkz/SynologyDrive/Photos/0001/0157.jpg
psfile://32hPdZdkz/SynologyDrive/Photos/0001/0158.jpg
psfile://32hPdZdkz/SynologyDrive/Photos/0001/0156.jpg

I added all those images to the gist above where it’s clearly seen they are totally different images

6

I’ll take a look at this later today: thanks for the report!

1 Like
7

I only see one of them in the gist.

1 Like
8

Sorry. git push failed. Fixed now

9

These images seem to have a precise (and same) captured-at.

PhotoStructure assumes if there are minutes or seconds or milliseconds set in the date, that it’s an actual timestamp, and not an estimated date stamp. Due to computational imagery, PhotoStructure has to dramatically relax minimum image correlation to make RAW and JPG pairs match.

This makes all the images be considered different:

exiftool -SubSecCreateDate="1985:12:06 00:00:00.000" 0*.jpg

And then deduping works:

$ photostructure info ~/Desktop/0156.jpg ~/Desktop/0157.jpg  | head
{
  fileComparison: 'These files represent different assets: low image correlation',
  variant: false,
  imageHashComparison: {
    imageCorr: 0.26,
    aRotation: 270,
    colorCorr: 0.53,
    meanCorr: 0.35,
    greyscale: false
  },

(perhaps I should add a heuristic that considers all dates less than, say, 1999 to be non-precise?)

10

Those photos were scanned and I added the date EXIF manually. I can’t know the time for the scanned photos so I put the artificial time

11

I think the heuristic could be that if you have 000 milliseconds, that’s an artificial time. I guess the 0.1% chance of the false positive heuristic is acceptable )

12

I think it’s simpler to just assume all times before 1999 are non-digital, and probably estimated. I’ll make that date a setting.

1 Like
13

Oops: I totally forgot about strictDeduping: just turn that on, and rebuild your library.

$ PS_STRICT_DEDUPING=1 ./photostructure info ~/Desktop/0156.jpg ~/Desktop/0157.jpg  | head
{
  fileComparison: 'These files represent different assets: low image correlation',
  variant: false,
  imageHashComparison: {
    imageCorr: 0.26,
    aRotation: 270,
    colorCorr: 0.53,
    meanCorr: 0.35,
    greyscale: false
  },

I’ll also add the “assume all dates before X are estimated” idea, too.

1 Like
14

Done! This is a comparison of the original images (with the 1985:12:06 19:35:56.762 captured-at):

$ node dist/library/info ~/Desktop/0156.jpg ~/Desktop/0157.jpg | head
{
  fileComparison: 'These files represent different assets: low image correlation (fuzzy dates require higher correlation)',
  variant: false,
  imageHashComparison: {
    imageCorr: 0.26,
    aRotation: 270,
    colorCorr: 0.53,
    meanCorr: 0.35,
    greyscale: false
  },

New setting documentation:


# +---------------------------+
# |  datesBeforeAreEstimated  |
# +---------------------------+
#
# PhotoStructure automatically interprets captured-at timestamps before this
# ISO-formatted datestamp as "estimated." PhotoStructure also considers
# timestamps whose minutes, seconds, and milliseconds are all 0 as
# "estimated."
#
# When assets have an estimated captured-at time, tighter image correlation is
# required to prevent different images as being incorrectly aggregated into
# the same asset.
#
# The default value is the release date for the Nikon Coolpix 950, which was
# one of the first  prosumer digital cameras with ~2MP resolution.
#
# PS_DATES_BEFORE_ARE_ESTIMATED="1999-02-15"

This will be in alpha.5.

1 Like
15

Is there a way to rebuild specific files only? ./photostructure sync-file --force leaves AssetFile.shown = 0. I am currently trying to ./photostructure sync --force --rebuild --exit-when-done but I think it’s an overkill to rebuild the entire library. Ideally I should be able to do something like ./photostructure --where "AssetFile.shown = 0" | ./photostructure sync-file --force --rebuild