PhotoStructure ignored the date metadata of some photos, marked them as taken in 2002

While I was browsing my library by date, I found a group of 62 assets, all tagged by the exact same date: Dec 8, 2002, 12:00 PM
These files have the correct date information inside the metadata and on the name, yet PhotoStructure still tagged them incorrectly.

Looking through the files, there doesnā€™t seem to be a pattern to them.
The dates vary between 2014, 2015 and 2016.
All were taken with my old Motorola XT1022 and uploaded to Google Photos using Picasa, but I have loads of other photos that fulfill these conditions that were tagged properly.

I am using PhotoStructure for Desktops 0.9.1 on Arch Linux.
Any logs I could provide to help debug this issue? Iā€™ll be mailing a sample of the affected pictures to support@photostructure.com shortly.

Edit: The photos have been emailed.

Thanks for reporting this!

You can run the info tool to see whatā€™s going on: https://photostructure.com/server/tools/#file-information

(add --verbose or even --debug for extra details), and look for the CapturedAt log messages.

You can also run exiftool -j -struct $path/to/file

Iā€™m happy to take a look at a sample image you email to me, too.

PhotoStructure takes the earliest valid time it finds in the metadata. In these filesā€™ case, CreateDate is set to 2002:12:08 12:00:00:

$ exiftool -CreateDate /home/mrm/Desktop/live/*.jpg
======== /home/mrm/Desktop/live/IMG_20141217_233300220-edited.jpg
Create Date                     : 2002:12:08 12:00:00
======== /home/mrm/Desktop/live/IMG_20141221_104133723-edited.jpg
Create Date                     : 2002:12:08 12:00:00
======== /home/mrm/Desktop/live/IMG_20150106_160343264-edited.jpg
Create Date                     : 2002:12:08 12:00:00
======== /home/mrm/Desktop/live/IMG_20150107_133241272-edited.jpg
Create Date                     : 2002:12:08 12:00:00
======== /home/mrm/Desktop/live/IMG_20150113_225416155-edited.jpg
Create Date                     : 2002:12:08 12:00:00
======== /home/mrm/Desktop/live/IMG_20150126_191717032-edited.jpg
Create Date                     : 2002:12:08 12:00:00
======== /home/mrm/Desktop/live/IMG_20150207_155319222-edited.jpg
Create Date                     : 2002:12:08 12:00:00
======== /home/mrm/Desktop/live/IMG_20160614_113451615.jpg
Create Date                     : 2002:12:08 12:00:00
======== /home/mrm/Desktop/live/IMG_20160614_114354045.jpg
Create Date                     : 2002:12:08 12:00:00
======== /home/mrm/Desktop/live/IMG_20160614_115211068.jpg
Create Date                     : 2002:12:08 12:00:00

You can remove the CreateDate tag from these files with exiftool: just run

exiftool -CreateDate= $path/to/file.jpg.

The next time PhotoStructure synchronizes those files, it should show the other date. Hereā€™s what info says after I had ExifTool remove CreateDate:

[
  {
    _PhotoStructureVersion: '1.0.0-alpha.0',
    capturedAt: {
      date: ExifDateTime {
        year: 2014,
        month: 12,
        day: 17,
        hour: 23,
        minute: 32,
        second: 59,
        millisecond: 0,
        tzoffsetMinutes: undefined,
        rawValue: '2014:12:17 23:32:59',
        zoneName: undefined
      },
      localCentiseconds: 2014121723325900,
      offset: undefined,
      src: 'tags:DateTimeOriginal'
    },

If this is a common issue in your library, would it be helpful to have the CapturedAtTags be a setting (so you could, say, remove CreateDate if you knew it was always going to be wrong)?

(How did it get set in the first place, I wonder?)

The current list is:

const CapturedAtTags = [
  // By the specification, DateTimeOriginal should be the time of the
  // shutter actuation, and CreateDate should be the time that the file
  // was written to the memory card (but not all mfrs follow the spec)
  // http://u88.n24.queensu.ca/exiftool/forum/index.php?topic=2568.0

  // <https://exiftool.org/TagNames/EXIF.html>

  // First "*original*"
  "SubSecDateTimeOriginal", // (fractional seconds for DateTimeOriginal)
  "SubSecCreateDate",
  "SubSecTimeDigitized", // (fractional seconds for CreateDate)
  "SubSecMediaCreateDate",

  "DateTimeOriginal", // (date/time when original image was taken)

  // Then "*create*"
  "OriginalCreateDateTime",

  "CreateDate", // (called DateTimeDigitized by the EXIF spec)
  "DateTimeCreated",
  "DateCreated", // XMP tag
  "DateTimeDigitized",
  "MediaCreateDate",

  "DateTime",

  // Then *modify*
  "SubSecTime", // (fractional seconds for ModifyDate)
  "ModifyDate", // (called DateTime by the EXIF spec)

  // Lightroom:
  // "HistoryWhen", // This is when history records happen, NOT when the file was captured!

  // Google Takeout (not really reliable, though)
  "photoTakenTime"

  // This is the Google Photos upload time. That's not something we want, so
  // it's commented out. DON'T ADD IT BACK, FUTURE ME!
  // "creationTime"
]

More details here:

Thanks for the fast replies, theyā€™re highly appreciated!

:eyes: :eyes: :eyes:

That would be very helpful, yes.
When PhotoStructure gains the ability to edit tags, it would be great if we could remove tags using the UI, bonus points if we could do it in bulk!

Quick question: Is the ā€˜Create Dateā€™ tag a frequently utilized in peopleā€™s libraries?
Maybe it should be weighted less than the ā€˜Image Date/Timeā€™ tag by default?

I wonder too. I have no idea of what could have caused this, other than a obscure bug inside Picasa (RIP!) or my old phoneā€™s firmware.

Given that @Lars_Noschinski saw this issue too, and in both cases, there were other tags that had good contents, I think itā€™s safe to just remove CreateDate from the list of tags that PhotoStructure looks at by default. (see below)

For extra credit, the list of tags will be configurable via the new capturedAtTags library setting:

# +------------------+
# |  capturedAtTags  |
# +------------------+
#
# The following tags are examined to parse a file's captured-at time. The
# earliest valid tag is returned. See
# <https://photostructure.com/faq/captured-at/> for details.
#
# PS_CAPTURED_AT_TAGS="[\"SubSecDateTimeOriginal\",\"SubSecCreateDate\",\"SubSecTimeDigitized\",\"SubSecMediaCreateDate\",\"DateTimeOriginal\",\"OriginalCreateDateTime\",\"DateTimeCreated\",\"DateCreated\",\"DateTimeDigitized\",\"MediaCreateDate\",\"DateTime\",\"SubSecTime\",\"ModifyDate\",\"photoTakenTime\"]"

This change will be in -alpha.3, which is a couple days out (I need to finish search).

1 Like

Would it be better to just include CreateDate at the end of the list? It would appear earlier it was midway in the list, but if it was ā€œlastā€ then itā€™s kind of an ā€œif all else failsā€.

Flip side, I can add it back if I want it, no biggie, just thought I would toss it out there.

2 Likes

I agree, it turns out that I had some examples that only had CreateDate, so removing that tag completely is a bit too blunt of a fix.

I added a special filter that discards CreateDate only if itā€™s this specific raw value.

Update: Iā€™ve added a new setting, badDates, which defaults to ["2002:12:08 12:00:00"] (this will ship with v1.0.0-beta.10).

1 Like

@LiveLM hereā€™s the parsing results for your files without this fix:

$ node dist/library/info --captured-at /home/mrm/Desktop/live2/* | grep Centiseconds
      localCentiseconds: 2002120812000000,
      localCentiseconds: 2002120812000000,
      localCentiseconds: 2002120812000000,
      localCentiseconds: 2002120812000000,
      localCentiseconds: 2002120812000000,
      localCentiseconds: 2002120812000000,
      localCentiseconds: 2002120812000000,
      localCentiseconds: 2002120812000000,
      localCentiseconds: 2002120812000000,
      localCentiseconds: 2002120812000000,

And with the fix:

$ node dist/library/info --captured-at /home/mrm/Desktop/live2/* | grep Centiseconds
      localCentiseconds: 2014121723325900,
      localCentiseconds: 2014122110413300,
      localCentiseconds: 2015010616034300,
      localCentiseconds: 2015010713324100,
      localCentiseconds: 2015011322541600,
      localCentiseconds: 2015012619171700,
      localCentiseconds: 2015020715531900,
      localCentiseconds: 2016061411345200,
      localCentiseconds: 2016061411435400,
      localCentiseconds: 2016061411521100,
1 Like

Dates look correct and they match the file names. Looks good to me!
When this fix makes itā€™s way into a release, Iā€™ll try making a new library containing the affected files to make sure they get tagged correctly.

1 Like

Hi,

Iā€™ve just started using Photostructure and imported a fairly large library of photos. Iā€™m using the Windows version V0.9.1 Beta. Iā€™m very impressed with it.
I noticed a whole bunch of photos (>1000) ended up with their dates at the 8 Dec 2002 when they were taken much later (2013-2015).
Using windows filemanager the correct date/time shows, but using Exiftool the Create date field is 8 Dec 2002 12:00pm but the Date/Time Original field shows the correct date/time.
From what I can see on the forums there is a common theme to the camera that took the photos when this problem appears. They all appear to be Motorola phones. In my case it is a Motorola XT1068.

So now I know what has happened, do I just remove the create dates from all of the 1000+ photos and then go through each photo in Photostructure one by one and tell it to resync or is there a way I can do this in a batch?
If I need to remove the create date, do I do this on the file in the Photostructure library or do I remove it in the original location(s)? Some of these photos are in several locations and Photostructure has done a great job of deduping them :grinning:.
Does the fix you mention above make any of this easier, and can it be applied to just the photos in the Photostructure library on this date or do I need to re-import all of my photos? It took about 4 days to do the initial import so I am hoping this will not be the case.

Anths

Welcome to PhotoStructure, @Anths !

Could you email me an example image that isnā€™t being parsed correctly? support@photostructure.com

You may want to try a v1.0.0 beta build, there are a ton of bugfixes and updates. Instructions are in the latest beta announcement.

Iā€™ll see if thereā€™s a way way to make this less time consuming.

Hi,

I have attached a sample of one of the photos that is not being parsed correctly.

Iā€™ll have a look at the v1.0.0 beta build.

Anthony

Thanks for your really quick reply. Much appreciated.

Being new to Photostructure I am not sure of all the features, but is it possible to define a small part of the Photostructure library to rescan without scanning the rest? Perhaps a way to scan a particular date range or by a particular tag or EXIF field?

Always feel free to ask if you canā€™t find something: more content on the forum makes it more likely for the Users Of Tomorrow to search and find an answer.

Good question! I answered this in a new topic:

I have quite a few scanned images. They have DateTimeOriginal set but it appears it was not used by PS. I estimated the dates on these old pictures after scanning and set them using Photoshop elements which I no longer use. Sometimes they have parts of the date missing. With exiftool it shows my DateTimeOriginal is ā€œ1998: : 00:00:00ā€. Thatā€™s two spaces after the first colon and three spaces after the second colon. I donā€™t see any messages about this being a bad format in the debug but PS doesnā€™t use it. Iā€™m going to script something to fix this with exiftool since I have the date I want in the filename, but I just wanted to mention in case you want to take a look at see if it ps could handle partial dates better. Photoshop Elements is a popular product even though I no longer use it so you could hit this again. If you want I can mail you a bad file.

I made the fixes and Iā€™m trying to quickly check the files. Syncing seems to miss some or maybe is still running I canā€™t tell. Iā€™m using docker and I connect to the container usually using portainer. When I try to run the manual sync commands as described at PhotoStructure | PhotoStructure Tools I always get:

empty libraryDataDirā‚€
{ā€œfatalā€:true,ā€œreasonā€:ā€œsync setup failed: empty libraryDataDirĀ¹ā‚€ā€,ā€œstatusā€:14}

Using either root or node acct I get the same error. Is there some environment I need to set up using docker after Iā€™ve connected to info photostructure commands?

thanks

Interesting, Iā€™ve never seen that format before: itā€™s always been something like ā€œ1998:0:0ā€ if only the year is set.

Iā€™ll add this as a format that PhotoStructure knows how to parse. Thanks for sharing!

Are you attaching to a currently running container by using docker exec? If youā€™re using docker run, you need to remember to include the bind mounts.

Also: make sure youā€™re using a :prealpha build (just for now, until I get this release out and set to :stable)ā€“it should have much better error messages when things arenā€™t set up as it expects, thanks to the new health check system.

Iā€™m connecting using exec to the existing container. Using portainer actually so Iā€™m at the bash prompt as root. ā€œsu node -cā€ doesnā€™t change the error.