Creation Time, Formatted Time, Total Confusion; or: Google Takeout Sidecar Files Are Misnamed

[tmeitner]

All righty.

I started to import my existing Google Photos Takeout files into Photostructure and noticed that a lot of the dates were screwy. I use an app to merge the JSON sidecar data and put it in the metadata of the image. I figured there was something wrong with that approach.

So, I wanted to start fresh, exporting a new Google Takeout of a handful of albums and importing them with the sidecar files into Photostructure, since PS can read the sidecar files anyway.

Doesn’t seem to make a difference - some dates are still screwy.

As an example, I pulled up the info on one. Maybe we can tag team this and figure out what’s going on. It might not be a bug, just something I’ve got to address in my files.

Below is the JSON info on one of the photos. Please enjoy my grandmother’s caption she wrote on the back of the photo…

When it imports into PS, however, it puts the time of the photo in May 2020, for some reason.

So, you know, I’m confused. I want it to be the “1950” date, as that’s how I have them organized in Google Photos. I want that organization to transfer over. Any thoughts as to what Photostructure is basing this date on and how I can fix that?

Thank you!

[mrm]

We’ll get this right: something about your images aren’t working with the current parsing heuristics for captured-at:

How does PhotoStructure capture captured-at?

If you DM or email me an example or two, along with their sidecars, I can see what’s going on.

[tmeitner]

You’re the man. Thanks. I’ll send over some.

[mrm]

Are you using v0.9.1, or a beta of v1.0?

Version 1.0.0 added JSON parsing, and seems to result in the correct value.

Here’s what it looks like on my box:

./photostructure info image.jpg
[
  {
...
    capturedAt: {
      date: 1950-01-02T03:20:00.000Z,
      localCentiseconds: 1950010119200000,
      offset: undefined,
      src: 'tags:photoTakenTime'
    }
...
  }
]

If you update to v1.0.0, you’ll need to “rebuild” your library to re-parse your image metadata (rebuilding should be scheduled automatically).

[tmeitner]

Yep, I’m on 1.0.0-beta.2. And this was from a fresh library startup after the upgrade to the beta (which I grabbed because of the JSON coverage and the face tagging).

[mrm]

Can you DM me the exact JPG and JSON filenames?

You can also peek into the inner machinery by running the info tool on your own box, and add either --info or --debug to the command:

https://photostructure.com/server/tools/#file-information

If you send me the output of that, I can snip out the relevant bit and explain what’s going on.

[tmeitner]

Alllll righty, let’s see…

1589854801991-1b68b4d1-8980-47a2-a59d-07f1fe791.jpg
1589854801991-1b68b4d1-8980-47a2-a59d-07f1fe79.json

No typos there. An extra “1” in the jpg name. Interesting. In fact, every single JSON file I’m looking at here has a one-letter difference. Maybe that’s supposed to work that way, heck if I know.

Okay, so for the info tool, I apologize for my ignorance. How do I run this? The link you sent over was for checking files, not the whole system. The command above it does throw info on the system, but it’s a docker command. I’m running on Ubuntu using the AppImage, so I’m not sure what the command is.

[mrm]

Ooof, that’s the issue. I’m expecting sidecars will be named with either (the full image name or basename of the image), followed by .json

This is because my own takeout sidecars are named. In this fashion. Are all of yours literally off by one, or are they sometimes different characters, or different suffixes?

If you don’t want to check, you can you email me a find -type f from the base of your Google Takeout, or just a subset of files, so I can poke at it.

[tmeitner]

I ran that command to take a look at the list. It appears that there are small handfuls of this happening throughout the folders. Most of the photos match their JSON files, but then there are little bunches where they’re just off by one letter/number.

It’s additionally super strange because the files all have normal file names, but then these ones that don’t match have weird, super-long file names with random characters. I wonder why that is.

So I guess my next question would be: if I go in and change the file names to match, will PhotoStructure automatically pick up on the update and rescan the info?

[mrm]

@tmeitner

if I go in and change the file names to match, will PhotoStructure automatically pick up on the update and rescan the info?

It should: if there are changes to either the file or the sidecar, the sync should re-import the file.

If you didn’t manually edit the sidecar names, though, I think it’d be better for PhotoStructure to properly handle this situation, though: I’ll think of a heuristic to make PhotoStructure consider these UUID-based-filenames have off-by-a-character sidecars and get that into the next beta build.

[mrm]

[tmeitner]

Seems like such a mess.

I appreciate the heck out of the work you’re doing here!

[mrm]

OK, this is coded up and will land in v1.0.0-beta.3

[tmeitner]

Well I am definitely looking forward to it. Thank you!

[mrm]

How’s beta.3 working out for you with this issue?

[tmeitner]

Hey man, thanks for following up!

I deleted my PhotoStructure library to start fresh with the same batch of photos with the JSON sidecars from Google Takeout, and… nope. It’s still a mess.

Every photo you see here was not taken in 2020/2021. Some of them were scanned in 2020. But a lot of the ones in 2021 weren’t scanned in 2021 nor were they taken in 2021.

I picked one at random, so let’s see what we can see. This picture is not a victim of the weird naming issues of the last time. Here’s what it looks like in PS:

I can see that the naming of the sidecar file is fine. But the modified date of these files is… May 8, 2021, which is what it is dated in PS:

And here are the contents of the JSON file attached to this particular photo:

{
  "title": "image-0142.jpeg",
  "description": "Lynne Meitner\n1947\n[unknown] Rest Cemetary",
  "imageViews": "0",
  "creationTime": {
    "timestamp": "1596250831",
    "formatted": "Aug 1, 2020, 3:00:31 AM UTC"
  },
  "photoTakenTime": {
    "timestamp": "-725791800",
    "formatted": "Jan 1, 1947, 3:10:00 PM UTC"
  },
  "geoData": {
    "latitude": 0.0,
    "longitude": 0.0,
    "altitude": 0.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "geoDataExif": {
    "latitude": 0.0,
    "longitude": 0.0,
    "altitude": 0.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  }
}

(Unrelated side note: I hope eventually we can get those descriptions to show up, because that’s how I preserved the writings on the backs of old photos when I uploaded them to Google Photos. I know it’s one of a billion things you’re working on - as long as they are preserved in the sidecars for now, we’ll get there.)

So… it appears the “file modified” information is what is getting pulled into PhotoStructure. Interesting.

[mrm]

Oof. This is yet another issue.

In this example, the JSON sidecars aren’t matching the image filename: I’d need to strip the “-1” and “-edited” image file suffixes to match with the sidecar.

The prior fuzzy matching I added only came into play for uuid-like filenames (I was conservative in the application of that heuristic as I was concerned about incorrect sidecar adoption).

I’ll try to get this fixed today.

And yes, I want description importing too!

[tmeitner]

Sorry I’m throwing so many bugs at you! (Or “you’re welcome for surfacing these bugs”? However you prefer to look at it.)

[mrm]

OK, this additional sidecar wonkiness will be handled in beta.4.