New Files Are Not Pulled Into the Library

[tkohhh]

It’s a good question, and one that I don’t know the answer to! @mrm will need to chime in on this one!

[mrm]

@tompagenet

Which is quite interesting. For a start, it’s been way longer than ~21 hours. Secondly, if I’m wrong and it did sync 21 hours ago, then that would be well after many additional pictures were added to my hard drive, so they should be in PhotoStructure (they don’t seem to be).

So what you’re looking at here is the Operations table: each sync is an “operation,” and as the sync progresses, it updates the operation’s status (including an optional “completedAt” column).

Each time you restart PhotoStructure, or a new sync starts chewing on a directory, a new operation row is upserted.

The “last started” column is the newest operation row for that given directory. The “last completed” column shows the most recent operation that has a non-null “completedAt” column.

@tompagenet

What 21 hours does look more like is the total amount of run time of PhotoStructure last synced

No, they’re absolute timestamps, and the “… ago” is just a nicer way to read it (compared to unixtime 1628969889.878 or ISO 2021-08-14T19:38:38.413Z).

@tompagenet

which if so would be less than ideal as I don’t leave the computer on all the time and so it would seldom update

I’ve (tried to) design PhotoStructure to handle “always on” and “sometimes on” gracefully, so when it starts up and as it does work, it saves its place so it can restart where it left off.

There are a couple issues that we could be looking at:

1. Sync isn’t being scheduled to run regularly (but it seems that it is running on your box, but this is what @tkohhh and I are seeing)

2. Sync, when revisiting a previously-imported directory, is missing new contents (this is what I expect is impacting you).

I’m writing a test harness that plays with time so an integration test can exercise the first issue, and I’ll also write another test harness that imports a directory with some assets, then copies new files into preexisting directories and new directories, and verifies they get picked up correctly. (I have tests for both of these cases as lower-level validations for the DirectoryIterator classes, but no high-level sync-level ones).

[tim_w_photography]

My sync is finally running, after doing nothing since May. As part of trying to to figure out what was going on I clicked the Restart Sync menu item in the Sync menu numerous times but never noticed any activity in the UI. I am running the AppImage.

Quite by accident I clicked the Hamburger menu, and found the old Sync options. Clicking Restart sync there has the progress icons now. Hopefully my photos since May will start to poor in.

[tompagenet]

Thanks Matthew - for this and everything else. I think I have got somewhere. Here is what the dialogue showed yesterday (14 August) at 1942 UTC +1:

Then this morning (15 August) at 1053 UTC+1 (the computer was pretty much off entirely between these two points in time):

So far so good - PhotoStructure is recognising that time has elapsed and is noting that the scan is stale. However a few moments later (1056 UTC +1) I get this:

And yet none of the images taken since 17 July have been brought in. It seems to have in all cases started and completed the sync within the same second; does this mean it isn’t really running at all?

[avdp]

Mine is also stale… Didn’t change the defaults, so it’s 16 hours past-due. This is on an always-on docker on unraid. I’ve not had new pictures, so it’s no consequence to me at this time, however, it seems odd.

Sync information

path	status	last started	last completed
/	stale	1 day, 16 hours ago	1 day, 16 hours ago
/photos	stale	1 day, 16 hours ago	1 day, 16 hours ago
/ps/library	stale	1 day, 16 hours ago	1 day, 16 hours ago

[tkohhh]

Just reporting that after updating to v1.1.0, my sync went stale again:

path            status   last started              last completed
/photos/	    stale    9 hours, 7 minutes ago    9 hours, 4 minutes ago
/ps/library     stale    9 hours, 4 minutes ago    9 hours, 4 minutes ago

[mrm]

Sorry about this: I’ll try to get to the bottom of it tomorrow.

[tkohhh]

Any updates here?

[mrm]

I’m looking into it today: I have a hypothesis.

[tkohhh]

[bdillahu]

OK, sorry, that’s just funny

[organizeme]

As for my own syncing status being stale?

[tkohhh]

Well, I’m sorry to report that my sync went stale again after the latest beta release.

OOOH IT’S RUNNING NOW! YAY!

[mrm]

@tkohhh

OOOH IT’S RUNNING NOW! YAY!

FWIW, there’s an issue with the progress panels not rendering in v2.0.0-beta.1 that I’m fixing now.

Thanks for the update!

Also: we’ll only know that it’s actually working properly after it’s been up for more than 24 hours.

(I spent the last several days writing a (large!) integration test that lets sync “travel through time” to recreate this issue, and it found (a couple!) issues that v2.0.0-beta.1 resolves).

[mrm]

(I’m totally using that gif for alpha and beta now)