New Files Are Not Pulled Into the Library

Bug
,
17

Thanks for the report and welcome to PhotoStructure, @odd7of8 !

As I was waiting for CI to go green tonight, I thought of a different way to schedule sync jobs and coded it into beta.10, which should resolve this issue.

I’m hoping to release beta.10 late tomorrow or Sunday. :partying_face:

image
image1620×2160 223 KB

18

image
image1079×1068 216 KB

19

LOL

I’ve been wrestling with batch-cluster for the past couple days to ensure sync-file health checks run properly and that stderr/stdout streams are flushed completely before resolving or rejecting tasks. When examining (tons) of people’s logs they send in, if their sync got “stuck” it seemed to be due to this issue.

1 Like
20

@mrm Leave something to do for Beta 11 :slight_smile:
There is enough in beta 10 to kick the tires on already, time to release!

21

Thank you for your work on this!

At least there’s an easy workaround to click Restart Sync until it can be fixed

22

OK OK V1.0.0-beta.10 released 🎉

5 Likes
23

For the people that upgraded to beta.10 or later: are you seeing the sync get rescheduled daily?

24

I’ll be looking at this tomorrow morning, as I dropped a bunch of new files earlier today!

25

At what time does the scheduled sync run? I dropped some new files yesterday that are not yet showing in Photostructure.

26

So at this point I can confirm that new photos are still not being pulled in on beta 12.

27

Here’s the output from info --sync-paths:

[
  {
    nativePath: '/photos/',
    uri: 'psfile://3HZ4UtEZv/',
    lastUpdatedAt: 2021072021454327,
    lastCompletedAt: 2021072021454327,
    lastStartedAt: 2021072021405619
  },
  {
    nativePath: '/ps/library',
    uri: 'pslib:/',
    lastUpdatedAt: 2021072021454343,
    lastCompletedAt: 2021072021454338,
    lastStartedAt: 2021072021454330
  },
  nextPathToSync: '/ps/library',
  restartSyncAt: '(sync is overdue!)'
]
28

OK. I’ve updated the sync rescheduler in beta.13 to run every hour with logging, so it’ll be way faster to debug.

2 Likes
29

Is there an ETA (other than soon™) on beta.13?

30

31

FYI, here’s what my sync-paths looks like currently:

[
  {
    nativePath: '/photos/',
    uri: 'psfile://3HZ4UtEZv/',
    lastUpdatedAt: 2021080517460945,
    lastCompletedAt: 2021080506421744,
    lastStartedAt: 2021080506421741
  },
  {
    nativePath: '/ps/library',
    uri: 'pslib:/',
    lastUpdatedAt: 2021080506421748,
    lastCompletedAt: 2021080506421746,
    lastStartedAt: 2021080506421745
  },
  nextPathToSync: '/photos/',
  restartSyncAt: '(sync is overdue!)'
]
32

Well, new photos did not show up this morning. Here’s the output of sync-paths:

[
  {
    path: '/photos/',
    library: false,
    next: false,
    status: 'synced',
    lastStarted: '12 hours, 47 minutes ago',
    lastCompleted: '12 hours, 44 minutes ago'
  },
  {
    path: '/ps/library',
    library: true,
    next: false,
    status: 'synced',
    lastStarted: '12 hours, 44 minutes ago',
    lastCompleted: '12 hours, 44 minutes ago'
  }
]

Just so I understand the logic… does sync only run if it’s considered stale, and it’s only considered stale if it last ran >24 hours ago?

33

So based on that report, sync ran to completion.

Right, and that interval is configurable.

If you click search, and click the saved “recently updated” search, and no new files are there, that must mean that sync didn’t pick up new files. I’ll look into that.

34

Actually, I think we’re fine (at least for now). 12 hours and 44 minutes ago was when I updated to beta.15, which would have kicked off a sync manually. New files WERE pulled in at that time. Then, I added some more files to see if they would show up on the next sync, erroneously thinking that would happen overnight. So I need to wait until the last sync was 24 hours ago.

Humbly I’d like to suggest that sync be scheduled absolutely, rather than relative to the last sync. Absolute scheduling produces reliable behavior, while relative scheduling could be affected by upgrades and manual syncs.

35

Well, it went stale again. I changed the syncIntervalHours to 6 in order to get through the troubleshooting process quicker.

Any ideas?

[
  {
    path: '/photos/',
    library: false,
    next: true,
    status: 'stale',
    lastStarted: '15 hours, 8 minutes ago',
    lastCompleted: '15 hours, 8 minutes ago'
  },
  {
    path: '/ps/library',
    library: true,
    next: false,
    status: 'stale',
    lastStarted: '15 hours, 8 minutes ago',
    lastCompleted: '15 hours, 8 minutes ago'
  }
]
36

I have some pretty big drives that took a long time to import–way longer than 24 hours–so to prevent them from being restarted before they completed, I made the interval count from last-completed, rather than last-started.

Sigh! I’ll check my test libraries and see if they’re misbehaving as well.