When should new files show up?

Scenario -
I have a Photostructure Node instance with 30k photos. It imported all those in one night.

Later, I add 100 new photos to the library. They’re recent photos. I want to show these to a family member the next day.

Currently they don’t show up. I added them at 10pm and at 8am the next day they’re not in the library.

How do I get these photos to show up more promptly? (or even when will they show?)

Thanks!

PhotoStructure’s sync process should wake up periodically, see what scan paths haven’t been visited for a while, and restart a sync for each of those paths.

This is controlled by the syncIntervalHours setting, which defaults to 24 hours. This topic was brought up a while back, too.

This had been handled by a main when sync got periodically restarted, but if sync is long-lived, I just realized I needed to add an additional “kick” to make sure daily scans happen when they should. This will be added in the next build.

Thanks for bringing this to my attention!

Great, thanks! I missed finding that setting in my search, ta for the link.

I wonder if it would be useful to be able to do a sync just for a folder, kind of like a “Resync this asset” but for a folder and sub-folders. For example, 95% of my library changes will be under the “2021” folder this year, no need to resync folders going back to 1980 as often.

1 Like

YUP: I think this would be super handy!

You can do this currently, if you don’t mind running things on the terminal and are using either PhotoStructure for Servers:

https://photostructure.com/server/tools/#manually-importing-files-and-directories

But making this “just work” would be much nicer. I’d thought of adding a “quick sync” option that visits the last N grandparent folders that new assets have been found in.

This would also handle it, I think:

Unfortunately this didn’t work. I did the following:

  1. 10pm Thurs added new photos to library (from Dec-2020 through to Apr-2021)
  2. 8am next morning Friday, no new photos showing
  3. 1pm Friday shutdown PS, changed syncIntervalHours to 10, started PS back up.
  4. 11pm Friday, no new photos showing.
  5. 9am Saturday, no new photos. Click “Restart sync”. All new photos show up correctly within 5 minutes.

So to me that says the periodic sync isn’t working (neither the 24 hour default, nor my reduced 10 hour setting). The workaround is okay, clicking Restart sync.

Did I change it in the right place? Why isn’t the 24 hour sync working?

Environment: Photostructure for Node, 1.0.0b3, library is in C:\PSLibraries\All.

Right. Sorry, I wasn’t very clear in the prior reply: thanks to your prior description of the problem, I looked at that section of code again, and found the bug:

This will be fixed in beta.4 (and then if you can run that test again, that’d be splendid).

Ah, gotcha, ta much. I’ll give it another go when beta 4 is out :slight_smile:
Pretty easy workaround in the meantime.

Related: Does PhotoStructure stay in sync with filesystem changes?

To add to this, this doesn’t really make much sense as a default on the windows version (or any non-server based version). I’ve added photos throughout a month, but they still haven’t been picked up.
I typically only use my PC with photostructure in the evenings for 4 hours.

I’ve also attempted removing and re-adding the folder, but no change.

Today, I just updated to beta 4.
Photostrucutre is doing… something. It’s write rate is ~100 MB/s in 5 sec bursts, but there’s no indication on the UI.
However, no new media has shown up.

I’m a huge fan of having an import folder that brings in new media, and then optionally deletes it.

Uh oh! Can you set PS_LOG_LEVEL=debug, restart the app, wait a couple minutes, shut down, and then send me your logs?

Also: I’ll be away from my desk until next week, so I’m afraid I won’t be able to take a look at to them until then. Sorry in advance for the latency…

I’m also getting the same thing, but not spikes, more like 200MB/sec consistently. A little worried it’s destroying my SSD. This is how my SSD has been looking since yesterday when I upgraded:


(note I didn’t rebuild my library)

I run three libraries and looks like the write-ahead log of the database “db.sqlite3-wal” for each library is getting hammered:

The file size seems normal.
image

When I run just one library I can see the spikes actually:

Apologies! I’ll look into this disk I/O issue tomorrow after I fix the docker binaries issue.

Was this bursty disk I/O new behavior with beta.5?

This behaviour was observed specifically on beta 5, but it could be beta 4 or 5 - didn’t spend long enough on beta 4 to notice.

OK, I think there were actually 2 issues going on:

  1. Database VACUUM and OPTIMIZE pragmas weren’t being called properly, which may have led to your large write-ahead-log file

  2. Filesystem root tags were having asset recounts every 5 seconds which was unnecessary. The next beta only rebuilds the filesystem roots when new mountpoints are seen.

  3. sync or sync-file may be crashing for some reason and getting automatically restarted. Restarting requires re-reading the library db which is a bunch of disk I/O. If you send me logs (or if you want to check yourself), we can see if there are any JavaScript heap out of memory or similar ilk.

Looking forward to the next beta. I’ve tried beta 7 and the bursty disk thrashing is there too.

In the meantime I’m leaving PS stopped since I don’t want to burn out the SSD :fire_extinguisher:

Logs coming up via dm.