Library not showing photos more recent than 2010

[awojtas]

Expected Behavior

I have photos from 1980-01-01 to today’s date. I expect PS to show these photos in the Library view e.g. under When and Folders.

Current Behavior

PS is only showing photos from 1980-01-01 to 2010-12-31. Photos in 2011 onwards are not showing.

Steps to Reproduce

A bit lengthy but I’m writing it all out just in case there’s something relevant here…

1. On my laptop I have 30k photos in Lightroom, a mix of jpgs and raws. I use a tool called Folder-Publisher by Jeffrey Friedl to automatically publish JPGs, automatically downsized to 3000 pixels max dimension to a server running PhotoStructure.
2. PhotoStructure for Servers (Docker Compose) 1.0.0-b2 is running on an i7 server with C: (fast SSD) containing the library, logs, cache and D: containing all photos. Re-organise is disabled because I’m happy with the folder structure.
3. The server folder D:\PSMedia contains about 30k photos from 1980 through to today. PS is configured to use this as the only source for media.
4. Given all the above, do a rebuild, and wait many hours. Eventually the “asset” count stops and it will only show photos up to 2010.
   More info below.

Environment

Operating system and version: Windows 10
PhotoStructure edition: PhotoStructure for Docker

Further info

Here’s how my PSMedia structure looks, with an example of a folder that isn’t imported:

But here’s how PS appears in the When view:

And the Folder view:

The About screen shows only 8706 assets:

Which is far short of the PSMedia’s contents, even allowing for the odd dupe:

I did the info command, and PS seems to think it’s got the photo, ignoredBecause is empty:

The bit that was cut off in that screenshot…

Help?

[mrm]

Just checking your logs now. There are…

$ find ~/Downloads/awojtas/Logs/ -name sync-\* | grep -v sync-file | wc -l
2053

…a lot of sync logs there. I’d expect 1, or 2, or maybe three. It looks like sync was flapping due to the error you found already, UNIQUE constraint failed: Progress.uri.

I’m surprised that you’re seeing an error, though, because I deleted that unique constraint on that column a while ago: it was removed by the 20200927_progress_v2 migration that should have been applied to your database.

Can you DM me a zipped copy of your db so I can see what’s going on? It’s sitting in $LIBRARY/.photostructure/models/db.sqlite3

(Basically sync won’t be able to continue without fixing this issue, which explains why there are a bunch of files that haven’t been imported yet)

[awojtas]

I’ve DM’d you a database

In case it’s relevant, the docker image was initially a 0.9 release before I changed it from server to server:beta and did a docker-compose pull. Did I miss something in that upgrade?

[awojtas]

Actually - no it’s not - this database is a completely clean start from 1.0.0b2. I take it back

I started with a completely blank library folder for this one.

[awojtas]

Noting that @mrm suggested starting with a new library so will be giving that a go. It’ll take a day or so to get to the point of knowing if it’s worked so will report back here.

I’ve updated to 1.0.0b3.

What I’ve done to create a new library:

1. Shutdown PS & docker-compose down
2. Delete the PSLibrary folder on disk
3. Re-create PSLibrary folder with Cache, Config, Logs sub-folders
4. Start up PS with docker-compose up -d
5. Go through the startup wizard of PS.

[mrm]

If you could make sure this is in your docker-compose that’d be a help…

    environment:
      - "PS_LOG_LEVEL=info"

(you could go with debug, but info should still help figure out what’s going on)

[awojtas]

Sure… it died trying it the first time with database lock errors (the sqlite database file never got written to disk). The second time I got this:

Third time lucky?

Edit: this is the line from web*.log that threw that message:
{“ts”:1621404120715,“l”:“error”,“ctx”:“Migration({“schema”:“models”,“db”:”/ps/library/.photostructure/models/db.sqlite3"})",“msg”:“Failed to apply migration models/20181030_models.sql”,“meta”:“Error: Failed to apply: CREATE INDEX IF NOT EXISTS asset_timeline ON Asset (capturedAt)\nWHERE\n shown = 1\n AND hidden = 0: SqliteError: code SQLITE_ERROR: no such column: capturedAt¹⁶\nSqliteError: no such column: capturedAt\n at Database.exec (/ps/app/node_modules/better-sqlite3/lib/methods/wrappers.js:9:14)\n at t.Migration.applySqlMigration (/ps/app/bin/web.js:9:675607)\n at async t.Migration.applyMigration (/ps/app/bin/web.js:9:675019)\n at async Object.t.asPromise (/ps/app/bin/web.js:9:571505)\n at async Object.t.thenOrTimeout (/ps/app/bin/web.js:9:120072)”}

[awojtas]

3rd time was the charm. I did a --force-recreate on the docker-compose up command for some extra oomph.
Syncing now, will leave overnight and check tomorrow

[awojtas]

Saw this while it is syncing. Maybe I spoke too soon. Is this a problem? The sync progress seems to still be running fine.

[awojtas]

PS - thanks for all the great support. I’ve just subscribed to plus.

[mrm]

Is this running docker from a Windows host?

I’ve not seen sqlite have so many issues before: I suspect the filesystem bridge in docker when hosted on Windows is just janky enough to make sqlite sad.

Can you run the node server version instead? I have a number of people running this way on large libraries on their Windows boxes without problems.

Thanks for subscribing!

[awojtas]

@mrm

Can you run the node server version instead? I have a number of people running this way on large libraries on their Windows boxes without problems.

Done and it’s working great! I went for docker because my goal is to have multiple libraries hosted off the same machine, and docker seemed easiest for that (I don’t know how to do that with Node yet). But Node seems to solve my problems so that’s a win!

I read up on docker compose bind volumes, apparently it uses CIFS behind the scenes when mapping Windows paths, which posters online were saying sqlite doesn’t get on with. I saw some suggested workarounds but didn’t investigate any.

Node has brought in 30k of photos overnight and seems to perform significantly better not just for the sync but also for serving up photos.

Happy to consider this resolved. Thanks!