Library remains empty after sync

Expected Behavior

Hi, I’m a new user - I’ve been watching the PhotoStructure website for many months and decided to finally jump in and give it a go. As I wanted search, I installed the 1.0.0.2 beta.

I expected to be able to install PhotoStructure, and point it at some photos, and then have them appear in the app.

Current Behavior

I installed PhotoStructure, it started and synced, and then none of my photos have appeared and it says my library is empty.

Steps to Reproduce

I installed PhotoStructure and configured it:

  • With a library at a new location D:\PhotoStructure
  • Pointing at a subset of my photos (just from 2014) at D:\PhotoStructurePictures (the folder path under here is D:\PhotoStructurePictures\Events\2014* where about a dozen folders are containing JPG images).
  • I’m using the Lite version so no reorganising of images

My photos started to be scanned -

Then the scan was complete -
Edit I had taken time to capture screenshots and add them in here, but on trying to save this bug, it says new users can only upload one screenshot, which is daft so you just get the first screenshot

But it always says the library is empty, and the about page says there are 0 “assets” (I don’t know if that’s bad, but it doesn’t seem good)
Edit I had taken time to capture screenshots and add them in here, but on trying to save this bug, it says new users can only upload one screenshot, which is daft so you just get the first screenshot

Note - it’s really off-putting to a new user who is just trying to raise a bug to have messages telling them they can’t upload multiple images, especially if they took the time to screenshot them for you.

Environment

Windows 10 20H2
Dell i7 laptop with 16GB RAM

PhotoStructure edition: PhotoStructure for Desktop

I suspect you’re bumping up against one of the many asset filters that PhotoStructure has:

I’m going to make logs visible in the UI in a future release, which should help make this much more discoverable. Without that, answering the question of “WTH is PhotoStructure doing” requires a terminal that’s running logtail, which is decidedly non-optimal.

Agreed, and apologies: this forum is powered by Discourse, and it has hundreds and hundreds of settings. There are some (like what you just bumped up against) that I haven’t correctly dialed in yet. I’ll see if I can fix that today.

Thanks for your time!

I suspect you’re bumping up against one of the many asset filters that PhotoStructure has

I read through that article and none of the reasons made sense for my library.

I’m going to make logs visible in the UI in a future release

That would be awesome. I spent ages looking on the forum trying to figure out how to see logs from the Windows desktop install.

I actually got it working - I uninstalled the Windows Desktop version and deployed a Docker Compose version, 0.9.1. This was dramatically slower to scan files, approx 5 minutes for 300 files rather than < 1 min. I pointed it at the exact same path as the Desktop version.

A noticeable difference immediately is that the thumbnails were showing for images during the sync:

And it works - I can browse my library and all the photos are there.

I tried changing to server:beta and doing a docker-compose pull, docker updated the container, but unfortunately it won’t start up now, Docker is showing it continually restarting:

2021-05-12 21_30_20-Dell server - AIDAN-DELL - Remote Desktop Connection

If you’d like me to try the Windows Desktop version again with some instructions on getting to the logs I’d be happy to try it?

There are some (like what you just bumped up against) that I haven’t correctly dialed in yet

I read the levels in Discourse, and to get to a level where I can post multiple images I have to spend something like 10 minutes reading posts.
The problem is, by the time I posted, I’d already spent over an hour (as an unregistered user) trying to read through forum posts to see if someone else had experienced the same problem. I eventually decided to raise a bug, and it was at that point I registered. So all my browsing before counted for naught.

Fixed that too, deleted the opened-by folder.

Now on 1.0.0b2 and it’s working great after a library rebuild.

I’m fine with Docker, I’m still curious why the Windows version didn’t work…

(forgot to mention - the Docker container is running on the same machine that the Windows Desktop version was installed on, just in case that wasn’t clear.)

About for the working Docker install:

Hello,

I have exactly the same problem: photostructure finds my pictures, scans them and when it finishes the library is empty.
I have no thumbnails visible during scan, but I see all the steps : thumbnailing, transcoding, etc.

Very important: looking at the logs I see that my files are not filtered, it happily reads them, reads the metadata and completes each asset.
Interestingly in the logs I see that it insists that my library is on a remote drive with this message:

2021-05-13T16:17:00.824Z web-38 info  ModelDbJanitor(/ps/library/.photostructure) Library on remote drive. Using local db replica. { src: '/ps/library/.photostructure/models/db.sqlite3',

The corresponding docker volume is on a local filesystem.

I’m using the latest beta docker image on Linux.

@venza sorry to hear this: can you email me your logfiles so I can take a look?

This is actually just the default when PhotoStructure runs in a docker container. You can override this by setting PS_FORCE_LOCAL_DB_REPLICA=false

Hello,
I sent the logs and my docker startup command line to the support email address.

I tried with the library on NFS and on the local filesystem with no change.

Thanks for the heads-up: I’ll check on it this morning.

I just looked, and SQLite is having problems writing to /ps/tmp. Is this volume bind-mounted to a local disk?

2021-05-14T03:47:03.050Z main-25 error DbBackup(/ps/tmp/local-db/models/db.sqlite3) Error: Could not back up db /ps/tmp/local-db/models/db.sqlite3 -> /ps/tmp/local-db/models/backup/db.sql
ite3: SqliteError: code SQLITE_IOERR: disk I/O error¹⁶
SqliteError: disk I/O error
    at Immediate.step (/ps/app/node_modules/better-sqlite3/lib/methods/backup.js:43:29)
    at processImmediate (internal/timers.js:461:21) { from: 'DbBackup._backup() failed', srcDb: '/ps/tmp/local-db/models/db.sqlite3' }

This is most likely due to slow windows disk I/O. I have to quintuple timeouts on Windows compared to Linux (on the same hardware and same disks) for I/O-bound tests: and importing is mostly likely bottle-necked by disk I/O. I think it’s just par for the course, unfortunately.

Me too: if you’re bored and have the time, and want to re-try with Windows, if you set PS_LOG_LEVEL to debug, reinstall and re-do your prior setup, and then send me the logs, I can take a look.

It’s Windows on desktop that was much faster than Docker

I tried - first up couldn’t even get it to start but luckily a reboot cleared that (Docker had been shutdown):

After rebooting, I could get further - got to this screen and clicked Continue:

Then saw the plans screen, clicked free, then it just showed a blank screen, although the menus worked:


Link to logs for this: Dropbox - File Deleted - Simplify your life

Deleted C:\Users\aidan\AppData\Roaming\PhotoStructure (note - I know this folder didn’t exist before I installed PS because I had gone looking to check) and this time it started, and I could get to the setup part.

Configured it like this:

And now it’s working. Thumbnails showing up and although the view says “library is currently empty”, there’s images in the views. I haven’t let it scan all 28,104 images but it’s making progress just fine:


It’s done 500 photos and 46 videos while writing this bug.

There’s a couple of things that I’ve changed since last time around which may have affected it:

  • I had 100s of photos without a DateCreated field, so PS would just show them based on the disk modified time (i.e. 2021-May), which was quite annoying, since most of them were from 2002-2004 era, and Photostructure would insist on putting them under “When 2021”. I used exiftool to find the Lightroom date and wrote that to “alldates”. These photos were 95% of the time from a Samsung SII (GT-I9100) phone.

  • Most of my videos from older smartphones didn’t have a proper datetime stamp embedded in them. All the DSLR ones did using QuickTime:CreateDate or the like. Again I used exiftool to write correct datetime stamps to the video files.

So - maybe it just got stuck on one of the files without a createdate, but like immediately? :woman_shrugging: It’s working now… back to Photostructure for Servers anyway.

Can you send me an example or two that PhotoStructure had to resort to using stat times? I wonder if there’s a captured-at tag hiding in there that we can use.

Thanks for the updates!

Here’s how photos with this problem appear in Lightroom, the “Date Created” field is empty.

To resolve it I use this command with exiftool:

And now the Date Created is populated, which means PhotoStructure will put it into the correct “When” bucket:

PS not 2002-2004, meant 2012-2014 - the Samsung S2 only came out in 2011 :slight_smile:

DMing you a couple of those photos now.

OK, well, here’s what beta.3 will think:

$ ./photostructure info ~/Downloads/awojtas/50331* --filter capturedAt

{
  a: {
    nativePath: '/home/mrm/Downloads/awojtas/50331730731_dba3316db6_o.jpg',
    capturedAt: {
      date: ExifDateTime {
        year: 2020,
        month: 9,
        day: 12,
        hour: 9,
        minute: 23,
        second: 16,
        millisecond: 0,
        tzoffsetMinutes: 720,
        rawValue: '2020:09:12 09:23:16',
        zoneName: 'UTC+12'
      },
      localCentiseconds: 2020091209231600,
      offset: 720,
      src: 'tags:ModifyDate'
    }
  },
  b: {
    nativePath: '/home/mrm/Downloads/awojtas/50331884062_1e7156785f_o.jpg',
    capturedAt: {
      date: ExifDateTime {
        year: 2020,
        month: 9,
        day: 12,
        hour: 9,
        minute: 23,
        second: 0,
        millisecond: 0,
        tzoffsetMinutes: 720,
        rawValue: '2020:09:12 09:23:00',
        zoneName: 'UTC+12'
      },
      localCentiseconds: 2020091209230000,
      offset: 720,
      src: 'tags:ModifyDate'
    }
  }
}

That seems to match what you were expecting, correct?

Yup spot on!

I just released beta.3, hopefully it’ll fix at least your captured-at issue.

I also fixed (at least one cause) of an erroneous “library is empty” message on initial run in beta.3, so that may not be an issue anymore.

I looked at the db you DMed me, and the schema doesn’t match up with what the migrations should result in. If you start with a new PhotoStructure library, that should result in a correct schema (feel free to send me a new db if it doesn’t…).

1 Like

We could probably mark this issue as resolved? The original problem has been sorted (although I’m not sure the exact root cause is known)

Will try that, ta