Photostructure Image Loading Speed

rgrandy · April 22, 2021, 3:29am

I am running alpha 7 on an Unraid Server in docker. I don’t have an SSD so everything is on some large traditional hard drives.
I have noticed that it can take a while for the image grid previews (quite a few seconds) to load on each screen and them when I click on an individual picture it can take up to 5-6 seconds to load (although this seems to vary depending on the image).
Is this expected timing? Is there anything I can do to speed this up?

Although, after having a window open with the program for a while, it seems like it speeds up loading everything.

mrm · April 22, 2021, 6:12pm

I’ve tried to design PhotoStructure to respond to most API and browsing events in <100ms (assuming reasonable file storage speed), so short answer: no, I want most pages to display instantly. 5 seconds makes me sad.

Especially during imports, and especially if your PhotoStructure container has been provisioned with lots of CPUs, PhotoStructure’s import processes can overwhelm the I/O of spinning rust devices, especially if they’re hosted on a low powered NAS.

Was this during an import?

rgrandy · April 22, 2021, 7:24pm

I hadn’t used photostructure in a while, although I do believe the docker was running for a few days already. After using it for a while it started to get a lot more snappy and the performance was more inline with what I might expect.

I’ll play around and see if I can replicate the behavior.

Rodger · April 25, 2021, 4:26am

+1 on this issue.

I’ve got 78,000 assets loaded, seems like all the processing and importing is done - and now it’s dog slow. Large thumbnails loads up pretty snappy, medium thumbnails pretty slow. Small thumbnails loads a few and then the rest are blank.

mrm · April 25, 2021, 5:11am

Can you tell me more about your setup?

Is you library stored on a local disk, or remote filesystem? What type of storage is it (spinning rust, SATA SSD, m2 SSD)? How is the disk formatted?

Depending on your browser’s screen size, small thumbnails can cause several hundred files to be requested almost simultaneously. This is no problem for SSDs, but can take a while on slower disks: but it absolutely shouldn’t just stop loading.

If you could send me debug logs of your web service that’d be helpful.

https://photostructure.com/faq/error-reports/#how-to-manually-send-your-logs

Rodger · April 25, 2021, 2:33pm

My library is stored on an external USB drive. According to StableBit Scanner the bus speed for this drive measures at ~240MB/s. (so about 2Gb/s).

The disk is a spinning HDD, formatted as NTFS.

The instructions for manually sending logs are unclear. Step 2 for ‘PhotoStructure for Desktops on Windows’ (part 3) says “click New” - but there are multiple options for clicking new on that window. Should it be new User variable, or new System variable?

Regardless, I may not need to send logs because I am seeing the disk utilization at 100%.

When I first open photostructure it says “Loading your library…” then “Setup timed out after 35 seconds.”. During this time the disk utilization gets pegged at 100% in windows task manager.

When I open photostructure the 2nd time, it loads successfully. However, after clicking around for a minute or so, it appears to stop loading. I previously assumed it had completely stopped, but when I monitor the disk usage I see that it’s pegged at 100% for up to 30 seconds before pictures show up. Seems like I wasn’t waiting long enough before.

Additional considerations I’ve noticed:
After clicking around through the pictures for a bit, when I decided to return to the root/main home screen - I was tapping ESC several times. Similar to navigating Steam in Big Screen mode - when you’ve navigated pretty deep and want to go back home you can just keep hitting ESC over and over until you’re at the main screen. I think what’s happening here is once I’ve already made it the home screen, if I quickly tap ESC 3 or 4 times then it’s queuing all of those requests up for more random pictures. Plus, since I used ESC to get back through the breadcrumbs - each one of those breadcrumb pages is still processing. For example, If I’m looking at a picture in When / 2019 / Nov, if I rapidly hit escape 5 times then it has to load a ton of pictures for When / 2019, then a ton of pictures for When, then a ton of pictures for the home pages 3 more times. If I’m correct on how that’s working, here’s my recommended fixes:

Don’t have ESC do anything when you’re on the main page/home page. If a user wants to refresh, perhaps use F5 instead since that’s a pretty standard refresh button.

Also, don’t load random pictures every single time when traveling through the breadcrumbs. For example, if I’m looking at When / 2016 and I see two pictures that I like - I can only look at one of them full screen. If I click one (of the two pictures) to view full screen, and then hit ESC - the other one is now lost because all new random pictures have been loaded. Not having to load a new batch of pictures EVERY time would also help with the disk-crushing utilization. Or, perhaps give users an option to choose between:
“automatic random picture refresh (recommended for SSD)”
“manual random picture refresh (recommended for HDD)”

Finally - perhaps there could be a better way to confirm whether or not photostructure is done processing. I’m only guessing it’s done because the number of assets has been stable for a while, and there is no scanning progress showing at the top of the screen. However, it’s confusing because the scanning progress wasn’t always showing previously when it WAS running. Also, I still have the option to right-click on the taskbar icon and choose “pause processing”. I think it should be grayed out when processing is completed, or changed to somehow indicate that processing is done.

mrm · April 29, 2021, 4:44am

Ah, thanks for the detailed response!

PhotoStructure’s random thumbnails on a large 4k display can result in many hundreds of small-file read ops, which on spinning rust HDDs is typically 20-50x slower than contiguous large-file reads.

That’s an interesting idea to have an “easy mode” for slow disks. I’ll think about that.

Until then, though, I’d recommend buying an SSD. A 1TB top-tier m2 NVME can be had for under $100, and it’s so nice to have everything load instantly.

PhotoStructure also supports putting the previews and DB on a separate path (on your new SSD!), and keep all your originals on much cheaper and larger HDDs.

(I’ll reply to your other concerns and suggestions tomorrow).

mrm · June 15, 2021, 3:46am

I looked into this last week and found a couple things:

Chrome is staggeringly faster than Firefox at rendering tiny thumbnails, at least on my Ubuntu desktop. Chrome renders the home page instantly or certainly within 300-500ms full-screened on my 4k display with tiny thumbs. Firefox takes 2-6 seconds to load the same page, and seems to lag-glitch randomly. Firefox is my primary browser, so this wasn’t fun for me to find out. I looked at the Firefox performance devtools, and it seems like it’s doing craptons of re-layout work, even though I’m using a grid. Interestingly, if you reduce the Firefox browser window to be ~75% of the 4k display, page layout speed increases dramatically.
I realized I could adopt lazy loading for all browsers except Safari: this change made it into beta.4. It didn’t seem to help speed up Firefox, though.

awojtas · June 15, 2021, 6:58am

I use a 4K, and Chrome is indeed sadly faster than Firefox (also my primary browser) for this.
Chrome loads the PS home with medium thumbs in 1.3sec vs Firefox 2.6-2.7sec. On a Dell XPS 15 i7-7700hq.

One reason why it looks janky on Firefox IMHO is that the “video duration” label is rendered before the thumbnail, so you see random “00:45” labels appear floating in the dark briefly before the thumbnails appear.

mrm · June 15, 2021, 6:20pm

UGH this bothered me too. I played with having the duration not render until the image is fetched, but honestly any “cleverness” just seems to make poor Firefox even slower.