New work scheduler, with both a new “single-threaded” mode and support for 20+ CPU servers, with improved docker and k8s support, and soft-timeout concurrency backoffs
Running the alpha installer will replace the currently installed version. PhotoStructure for Desktops will detect that it’s an alpha build, and automatically upgrade you to newer alpha or beta builds, and then “stick” to stable builds (once v2.1-stable is released).
PhotoStructure for Docker users
Shut down PhotoStructure
Run docker pull photostructure/server:alpha in a terminal
Change your docker run command to use photostructure/server:alpha
PhotoStructure for Docker Compose users
docker-compose down
Edit your docker-compose.yml to use image: photostructure/server:alpha
docker-compose pull ; docker-compose up -detach
PhotoStructure for Node users
Make sure you’ve got Node.js 16.15.0 or later installed. If you’re using nvm, run nvm install 16 ; nvm alias default node to get the latest.
Shut down PhotoStructure
cd into your photostructure-for-servers directory, and run git checkout alpha ; ./start.sh
I’m seeing “Cannot find module ‘stream/promises’”
Please upgrade to Node 16 or later. Instructions are above.
OMG I upgraded and my screen is blank!
If you’re accessing your library via http, know that newer versions of Chrome and Firefox will include a new Upgrade-Insecure-Requests header. If you open your browser’s developer console, you’ll see something like this:
I’ll get these sorted as soon as I can. If you see other bugs, please post here or in discord!
papaparse not found (fixed in alpha.1)
Docker build takes 40 minutes for GitHub Actions to release (the fix requires extracting the prebuild step into a new published base-tools image) (fixed in next build)
The docker v2.1.0-alpha.1 build doesn’t seem to have been tagged with :alpha (fixed in next build)
Windows fails to launch with “SYSTEMROOT not set” (issue with batch-cluster and new Node 16 windows fork behavior, fix soon)
look into @avdp’s SQLite corruption issue (can you start with a new library, and if it recurs, send me debug logs, please?)
About page has a caution icon by the library directory on docker
I got a sqlite corruption while navigating through the UI. Restarting the container (on UNRAID) cleared out the issue for now. Nevertheless it seems the sqllite issues are not behind us with 2.1.0-alpha
{"toast":{"text":"Well, darn, that didn't work","uuid":"SqliteError: SQLITE_CORRUPT: database disk image is malformed","details":"SqliteError: SQLITE_CORRUPT: database disk image is malformed","type":"warning","button1Text":"Return home","button1Url":"/tag#dismiss","button1Class":"info","onClickUrl":"#dismiss"}}
Bug, I think: Sync => Pause processing/Resume & Restart Sync is not doing anything for me. This is an issue when you have a big library and the process got interrupted. Even after a restart (app and system) it doesn’t continue with the import/scan process. A “forced” restart sync from the menu doesn’t help either.
It seems “exposeNetworkwithoutAuth” is also not working…it generates an URL, but the page stays white. Tested on iPad, Android, Win 10 with Chrome, Safari, Firefox and Edge.
No worries at all: I appreciate your help in banging on the alpha build, and I suspect this will trip up more people.
Is there a safe way I can default to making things “work by default”, I wonder?
For reference, v2.1 now has the following CSP/CORS/ “web security” settings:
upgradeInsecureRequests, if true, adds the upgrade-insecure-requests CSP directive. If false, the directive is omitted. CSP linters suggest that it be included.
cspReportOnly, if true, uses the Content-Security-Policy-Report-Only header rather than the standard Content-Security-Policy. CSP violations are only reported. This should only be set to true temporarily for debugging CSP errors.
enableWebSecurity: this defaults to true if you use --expose or exposeNetworkWithoutAuth=true. If explicitly set to false, it changes the default of cspReportOnly to true and upgradeInsecureRequests to false. If you’re accessing your PhotoStructure library via the web interface, you want this set to true.
So, with that out of the way–perhaps I default upgradeInsecureRequests to false?
After explaining this, I think enableWebSecurity is too confusing and complicated: I’m going to delete it.
Related: did people know that there’s a trustProxy setting? It currently defaults to false because I’m chicken, but I think defaulting it to loopback might be almost as safe of a default. More defaults about trustProxy are here.
Upgrade insecure requests assumes that you are capable of serving over https. How many of your users do you think are using https? I would leave it out until such a time as Photostructure supports https out of the box.
@mrm - by the way, this was with a brand new library (I always wipe everything out between version upgrades). When I restarted the container it continued where it left off and it has since complete successfully. I will retry from scratch again this weekend, just to see if I reproduce.
I take that back. After some testing it works for me, but I changed now from CIFS to NFS, which might or might not have something to do with it.
Is it just me or is the import process much slower than in other versions? It’s running now for 2 hours or so and it feels significantly slower.
Update: I might understand now why it feels slow. It seems that (obviously) videos take much more time to import. Once that is done, the progress on the photos is a lot faster…so it’s a perception thing…
I have now imported my library with 36k images and ~1k videos. The sync-report is quite helpful, thank you!
Some observations:
After ~26 hours and 19k processed pictures the process just stopped and I had to force a re-sync, after another 12 hours the remaining pictures got processed
I have quite a few pictures that got rejected with “dimensions too small” or “file too small”
Seems like “wmv” is not supported => “not supported mime type, dimensions too small”
Seems lime “bmp” is not supported => “not photo or video extension”
I have quite a few pictures that failed with “Cannot build previews: none of the files are valid: VipsJpeg: Invalid SOS parameters for sequential JPEG²ⶔ (happy to provide an example file per PM message)
Building on my observations a few questions/comments:
Many videos got rejected because they are too big or too small. In fact I would actually prefer to not process videos at all, but “link” to the original source instead and displaying a thumbnail only. This would save a lot of processing time and keep the overall footprint small. I understand that not all users would like to have it like that, but as an option I would appreciate that
It would be helpful too understand what parameters to tweak if you have a lot of CPU power, if you have a lot of RAM, or even both
Adding folders to exclude (also ex-post) as per my PM would be awesome
Overall it runs so far very stable and does what it should.
Many videos got rejected because they are too big or too small. In fact I would actually prefer to not process videos at all, but “link” to the original source instead and displaying a thumbnail only.
You can tell photostructure not to transcode videos. Either set transcodeVideos=false in the settings file, or set the PS_TRANSCODE_VIDEOS environment variable to false. (depending on how you’re running photostructure)
Please do send me any photos or videos whose dimensions are not being extracted correctly!
As @avdp stated, there’s a PS_TRANSCODE_VIDEOS which is the big switch to disable all transcoding, but you can configure which video types get transcoded, and at what bitrate, via the following settings:
