Support for S3 storage

[copied from the subReddit ]

I see on PhotoStructure’s “What’s Next” page that " Non-local imports (via S3 or other URL-accessible sites)" is coming, someday.

That would be AWESOME. I self-host and currently pay about $40/month for volume space for my assets on Digital Ocean. If PhotoStructure could address those assets via DO’s Spaces, I’d only be spending like $5/month.

I realize it’s a feature you, yourself, want, so it’ll likely come someday. I can wait. I’m just giving you feedback that I want it, too. :slight_smile:

Howdy!

Will the new originalsDir and something like rclone --mount suffice, do you think?

https://photostructure.com/about/2020-release-notes/#more-storage-flexibility

Since you’re suggesting it, I bet it works and solves my need. I’ll report the results back here if I’m smart enough to get it working.

1 Like

I think what you’re wanting to set up here will be a common setup: if you get stuck or see issues, I’d be happy to help.

How much space do you need? Not sure if you’re willing to switch from DigitalOcean, but BuyVM provide block storage at $5/TB/month, in 256 GB increments (so $1.25 per 256 GB), and the VPS just sees it as another regular hard drive. No S3 API needed, just mount it like any other drive. You can use whatever file system you like (ext4, zfs, btrfs).

The downside is that it can only be used with BuyVM’s VPSes. Their $15/month (4 GB RAM, 80 GB SSD, unmetered bandwidth @ 1 Gb/s) and above plans include dedicated CPU usage, so you can use the CPU 100% with no problems. DigitalOcean don’t provide dedicated CPU unless you get their “CPU-Optimized Droplets” which start at $40/month.

I swear I don’t work for them; I’m just a happy customer :stuck_out_tongue:

I think I did figure out how to do this, but Photostructure doesn’t seem to like it.

I got rclone to mount an Amazon S3 bucket containing a few pics and Photostructure launches and shows me the initial Settings and EULA screen, and it even creates a .photostructure dir in the S3 bucket, but then it crashes and kills the mount when I click the “Save” button on the EULA screen (i.e., the photostructure container shuts down and the rclone mount dies with “Transport endpoint is not connected”).

FYI, I’m using this to set up Photostructure on a Digital Ocean droplet.

Here’s the Photostructure container log from startup to “crash”:

Upgrading your library database...
PhotoStructure is ready: <http://localhost:1787/>
2021-01-03T14:40:28.726Z main-13 error Caught error from file write stream Error: EBADF: bad file descriptor, write
2021-01-03T14:40:32.282Z sync-33 error Caught error from file write stream Error: ENOTCONN: socket is not connected, close
WatchedChild.onError() {
ctx: {
src: 'ChildService(sync).onStdout()',
fatal: true,
ignorable: false,
errToS: 'sync-file: internal error: Error: onStderr({"error":"Health checks failed¹","problems":["Cannot write to /ps/library: undefined: _nativeCopyFile(/ps/library/.tmp-swr2ku/write-test.jpg.gz): {\\"src\\":\\"/ps/app/public/images/splashbg02-1024w.jpg.gz\\",\\"dest…'
},
src: 'ChildService(sync).onStdout()',
error: l [Error]: ChildService(sync).onStdout()sync-file: internal error: Error: onStderr({"error":"Health checks failed","problems":["Cannot write to /ps/library: undefined: _nativeCopyFile(/ps/library/.tmp-swr2ku/write-test.jpg.gz): {\"src\":\"/ps/app/public/images/splashbg02-1024w.jpg.gz\",\"dest…¹⁶
at C.onError (/ps/app/bin/main.js:3:133290)
at A.onStdout (/ps/app/bin/main.js:3:130238)
at s.onData (/ps/app/bin/main.js:3:137340)
at /ps/app/bin/main.js:3:203449
at Array.forEach (<anonymous>)
at s.onChunk (/ps/app/bin/main.js:3:203396)
at Socket.<anonymous> (/ps/app/bin/main.js:3:203615)
at Socket.emit (events.js:315:20)
at Socket.EventEmitter.emit (domain.js:486:12)
at addChunk (_stream_readable.js:309:12) {
cause: undefined,
retriable: true,
fatal: false
}
}
2021-01-03T14:40:32.961Z web-27 error Caught error from file write stream Error: ENOTCONN: socket is not connected, close
WatchedChild.onError() {
ctx: {
src: 'ChildService(web).onStdout()',
fatal: true,
ignorable: false,
errToS: 'Cannot write to /ps/library: undefined: Cannot mkdirp /ps/library/.tmp-7rx9wu¹⁶⁵undefined'
},
src: 'ChildService(web).onStdout()',
error: l [Error]: ChildService(web).onStdout()Cannot write to /ps/library: undefined: Cannot mkdirp /ps/library/.tmp-7rx9wuundefined¹⁶
at C.onError (/ps/app/bin/main.js:3:133290)
at A.onStdout (/ps/app/bin/main.js:3:130238)
at s.onData (/ps/app/bin/main.js:3:137340)
at /ps/app/bin/main.js:3:203449
at Array.forEach (<anonymous>)
at s.onChunk (/ps/app/bin/main.js:3:203396)
at Socket.<anonymous> (/ps/app/bin/main.js:3:203615)
at Socket.emit (events.js:315:20)
at Socket.EventEmitter.emit (domain.js:486:12)
at addChunk (_stream_readable.js:309:12) {
cause: undefined,
retriable: true,
fatal: false
}
}
{"fatal":true,"exit":true,"status":12,"pid":13,"ppid":6,"error":"ChildService(web).onStdout(): Error: ChildService(web).onStdout()Cannot write to /ps/library: undefined: Cannot mkdirp /ps/library/.tmp-7rx9wuundefined¹⁶"}

Shutting down PhotoStructure...
Verifying & backing up /ps/tmp/local-db/models/db.sqlite3...

I’m not fishing for free tech support here, just letting you know my results. Though if anyone does want to help me figure this out, I’d appreciate it. I’d love to get my asset library on S3. I may check out Daniel’s suggestion of BuyVM (below), but then I’d have to figure out how to set up everything on BuyVM (“everything” = photostructure, traefik, syncthing, sftp server; I had a friend create Psodo for me so I’d have to kind of start from scratch on a non-Digital-Ocean platform - though I think cloud-init is pretty universal).

Until originalsDir is a thing, I think this may not work well for a library directory proper: I expect rsync --mount to be overwhelmed by the I/O needed for files inside the .photostructure directory.

I guess I’m not understanding what originalsDir will do. In my current setup, I have one directory containing my asset library, so whether I choose “yes, please copy my photos into my Photostructure library” or “No thanks, I like my photos and videos where they already are,” the results are the same.

Will originalsDir allow a single backup of the asset library, or will it merely separate the .photostructure directory from the asset library?

:point_up: It does this.

Another solution could be to make the PhotoStructure library a local directory, and then make the year subdirectories symlinks to the bind mount, but that’s a pretty grotty hack.