1.0.0-beta.2 error: "Failed to scan system volumes"

[mrm]

I also just wrote this up:

Support for systems with many external drives

It seems that several beta users have a bunch (5+) external drives and remote filesystems mounted on their machines that run PhotoStructure. Currently, PhotoStructure (even v1.0.0-beta.2) collects all mounted volume metadata. This collection can take a while, which led to posts like this. I just had a user ask: why the heck does photostructure care about any of my other drives? and this is absolutely a fair question! Volume metadata extraction is expensive, so PhotoStructure actually only…

[Zirro]

Thanks! I’ve tried increasing these values in the settings, but doing so revealed a timeout in another module:

{"ts":1621151318886,"l":"warn","ctx":"Deferred(rpc.RequestResponse(volumes)#35NctjzJs-3)","msg":".reject()","meta":"TIMEOUT: rpc.RequestResponse(volumes)#35NctjzJs-3 after 30001ms"}
{"ts":1621151318897,"l":"warn","ctx":"Error","msg":"onError(): TIMEOUT: rpc.RequestResponse(volumes)#35NctjzJs-3 after 30001ms\nTIMEOUT: rpc.RequestResponse(volumes)#35NctjzJs-3 after 30001ms\n    at Object.t.asError (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:563913)\n    at p.reject (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:105243)\n    at Timeout.<anonymous> (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:104507)\n    at listOnTimeout (internal/timers.js:554:17)\n    at processTimers (internal/timers.js:497:7)","meta":{"event":"nonFatal","message":"volumes() failed"}}
{"ts":1621151318898,"l":"warn","ctx":"Error","msg":"onError(): TIMEOUT: rpc.RequestResponse(volumes)#35NctjzJs-3 after 30001ms\nTIMEOUT: rpc.RequestResponse(volumes)#35NctjzJs-3 after 30001ms\n    at Object.t.asError (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:563913)\n    at p.reject (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:105243)\n    at Timeout.<anonymous> (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:104507)\n    at listOnTimeout (internal/timers.js:554:17)\n    at processTimers (internal/timers.js:497:7)","meta":{"event":"nonFatal","message":"unhandledRejection"}}

[mrm]

@Zirro

onError(): TIMEOUT: rpc.RequestResponse(volumes)#35NctjzJs-3 after 30001ms\nTIMEOUT: rpc.RequestResponse(volumes)#35NctjzJs-3 after 30001ms\n    at Object.t.asError (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:563913)\n    at p.reject (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:105243)\n    at Timeout.<anonymous> (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:104507)\n    at listOnTimeout (internal/timers.js:554:17)\n    at processTimers (internal/timers.js:497:7)","meta":{"event":"nonFatal","message":"volumes() failed"}}
{"ts":1621151318898,"l":"warn","ctx":"Error","msg":"onError(): TIMEOUT: rpc.RequestResponse(volumes)#35NctjzJs-3 after 30001ms\nTIMEOUT: rpc.RequestResponse(volumes)#35NctjzJs-3 after 30001ms\n    at Object.t.asError (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:563913)\n    at p.reject (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:105243)\n    at Timeout.<anonymous> (/Applications/PhotoStructure.app/Contents/Resources/app.asar/sync.js:9:104507)\n    at listOnTimeout (internal/timers.js:554:17)\n    at processTimers (internal/timers.js:497:7)","meta":{"event":"nonFatal","message":"unhandledRejection"}}

Thanks for reporting this, and sorry for the glitch! beta.3 will make that timeout defer to your overridden value.

Related:

from

Uncertain, lazy, forgetful, & impatient: It’s what you want your code to be.

[Rodger]

This “failed to scan system volumes” error has come back in beta.3. I confirmed my system settings.toml are still:

commandTimeoutMs=60000
mountpointsTtlMs=60000
volumeMetadataTtlMs=1200000

[mrm]

Thanks for sending me your logs. It’s taking 20+ seconds to run

Get-PSDrive -PSProvider FileSystem

I suspect one or more of your drives are spinning down, and it’s taking a while to spin all 9 of your platters back up to speed. It definitely adds some pressure for me to build out the more efficient volumes algorithm.

[stirlow]

I am having this error on v0.9.1. MacOS 11.4 Mac mini (M1, 2020).

I have 3 external drives connected however I know they’re spinning as I have just opened the drives in finder. Interestingly this bug occurred immediately after the welcome/setup dialog. It then prevents the app from starting correctly after that. Even clicking “preferences” seems to cause a crash or close of the app.

EDIT: One of these drives is used for Time Machine. I assume it might be this bug: Apple silicon support - #4 by mrm

[mrm]

@stirlow please try the beta build: I’ve fixed a bunch of issues with m1 and Big Sur.

[tommyB]

Hi, I was getting this error as well - I’ve added the following to my .bash_profile in Linux and this stops the error :

PHOTO STRUCTURE

export PS_COMMAND_TIMEOUT_MS=120000
export PS_MOUNTPOINTS_TTL_MS=120000
export PS_VOLUME_METADATA_TTL_MS=120000
export PS_SETUP_TIMEOUT_MS=120000

However it now doesn’t seem to be pulling in any additional directories I add after the initial populating with the contents of ~/Pictures ?

I’m on 1.0.0-beta.7-x86_64 AppImage on Elementary OS 5.1 (Linux)

Tom

[tommyB]

Update : reboot fixed the syncing - it looks like the shutdown is not always clean and on restart the sync may not be in a good state. Is there a recommended way to kill all parts (and restart) on Linux if it ends up hung in some way ?

[mrm]

@tommyB

Is there a recommended way to kill all parts (and restart) on Linux if it ends up hung in some way ?

First off: howdy, welcome to PhotoStructure!

Second: crap, sorry that the volume scanning glitched on you. The desktop build is really only supported on Ubuntu 20.04; I’ve never tried it on Elementary. If you can run it under docker, that might work better.

Third: all PhotoStructure processes register a sigint handler, so running something like killall -vw PhotoStructure will shut them down gracefully.

I haven’t had to (personally) do that (killall) for a year, though: I suspect something about Elementary is causing PhotoStructure’s gears to grind. If you launch the appimage from a terminal, and add --verbose to the end of the command, it’ll run with verbose info logging that may be enlightening.

[shrippen]

I’m also gettings this error on Beta9.

{“fatal”:true,“exit”:true,“status”:12,“pid”:26,“ppid”:19,“error”:“ChildService(sync).onStdout(): Error: ChildService(sync).onStdout()Failed to scan system volumes.¹⁶”}

I am running as a docker on Unraid.

[mrm]

@shrippen

I’m also gettings this error on Beta9.

Oof, apologies!

Can you shell into your docker container running PhotoStructure and email or dm me the output of

./photostructure info --debug --volumes

Thanks!

[mrm]

Thanks for the DM: that run was successful (which is good news and bad news: good that it normally (?) works, bad news that this is a heisenbug.

I’m not getting the lighter fluid quite yet, but I’ll think about how try to prevent this in the future.

[Rodger]

This issue still persists in 1.0.0-beta.11.

PhotoStructure for Desktops, Windows 10.

(Failed to scan system volumes)

[mrm]

Sorry about that: can you send me your logs so I can see what’s failing?

[Rodger]

Just FYI, my VPN network drive is currently offline - and has been for about a week.

Knowing that, would you still need to see logs? Or would this crashing be expected behavior when a network drive is offline?

Side note- it would be super awesome if someday in a future version users could opt out of having PhotoStructure constantly scanning all of their volumes. I understand the premise, but in my case my pictures will always be on the same drive… and I promise the drive letter will never change.

Cheers,

Rodger.

[mrm]

@Rodger

Knowing that, would you still need to see logs?

That’d be great to isolate what’s actually failing.

@Rodger

would this crashing be expected behavior when a network drive is offline?

No: that should be an expected situation, and PhotoStructure should handle it gracefully: and certainly not crash.

@Rodger

it would be super awesome if someday in a future version users could opt out of having PhotoStructure constantly scanning all of their volumes

You can set volumeMetadataTtlMs to something gigantic: a day in milliseconds is 86400000:

By default, volumes should only be scanned by main, once, and then shared across processes via rpc until a new volume is mounted or an hour elapses (whatever comes first).

[mrm]

@Rodger

Side note- it would be super awesome if someday in a future version users could opt out of having PhotoStructure constantly scanning all of their volumes

I was thinking about this a bit more, and the only reason why I scan volume metadata is to make sure PhotoStructure pauses the sync if disk space gets tight.

If there’s plenty of disk space available on the cache and library volumes, PhotoStructure can safely cache volume metadata for a long while, because PhotoStructure knows it isn’t in a danger zone.

I also just realized that PhotoStructure can gracefully ignore any errors fetching volume metadata for volumes that aren’t being read from or written to. I’ll see how doable that is.

@Rodger thanks again for the suggestion!

[mrm]

volume handling was extensively reworked (again) in beta.13. @Rodger are you still seeing this issue after upgrading?

[Rodger]

Beta.13 was still crashing daily, but I apologize I didn’t jot down or notice the exact error. I just upgraded to beta.14, and will play close attention.