If you had to wipe your PS library, I wonder if there was something amiss with your library database?
If you still have the db.sqlite3 file from the prior library, can you email it to me so I can take a look?
Cheers, and thanks for the update!
Windows 10.
Its slightly better today, not sure what changed. You’d mentioned 2.1 would be able to destroy my 5950X, do I need to apply any settings?
Edit: Apparently I was impatient, it hit 70% for a little bit…nice
Hi @mrm ,
Thank you for the next alpha version! I have sent you a report and a screencast to demonstrate my issues. I hope it helps for alpha 3 
woah woah woah PhotoStructure utilizes hardware, it doesn’t destroy it 
I’m glad it got to 70%. Know that it actually throttles additional work if aggregate CPU load exceeds whatever cpuLoadPercent is set to (it defaults to 75%), so it may exceed that momentarily, but probably only during ffmpeg transcoding.
Any chance we’ll soon see the ability for PhotoStructure to pull in Mac Photos app libraries?
I looked into this a bit before I shipped alpha.2.
Photostructure now knows how to ask macOS where the system photos library is, and asks to read the contents–it’s why you get the “…photoslibrary ENOENT” error at startup now.
For all other directories, like the desktop directory and /volumes, simply asking to read the directory will make macOS pop up a dialog asking the user of it’s OK for PhotoStructure to read the contents of that directory. If the user accepts, the readdir succeeds.
Unfortunately, even though I’ve signed PhotoStructure with read access to the library via entitlements, this falls unless you manually add that directory in the control panel as being accessible by PhotoStructure.
I’ll spend some more time researching why macOS is denying access to the photos library and hopefully I’ve just missed something obvious.
Turns out macOS only allows API access to photo assets, which is a huge PITA. Especially irritating, because they added an signing entitlement that would lead the unsuspecting dev to think they’d be granted read-access to photo libraries.
A workaround (for now) would be to create a symlink to your masters directory, and import via the symlink.
To do that, open a terminal, and run
cd $HOME
ln -s "$(defaults read read com.apple.photolibraryd SystemLibraryPath)/Masters" photo-library
and then tell PhotoStructure to scan $HOME/photo-library.
If you’re running (or ran) an older version of Apple Photos, and don’t have a Masters directory, it might be named “Originals”.
Hi @mrm,
Are you expecting scan scheduling to be fixed in this release? My library got to almost 3 days in the “last completed” field before I hit the “Restart sync” menu item. In the meantime I had manually scanned a few recent folders in case that makes a difference.
I mean, I expected things to work, but between v2.0 and v2.1 I broke sync rescheduling, and will fix that before I ship v2.1-stable.
Apologies for the inconvenience!
Curiously, after I hit the Restart sync option, nothing seemed to happen. I took the opportunity to reboot the host for other reasons, now I have lots of files owned by root again. Not sure whether it’s the restart sync or the reboot, but most other times I’ve seen this have been in a similar scenario. I might be getting closer to working out when it happens.
What’s your suggestion for triggering a full resync?
Selecting sync from the nav menu:
should cancel the current sync and restart. Can you DM me your host OS, version of Docker, and any other system configuration information you can share?
Curiously the amount of photos processed and to be processed seems too keep resetting. 2 days and only 19000 assets added
All files local on a spinner
5950x
2080 RTX
Any thoughts?
Thanks, I did that and I saw plenty of activity and the sync reports look populated, but none of the new photos I expected have appeared - I’m specifically looking in the “When” for June 2022.
I see a large sync report file and two smaller ones with an error:
/photostructure/library/.photostructure/sync-reports/2022-06-21$ ll
total 11603
drwxrwxrwx 2 user user 5 Jun 21 05:06 ./
drwxrwxrwx 31 user user 33 Jun 21 02:06 ../
-rwxrwxrwx 1 user user 80310812 Jun 21 05:06 14-07-03-sync-report.csv*
-rwxrwxrwx 1 user user 477 Jun 21 05:06 17-07-07-sync-report.csv*
-rwxrwxrwx 1 user user 1623 Jun 21 05:07 17-07-16-sync-report.csv*
$ tail 14-07-03-sync-report.cs v
1655787984451,/media/photos/2017/2017 02 17/D_IMG_1376.JPG,enqueued,,,
1655787984451,/media/photos/2017/2017 02 17/D_IMG_1377.JPG,enqueued,,,
1655787985843,/media/photos/2017/2017 02 17/D_IMG_1376.JPG,new,,http://127.0.0.1:1787/asset/94895,1392
1655787986058,/media/photos/2017/2017 02 17/D_IMG_1377.JPG,new,,http://127.0.0.1:1787/asset/93224,1607
1655787986558,/media/photos/2018/2018 07 11 XXXXXXX/M_IMG_5757.MOV,new,,http://127.0.0.1:1787/asset/138977,8255
1655787987172,/media/photos/2017/2017 02 17/D_IMG_1378.JPG,enqueued,,,
1655787987173,/media/photos/2017/2017 02 17/D_IMG_1379.JPG,enqueued,,,
1655787988182,/media/photos/2018/2018 07 11 XXXXXXX/M_IMG_5758.MOV,enqueued,,,
1655788024135,/media/photos/2017/2017 02 17/D_IMG_1378.JPG,new,,http://127.0.0.1:1787/asset/91993,36963
1655788024347,/media/photos/2017/2017 02 17/D_IMG_1380.JPG,enqueued,,,
$ tail 17-07-07-sync-report.csv
ts,path,state,details,url,elapsedMs
1655788027166,/media/photos/2017/2017 02 17/D_IMG_1380.JPG,failed,"BatchCluster has ended, cannot enqueue {""id"":9967,""method"":""buildAssetPreviews_"",""args"":[{""assetId"":90019,""assetFiles"":[{""$ctor"":""models.AssetFile"",""id"":102482,""assetId"":90019,""shown"":true,""uri"":""psfile://2KjKKRiCY/2017/2017%2002%2017/D_IMG_1381.JPG"",""mountpoint"":""/m…:false,""recountAllTags"":false}]}",http://127.0.0.1:1787/asset/90019,2819
$ tail 17-07-16-sync-report.csv
ts,path,state,details,url,elapsedMs
1655788036449,/media/photos/2017/2017 02 17/D_IMG_1380.JPG,enqueued,,,
1655788037033,/media/photos/2018/2018 07 11 XXXXXXX/M_IMG_5758.MOV,failed,"BatchCluster has ended, cannot enqueue {""id"":9964,""method"":""buildAssetPreviews_"",""args"":[{""assetId"":138977,""assetFiles"":[{""$ctor"":""models.AssetFile"",""id"":411060,""assetId"":138977,""shown"":true,""uri"":""psfile://2YqPvS8aJ/2018/2018%2007%2011XXXXXXXXXX…:false,""recountAllTags"":false}]}",http://127.0.0.1:1787/asset/138977,48851
1655788037149,/media/photos/2017/2017 02 17/D_IMG_1379.JPG,failed,"BatchCluster has ended, cannot enqueue {""id"":9965,""method"":""buildAssetPreviews_"",""args"":[{""assetId"":91993,""assetFiles"":[{""$ctor"":""models.AssetFile"",""id"":411067,""assetId"":91993,""shown"":true,""uri"":""psfile://2YqPvS8aJ/2017/2017%2002%2017/D_IMG_1378.JPG"",""mountpoint"":""/m…:false,""recountAllTags"":false}]}",http://127.0.0.1:1787/asset/91993,49975
On the plus side, I don’t have a pile of files owned by root!
hmm I’m running 2.1.0 and 1.1.0 at the same time over the same picture folder.
However in the 2.1.0 I can click update library but deleted files from a folder won’t get removed, only new files are added.
On the 1.1.0 version it works as expected, old files are removed and the new files are added…
Another strange behavior is that in some folders 1 or 2 images are missing compared to the 1.1.0 version. I can’t figure out why those images are missing in the 2.1.0 since they are like all the other images in this folder
another one is that on the other hand a folder in the v 1.1.0 is described to have 20 assest however when I click the folder 21 images are displayed… So counting is also a problem in the stable version of 1.1.0
Re: this, do you think it would be a better behavior to have Photostructure natively import the masters directory? I’m currently using GitHub - boredazfcuk/docker-icloudpd: An Alpine Linux 3.13.5 container for the iCloud Photos Downloader command line utility to keep a downloaded copy of my iCloud Photo Library up to date, but it has some limitations. One such issue is it exports the live photos as two separate files. I’m not sure if that’s how they are stored in the existing Apple masters folder on MacOS.
If MacOS is truly the better experience for importing an iCloud Photo Library, then my next question would be: Can I run PS on both Mac and Docker and have them both maintain the library? Use Docker to be my daily driver but also intermittently kick off the MacOS desktop version to add the latest iCloud Photos?
Thanks.
Unfortunately live photos have been encoded in different ways on different iPhones, but at least with iPhone 11-13, they seem to be encoded as two distinct files with matching content UUID metadata headers (which I’ll be using for deduplication when I add proper support for live photos).
As long as the volumes you scan have .uuids, and you don’t run PhotoStructure concurrently on both the mac and docker, the URIs should match up across OSes, and things should “just work”.
I initially wrote that cross-machine feature so you could tote a single (big) external HDD around to all your computers and sweep them all into one directory structure, and then mount the HDD on (or shuck it into) your server.
ok here another one. When I delete an image in a folder externally so via a file Explorer etc. and do a sync it does not get deleted in PS. When I try to download the file over the web interface it results in an error (of course since the physical file is missing only the thumbnail is present)
And I can’t even remove it by using the command: photostructure sync /photos/_neu --force
The only way is to delete it over the web interface delete button, but this can’t be the solution?
@GekoCH thanks for reporting (as always!)
I’d like to know exactly how your files and setup are so I can reproduce this and fix the issue.
There’s, say, /photos/_neu/photo.jpg, you import it with sync, then delete the file via the file explorer, and navigate back to the asset?
@mrm Ok I try to explain it a bit better
PS runs in a docker.
The folders config, library, logs, tmp are located on an SSD.
The folder with all the photos are on an other drive and I added this drive with UNRAID as a path.
I first do a complete sync / setup for the first time over the /photos folder.
Then I went into the folder called “_neu” since this is the folder my camera does upload all new pictures made during the day.
Once we deleted the bad ones (over a SMB share (not in PS)) and did some color grading they get separated into new folders like /photos/event1 /photos/event2 etc.
Therefore they are removed from the /photos/_neu folder.
When I then hit the Sync button the new /photos/event1 and /photos/event2 are discovered but the deleted (or moved) files form the /photos/_neu are not removed from the PS liabray.
I hope this explains the whole process a bit better…
Hi Matthew, I think it is related to the PS_COMMAND_TIMEOUT_MS setting. I get the same BatchCluster notification when I set it to 60s.