First sync is behaving very odd

ISSUE 1: Photos missing after Sync completed
I have 117,000 unique photos. I previously tried PS on my linux partition and it only found 91,000 assets. I had already de-duped by photos with two different apps and am confident I don’t have 25,0000 duplicates. So I started over again in windows…

ISSUE 2: New Library, Count of Photos Processed/remaining to be processed make no sense.

After two days of syncing, It has found 91,272 assets…but for the last twelve hours, the number of photos remaining to be processed has moved marginally (from 19,500 to 19,042) while the number of processed photos keeps growing and far exceeds the number of photos I have by some margin (156,028). Can anyone explain this and advise what I should do to get a library that matches the assets I actually have?

My Library File structure is:
OUTPUT/ COUNTRY/YYYY-MM-DD/Country-000003.raf
PHOTOS/COUNTRY/YYYY-MM-DD/Country-000001.raf
VIDEOS/COUNTRY/YYYY-MM-DD/Country-000002.raf

PS for desktop, Windows 10 and Ubuntu 21.04

Welcome to PhotoStructure @kyle! Apologies that PhotoStructure is misbehaving for you.

PhotoStructure has a bunch of file filters that can prevent assets from being imported into your library. PhotoStructure | Why didn't my file get imported into my Library? should help.

I suspect it skipped over a large chunk of files, rather than marking them as deleted. You can get a high level view of how many assets versus duplicate variations PhotoStructure has found via the about page: PhotoStructure | What are “Library metrics”?

Oof, that’s no good. Can you shut down PhotoStructure, set logging to debug, restart, and after 10 minutes or so, send me your logs? Details are here: PhotoStructure | What are in my logs and error reports?

Thanks for your quick reply. The log folders are too big to email, even compressed. I had clicked “send us your recent logs” under help but it doesn’t appear to do anything? Should I just send one of the main, one sync and one sync-file logs? If so, let me know where to send it. FYI In the logs I’m seeing countless errors like this: {“ts”:1634638545404,“l”:“error”,“ctx”:“AssetFileImporter(F:\PHOTOS\Turkey\2019-03-30\Turkey-114518.dng)”,“msg”:"_apply(): unexpected missing asset file id",“meta”:{“path”:“F:\PHOTOS\Turkey\2019-03-30\Turkey-114518.dng”,“af”:{"$ctor":“models.AssetFile”,“aperture”:2.8,“assetId”:84115,“capturedAtLocal”:2019033009004189,“capturedAtOffset”:-360,“capturedAtPrecisionMs”:10,“capturedAtSrc”:“tags:SubSecDateTimeOriginal (tz inferred from Turkey-114517.dng tags.MetadataDate)”,“fileSize”:20277292,“focalLength”:“6.6 mm”,“height”:4032,“iso”:20,“lensId”:“LensID:iPhone 7 Plus back camera 6.6mm f/2.8”,“make”:“Apple”,“meanHash”:“4ODg4ODw8PB2BRcdDQ8PDw8HN25IAHDg”,“mimetype”:“image/x-adobe-dng”,“mode0”:2555,“mode1”:2527,“mode2”:1389,“mode3”:2267,“mode4”:3145,“mode5”:1385,“mode6”:511,“model”:“iPhone 7 Plus”,“mountpoint”:“F:\”,“mtime”:1633359190718,“rotation”:90,“sha”:“L9dwuWKDhs6iQ52ett0aWKsIlhH4zvQi”,“shutterSpeed”:“1/4000”,“uri”:“psfile://2he2FjwPX/PHOTOS/Turkey/2019-03-30/Turkey-114518.dng”,“version”:11,“width”:3024}}

Re the duplicates, here are the current library metrics…

91,374 assets
91,305 image files
69 video files
2,954 tags

Ah, sorry about that: that uses Sentry to send the last 100ish log entries, and that turned out to never be enough context for me to see what was going on, so it’s been deleted from the next build.

This issue I’ve fixed in v2.0: I’ll try to get a beta build out soon.

Sure, that’d be plenty: hello@photostructure.com

WOAH no dupes! (I wish my library was that squeaky-clean!)

(Here’s one of my test libraries:)
image

Thanks for your time-I’ve sent logs. I really hope I can get this working as you have absolutely nailed the UI for photostructure-love it!