Increasing Confidence in the Import

First of all - Love PhotoStructure - Ethos, UX, Execution and the automagic categorization. It’s the tool I’ve been looking for in many ways. So many things are done right including the safe import.

The challenge I have is that the final part of importing to me is removing the “originals” and using the new “Library” and associated File/Folder Structure as the source of truth.

I’ve read the healthy discussion on whether PS should do that or not. I am happy to manually delete a directory but I need have confidence that everything I’m deleting has been imported.

Additionally, there needs to be an easy way to cp/mv anything that was not imported (excluding dupes) a command line tool is okay for this.

I know there is a command line based approach to to get list of unimported files but going through that manually when you have 10ks of photos is not serviceable.

If there is already an easy way to do this or even an import workflow that is better - happy to hear.

Howdy @sandeep , welcome to PhotoStructure!

Thanks, that’s really appreciated! Can I use this quote on the public website?

I’m really hoping the new sync report in v2.1 will help answer this question.

Like you said, with 10k+ files, you need some industrial-strength tools to filter and sort, and rather than building that, I’m just emitting a CSV that you can then use Excel, OpenOffice, or any other spreadsheet tool to

  • filter on “rejected”, which will list the files that sync didn’t import (and the reason),
  • filter on “failed” and “timeout”, which are the files that PhotoStructure tried to import but couldn’t, and
  • filter on “skipped”, which are the directories that PhotoStructure didn’t even walk into, due to it being considered “ignorable” (and the details column will explain why it was skipped/ignorable).

Ah! That’s a great idea!

The list tool seems to be a good candidate to add this feature, and a glob pattern would make this easy to work with. You’d run something like

photostructure list --unimported /path/to/photos/**/*

(please do suggest a better switch name for this: --missing-from-library perhaps? --not-in?

Feel free re quote.

I think Sync Report will work. I’m seeing a bunch of files that say they are not imported but when I look at the info it doesn’t show any rejection.

FWIW this is what I’m doing now:

find "/Desktop/Pictures/Batch 1" -type f | sort > ./all.txt

./photostructure list | grep -E "^/Desktop/" | sort > ./imported.txt

echo “Total Issues:”
comm -2 -3 all.txt imported.txt| wc -l

echo “Known Issues unwanted, Unimportable, etc.  :”
 comm -2 -3 all.txt imported.txt| grep -e "ini" -e "db" -e "xmp" -e "gz" -e "psd" -e "txt" -e "DS_Store"  -e “pto” -e “tiff” | wc -l

echo “PSD, TXT, TIFF and PTO Files to move:”
comm -2 -3 all.txt imported.txt| grep -e "psd" -e "txt" -e “tiff” -e”pto” | sort > ./1-Files_to_Move_Manually.txt

comm -2 -3 all.txt imported.txt| grep -e "psd" -e "txt" -e “tiff” -e”pto”| wc -l

echo “Burst Shots and Other Non Imports”
 comm -2 -3 all.txt imported.txt| grep -v -e "ini" -e "db" -e "xmp" -e "gz" -e "psd" -e "txt" -e "DS_Store"  -e “pto” -e “tiff” | > ./3-Potenial_Burst_Shots_not_imported.txt

 comm -2 -3 all.txt imported.txt| grep -v -e "ini" -e "db" -e "xmp" -e "gz" -e "psd" -e "txt" -e "DS_Store"  -e “pto” -e “tiff” | wc -l

The output is

“Total Issues:”
   12331
“Known Issues unwanted, Unimportable, etc. :”
   11417
“PSD, TXT, TIFF and PTO Files to move:”
      46
“Burst Shots and Other Non Imports”
     914

Looking into will take some time but a big chunk seem to be from “Burst Mode” style shooting.

I suspect some of the upcoming items around handling custom file types; Burstmode and cleaning up after import will help.

1 Like

Looks like many of the Darktable created xmp sidecar files are not getting imported. Hitting multiple filters :point_down:

photostructure-for-servers % ./photostructure info "/Photo Feb 28, 9 42 00 PM.jpg.xmp"
[
  {
    _PhotoStructureVersion: '1.1.0',
    capturedAt: { localCentiseconds: undefined },
    filters: {
      accepted: [
        'notHiddenCheap',
        'noMediaFilter',
        'hasMimeType',
        'notRejected',
        'minVideoDurationFilter'
      ],
      rejected: [
        'exifExtFilter',
        'fileSizeFilter',
        'supportedMimeTypeFilter',
        'minDimensionsFilter'
      ]
    },

...

This is related to what @mrm talks about here Non Destructive Edits and what to do with this in the future.

For now is there a way to get PS to keep these XMPs together with the source file in the import/organization so that I can use Darktable on the PS Library.

The remainder - I can wait for burst mode or handle on my own.

Sidecars aren’t imported by themselves.

For every photo and video that PhotoStructure imports, it looks for any relevant sidecar files, and copies those alongside the imported file. Versions prior to v2.1 had a couple defects that could prevent some sidecars from being copied into the library, but I believe I have addressed that issue in v.2.1-alpha.1.

I’ll add this to the info command now: it’ll be in the next build.