Scanning nested folders

TheSpatulaOfLove · September 6, 2021, 2:37pm

Good morning.

I have been toying with Photostructure for a couple weeks on my Unraid server and tried to do a full sort this weekend. The results seem like it didn’t find everything I anticipated.

I created a share, then proceeded to dump many many folders under that share, some with quite deep nesting. In addition, I also threw quite a few old iPhoto libraries under the nesting.

My question is, how deep does Photostructure look within the nesting for images?

mrm · September 6, 2021, 3:35pm

It should descend indefinitely!

That said, there are a bunch of reasons why it may not want to walk into a directory.

It may have found a NoMedia file. Read more here: PhotoStructure | How do I prevent a directory from being imported into my library?
There is a basic symlink loop detector in PhotoStructure’s directory walker: if it finds the same name element in the path more than 7 times (by default) it won’t descend into child directories.

You can customize that here:

github.com

photostructure/photostructure-for-servers/blob/1f6d907769fd2b01900eb88ce64bf0d732998aa0/defaults.env#L2068

    
      
          
          

          
# +----------------------------+
          # |  maxDuplicatePathElements  |
          # +----------------------------+
          #
          # How many times can a given path element exist in a directory before it is
          # considered within an infinite filesystem loop, and should be skipped from
          # import?
          #
          # PS_MAX_DUPLICATE_PATH_ELEMENTS="7"
          
          

          
# +---------------------+
          # |  overwriteOriginal  |
          # +---------------------+
          #
          # Should changes made through the UI, like rotations, captions, and keywords,
          # overwrite the original file? This is potentially dangerous, as your original
          # may be lost if the disk has errors, or there are issues in rewriting the
          # file contents. If this is set to false, the original file will be retained

PhotoStructure has a metric crapton of directory patterns that are “ignorable”, to prevent it from walking through system directories and other places of ill repute (like node_modules). You can force-include a folder via this setting:

github.com

photostructure/photostructure-for-servers/blob/1f6d907769fd2b01900eb88ce64bf0d732998aa0/defaults.env#L145

    
      
          # PS_FORCE_LOCAL_DB_REPLICA="false"
          
          

          
# +----------------+
          # |  neverIgnored  |
          # +----------------+
          #
          # Paths to files or directories that should not be ignored, even if they are
          # hidden or have .nomedia files.
          #
          # PS_NEVER_IGNORED=""
          
          

          
# +----------------+
          # |  originalsDir  |
          # +----------------+
          #
          # This is the directory that PhotoStructure uses to store original images when
          # "copyAssetsToLibrary" is enabled. Absolute paths are supported. Relative
          # paths are evaluated from your libraryDir. This setting defaults to ".",
          # which is the same as your PhotoStructure library directory.

TheSpatulaOfLove · September 6, 2021, 4:34pm

I don’t think there are any NoMedia type files in the structures. I literally copy/pasta old hard drives into this giant blob of crap to have PhotoStructure tear through. I will double check to see if there are any residual directory names that have a dot in front.

The other thing I may try is to change the 7x thing. That may be a possibility since I was grabbing a lot of old Mac directories that are similarly structured. Is there any risk to changing this to say 14 or 21? (Resource limitation?)

On a final note, is there some kind of summary that tells me how many files were scanned/deduped/copied so I can see if it missed anything in the giant mess I had it scanned through?

mrm · September 6, 2021, 6:50pm

There’s nothing magical about the default value: I just picked a value large enough to hopefully avoid false positives (it means you can’t have the same directory name repeated seven times, which seems even more “giant blob of crap”-esque than my terrible mess.

If you know you don’t have any circular references, you can make it as large as the deepest folder structure that you have.

There is the “library metrics” on the about page, and if you want to get fancy (and you’re comfortable with the terminal), you can use list: https://photostructure.com/server/tools/#path-names-in-your-library:

./photostructure list | grep /path/to/giant/blob/of/gold

will return the images that were imported. To get the list that wasn’t imported, see https://photostructure.com/server/tools/#which-files-in-a-directory-didnt-get-imported

TheSpatulaOfLove · September 6, 2021, 9:35pm

I smell a feature request brewing…

Thank you for your rapid responses.