PhotoStructure’s directory walker was rewritten in v2.1 to add glob support.
What’s this?
PhotoStructure comes built-in with several hundred exclusion patterns to avoid walking into system and application support directories. Read more about this here.
If any of these patterns cause your files to be ignored, or you’re already using some other naming convention to exclude directories (instead of, for example, NoMedia), the new-to-v2.1 excludeGlobsOmit and excludeGlobsAdd settings should give you the flexibility to teach PhotoStructure to import just the right set of files.
Prior approaches
Versions prior to v2.1-alpha.4 used a pleasing melange of regular expressions, string matching, and subdirectory matching to detect system and application support directories, along with a neverIgnored setting which was a simple set of pathnames.
In trying to adjust the code to be easier to explain and configure, an initial attempt in alpha.2/alpha.3 was implemented using a new setting called globs, but that implementation didn’t work well with the built-in exclusion patterns and led to a couple bugs and some deal of confusion, hence this new approach.
New approach in v2.1.0-alpha.4: one set of exclusion patterns
-
PhotoStructure’s set of exclusion patterns and RegExps have all been converted to glob patterns, the
excludeGlobs. -
During import, all files and directories will be matched against all
excludeGlobspatterns. Any matching files or directories will be excluded from being imported from your library. -
These patterns can be adjusted via new
excludeGlobsAddandexcludeGlobsOmitsettings, which let you add and remove glob patterns fromexcludeGlobs.
What’s a “glob”?
“Globs” are patterns that may include asterisks, like /a/*/c (which matches any sub-directory within /a that includes a file or sub-directory named c), and other “magic” characters to match desired patterns.
Magic glob characters
-
*Matches 0 or more characters in a single path portion -
?Matches 1 character -
[...]Matches a range of characters, similar to a RegExp range. If the first character of the range is!or^then it matches any character not in the range. -
!(pattern|pattern|pattern)Matches anything that does not match any of the patterns provided. -
?(pattern|pattern|pattern)Matches zero or one occurrence of the patterns provided. -
+(pattern|pattern|pattern)Matches one or more occurrences of the patterns provided. -
*(a|b|c)Matches zero or more occurrences of the patterns provided -
**If a “globstar” is alone in a path portion, then it matches zero or more directories and subdirectories searching for matches. It does not crawl symlinked directories.
This list is courtesy of glob. FWIW, PhotoStructure uses picomatch with default options.
Notes
Always suffix directory patterns with a forward-slash
To disambiguate between files and directories, PhotoStructure applies glob patterns that end with a forward-slash to directories, and all other patterns to files.
Directories will be matched as they are visited, so **/node_modules/ will exclude all Node.js node_modules directories, regardless of depth.
Always use forward-slashes in your patterns (even on Windows)
Back-slashes are for escaping characters. Windows pathnames will have backslashes replaced with forward slashes before matching against glob patterns.
Patterns for root directories
To match a root directory on macOS or Linux, prefix the pattern with a forward-slash (like /tmp/)
To match a root directory on Windows, either include the drive letter, or use * (like C:/tmp/ or */tmp/).
Use ./photostructure info --globs to list patterns
Inspect the excludeGlobs file and directory arrays via the info tool:
$ ./photostructure info --globs
{
excludeGlobs: {
file: [
{ glob: '**/.*', desc: 'hidden file' },
{ glob: '**/facetile*', desc: 'face thumbnail' }
],
dir: [
{ glob: '**/__MACOSX/', desc: 'macOS resource fork' },
{ glob: '**/_includes/', desc: 'code' },
{ glob: '**/.*/', desc: 'hidden dir' },
...
Add --flat and --filter arguments if you want just the array of directory globs:
$ ./photostructure info --globs --filter excludeGlobs.dir.glob --flat
[
'**/__MACOSX/',
'**/_includes/',
'**/.*/',
'**/@eaDir/',
'**/@Recycle/',
'**/@SynoResource/',
...
Here’s an example of removing the __MACOSX pattern:
PS_EXCLUDE_GLOBS_OMIT='**/__MACOSX/' ./photostructure info --globs --filter excludeGlobs.dir.glob --flat
[
'**/_includes/',
'**/.*/',
'**/@eaDir/',
'**/@Recycle/',
'**/@SynoResource/',
...
Remember that other filters still apply
Note that globs only overrides PhotoStructure’s built-in file and directory exclusion patterns. All of the other filters, including
-
requireMakeModel, -
rejectRatingsLessThan, -
keywordBlocklist, -
minImageDimension, -
minVideoDimension, -
minVideoDurationSec, -
maxVideoDurationSec, -
minAssetFileSizeBytes, maxAssetFileSizeBytes
(note this list is current as of v2.1, and should be expected to change in future releases)
Examples
exclude custom directories
If you always put random screenshots that you don’t want in your library in a “screenshots” directory, add this pattern to your excludeGlobsAdd:
**/screenshots/
Note that the pattern ends with a forward-slash, /: this tells PhotoStructure that this pattern is for directories, not for files.
exclude a specific file extension
If for some reason you don’t want to import any .CR3 file, add this to your excludeGlobsAdd:
**/*.cr3
include hidden directories
If you want PhotoStructure to scan directories that start with a period, add
**/.*/
to your excludeGlobsOmit setting.
I added a pattern to excludeGlobsAdd, but files that should match that glob are still in my library!
First, make sure you structured the glob correctly:
- if you want to exclude directories, the pattern must end with a forward slash (even on Windows)
- glob patterns always use forward-slashes, even on Windows
Second, know that PhotoStructure needs to do a sync for the affected directories before your glob will come into play. There’s a “cleanup” step after sync’ing a directory: any file from that directory hierarchy that wasn’t updated since the start of the sync will be re-examined, and, most likely, removed from the library (the original file won’t be touched: just the asset and asset file db rows, the preview images, and, if present, the transcoded video).
This cleanup only runs at the end of syncing a directory. If you add a glob but never re-sync the impacted files, this cleanup step won’t run, and your new exclusion glob won’t be in effect.
If you re-add an existing parent directory of files that you want to remove to your scan paths and resync, this should “force” the cleanup step to run, and respect your exclusion glob.
excludeGlobs for v2.1.0-alpha.4
Here’s a list of globs shipping with version 2.1.0-alpha.4. Note that these may be adjusted in future versions. Feel free to suggest more default patterns, or edits to existing patterns!
{
"excludeGlobs": {
"file": [
{
"glob": "**/.*",
"desc": "hidden file"
},
{
"glob": "**/facetile*",
"desc": "face thumbnail"
}
],
"dir": [
{
"glob": "**/__MACOSX/",
"desc": "macOS resource fork"
},
{
"glob": "**/_includes/",
"desc": "code"
},
{
"glob": "**/.*/",
"desc": "hidden dir"
},
{
"glob": "**/@eaDir/",
"desc": "Synology thumbnail"
},
{
"glob": "**/@Recycle/",
"desc": "QNAP trash"
},
{
"glob": "**/@SynoResource/",
"desc": "Synology metadata"
},
{
"glob": "**/*.lrdata/",
"desc": "LightRoom data"
},
{
"glob": "**/*.photo?(s)library/(attachments|database|external|preview?(s)|resource?(s)|scopes)/",
"desc": "Apple Photo"
},
{
"glob": "**/*.sparsebundle/bands/",
"desc": "Time Machine backup"
},
{
"glob": "**/#recycle/",
"desc": "Synology trash"
},
{
"glob": "**/#snapshot/",
"desc": "fs snapshot"
},
{
"glob": "**/$Recycle.Bin/",
"desc": "Windows trash"
},
{
"glob": "**/3rdParty/",
"desc": "code"
},
{
"glob": "**/appdata/(local|locallow|roaming)/",
"desc": "Windows default AppData"
},
{
"glob": "**/Application Data/",
"desc": "Windows application support"
},
{
"glob": "**/Application Support/",
"desc": "Windows application support"
},
{
"glob": "**/Applications/",
"desc": "macOS Applications dir"
},
{
"glob": "**/arangodb/",
"desc": "3rd party"
},
{
"glob": "**/cache/",
"desc": "Cache dir"
},
{
"glob": "**/CacheClip/",
"desc": "Movie cache dir"
},
{
"glob": "**/caches/",
"desc": "Cache dir"
},
{
"glob": "**/cmake/",
"desc": "code"
},
{
"glob": "**/com.apple.TimeMachine.localsnapshots/",
"desc": "macOS"
},
{
"glob": "**/contents/(frameworks|plugins|resources|sharedsupport)/",
"desc": "macOS application"
},
{
"glob": "**/cpan/",
"desc": "code"
},
{
"glob": "**/cygwin?(64)/(bin|cygdrive|dev|etc|games|include|lib?(32|64|x64)|local|locale|man|proc|sbin|share|src|tmp|usr|var)/",
"desc": "Cygwin"
},
{
"glob": "**/data/$of/",
"desc": "windows backup"
},
{
"glob": "**/DefinitelyTyped/",
"desc": "TypeScript type"
},
{
"glob": "**/Desktop DB/",
"desc": "macOS metadata"
},
{
"glob": "**/Desktop DF/",
"desc": "macOS metadata"
},
{
"glob": "**/Desktop.ini/",
"desc": "macOS metadata"
},
{
"glob": "**/dev/(block|bsg|bus|char|cpu|disk|dri|fd|hugepages|input|lightnvm|mapper|mqueue|net|pts|shm|snd|ubuntu-vg|usb|vfio)/",
"desc": "Linux /dev backup"
},
{
"glob": "**/DisplayDriver/",
"desc": "Windows driver"
},
{
"glob": "**/DLLs/",
"desc": "Windows library"
},
{
"glob": "**/docs/(admin|content|general|generated|sql)/",
"desc": "code"
},
{
"glob": "**/dyld/",
"desc": "macOS library"
},
{
"glob": "**/ehthumbs.db/",
"desc": "Windows thumbnail"
},
{
"glob": "**/go/(bin|blog|cmd|doc|lib|misc|pkg|src|test|vt)/",
"desc": "code"
},
{
"glob": "**/i18n/",
"desc": "i18n code"
},
{
"glob": "**/ImageMagick*/",
"desc": "ImageMagick source"
},
{
"glob": "**/iMovie Cache/",
"desc": "Apple iMovie cache"
},
{
"glob": "**/Install macOS*/",
"desc": "macOS installer"
},
{
"glob": "**/Install OS X*/",
"desc": "macOS installer"
},
{
"glob": "**/iTunes/",
"desc": "Apple iTunes"
},
{
"glob": "**/iTunes Cache/",
"desc": "Apple iTunes"
},
{
"glob": "**/iTunes Media/",
"desc": "Apple iTunes"
},
{
"glob": "**/lfs/(incomplete|objects)/",
"desc": "Git LFS object"
},
{
"glob": "**/lib/",
"desc": "code"
},
{
"glob": "**/lib/(firmware|modules|systemd|udev|x86_64-linux-gnu)/",
"desc": "Linux system package"
},
{
"glob": "**/library/(accessibility|accessibilitybundles|accounts|address book plug-ins|apple|applemediaservices|application support|assetcache|assets|assetsv2|assettypedescriptors|assistant|audio|awd|bridgesupport|bundles|cachedelete|caches|cardkit|classroom|colorpickers|colors|colorsync|components|compositions|configurationprofiles|contextual menu items|coreaccessories|coreanalytics|coreimage|coremediaio|coreservices|cryptotokenkit|defaultsconfigurations|desktop pictures|developer|dictionaries|differentialprivacy|directoryservices|display|displays|distributedevaluation|documentation|driverextensions|dtds|duetactivityscheduler|extensions|fdr|featureflags|filesystems|filters|fonts|frameworks|gpubundles|graphics|hidplugins|identityservices|image capture|input methods|installersandboxes|internet plug-ins|internetaccounts|isp|itunes|java|kerberosplugins|kernelcollections|kernels|keyboard layouts|keychain|keychains|lasecureio|launchagents|launchdaemons|lexicons|linguisticdata|locationbundles|loginplugins|logs|mediastreamplugins|messages|messagetracer|modem scripts|monitorpanels|multiverseplugins|networkserviceproxy|onboardingbundles|opendirectory|openssl|osanalytics|pairedsyncservices|password server filters|pdf services|perl|preferencebundles|preferencepanes|preferences|preferencessyncbundles|printers|privateframeworks|privilegedhelpertools|python|quicklook|quicktime|receipts|recents|ruby|runningboard|sandbox|screen savers|screenreader|script editor plugins|scriptingadditions|scripts|security|services|sounds|speech|speechbase|spotlight|stageddriverextensions|stagedextensions|startupitems|syncservices|systemconfiguration|systemextensions|systemmigration|systemprofiler|tcl|templates|textencodings|textinput|trial|updates|user pictures|user template|usereventplugins|usernotifications|video|videoprocessors|webserver|widgets|xpc)/",
"desc": "macOS Library"
},
{
"glob": "**/libs/",
"desc": "code"
},
{
"glob": "**/local/(bin|cygdrive|dev|etc|games|include|lib?(32|64|x64)|local|locale|man|proc|sbin|share|src|tmp|usr|var)/",
"desc": "FHS system"
},
{
"glob": "**/lost+found/",
"desc": "fsck recovered file"
},
{
"glob": "**/macOS Install*/",
"desc": "macOS installer"
},
{
"glob": "**/MinGW*/",
"desc": "code"
},
{
"glob": "**/mnt/(cache|crash|games|lib|local|lock|log|logs|mail|run|snap|spool|tmp)/",
"desc": "Linux /mnt"
},
{
"glob": "**/msys*/",
"desc": "code"
},
{
"glob": "**/msys*/(clang*|dev|etc|mingw*|tmp|ucrt*|var)/",
"desc": "code"
},
{
"glob": "**/Network Trash/",
"desc": "AFP trash"
},
{
"glob": "**/nix/store/",
"desc": "NixOS package"
},
{
"glob": "**/node_modules/",
"desc": "Node.js library"
},
{
"glob": "**/opt/(bin|cygdrive|dev|etc|games|google|include|lib?(32|64|x64)|local|locale|man|proc|sbin|share|src|tmp|usr|var|x11)/",
"desc": "FHS package"
},
{
"glob": "**/osv/(apps|arch|compiler|external|include|java|modules|musl|tests)/",
"desc": "code"
},
{
"glob": "**/packages/(*.mpkg|*.pkg)/",
"desc": "macOS package"
},
{
"glob": "**/Perl?(64)/(bin|eg|etc|html|lib|site)/",
"desc": "code"
},
{
"glob": "**/pg/pgsql/",
"desc": "code"
},
{
"glob": "**/pkg/(acceptance|ccl|cli|cmd|server|sql|storage|ui|util|workload)/",
"desc": "code"
},
{
"glob": "**/pkgconfig/",
"desc": "code"
},
{
"glob": "**/pkgs/",
"desc": "Python code"
},
{
"glob": "**/proc/(acpi|asound|bus|driver|fs|ipmi|irq|net|scsi|self|sys|sysvipc|thread-self|tty)/",
"desc": "Linux /proc backup"
},
{
"glob": "**/Program Files?( x86)/",
"desc": "Windows program"
},
{
"glob": "**/ProgramData/",
"desc": "Windows application support"
},
{
"glob": "**/ps/(cache|config|logs|tmp)/",
"desc": "PhotoStructure docker dir"
},
{
"glob": "**/Python*/(dlls|doc?(s)|include|lib?(s)|scripts|tcl|tools)/",
"desc": "code"
},
{
"glob": "**/resources/media/face/",
"desc": "face thumbnail"
},
{
"glob": "**/Ruby*/(bin|include|lib|msys*|packages|share|ssl)/",
"desc": "code"
},
{
"glob": "**/rubygems-*/",
"desc": "Ruby code"
},
{
"glob": "**/sdk/(build-tools|emulator|extras|patcher|platform-tools|platforms|sources|tools)/",
"desc": "code"
},
{
"glob": "**/site-packages/",
"desc": "Python code"
},
{
"glob": "**/spotifycache/",
"desc": "Spotify cache"
},
{
"glob": "**/src/**/(bin|ci|dist|doc?(s)|etc|lib?(s)|main|spec?(s)|src|test?(s)|tools|util)/",
"desc": "code"
},
{
"glob": "**/SteamApps/",
"desc": "Game files"
},
{
"glob": "**/System/(applications|developer|driverkit|iossupport|library)/",
"desc": "macOS System"
},
{
"glob": "**/System Volume Information/",
"desc": "Windows system metadata"
},
{
"glob": "**/System32/",
"desc": "Windows system"
},
{
"glob": "**/temp/",
"desc": "temporary file"
},
{
"glob": "**/Temporary Items/",
"desc": "temporary file"
},
{
"glob": "**/test/fixtures/",
"desc": "code"
},
{
"glob": "**/test_suite/",
"desc": "code"
},
{
"glob": "**/testutil?(s)/",
"desc": "code"
},
{
"glob": "**/third_party/",
"desc": "code"
},
{
"glob": "**/Thumbnail?(s)/",
"desc": "thumbnail"
},
{
"glob": "**/Thumbs.db/",
"desc": "Windows thumbnail"
},
{
"glob": "**/tmp/",
"desc": "temporary file"
},
{
"glob": "**/Trash/",
"desc": "trash dir"
},
{
"glob": "**/usr/(bin|cygdrive|dev|etc|games|include|lib?(32|64|x64)|local|locale|man|proc|sbin|share|src|tmp|usr|var)/",
"desc": "FHS system"
},
{
"glob": "**/var/(cache|crash|games|lib|local|lock|log|logs|mail|run|snap|spool|tmp)/",
"desc": "Linux /var"
},
{
"glob": "**/Windows/(boot|containers|cursors|fonts|help|installer|logs|microsoft.net|servicing|softwaredistribution|system?(32)|syswow64|temp)/",
"desc": "Windows system dir"
},
{
"glob": "**/Windows10Upgrade/",
"desc": "Windows upgrade"
},
{
"glob": "**/Xcode.app/",
"desc": "macOS code"
},
{
"glob": "/bin/",
"desc": "command binary"
},
{
"glob": "/Dell/",
"desc": "Windows driver"
},
{
"glob": "/dev/",
"desc": "device file"
},
{
"glob": "/Drivers/",
"desc": "Windows driver"
},
{
"glob": "/etc/",
"desc": "system configuration file"
},
{
"glob": "/home/mrm/.config/",
"desc": "env.PS_CONFIG_DIR"
},
{
"glob": "/home/mrm/snap/",
"desc": "Snap software"
},
{
"glob": "/initrd/",
"desc": "initial ramdisk"
},
{
"glob": "/Intel/",
"desc": "Windows driver"
},
{
"glob": "/lib?(32|64|x64)/",
"desc": "system library"
},
{
"glob": "/lost+found/",
"desc": "fsck recovered file"
},
{
"glob": "/Microsoft/",
"desc": "Windows driver"
},
{
"glob": "/NVIDIA/",
"desc": "Windows driver"
},
{
"glob": "/proc/",
"desc": "process metadata"
},
{
"glob": "/sbin/",
"desc": "system binary"
},
{
"glob": "/snap/",
"desc": "snap software"
},
{
"glob": "/sys/",
"desc": "system metadata"
},
{
"glob": "/Windows/",
"desc": "Windows system"
},
{
"glob": "/Windows.old/",
"desc": "Windows system"
}
]
}
}
See also
-
PhotoStructure | Why didn't my file get imported into my Library?
-
PhotoStructure | How do I prevent a directory from being imported into my library?
Update 2022-06-27
The prior globs implementation caused confusion and new import bugs. Glob patterns are now only used for exclusions.
Click the orange pencil icon on this post to view the prior approach.