Thanks for starting working on it. Can’t wait… How do you plan to handle terms with spaces or colons?
The query parser will look for double or single quotes, so
who:"Sam whatshisface" will work.
I’ve got the grammar nailed down.
Update 2021-03-24: I was able to add
after: namespaces, but as the
MATCH operator doesn’t have a
NOT MATCH sibling, I had to remove the
- query negation from the grammar.
Update 2021-03-24: OMG I figured out how to do negation, but only for namespaced query terms.
Update 2021-04-10: The grammar has changed sufficiently to delete most of this post: see the instructions on the website instead.
I was able to add
--query support to the
list tool, so queries are going to be available to both the UI and the CLI:
$ ./photostructure list --help Usage: list [options] List paths in a Library. Options: ... --query <TERMS...> Apply the given query and return all matching asset file paths. Must be the last argument. Incompatible with --where. -j --json Emit a JSON object for every asset file with "hidden" and "shown" flags --dump Emit a JSON object that includes all column values --where <WHERE> Apply a SQL WHERE clause to filter the rows returned. (default: "") --orderby <ORDERBY> Apply a SQL ORDER BY clause to filter the rows returned. (default: "")
I had a bit of a hitch in my search giddyup: SQLite doesn’t support multiple
MATCH clauses, so I had to teach PhotoStructure how to condense these queries down to disjunctive normal form. This seems like it works now, and I’m building out the frontend now.
If you’ve emailed me logs or asked for support for the last day or two, know that I’ll be getting to your message soon.
Have you considered using INTERSECT instead? Then you could use one MATCH per subquery
I may be misunderstanding you, but this would still mean I have to reduce to conjunctive normal form (rather than disjunctive normal form), right?
FWIW, I’ve got the SQL generator working…
Well, I assumed you have multiple AND conditions that’s why I suggested INTERSECT but I think if you rewrite the generating code to use INTERSECT for AND clause and UNION ALL for OR clause, you don’t need to convert to any normal forms and use it as is. But obviously my idea requires at least performance check
Yeah, if UNION turns out to have horrible performance, I will certainly try INTERSECT. Thanks!
Goodness knows that I’ve been surprised by seemingly equivalent SQL that was 10x+ faster (especially when query planner wasn’t very sophisticated).
Have you tried to do UNION ALL and then DISTINCT at the end of the query? They say UNION ALL has better performance than just UNION
I’ll check that out, thanks.
@mrm I’ve checked the documentation for your search feature and it looks really awesome. I suggest to consider more convenient syntax for date ranges which, in my opinion, going to be one of the most used filters.
Please also consider some libraries for parsing dates like
I would be nice if you also think how to fix one annoying issue all DAMs have. If you have hierarchical keywords
a/b/c and you look for
a/b there should be a way to distinguish cases when you want to find exactly that keyword without its descendants. Maybe it could be solved by
kw:a/b AND -kw:a/b/* but it would be nice to have neater syntax for that case
This is interesting to me, because I actually don’t think I’ll use date ranges very much. I’m much more interested in searching by people and keywords, especially in combination.
Obviously neither of us is right or wrong, but it does underscore that there are lots of different preferred use cases.
Thanks for those links!
I thought non-ISO-formatted dates would be neat, too, and did a bit of looking for a date parser that supported internationalization, but (like most of npm), were either English-only, had no tests, had been abandoned, or all of the above.
I think just a could regexs would get year, month, week, and day parsing. Here’s a google sheet: please ask for edit access if you’d like to add a language or other examples.
tag:/a/b/c suffice (where /a/b/c is the entire path of the tag)?
Just as more feedback… I do want to deal with date ranges a lot, but personally never seem to use the “natural language” type date formats (last week, etc.). Maybe it’s the programmer in me that finds them too ambiguous (or I’m just too old :-)).
Anyhow, I certainly don’t mind if they are there for folks that like them, I can see the use, but just to toss in my 2 cents…
Sorry, I didn’t explain better about tags
What if I want to find
tag:/a/b but I don’t want to include
tag:/a/b/c or any other descendants of
Real use case example
kw:dog but you want to exclude
kw:dog/Rex etc. So basically, you want to find only unknown dogs.
Hope this example explains my concern better
I’m also a programmer and
yyyy-MM-dd is the only thing that I prefer most of the time, but as your product’s supposed main audience is non-IT, I assume
date:"last summer" is more preferable to type than
after:2020-06-01 AND before:2020-09-01
BTW, relative dates you can find even in a very IT stuff
Ah! I already added operators for date terms: perhaps
tag:=/keywords/dog means only look at that tag, and not match descendant tags: would that work?
Edit 20210410: deleted the remainder of the post, as it has been superseded by the documentation.