6
\$\begingroup\$

I am putting together a little module for oss release that will let you parse a boolean expression consisting of and/AND/or/OR's (no brackets yet) and output a complete elasticsearch query.

Boolean expression logic:

Right now it uses OR as the basis and puts everything on top of that as ANDs. This means that AND binds left to right.

I lack input on:

  1. The quality of the output elasticsearch query - can it be simplified? Are there better approaches?
  2. The way I interpret the boolean expression.

def string_to_query(s):
    s = s.lower()
    tokens = [' '.join(t.split()) for t in s.split('or')]
    or_terms = []
    while tokens:
        leaf = tokens.pop()

        and_terms = leaf.split('and')
        if len(and_terms) < 2:
            term = and_terms[0]
            or_terms.extend([
                {"match": {"Review.Text": {
                    "query": term, "operator": "and"}}},
                {"match": {"Review.Title": {
                    "query": term, "operator": "and"}}}
            ])
        else:
            filters = [
                {"bool": {
                    "should": [{"match": {"Review.Text": {
                        "query": term, "operator": "and"}}},
                        {"match": {"Review.Title": {
                            "query": term, "operator": "and"}}}]

                }} for term in and_terms]
            or_terms.append(
                {"bool": {
                    "must": filters
                }})

    return {"query":
            {"bool": {
                "should": or_terms
            }}}


query = string_to_query(
    'dog and dog food or cat and cat food'
)

assert query == {
"query": {
    "bool": {
    "should": [
        {
        "bool": {
            "must": [
            {
                "bool": {
                "should": [
                    {
                    "match": {
                        "Review.Text": {
                        "operator": "and",
                        "query": "cat "
                        }
                    }
                    },
                    {
                    "match": {
                        "Review.Title": {
                        "operator": "and",
                        "query": "cat "
                        }
                    }
                    }
                ]
                }
            },
            {
                "bool": {
                "should": [
                    {
                    "match": {
                        "Review.Text": {
                        "operator": "and",
                        "query": " cat food"
                        }
                    }
                    },
                    {
                    "match": {
                        "Review.Title": {
                        "operator": "and",
                        "query": " cat food"
                        }
                    }
                    }
                ]
                }
            }
            ]
        }
        },
        {
        "bool": {
            "must": [
            {
                "bool": {
                "should": [
                    {
                    "match": {
                        "Review.Text": {
                        "operator": "and",
                        "query": "dog "
                        }
                    }
                    },
                    {
                    "match": {
                        "Review.Title": {
                        "operator": "and",
                        "query": "dog "
                        }
                    }
                    }
                ]
                }
            },
            {
                "bool": {
                "should": [
                    {
                    "match": {
                        "Review.Text": {
                        "operator": "and",
                        "query": " dog food"
                        }
                    }
                    },
                    {
                    "match": {
                        "Review.Title": {
                        "operator": "and",
                        "query": " dog food"
                        }
                    }
                    }
                ]
                }
            }
            ]
        }
        }
    ]
    }
}
}
\$\endgroup\$
  • \$\begingroup\$ Does this currently work as intended? \$\endgroup\$ – Mast Mar 21 '18 at 15:42
  • 1
    \$\begingroup\$ Yeah i would say so. \$\endgroup\$ – Johannes valbjørn Mar 21 '18 at 16:42
  • 3
    \$\begingroup\$ Hmm, "please correct me on any stack overflow newbie errors" makes me seriously doubt your code works as you intend. \$\endgroup\$ – Peilonrayz Mar 21 '18 at 17:01
  • 1
    \$\begingroup\$ It was a comment made for any errors mde in etiquette on this forum. I am positive my code works as intended. See the assertion in the code block where i show what the output is. Copy paste in a python interpreter and voila . \$\endgroup\$ – Johannes valbjørn Mar 21 '18 at 17:28
  • \$\begingroup\$ My question is 1. Whether the output elasticsearch query is overly complex and could be simplified? And 2. Whether the output reflected a correct and intuitive understanding of the boolean string input \$\endgroup\$ – Johannes valbjørn Mar 21 '18 at 17:32
4
\$\begingroup\$

Your usage of split makes your function rather fragile:

>>> string_to_query('doctor and heart')
{'query': {'bool': {'should': [{'bool': {'must': [{'bool': {'should': [{'match': {'Review.Text': {'operator': 'and',
                                                                                                  'query': ''}}},
                                                                       {'match': {'Review.Title': {'operator': 'and',
                                                                                                   'query': ''}}}]}},
                                                  {'bool': {'should': [{'match': {'Review.Text': {'operator': 'and',
                                                                                                  'query': ' '
                                                                                                           'heart'}}},
                                                                       {'match': {'Review.Title': {'operator': 'and',
                                                                                                   'query': ' '
                                                                                                            'heart'}}}]}}]}},
                               {'match': {'Review.Text': {'operator': 'and',
                                                          'query': 'doct'}}},
                               {'match': {'Review.Title': {'operator': 'and',
                                                           'query': 'doct'}}}]}}}

Which is equivalent to: "(the empty string AND heart) OR doct" rather than "doctor AND heart".

An other use-case to consider is the use of "and" or "or" as words to search for rather than operators (as "Tom and Jerry", I don't want to search for documents containing "Tom" and "Jerry" separately, but for documents containing the phrase "Tom and Jerry").

Usually, for these kind of problems, an intermediate representation produced by an ad-hoc parser is way better and simpler to convert to the end result. Here I suggest producing a list of lists, since you don't (yet) consider priorisation of clauses using parenthesis. Thus:

[
   [A, B, C],
   [D, E],
   [F],
]

Would be equivalent to "(A and B and C) or (D and E) or F". Which can then easily be converted to the elasticsearch query DSL using simple list comprehensions. The catch, however is that each clause can be complete sentences and must apply to two fields: "Review.Text" and "Review.Title". This is where the multi-match query can simplify the whole writing: each clause A, B, C, D, E, and F would be converted to

{'multi_match': {
    'query': clause,
    'type': 'phrase',
    'fields': ['Review.Text', 'Review.Title'],
}}

With all the advantages of the multi-match query such as giving more weight to a single field.


The following rewrite extend the supported syntax to allow double quotes to mean "perfect match":

import re


class ClauseParser:
    def __init__(self, tokenizer, *operators):
        self._tokenizer = tokenizer
        self._operators = set(operators)
        self._found_operator = None

    def __iter__(self):
        for token in self._tokenizer:
            token_value = token.group(0)
            if token.group(2) in self._operators:
                self._found_operator = token_value
                return
            yield token_value

    @property
    def operator(self):
        found_operator = self._found_operator
        self._found_operator = None
        return found_operator


def parser(tokenizer):
    clause_parser = ClauseParser(tokenizer, 'and', 'or')
    current_group = []
    while True:
        current_group.append(' '.join(clause_parser))
        found_operator = clause_parser.operator
        if found_operator != 'and':
            yield current_group
            if found_operator is None:
                return
            current_group = []


def convert_and_clauses(clauses):
    return [
        {'multi_match': {
            'query': clause,
            'type': 'phrase',
            'fields': ['Review.Text', 'Review.Title'],
        }} for clause in clauses
    ]


def string_to_query(phrase):
    tokenizer = re.finditer(r'"([^"]+)"|(\w+)', phrase)
    query = list(parser(tokenizer))

    or_clauses = {'bool': {'should': [
        {'bool': {'must': convert_and_clauses(clauses)}}
        for clauses in query
    ]}}

    return {'query': or_clauses}

Example usage:

>>> string_to_query('doctor and heart')
{'query': {'bool': {'should': [{'bool': {'must': [{'multi_match': {'fields': ['Review.Text',
                                                                              'Review.Title'],
                                                                   'query': 'doctor',
                                                                   'type': 'phrase'}},
                                                  {'multi_match': {'fields': ['Review.Text',
                                                                              'Review.Title'],
                                                                   'query': 'heart',
                                                                   'type': 'phrase'}}]}}]}}}
>>> string_to_query('"Tom and Jerry" or "Road runner and vil coyote"')
{'query': {'bool': {'should': [{'bool': {'must': [{'multi_match': {'fields': ['Review.Text',
                                                                              'Review.Title'],
                                                                   'query': '"Tom '
                                                                            'and '
                                                                            'Jerry"',
                                                                   'type': 'phrase'}}]}},
                               {'bool': {'must': [{'multi_match': {'fields': ['Review.Text',
                                                                              'Review.Title'],
                                                                   'query': '"Road '
                                                                            'runner '
                                                                            'and '
                                                                            'vil '
                                                                            'coyote"',
                                                                   'type': 'phrase'}}]}}]}}}
>>> string_to_query('cat and cat food or dog and dog food')
{'query': {'bool': {'should': [{'bool': {'must': [{'multi_match': {'fields': ['Review.Text',
                                                                              'Review.Title'],
                                                                   'query': 'cat',
                                                                   'type': 'phrase'}},
                                                  {'multi_match': {'fields': ['Review.Text',
                                                                              'Review.Title'],
                                                                   'query': 'cat '
                                                                            'food',
                                                                   'type': 'phrase'}}]}},
                               {'bool': {'must': [{'multi_match': {'fields': ['Review.Text',
                                                                              'Review.Title'],
                                                                   'query': 'dog',
                                                                   'type': 'phrase'}},
                                                  {'multi_match': {'fields': ['Review.Text',
                                                                              'Review.Title'],
                                                                   'query': 'dog '
                                                                            'food',
                                                                   'type': 'phrase'}}]}}]}}}
| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ This is exactly what we needed!!!! Your solution is much appreciated! Would you be ok with releasing this code in a generalised way (fields should be given as args) under a MIT license to pypi ? I could take care of packaging, readme, tests and upload to pypi / github and list you as author \$\endgroup\$ – Johannes valbjørn Mar 22 '18 at 10:47
  • \$\begingroup\$ @Johannesvalbjørn Sure. What I’ve made in the past is to create a pull-request so my name is associated exactly to the work provided and not necessarily to the whole project. \$\endgroup\$ – 301_Moved_Permanently Mar 22 '18 at 10:55
  • \$\begingroup\$ good idea!!! i've created a public repo, youre more than welcome to create a PR to it : github.com/trustpilot/python-stringtoesquery \$\endgroup\$ – Johannes valbjørn Mar 22 '18 at 11:03
1
\$\begingroup\$

I have come up with another solution that yields much simpler results. It uses query_string searching and the builtin boolean expressions over a set of defined fields:

def string_to_query(s):
    s = s.lower()
    tokens = [' '.join(t.split()) for t in s.split('or')]
    or_terms = []

    while tokens:
        leaf = tokens.pop()

        and_terms = leaf.split('and')
        if len(and_terms) < 2:
            term = and_terms[0]
            or_terms.append('"{}"'.format(term.strip()))
        else:
            and_terms = ['"{}"'.format(term.strip()) for term in and_terms]
            and_string = "( " + " AND ".join(and_terms) + " )"
            or_terms.append(and_string)

    query_string = " OR ".join(or_terms)
    return {
        "query": {
            "query_string": {
                "fields": ["Review.Title", "Review.Text"],
                "query": query_string
            }
        }
    }

query = string_to_query(
    'dog and dog food or cat and cat food'
)

assert query == {
"query": {
    "query_string": {
    "fields": [
        "Review.Title",
        "Review.Text"
    ],
    "query": "( \"cat\" AND \"cat food\" ) OR ( \"dog\" AND \"dog food\" )"
    }
}
}
| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.