Elasticsearch integration for guillotina. Supports Elastic search 7.x and 8.x

pip install guillotina_elasticsearch defaults to Elasticsearch 8.x support.

config.yaml can include elasticsearch section

elasticsearch:
  index_name_prefix: "guillotina-"
  connection_settings:
    hosts:
      - "http://127.0.0.1:9200"
    sniffer_timeout: 0.5
    sniff_on_start: true
  security_query_builder: "guillotina_elasticsearch.queries.build_security_query"

Example custom security_query_builder settings:

async def security_query_builder(container, request):
    return {
        'query': {
            'bool': {
                'filter': {
                    'bool': {
                        'should': [{'match': {'access_roles': "foobar"}}],
                        'minimum_should_match': 1
                    }
                }
            }
        }
    }

Setup your python virtual environment for version >=3,8. Tested with 3.8, 3.9 and 3.10

# Linux
pip install -e ".[test]"
pytest tests/

By default the tests run an ES fixture with version 8. If you want to run the tests for ES version 7, change the image version in the conftest.py

POST SITE_URL/@catalog

{}

DELETE SITE_URL/@catalog

{}

guillotina_elasticsearch comes with a es-migrate guillotina command:

./bin/g es-migrate

This command will perform a live migration on the index. It does this by performing the reindex on a new index while the other one is still active.

New index and delete requests are performed on both indexes during live migration.

It is also smart about how to migrate, doing a diff on the mapping and only reindexing the fields that changed.

In this version, the library elasticsearch 7 has been upgraded to elasticsearch 8. There are some changes that need to be taken into account in the settings of old elasticsearch config files.

• The hosts field in the guillotina's configuration file, need to include the scheme: http or https
• The sniffer_timeout in the guillotina's configureation file, can not be None
• The doc_type has been removed. Specifying types in requests is no longer supported.
• The include_type_name parameter is removed.

The elasticsearch field of the config.yaml file is directly passed to instantiate AsyncElasticsearch. The class definition is the same of the synchronous one, to know how to configure your ES take a look at: https://elasticsearch-py.readthedocs.io/en/v8.12.0/api/elasticsearch.html#elasticsearch.Elasticsearch

• ES 6 does not have doc types support
• aioes deprecated
• IElasticSearchUtility changes:
  • query: doc_type param no longer used
• IElasticSearchUtility.conn changes:
  • put_mapping
  • put_settings
  • put_alias
  • get: needs doc_type=DOC_TYPE
  • bulk: needs doc_type=DOC_TYPE
  • conn.transport.get_connection(): ._session -> .session, ._base_url -> .base_url
  • conn.transport.get_connection().[method] -> need to include content-type: application/json

If container es (elasticsearch) fails to start when running tests, you should increase max_map_count. command

# Linux
sudo sysctl -w vm.max_map_count=262144

All mappings are flattened when they are applied to Elastic Search.

Elasticsearch doesn't support different mappings for different types.

If there are conflicting mappings, this plugin will choose the mapping in the addon last applications listed which follows the guillotina practice of allowing the last listed application to override the settings of the previously list application

elasticsearch:
  index_name_prefix: "guillotina-"
  connection_settings:
    hosts:
      - "127.0.0.1:9200"
    sniffer_timeout: 0.5
    sniff_on_start: true
    use_ssl: true
    http_auth: admin:admin