Skip to content

Improve block loader fallback to source when source mode is synthetic. #115394

Open
@martijnvg

Description

Sometimes MappedFieldType#blockLoader(...) implementations fallback to an implementation that uses source. For example when a field has doc values or stored fields disabled, when ignore above or ignore above have been configured. Meaning it would read the _source field and then extract the relevant field out of it and use that as value to be returned by the block loader.

When synthetic source is enabled then instead the source gets computed from many doc value or stored fields, and then the relevant field gets extracted. This is very slow and this should be improved. The interesting part with synthetic source is that we don't need to compute the source in order to provided a fallback values as part of block loaders returned by MappedFieldType#blockLoader(...).

Synthetic source details relevant to block loader fallback logic:

  • A field value exceeds the configured ignore above, then the value is stored in a separate stored field with the suffix _original.
  • A field value is malformed, then the value gets stored in a stored field with the same name as is defined in the mapping. Regardless of whether stored fields are enabled.
  • A field is has no stored or doc values fields. Then it gets stored in _ignored_source stored field.
  • Ignored source is the fallback for synthetic source to avoid content in source getting lost. So for example if a an object field is disabled or number of allowed mapped fields is exceeded, the field values / content should end up in ignored source.

In case of synthetic source the block loaders returned by MappedFieldType#blockLoader(...) can be made aware if these details and instead of returning a BlockSourceReader based implementation, return an implementation that uses the right stored field or uses ignored source.

Tasks:

  • Handle ignore above more efficiently.
  • Handle ignore malformed more efficiently.
  • Handle the reading field values that are stored in ignored source.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions