This issue is to track the progress and discussion around using the newly merged dynamic ECS template from elastic package: elastic/elastic-package#1073

Status on specific tests can be found in the issue comments below, but as an overview:

• Custom input testing - Done, working without issues.
• SEI Integrations - Done, working without issues
• Endpoint - Initial testing performed, only a minor issue requires investigation.
• APM - Initial testing performed, no apparent technical issues, but needs some discussion around implementation
• Observability Integrations - Done, currently looking at next possible steps.
• Cloud Defense - Initial testing done, outside of integration package, no issues yet
• Cloud Security Posture - Initial testing done, outside of integration package, no issues yet

This was initially created for common usecases in custom input packages, that do not have any out of the box ECS mapping, and to cover beat processors like add_host_metadata that produces ECS fields that might not already be mapped: elastic/elastic-package#1018

However this later evolves into replacing the need for adding ECS fields manually in each integration package, allowing all integration packages to map ECS field types automatically from a single source, to make it all more consistent.

This feature is currently in a TESTING phase until further notice, and package developers could use the below process to test their own packages with the dynamic template:

1. Add the required field in _dev/build/build.yml

dependencies:
  ecs:
    reference: git@8.6
    import_mappings: true

2. Remove the ecs.yml file from each datastream in the package.
3. Ensure that the base-fields.yml still has mappings for @timestamp and data_stream fields

- name: data_stream.type
  type: constant_keyword
  description: Data stream type.
- name: data_stream.dataset
  type: constant_keyword
  description: Data stream dataset name.
- name: data_stream.namespace
  type: constant_keyword
  description: Data stream namespace.
- name: event.module
  type: constant_keyword
  description: Event module
  value: ti_abusech
- name: event.dataset
  type: constant_keyword
  description: Event dataset
  value: ti_abusech.malware
- name: "@timestamp"
  type: date
  description: Event timestamp.

4. Update format_version in your root manifest.yml to 2.3.0
5. Ensure you build the latest elastic-package
6. Either spin up the stack and test the package manually, or if you have configured system test

Related issues:

elastic/elasticsearch#85692
#4961
elastic/ecs#1869
elastic/elasticsearch#89743
elastic/elastic-package#1018
elastic/elastic-package#1073