Summary: This Issue proposes a discussion around the potential benefits and drawbacks of transitioning filebeat's state management across all inputs to a unified, open source key-value (KV) store. The current implementation uses custom logic and a simple registry file across all inputs, causing readability, complexity and maintenance overhead, specially when the files start to grow to very large sizes. By moving to a unified KV store, we could potentially streamline state management and improve maintainability & readability, but it also introduces new challenges-especially around migration, coordination, and performance.

Purpose: Currently, Filebeat relies on custom libbeat/statestore (generic), v2 cursor.StateStore(generic), entityanalytics/kvstore (specific) etc. to name a few, implementations, with each input managing its state in slightly different ways. While this works, it results in the following issues:

• Inconsistency: Multiple state management implementations across multiple inputs lead to complexity in maintaining consistency.
• Redundancy: Different teams managing different parts of the state logic often duplicate efforts or apply variations of the same solution.
• Coordination Overhead: Any changes or improvements to certain state handling implementations require coordination with multiple teams, slowing down development.
• Technical Debt: As new features and fixes are added, the disparity between implementations grows, increasing the cost of future changes.

Moving to a unified open source KV store could offer a more scalable and maintainable solution, but there are trade-offs that need to be considered.

• Consistency Across Inputs: A single KV store would ensure a uniform approach to state management, reducing the number of unique implementations and making it easier to debug issues.
• Easier Maintenance: Having one place for state management logic simplifies the process of updates and fixes, reducing the effort required to coordinate across teams. Using an open source kv-store also gives us a bunch of standard functionalities out of the box which removes the need for iterative developments and prevents issues of missing features.
• Future Proofing: A unified store could provide flexibility in scaling to new inputs or features that may emerge over time without introducing additional state logic per input.
• Scalability: An open source, unified kv store will potentially provide easier scalability, if required, than having several custom solutions lying all across the board.

Challenges & Considerations:

• Migration complexity: We would need migration logic to ensure state is properly moved from the existing custom registry files to the unified KV store. This would require careful consideration to avoid state loss or corruption during upgrade.
• Performance implications: An open source KV store may introduce performance overhead depending on the implementation. We'll need to assess whether it can handle the throughput and low-latency demands of Filebeat across different environments.
• Coordination across teams: Moving to a unified solution would require coordination across different teams and a refactor of existing logic, which could delay other projects or features.
• Failure handling: A centralized store could become a single point of failure. Redundancy mechanisms and proper error handling will be essential to prevent issues from cascading. Although the current implementation also has a single point of failure, i.e. the registry file with the baggage of several different state management implementations.

Some popular open source KV store libraries :

• Bitcask
• Badger
• Diskv
• Etcd
• Lotusdb
• Pogreb
• cznic/kv (seems to be the only key-ordered KV store in go atm as suggested by @efd6)

The goal of this Issue is to open up further discussions on simplifying the state management logic across the board and promote a more maintainable solution if possible for the future. There are several challenges and obstacles to getting this implemented across the board, hence wanted your feedback on this. Maybe we could start small with a few selected inputs and based on the success/failure move foreword ?

Please do share your thoughts on this.