Skip to content

change: Consider disabling catalog caching by default and enabling it on a case-by-case basis in Meltano Hub #8970

Open
@edgarrmondragon

Description

Feature scope

Configuration (settings parsing, validation, etc.)

Description

Rationale

Catalog caching address the time and stability aspect of syncing dynamic sources, but causes issues when the catalog becomes stale and breaks users' expectations.

After the implementation of on-demand cache invalidation via --refresh-catalog (#8580), and the fortunate detail of it adding a _use_cached_catalog extra setting to extractors, we can consider setting it to false by default and going through the Hub updating adding use_cached_catalog: true to those extractors that most definitely need it: all SQL sources, Salesforce and any other dynamic sources.

This could be a breaking change, so we wanna be careful.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Type

    Projects

    • Status

      Discussed

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions