Closed
Description
Related topics:
Meltano Version
2.3.0 / 2.4.0 / 2.6.0
Python Version
3.8
Bug scope
CLI (options, error messages, logging, etc.)
Operating System
Windows 10, Ubuntu
Description
Catalog isn't updating properly when using run. It's possible this is just with the run command and/or just with Windows I haven't verified anything beyond this setup. Repro steps below
Code
1. Get a postgres instance up my goto is `podman run -e POSTGRES_PASSWORD=postgres -p 5432:5432 -h postgres d postgres`
2. Create a new meltano project, `meltano init catalog_bug`
3. Add this `meltano.yml`
version: 1
default_environment: dev
plugins:
extractors:
- name: tap-postgres
namespace: tap_postgres
pip_url: git+https://github.com/AutoIDM/tap-postgres.git
executable: tap-postgres
capabilities:
- state
- catalog
- discover
settings:
- name: sqlalchemy_url
kind: password
select:
- autoidm-testh.col1
loaders:
- name: target-jsonl
variant: andyh1203
pip_url: target-jsonl
environments:
- name: dev
- name: staging
- name: prod
project_id: 0386f428-f486-4e8c-8ea3-75146356d2f1meltano install(Note a.envvar ofTAP_POSTGRES_SQLALCHEMY_URL="postgresql://postgres:postgres@localhost:5432/postgres"maybe be helpful!)- Create some test data
CREATE TABLE autoidm.testh (
col1 text NULL
);
insert into autoidm.testh (col1)
VALUES('row1');
- Run meltano
meltano run tap-postgres target-jsonl - Alter the table schema
alter table autoidm.testh
add column col2 text NULL;
insert into autoidm.testh (col1, col2)
VALUES('row2', 'row2');
- Manually update
meltano.ymlto include the new column
select:
- autoidm-testh.col2
- Run
meltano run tap-postgres target-jsonl, note the new output data
{}
{}
``
Two empty rows as col2 doesn't exist
10. Run `rm -r .meltano\run\tap-postgres\`
11. Run `meltano run tap-postgres target-jsonl` everything works now
Metadata
Assignees
Labels
Type
Projects
Status
Discussed