add ability to switch off/on generation of Parquet Files #1074
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1074 +/- ##
============================================
- Coverage 51.49% 51.47% -0.03%
+ Complexity 717 716 -1
============================================
Files 95 95
Lines 5909 5916 +7
Branches 787 790 +3
============================================
+ Hits 3043 3045 +2
- Misses 2578 2579 +1
- Partials 288 292 +4 ☔ View full report in Codecov by Sentry. |
mozzy11
force-pushed
the
isolate_fhir_sync
branch
from
2f13701 to
9ffab3a
Compare
|
cc @bashir2 |
7 tasks
|
cc @bashir2 |
bashir2
reviewed
Jul 11, 2024
There was a problem hiding this comment.
Thanks @mozzy11 for this change. I had a look and made some comments but in general I feel we need to think deeper about the implications of skipping Parquet file generation; I feel there are still scenarios not covered in your change (beyond what I have commented below) but need to think more about this.
pipelines/batch/src/main/java/com/google/fhir/analytics/FhirEtlOptions.java
Outdated
Show resolved
Hide resolved
pipelines/batch/src/main/java/com/google/fhir/analytics/FetchSearchPageFn.java
Outdated
Show resolved
Hide resolved
pipelines/controller/src/main/java/com/google/fhir/analytics/DataProperties.java
Outdated
Show resolved
Hide resolved
pipelines/controller/src/main/java/com/google/fhir/analytics/DataProperties.java
Outdated
Show resolved
Hide resolved
Co-authored-by: Bashir Sadjad <bashir@google.com>
…lOptions.java Co-authored-by: Bashir Sadjad <bashir@google.com>
mozzy11
changed the title
mozzy11
force-pushed
the
isolate_fhir_sync
branch
6 times, most recently
from
fab000c to
ac3c1ff
Compare
mozzy11
force-pushed
the
isolate_fhir_sync
branch
from
72c160f to
345d321
Compare
|
cc @bashir2 . i have worked on the review comments |
|
Hello @bashir2 . I addresed the comments above |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1073
Add ability to switch of creation of a parquet DWH in case of syncying betwen FHIR servers
Added a flag to the
createParquetDwhto the controller to switch off/on creation of parquet DWHE2E test
Adedd e2e tests for synching from a hapi fhir sever to another using the pipeline controller for both FULL and INCREMENTAL modes while swtching on/off creation of parquet DWH
TESTED:
Testes Locally syncying between FHIR server while the parquet DWH is switched off/on
Checklist: I completed these to help reviewers :)
I have read and will follow the review process.
I am familiar with Google Style Guides for the language I have coded in.
No? Please take some time and review Java and Python style guides.
My IDE is configured to follow the Google code styles.
No? Unsure? -> configure your IDE.
I have added tests to cover my changes. (If you refactored existing code that was well tested you do not have to add tests)
I ran
mvn clean packageright before creating this pull request and added all formatting changes to my commit.All new and existing tests passed.
My pull request is based on the latest changes of the master branch.
No? Unsure? -> execute command
git pull --rebase upstream master