Read the Incremental Update for v0.27 to learn what's new in this release!

What's Changed

• Avoid loss of precision in deserializing decimals from JSON in python by @ryzhyk in #2511
• docker: point compose file to Feldera version 0.26.0 by @gz in #2513
• [adhoc] Fix serialization of time values. by @ryzhyk in #2512
• Fix a bug in the text output that sometimes doesn't send the result headers by @gz in #2510
• [SQL] Constant string expressions should not be considered monotone by @mihaibudiu in #2515
• [adhoc] Eliminate negative weights in outputs. by @ryzhyk in #2521
• dbsp_adapters: Fix hang because a queue with 0 records could be nonem… by @blp in #2522
• dbsp: Speed up sharding by implementing N-way merge. by @blp in #2523
• [SQL] Propagate monotonicity information through flatmap operators by @mihaibudiu in #2538
• [SQL] NULL values for columns with LATENESS are never considered late by @mihaibudiu in #2518
• [SQL] Do not ignore FILTER for ARRAY_AGG by @mihaibudiu in #2542
• Update markdown for docusaurus v3. by @gz in #2543
• [SQL] Do not carry in joins fields that are already available in keys by @mihaibudiu in #2540
• Improve async merger by @blp in #2537
• Adjust console banner with new logo. by @gz in #2550
• Add pre commit config by @gz in #2546
• Fixup ASOF JOIN docs. by @ryzhyk in #2548
• py: clear the view_tx queues when shutting down a pipeline by @abhizer in #2536
• Remove neighborhoods and quantiles. by @ryzhyk in #2554
• dbsp: Summarize operator metadata into root in circuit profiles. by @blp in #2553
• Upgrade the pub-sub crate to 0.29.1. by @ryzhyk in #2557
• [SQL] Variant support for binary values by @mihaibudiu in #2551
• Disable tracing by default. by @ryzhyk in #2564
• [sql] Prepare to publish sqllib on crates.io. by @ryzhyk in #2562
• Apply new Feldera brand theme to WebConsole by @Karakatiza666 in #2497
• [SQL] Propagate monotonicity information through ASOF joins by @mihaibudiu in #2566
• [Benchmarks] make batch size configurable by @mihaibudiu in #2490
• [SQL] Remove dependency on json crate by @mihaibudiu in #2561
• [SQL] Implement max and min value for DECIMAL by @mihaibudiu in #2552
• py: publish to pypi by @abhizer in #2534
• Rename www.feldera.com/docs -> docs.feldera.com. by @gz in #2569
• Pre commit adjustments by @gz in #2573
• Reduce WebConsole app size, add page load indicators by @Karakatiza666 in #2533
• pipeline-manager: logs endpoint by @snkas in #2500
• fda: New command heap-profile. by @blp in #2559
• [Rust] update faker crate; version 2.10 is not backwards compatible with 2.9 by @mihaibudiu in #2576
• [WebConsole] Preload Pipeline Edit page code while on welcome screen by @Karakatiza666 in #2575
• docs: Update the concepts page. by @blp in #2570
• refactored python tests by @rivudhk in #2519
• Rename links feldera.com/docs -> docs.feldera.com by @gz in #2583
• dbsp_adapters: remove dependency on prometheus crate by @abhizer in #2408
• pipeline-manager: fix logs ending test by @snkas in #2587
• [SQL] Extract temporal filters from more kinds of comparison expressions by @mihaibudiu in #2590
• Move top left Feldera logo to top left by @Karakatiza666 in #2577
• [SQL] More conservative ASOF join monotonicity inference by @mihaibudiu in #2572
• Set default_workers() to 8. by @gz in #2488
• [SQL] Support casts between VARIANT and ARRAY and MAP by @mihaibudiu in #2592
• Datagen fixes by @gz in #2599
• Update what-is-feldera.md by @lalithsuresh in #2601
• fda improvements by @gz in #2600
• Adjust README.md to reflect new brand by @gz in #2603
• Implement fda log command. by @gz in #2605
• S3 connector docs by @ryzhyk in #2606
• docs: update paper link in README.md by @lalithsuresh in #2610
• README: Update architecture image with new Feldera logo. by @blp in #2611
• docker: make sqllib crate available to sql-compiler, which got recently moved to crates/ by @lalithsuresh in #2614
• [docs] Add link to the sandbox. by @ryzhyk in #2618
• Multiple WebConsole fixes by @Karakatiza666 in #2622
• release: bump project version to 0.27.0 by @gz in #2625

Full Changelog: v0.26.0...v0.27.0

Read the Incremental Update for v0.26 to learn what's new in this release!

What's Changed

• Aggregation tests in python by @rivudhk in #2240
• [SQL] Use linear aggregates more often by @mihaibudiu in #2356
• Get rid of the changelog. by @gz in #2366
• Fix datagen case sensitivity issue & reaching unreachable code by @gz in #2369
• Switch to released versions for rkyv, ijson, size-of and rust-decimal by @gz in #2371
• Add option to configure circular buffer threshold when parsing Change Stream by @Karakatiza666 in #2345
• pipeline-manager: log GET /healthz at trace level by @snkas in #2372
• [SQL] Do not CSE operators that are followed by GC operators by @mihaibudiu in #2370
• [SQL] Support comparisons between intervals by @mihaibudiu in #2368
• Forgot to rename something. by @gz in #2375
• Add Flink hashmap test by @aehmttw in #2268
• json: key_fields option. by @ryzhyk in #2373
• [SQL] Improved debugging graphs by @mihaibudiu in #2376
• [SQL] aggregate_linear_postprocess is incremental-only by @mihaibudiu in #2379
• [SQL] Do not pass source position information to Rust UDF functions by @mihaibudiu in #2378
• [SQL] Fix arg_min aggregate by @mihaibudiu in #2386
• Initial machinery for ad-hoc queries. by @gz in #2381
• A CLI shell for adhoc queries. by @gz in #2388
• [SQL] Allow joins to index by nullable keys by @mihaibudiu in #2389
• [adapters] Avro input format support. by @ryzhyk in #2391
• Enable Playwright unit tests (not component) by @Karakatiza666 in #2392
• The Makefile embedded in the profile rust program needs tabs by @mihaibudiu in #2395
• Enable nexmark q16; always generate a profile by @mihaibudiu in #2396
• Send notification for failed CI jobs to slack. by @gz in #2403
• Enable ingesting insert_delete changes by copy-paste in Change Stream by @Karakatiza666 in #2377
• Increase and make HTTP timeouts more specific. by @gz in #2405
• [demo] Fix Debezium Postgres CI job. by @ryzhyk in #2406
• [adapters] Disable FT Kafka connector tests. by @ryzhyk in #2393
• Bump quinn-proto from 0.11.6 to 0.11.8 by @dependabot in #2415
• Add basic PostHog tracking to WebConsole by @Karakatiza666 in #2411
• Add requirements.txt to demo projects by @xiasongh in #2416
• Prepare for UDF UI: add file tabs to Pipeline Edit view by @Karakatiza666 in #2410
• [avro] Confluent JDBC sink connector Avro format. by @ryzhyk in #2418
• Update delta-rs crate. by @gz in #2404
• Fix working directory for TPC-H benchmarks by @mihaibudiu in #2398
• Distinguish between table and projected schemas. by @gz in #2422
• [avro] Confluent JDBC avro output format unit test. by @ryzhyk in #2428
• docs: update helm and image version to v0.9.0 by @snkas in #2409
• Fix nightly build by @gz in #2427
• Upgrade redpanda container by @ryzhyk in #2433
• pipeline-manager: refactor API of API key by @snkas in #2413
• tiktok: add query with hopping, tumbling and rolling aggregates by @abhizer in #2414
• [avro] Make Avro parser less strict. by @ryzhyk in #2431
• Add tests for average and sum aggregate for decimal type by @rivudhk in #2436
• [SQL] Remove dependency on sqlx by @mihaibudiu in #2439
• py: wait for a failed pipeline to shutdown by @abhizer in #2435
• Run TPC-H benchmark using file adapter by @mihaibudiu in #2426
• [kafka] Improve flow control in Kafka output. by @ryzhyk in #2440
• Initial skeleton for rust variant type. by @gz in #2441
• Fix nightly test by disabling python, fix benchmark CI... by @gz in #2442
• Disable flaky kafka test. by @gz in #2444
• py: fix build error introduced in 60390d3 by @abhizer in #2443
• Stream the adhoc query results by @gz in #2429
• Disable tpch benchmarks. by @gz in #2446
• Improve auto-complete in shell. by @gz in #2447
• Update README.md batches by @gz in #2448
• pipeline-manager: debug log connection configuration instead of string by @snkas in #2434
• Percentage calculation fix for profiles by @blp in #2449
• [SQL] Support for VARIANT types by @mihaibudiu in #2421
• [adapters] Fix deadlock around catalog handle. by @ryzhyk in #2450
• [avro] Add documentation for the Avro format. by @ryzhyk in #2452
• Adhoc insert by @gz in #2454
• [docs] Update Debezium connector docs. by @ryzhyk in #2456
• [docs] Confluent JDBC sink connector docs. by @ryzhyk in #2458
• [SQL] Document how to create a timestamp from Unix epoch by @mihaibudiu in #2461
• [SQL] Support for JSON by @mihaibudiu in #2453
• pipeline-manager: runner rework by @snkas in #2457
• Update README.md by @lalithsuresh in #2466
• Make ad-hoc insert synchronous. by @gz in #2460
• Adjust benchmark scripts to new setup. by @gz in #2470
• Step size by @blp in #2445
• pipeline-manager: implement /metrics endpoint by @abhizer in #2455
• [adapters] Remove debug output by @ryzhyk in #2473
• dbsp_adapters: Fix adhoc insertion. by @blp in #2477
• Fda improvements by @gz in #2476
• Allow inserts for non-materialized tables. by @gz in #2482
• Fix a bug where datagen ignored null_percentage for structs. by @gz in #2485
• Add exec command to fda. by @gz in #2486
• [SQL] Allow field access in nullable structs by @mihaibudiu in #2484
• [SQL] Fixes 3 bugs: LATENESS unsafe code, WATERMARK uncompilable code… by @mihaibudiu in #2489
• [manager] Improve reporting when the pipeline is dead. by @ryzhyk in #2491
• (de)serialization for the VARIANT type by @ryzhyk in #2469
• [docs] Fix references to Debezium in the docs. by @ryzhyk in #2495
• [docs] Fix broken doc link by @ryzhyk in #2496
• [adapters] Shutdown pipeline on a fatal error. by @ryzhyk in #2481
• Fix code editor not becoming readonly when starting pipeline by @Karakatiza666 in #2499
• Correctly deserialize timestamps from arrow. by @ryzhyk in #2501
• dbsp: Fix performance with small input batches. by @blp in #2505
• release: bump project version to 0.26.0 by @gz in #2508

New Contributors

• @xiasongh made their first contribution in #2416

Full Changelog: v0.25.0...v0.26.0

Read the Incremental Update for v0.25 to learn what's new in this release!

What's Changed

• [SQL] Correct supported range of decimal values by @mihaibudiu in #2016
• [SQL] stddev result may always be null by @mihaibudiu in #2266
• Change cpu_seconds to cpu_msecs by @aehmttw in #2271
• Added cloud benchmark script by @aehmttw in #2080
• update README for tiktok demo / benchmarks by @abhizer in #2277
• [tiktok] Make Kafka endpoint configurable. by @ryzhyk in #2282
• [SQL] Fix a couple more bugs; compile TPCDS by @mihaibudiu in #2292
• [SQL] Start to compute waterline upper bounds only when some data has… by @mihaibudiu in #2293
• Advance Calcite version by @mihaibudiu in #2299
• Add color to profile nodes as a function of time used by @mihaibudiu in #2298
• Fix doc link. by @gz in #2306
• Update connectors doc. by @gz in #2308
• [SQL] Implement the NOW table as a generator by @mihaibudiu in #2301
• Add throughput and memory graphs by @Karakatiza666 in #2307
• Fix broken link. by @gz in #2317
• [SQL] Implement temporal filters by @mihaibudiu in #2309
• Google Pub/Sub input connector by @ryzhyk in #2312
• Update serde-json dependency. by @gz in #2320
• Enable query q13 in nexmark by @mihaibudiu in #2295
• Fix metrics resetting when pausing by @Karakatiza666 in #2325
• Generalize window to support open/closed ranges. by @ryzhyk in #2326
• dbsp: aggregate_linear_postprocess operator. by @ryzhyk in #2323
• Set default crypto provider. by @ryzhyk in #2328
• py: convert SQL types to proper pandas type by @abhizer in #2305
• Fix WebConsole unresponsive under load with change stream by @Karakatiza666 in #2297
• A CLI for interacting with the Feldera REST API by @gz in #2324
• Added TPC-H benchmark suite by @aehmttw in #2276
• [SQL] Do not perform % on negative numbers by @mihaibudiu in #2334
• [SQL] Fix bug in LAG implementation by @mihaibudiu in #2338
• [SQL] Add TO_INT function to convert a byte array to an integer by @mihaibudiu in #2340
• Test for issue2316 by @mihaibudiu in #2341
• pipeline-manager: manual database tests by @snkas in #2329
• pipeline-manager: restrict maximum length and pattern of names by @snkas in #2331
• [SQL] Fix lag bug again by @mihaibudiu in #2349
• Update real-time clock periodically by @ryzhyk in #2336
• [SQL] Aggregates should use checked arithmetic by @mihaibudiu in #2352
• Support BINARY, VARBINARY in datagen. by @gz in #2355
• [Datagen] time, date, timestamp range support with strings. by @gz in #2357

Full Changelog: v0.24.0...v0.25.0

What's Changed

• Add information about source code position to table properties. by @mihaibudiu in #2191
• [SQL] Optimize programs based on foreign-key joins by @mihaibudiu in #2205
• Point to tutorial instead of demo by @ryzhyk in #2209
• Update stale text in the tutorial. by @ryzhyk in #2210
• docker: point compose file to Feldera version 0.23.0 by @lalithsuresh in #2203
• bench: update tiktok benchmark by @abhizer in #2141
• dbsp: Reduce memory use for merges with output smaller than the inputs. by @blp in #2156
• Benchmark by @blp in #2206
• Benchmark by @blp in #2211
• Update the Hopsworks demo for the new API. by @ryzhyk in #2202
• Temporarily disable nexmark Q15 by @aehmttw in #2213
• benchmark: Make output from Feldera SQL runs appear without buffering. by @blp in #2215
• [SQL] waterlines need a delay by @mihaibudiu in #2217
• Increase timeouts in S3 tests by @ryzhyk in #2223
• [SQL] Support functions DATE_TRUNC, TIME_TRUNC, TIMESTAMP_TRUNC by @mihaibudiu in #2227
• [SQL] Nexmark: format_date cannot be gc-ed, use just DATE by @mihaibudiu in #2233
• Datagen bug-fixes by @gz in #2231
• [SQL] Improve key analysis for equijoins by @mihaibudiu in #2232
• [SQL] Specify and enforce the type of elements in arrays, values in m… by @mihaibudiu in #2218
• Improve UX of SQL and Rust compilation status by @Karakatiza666 in #2221
• Fix default change stream behavior not following list end by @Karakatiza666 in #2238
• [DBSP]: Change radix value for radix trees to 2. by @ryzhyk in #2225
• Multiple WebConsole fixes by @Karakatiza666 in #2239
• Minor improvements to profiles by @blp in #2241
• Disable q16 by @aehmttw in #2245
• [SQL] Restrict tumbling window to 'short' SQL intervals by @mihaibudiu in #2247
• [SQL] For the first step use the minimum possible waterline everywhere by @mihaibudiu in #2246
• [SQL] Infer monotonicity from comparisons of the shape col + constant >= expression by @mihaibudiu in #2248
• pipeline-manager: less verbose logging at lower level by @snkas in #2252
• Enable creating pipeline with from a preset demo or a custom program via link by @Karakatiza666 in #2255
• Add Nexmark connector by @blp in #2226
• build(deps): bump openssl from 0.10.64 to 0.10.66 by @dependabot in #2079
• dockerfile: copy nexmark crate for compiler by @snkas in #2263
• pipeline-manager: info log level for feldera crates and warn for others by @snkas in #2262
• [adapters] Speed up pausing/unpausing Kafka. by @ryzhyk in #2260
• benchmark: Fix bug in feldera-sql runner for non-Nexmark benchmarks. by @blp in #2264
• Script to build compiler with released/unreleased Calcite by @mihaibudiu in #2265
• [SQL] Give warnings about the 'materialized' property for views by @mihaibudiu in #2256
• benchmark: Make tiktok generation script use default MAX_EVENTS. by @blp in #2267
• Add option to collect circuit profile in benchmark by @aehmttw in #2224
• [SQL] Support for ASOF join by @mihaibudiu in #2214
• Fix monitoring command. by @gz in #2259
• [SQL] Add table functions to the SQL TOC by @mihaibudiu in #2269
• scripts: Pass --events to Feldera SQL benchmark. by @blp in #2270
• docker: install git so sql-compiler's build.sh runs by @lalithsuresh in #2272
• release: bump project version to 0.24.0 by @gz in #2273

Full Changelog: v0.23.0...v0.24.0

We've majorly simplified the REST API to only have Pipelines as entities. We've also massively simplified the Web Console around that concept (#2071). This makes it so that a pipeline is described with a single SQL file that embeds both the program and the connectors. Please check out any of the available examples in the demos/ folder to see how to use the new API.

What's Changed

• docker: point compose file to Feldera version 0.22.0 by @gz in #2139
• Fast(er) datagen by @gz in #2140
• Enable CPU profiler by default. by @ryzhyk in #2137
• Simplify datagen config by @gz in #2148
• py: allow specifying update format in SQLContext.input_json by @abhizer in #2144
• [SQL] Enable 3 more Nexmark queries using SQL user-defined functions by @mihaibudiu in #2152
• Update README.md by @lalithsuresh in #2150
• Update README.md by @lalithsuresh in #2157
• Fix CI error with bench-publish by @aehmttw in #2159
• Update README.md by @lalithsuresh in #2160
• Rework API by @snkas in #2071
• Add basic implementation of visualization of change streams by @Karakatiza666 in #2166
• papers: migrate papers/ folder to feldera/papers repo by @lalithsuresh in #2167
• repo: cleanup top-level folder by @lalithsuresh in #2170
• Reworked debezium postgres demo by @ryzhyk in #2168
• Add instructions to the Postgres Debezium demo. by @ryzhyk in #2172
• build: bring back Earthly caching by @lalithsuresh in #2173
• demo: Update Hopsworks demo for the new API. by @ryzhyk in #2175
• docs: Update tutorial text for WebConsole changes. by @ryzhyk in #2171
• WebConsole: fix eggress query doesn't work with auth disabled by @Karakatiza666 in #2176
• docker: use mold linker for compiler-server by @lalithsuresh in #2180
• docker: fix mold download link by @lalithsuresh in #2181
• web-console: do a pass over wording and status box by @lalithsuresh in #2182
• Improve change stream UI by @Karakatiza666 in #2178
• Pull demos from API on home page reload by @Karakatiza666 in #2183
• Improve demos layout by @Karakatiza666 in #2184
• pipeline-manager: demo title and list sorted by file path by @snkas in #2185
• Added memory tracking for Flink by @aehmttw in #2126
• dbsp_nexmark: Exit without hanging if a DBSP thread panics. by @blp in #2130
• web-console: simplify the examples page by @lalithsuresh in #2192
• Improve program status behavior by @Karakatiza666 in #2194
• Fix login with Google after logout by @Karakatiza666 in #2195
• py: refactor python api to use PipelineBuilder by @abhizer in #2177
• Revert "build: bring back Earthly caching" by @lalithsuresh in #2196
• Fix profile picture not displayed by @Karakatiza666 in #2197
• Add null check by @Karakatiza666 in #2199
• benchmark: Convert SQL benchmarks for Nexmark and Tiktok to new API. by @blp in #2198
• release: bump project version to 0.23.0 by @lalithsuresh in #2200

Full Changelog: v0.22.0...v0.23.0

What's Changed

• Fix aggregating metrics when multiple connectors per relation by @Karakatiza666 in #2089
• py: add SQLContext.input_json() to send json data by @abhizer in #2088
• [SQL] validate table properties syntax by @mihaibudiu in #2091
• demo: new faster data generator for Hopsworks Tiktok RecSys by @abhizer in #2085
• [SQL] Fix bugs related to SQL UDFs by @mihaibudiu in #2099
• [README] fix broken documentation links by @kshuta in #2109
• Adds a new datagen input adapter for tables. by @gz in #2107
• docs: fix getting started link in README by @lalithsuresh in #2114
• [SQL] FOREIGN KEY validation and metadata construction by @mihaibudiu in #2108
• Connector orchestration by @ryzhyk in #2113
• Debezium Postgres demo. by @ryzhyk in #2116
• [SQL] Document and test TIMESTAMPADD function; improve Calcite optimizer structure by @mihaibudiu in #2118
• Test aggregates by @rivudhk in #2103
• Optimize asynchronous merger by @blp in #2115
• Revert "Test aggregates (#2103)" by @rivudhk in #2123
• dbsp: Don't reserve excess capacity for vector merge results. by @blp in #2127
• Merger revert by @blp in #2135
• release: bump project version to 0.22.0 by @gz in #2138

New Contributors

• @kshuta made their first contribution in #2109
• @rivudhk made their first contribution in #2103

Full Changelog: v0.21.0...v0.22.0

What's Changed

• pipeline-manager: remove service probing by @snkas in #2002
• Add tracing for dbsp steps and general docs for monitoring feldera by @gz in #1999
• [SQL] Improved monotonicity analysis for filter operators by @mihaibudiu in #2006
• DBSP: Asof-join operator. by @ryzhyk in #1998
• Merge new WebConsole layout prototype by @Karakatiza666 in #1994
• Automate Flink testing in CI by @aehmttw in #2005
• dbsp: test using window to implement SQL now. by @ryzhyk in #2015
• demo: Hopsworks Tiktok RecSys pipeline by @abhizer in #1992
• [SQL] Fix bugs in implementations of LIMIT by @mihaibudiu in #2020
• Add docs for the new WebConsole project by @Karakatiza666 in #2021
• Disable sveltekit build by @Karakatiza666 in #2024
• Re-enable and fix sveltekit build by @Karakatiza666 in #2025
• Disable coloring in maven builds by @mihaibudiu in #2026
• [SQL] Fix bug for primary key fields that need cloning by @mihaibudiu in #2029
• Update README.md by @gz in #2035
• [SQL] Preliminary implementation of the now() function by @mihaibudiu in #2019
• WebConsole: Fix pipeline relation metrics not aggregated … by @Karakatiza666 in #2036
• [SQL] Fix bugs in window bound computations for INTERVAL types by @mihaibudiu in #2034
• [SQL] Fix handling of monotonicity for tables with primary keys by @mihaibudiu in #2041
• Make sure dbsp and adapters metrics verison match. by @gz in #2044
• Update grafana_dashboard.json with some storage metrics by @gz in #2046
• [SQL] Correctly handle nullable types in ARG_MAX by @mihaibudiu in #2047
• [SQL] Use waterline and not waterline_monotonic by @mihaibudiu in #2048
• Fix bun build in deploy/Dockerfile by @Karakatiza666 in #2050
• Generalize SQL benchmarks to make it easier to add more by @aehmttw in #2013
• Reset min-storage-bytes to default by @aehmttw in #2053
• [SQL] Have the compiler synthesize the 'NOW' table by @mihaibudiu in #2049
• [SQL] Improved monotonicity analysis for joins followed by filters by @mihaibudiu in #2014
• Make Flink run all queries by @aehmttw in #2054
• TikTok demo: Fixes and refactoring. by @ryzhyk in #2055
• dbsp_adapters: Wait a bit before disconnecting on pause in url adapter. by @blp in #2057
• Improve profiling output to make storage performance easier to understand by @blp in #2059
• nexmark: Use same metrics version as the rest of the crates. by @blp in #2061
• dbsp: Name background threads so system tools can distinguish them. by @blp in #2062
• Update program.sql by @gz in #2064
• Fix port in documentation. by @gz in #2065
• [SQL] Two optimizations by @mihaibudiu in #2068
• Storage optimizations by @blp in #2066
• Revert "WebConsole: Fix pipeline relation metrics not accumulated when there are multiple connectors per relation" by @lalithsuresh in #2069
• Benchmark with 100M events by @aehmttw in #2077
• Add metric for records dropped due to lateness by @mihaibudiu in #2074
• Read input connector names from SQL by @aehmttw in #2076

Full Changelog: v0.20.0...v0.21.0

What's Changed

• Post release v0.19.0 by @gz in #1944
• Improve py docs by @abhizer in #1942
• dbsp: Fix build of galen benchmark. by @blp in #1928
• py: max_queue_records param to connect_source_kafka by @ryzhyk in #1948
• Improved monotonicity analysis by @mihaibudiu in #1949
• Switch binary download port from 9090 to 8085. by @gz in #1951
• pipeline-manager: SQL MAP type by @snkas in #1941
• benchmark: API key and Kafka option arguments by @snkas in #1954
• [SQL] Do not sort collection after TOP K unless it is needed at the output by @mihaibudiu in #1958
• Fix for numpy regression. by @ryzhyk in #1961
• Update earthly version/install. by @gz in #1960
• build(deps): bump scikit-learn from 1.2.1 to 1.5.0 in /demo/demo_notebooks by @dependabot in #1888
• [SQL] Reorganize compilation stages and fix 3 bugs by @mihaibudiu in #1966
• [SQL] support MATERIALIZED table and views by @mihaibudiu in #1959
• [SQL] Improve monotonicity analysis by @mihaibudiu in #1970
• Do not automatically distinct ouputs. by @ryzhyk in #1972
• Switch from log to tracing. by @gz in #1976
• [SQL] Improve monotonicity for aggregations by @mihaibudiu in #1974
• ci: run python tests by @abhizer in #1873
• Nexmark sql output by @blp in #1979
• Minor profiling-related improvements by @ryzhyk in #1971
• WebConsole: Drop all Data Services functionality by @Karakatiza666 in #1978
• Post-fix for unsuccessful rebase in PR1971. by @ryzhyk in #1981
• Expose dbsp metrics in pipeline metrics endpoint. by @gz in #1969
• Fix docker command by @aehmttw in #1984
• Use a single tokio runtime instead of n. by @gz in #1983
• docs: feldera enterprise by @snkas in #1991
• docs: remove deploy from sidebar and fix secret management links by @snkas in #1993
• Improve Kafka input performance for small messages by @blp in #1967
• docs: eks cluster storage class check by @snkas in #1996
• dbsp: Change threshold for spilling to storage from rows to bytes. by @blp in #1917
• Make profiles downloadable via API by @blp in #1988
• benchmark: Make poller threads configurable for SQL benchmarks. by @blp in #1997
• Enable playwright tests by @Karakatiza666 in #1952
• Add script to run flink on CI by @aehmttw in #1985
• [SQL] a few small optimizations: combine join/filter/map and map/mapindex by @mihaibudiu in #1986
• Disable navigation from materialized to non-materialized view by @Karakatiza666 in #2001
• release: bump project version to 0.20.0 by @gz in #2007

Full Changelog: v0.19.0...v0.20.0

What's Changed

• [SQL] Support for SQL ARRAY_AGG aggregate function by @mihaibudiu in #1862
• SQL benchmarks graph by @aehmttw in #1857
• [SQL] Do not attempt to simplify expressions that will cause panics a… by @mihaibudiu in #1869
• [SQL] Hopping and tumbling windows using table functions by @mihaibudiu in #1855
• Per-thread Storage cache. by @gz in #1853
• avro: make serializing schema ID configurable by @abhizer in #1871
• [SQL] Move simulator to separate project by @mihaibudiu in #1877
• [SQL] Fix bug in dead code elimination by @mihaibudiu in #1879
• Upgrade delta-rs to 0.18. by @ryzhyk in #1878
• [SQL] Second attempt to fix build by @mihaibudiu in #1880
• join_flatmap by @ryzhyk in #1881
• benchmark: Handle case where Kafka requires different access `host:po… by @blp in #1876
• [SQL] Use join_flatmap when possible by @mihaibudiu in #1882
• Log SQL metrics by @aehmttw in #1875
• pipeline-manager: reduce logging in integration tests by @snkas in #1885
• [SQL] Support hopping windows for nullable timestamp columns by @mihaibudiu in #1887
• [SQL] Monotonicity analysis for hopping windows by @mihaibudiu in #1889
• Run rust tests separately by @aehmttw in #1893
• [SQL] Do not assume that user-defined types are structs by @mihaibudiu in #1897
• [SQL] Introduce syntax to specify connector properties in SQL by @mihaibudiu in #1896
• Avro: Serialize non-nullable fields as nullable. by @ryzhyk in #1892
• Add a dialog to configure pipeline's runtime resources by @Karakatiza666 in #1027
• Add health popup containing all system errors to the header by @Karakatiza666 in #1856
• [SQL] Remove unsupported SQL construct (FILTER and OVER) from documentation by @mihaibudiu in #1900
• py: SQLContext.register_type by @ryzhyk in #1899
• pipeline-manager: demos supplied via local directory by @snkas in #1890
• py: add register_local_view by @abhizer in #1884
• [SQL] Fix bug in CSE pass by @mihaibudiu in #1903
• SQL storage metrics by @aehmttw in #1886
• Adds a new spine that does compaction in a thread. by @gz in #1863
• [SQL] Prevent SQRT(negative value) in STDDEV due to unstable FP computations by @mihaibudiu in #1908
• py: add_lateness. by @ryzhyk in #1910
• [SQL] Give warnings for LATENESS statements without corresponding views by @mihaibudiu in #1909
• [SQL] Preliminary support for MAP types by @mihaibudiu in #1905
• dbsp: Fix hidden dependency on vec-based batches in input_upsert. by @blp in #1913
• Avoid head-of-line blocking in async spine. by @gz in #1912
• Log more SQL metrics by @aehmttw in #1915
• Convert all DBSP batches to possibly use storage by @blp in #1867
• Sampled histograms by @blp in #1916
• dbsp_adapters: Fix typo in comment. by @blp in #1918
• Handle negative fuel in merger.rs. by @ryzhyk in #1923
• Integration test logging by @ryzhyk in #1921
• Switch from crossbeam-channel to std::sync::mpsc::channel. by @gz in #1924
• Enable storage in testing by @gz in #1927
• py: implement SQLContext.wait_for_completion by @abhizer in #1872
• [SQL] proper unnest check for null arguments by @mihaibudiu in #1930
• docs: simplify the fraud detection preamble by @lalithsuresh in #1929
• Bump delta by @ryzhyk in #1934
• py: add hopsworks integration demo by @abhizer in #1901
• Fix UI regressions by @Karakatiza666 in #1940
• Release v0.19.0 by @gz in #1943

New Contributors

• @aehmttw made their first contribution in #1857

Full Changelog: v0.18.0...v0.19.0

Added

• [SQL] CREATE LOCAL VIEW statement to declare views which are not outputs. (#1851)
• Documentation: added the fraud detection use case to the docs section of the website.
• API: storage class can be configured via pipeline resources (#1829)
• [Python] Added support for Kafka connector via Python SDK (#1807)
• [Python] Added support for HTTP GET connector via Python SDK
• [Python] Add support for Avro format (#1836)
• [Python] allow specifying pipeline resource configuration and compilation profile via SQLContext (#1835)
• [Python] Add function to wait for the pipeline to become idle (#1841)
• WebConsole: DeltaLake input (#1744) and output (#1584) conenctors
• WebConsole: Add output buffer configuration UI (#1591)

Fixed

• WebConsole: Fix crash when displaying anonymous output columns (#1757)