This is a pre-release to test new AWS SDK version #1799

What's Changed

• sqlserver: special url for proxy health-checks by @mialinx in #1709
• try to switch to docker compose v2 by @chipitsine in #1765
• add universal backup tool by @danilkaz in #1689
• aof datadir clean fix by @kronwerk in #1767
• MySQL: Add xbstream unarchiver to wal-g by @ostinru in #1744
• Update README.md by @reshke in #1772
• Fixed working without metadata by @debebantur in #1773
• clean only specified files and folder from data dir by @kronwerk in #1776
• fix checkout on Windows by @chipitsine in #1777
• delete single redis backup by @Believemenot in #1786
• 1. add backup-info cmd for redis; 2) some refactor by @kronwerk in #1787
• Test postgres/generic_meta_interactor.go/SetIsPermanent by @pevel-dev in #1788
• add redis used_memory and used_memory_rss to sentinel on backup creation by @kronwerk in #1790
• greenplum: separate restore points fetch & find operations - so we can reuse fetched metadata. by @ostinru in #1793
• Correction for checksum verification PostgreSQL by @boosterKRD in #1779
• Bumped aws sdk version by @ItielOlenick in #1799
• sort detailed backup list too by @kronwerk in #1803
• Bump golangci/golangci-lint-action from 6.0.1 to 6.1.1 by @dependabot in #1801
• fix: fix slice init length by @cuishuang in #1800
• Cloudberry DB initial support by @ostinru in #1796
• Add makezero and asciicheck linters by @ostinru in #1812
• Continue Greenplum garbage delete when no segment data found by @ostinru in #1814
• Remove unexpected '\r\n' sequence at the end of multiline commands in the config by @noname0443 in #1817
• Greenplum 7 support by @ostinru in #1818
• clean data on rdb restore also by @kronwerk in #1819
• Add JournalSize to backup sentinels by @noname0443 in #1806
• Fixes for partial restore by @debebantur in #1816
• Update go version to resolve CVE-2024-34156 by @annielzy in #1820
• add whitelist for mongodb datadir deletion protection by @NotAmigo in #1808

New Contributors

• @chipitsine made their first contribution in #1765
• @Believemenot made their first contribution in #1786
• @pevel-dev made their first contribution in #1788
• @boosterKRD made their first contribution in #1779
• @ItielOlenick made their first contribution in #1799
• @cuishuang made their first contribution in #1800
• @annielzy made their first contribution in #1820

Full Changelog: v3.0.3...v3.0.4

Major feature of this release is full support for OrioleDB.
WAL-G supported block-level incremental backups since v0.1.3, but it previously treated OrioleDB data as a collection of unknown files. Now WAL-G understands if OrioleDB is installed into cluster and makes efficient backup copies of OrioleDB data. Thanks to @homper and other Supabase engineers.

Additionally, this release includes two new Postgres commands catchup-send and catchup-receive. These commands are useful when you need to bring a lagging replica up to date without pushing a new backup to the storage. In essence, they work like pg_rewind but in reverse. Perhaps we should rename them to pg_wind. Thanks to @vbp1 for drawing my attention to the problem of current catchup implementation and architecting new design.

We've also added aof-based backups for redis (@kronwerk).

This release also mitigates several CVEs in dependencies (#1675, #1715) and fixes assorted bugs. Thanks to all the WAL-G community :)

What's Changed

• Allow users to expose debug & pprof endpoints for wal-g-gp by @ostinru in #1748
• Delete greenplum backups with higher concurrency by @ostinru in #1750
• Upgrade golang.org/x/net to resolve CVE-2023-45288 by @jorsol in #1737
• redis aof backup and fetch, basic integration tests by @kronwerk in #1749
• Added missing test to parallel by @debebantur in #1730
• Add extendBackupCursor MongoDB flag to make extension optional by @NotAmigo in #1752
• Update CODEOWNERS by @VGoshev in #1753
• 1. change misleading class name; 2. add gomt check. by @kronwerk in #1754
• up golang version to 1.21 by @kronwerk in #1755
• [orioledb] Using latest orioledb in tests and new pageHeader field order by @homper in #1758
• Reduce log clutter from storages by @x4m in #1760
• behave tests for redis aof backup by @kronwerk in #1759
• clean all redis data before aof backup, not only aof folder by @kronwerk in #1763
• fix LoadCustomCABundleError when setting WALG_S3_CA_CERT_FILE by @berler in #1764

New Contributors

• @jorsol made their first contribution in #1737
• @berler made their first contribution in #1764

Full Changelog: v3.0.2...v3.0.3

This pre-release is intended to preview OrioleDB support.

Also it fixes some bugs, notably #1736: reading calls to S3 could be really slow due to unintended bucket listings. The bug seems to be introduced in 3.0.1 pre-release.

What's Changed

• Fixes for working with s3 by @debebantur in #1718
• add test cases for tryFindHighestTimelineID by @annetannet in #1720
• Migrate to docker compose v2 by @debebantur in #1727
• сhange redis installation in tests from pkg to repo by @kronwerk in #1732
• Create MockObject for storage.Object and add backup tests with gomock by @alexnik10 in #1731
• wal-g hangs when s3 access_id or access_secret_key contains line break by @mialinx in #1733
• Add tests for FindTargetByName by @RedSquid2212 in #1729
• Refactor mongodb and redis func tests docker images by @zykovd in #1735
• Fix link_libsodium.sh by @garry-t in #1739
• Fix "backup-push Connection cancelled by remote host" error on Windows by @rail-rain in #1745
• Small error cleanup in catchkup-receive by @x4m in #1701
• Orioledb basic support by @homper in #1698
• Disable storage validation by default by @x4m in #1747
• [mongodb] add opportunity set timeout for recovery oplog by @Apelsin234 in #1746

New Contributors

• @annetannet made their first contribution in #1720
• @kronwerk made their first contribution in #1732
• @alexnik10 made their first contribution in #1731
• @RedSquid2212 made their first contribution in #1729
• @zykovd made their first contribution in #1735
• @garry-t made their first contribution in #1739
• @rail-rain made their first contribution in #1745

Full Changelog: v3.0.1...v3.0.2

This is a technical pre-release aimed at mitigating several CVEs in dependencies (#1675, #1715). These vulenrabilities have no known ways to be exploited against WAL-G. But these days we should keep CVE scanners calm.
Also this pre-release gives access to several features that didn't make it into 3.0.0 because were not tested enough (#1662, #1646, #1652).
As usual, handful of bugs fixed.

Thanks to everyone who contributed!

What's Changed

• Support DigitalOcean Spaces (NoSuchKey during ListFolder) by @turt2live in #1646
• Skip archive_command check on standby by @x4m in #1662
• fixes for ao check config by @debebantur in #1664
• Fix for GP incremental backups by @debebantur in #1666
• Add unit tests for internal.databases.greenplum.buildFetchCommand by @denisSurkov in #1661
• Catchup-send and catchup-receive commands by @x4m in #1652
• Collecting statistics about read/written bytes by @debebantur in #1670
• Add Docs for undocumented debug env variables by @sebastiangrill in #1640
• sanikadze:doc_gp_gp_pitr by @Sanikadze in #1672
• Implement lockFile for windows by @passcod in #1673
• upgrade golang.org/x/net to resolve CVE-2023-39325 by @devinturner in #1675
• Fix WALG_STATSD_EXTRA_TAGS by @nick-kentik in #1676
• Fix gp tests by @debebantur in #1677
• add path to recovery.conf for greenplum by @avkashin in #1679
• Clarified dev installing instruction on Ubuntu for go 1.16+ by @homper in #1681
• added IsTablespaceMapExists test by @NiripsaKakVsegda in #1680
• split mongodb restore on stages under flag by @Apelsin234 in #1685
• add support to redis acl user by @NotAmigo in #1682
• Add glob pattern support in storagetools by @ReshetnikovPavel in #1684
• add PGSSLCERT, PGSSLKEY, PGSSLROOTCERT support by @NiripsaKakVsegda in #1687
• WALG-1627. MySQL fix file double Close() on file by @ostinru in #1628
• add create upload input unit tests by @danilkaz in #1690
• add tests for SetUserData in postgres/generic_meta_interactor by @SixBlueBlack in #1697
• Fix lll lint violation, update golangci version by @serprex in #1686
• Exclude standby.signal by @munakoiso in #1702
• WALG-1665. User proper DSNs in MySQL documentation & validate DSN format it earlier by @ostinru in #1692
• unit test for internal/backup.go/FetchMetadata by @ValeriaIM in #1695
• Bump golangci/golangci-lint-action from 4.0.0 to 5.3.0 by @dependabot in #1703
• Save isIncremented attribute by @debebantur in #1704
• Add charset converter to convert non utf-8 char by @anisurrahman75 in #1705
• Bump golangci/golangci-lint-action from 5.3.0 to 6.0.1 by @dependabot in #1706
• WALG-1683. Save CPU arch / OS to mysql sentinel. by @ostinru in #1693
• added without metadata to gp by @debebantur in #1708
• Tests for isPagedFile by @Themplarer in #1710
• Added validation for S3 credentials by @Themplarer in #1711
• fixed hex parsing in timeline by @debebantur in #1712
• Added retention period for s3 by @debebantur in #1713
• Add few unit test on azure ConfigureStorage by @Loraan in #1714
• Add Uploader interface mock, test on UploadSentinel and make mock command by @Loraan in #1716
• Update modules in order to resolve CVEs. by @montaguethomas in #1715

New Contributors

• @turt2live made their first contribution in #1646
• @sebastiangrill made their first contribution in #1640
• @Sanikadze made their first contribution in #1672
• @passcod made their first contribution in #1673
• @devinturner made their first contribution in #1675
• @avkashin made their first contribution in #1679
• @homper made their first contribution in #1681
• @NiripsaKakVsegda made their first contribution in #1680
• @NotAmigo made their first contribution in #1682
• @danilkaz made their first contribution in #1690
• @SixBlueBlack made their first contribution in #1697
• @ValeriaIM made their first contribution in #1695
• @anisurrahman75 made their first contribution in #1705
• @Themplarer made their first contribution in #1710
• @Loraan made their first contribution in #1714
• @montaguethomas made their first contribution in #1715

Full Changelog: v3.0.0...v3.0.1

Hello everyone!

This is a much-anticipated release. In the future, we will strive to avoid accumulating such a large number of changes. We will code frequently, ship frequently, and release frequently!

Noteworthy Changes

• Every database engine now benefits from failover storage. This technique can save the day if one of the storage systems goes down. However, do not forget to monitor them anyway.
• For Postgres, we have a WAL archiving daemon available. Its primary purpose is to improve the efficiency of parallel archiving. Additionally, WAL-G supports the partial restoration of a cluster. If you have large, unnecessary tables, you can exclude them from the restoration process, the Postgres will skip over the WAL records for these tables.
• Greenplum's PITR (Point-in-Time Recovery) is undergoing its maturation process. Partial restore is supported in a similar way to Postgres. We now have tools to verify the integrity of AO\AOCS files in storage without actually restoring backups. This technique is necessary to ensure developers can sleep well, as Greenplum backups are relatively new, and extra precautions need to be taken to ensure they are safe and reliable.
• MongoDB backups can now benefit from binary backups, which significantly reduce the impact of backups on a running workload.
• MySQL now supports incremental backups, also known as delta backups. However, this feature is still in beta mode and the API may change in the future.
• Also, we now have beta support for etcd. Changes necessary for real PITR have not yet been merged into etcd, and can be found at etcd-io/etcd#17233.

Detailed changes log

• Correct the --without-files-metadata docs by @usernamedt in #1327
• expose zstd compression by @heinerstilz in #1326
• [WIP] Multithread downloading directory by @hariton27sy in #1297
• Add WALG_GP_AOSEG_SIZE_THRESHOLD setting by @usernamedt in #1329
• Respect WALG_NETWORK_RATE_LIMIT globally by @usernamedt in #1330
• Restrict rate limiting only to backup-push/backup-fetch commands by @usernamedt in #1331
• Initial mvp mongodb binary backup by @KhurtinDN in #1285
• Add delta backups for Greenplum by @usernamedt in #1333
• Fix sorting in detailed mongodb backup list by @KhurtinDN in #1334
• Add @VGoshev and @KhurtinDN as codeowners of MongoDB part of Wal-G by @VGoshev in #1335
• Add tests, docs and minor fixes for Greenplum delta backups by @usernamedt in #1337
• Fix mongodb binary backup during mongod do checkpoint by @KhurtinDN in #1342
• Add rate limiter for mongodb by @KhurtinDN in #1343
• Add data catalog size for PostgreSQL/Greenplum backups by @usernamedt in #1344
• Add minor improvements for Greenplum by @usernamedt in #1345
• postgresq.md add cmake installation by @rogaliiik in #1348
• Support ARM64 builds on MacOS by @usernamedt in #1349
• Implement delete target backup for Greenplum by @usernamedt in #1350
• Add delete garbage command for Greenplum by @usernamedt in #1351
• Add limiters to Greenplum segment backup-push/backup-fetch by @usernamedt in #1352
• Fix failing docs builds by @usernamedt in #1354
• Fix distro name for ubuntu-20.04 in arm release by @williancolognesitrimble in #1355
• Bump golangci/golangci-lint-action from 3.2.0 to 3.3.0 by @dependabot in #1361
• Add minimal mongod config path option by @KhurtinDN in #1360
• Fix closing mongodb backup cursor by @KhurtinDN in #1359
• Replace mongodb binary backup to tar archive format by @KhurtinDN in #1357
• Fix binary backup of mongodb 4.2 by @KhurtinDN in #1364
• Add daemon mode for wal-push command by @rogaliiik in #1353
• Change error to warning when creating symlink, that already exists by @IncubusRK in #1320
• [Greenplum] do not fail backup on increment reading error by @usernamedt in #1365
• Only dump relevant env variables with LOG_LEVEL=DEVEL by @LeGEC in #1367
• Do not print secrets in debug logs by @LeGEC in #1368
• Bump golangci/golangci-lint-action from 3.3.0 to 3.3.1 by @dependabot in #1380
• enable using aws iam roles by @Qwiz in #1377
• notify_d watchdog for daemon service by @rogaliiik in #1371
• Use pure go implementation for zstd by @LeGEC in #1376
• Restore debug logging improvements by @usernamedt in #1389
• postgresql: fix data path mismatch on local backups by @LeGEC in #1383
• Fix Greenplum deleted AO segment file error handling by @usernamedt in #1391
• Implement max file size for mysql by @bularond in #1374
• Fix non-string config settings handling by @usernamedt in #1393
• Makefile: change compilation and linking of libbrotli to optional (and off by default) by @LeGEC in #1379
• add unit tests for the WriteBackupListDetails function by @tewboat in #1395
• Add recovery_target_timeline setting to Greenplum restore config by @usernamedt in #1398
• Add detailed json backup-list for Greenplum by @usernamedt in #1399
• Add restore-point-list command for gp by @rogaliiik in #1397
• Update golangci-lint version to 1.50.1 by @serprex in #1381
• Add restore-point-ts flag for Greenplum backup-fetch by @usernamedt in #1403
• Greenplum: populate pg_hba.conf on master host during restore by @usernamedt in #1404
• enable linters for cmd/redis internal/databases/redis and pkg/storages by @teewar2 in #1407
• Fix wal-g-mongo mongod restore procedure by @KhurtinDN in #1414
• Add mongo-rs-members parameter to binary mongo restore by @KhurtinDN in #1418
• [MongoDB] add opportunity restore mongodb backup with + 1 version by @Apelsin234 in #1417
• Add mongo-rs-name and mongo-rs-members parameters to binary mongo restore by @KhurtinDN in #1419
• add unit test for greenplum/backup_fetch_handler by @teewar2 in #1406
• fix bug with mixed up arguments by @Apelsin234 in #1421
• Postgres requires data dir to be 0700 or 0750. by @Aevin1387 in #1420
• Add support mongodb 6.0 by @KhurtinDN in #1416
• Fix connect to mongodb during restore procedure by @KhurtinDN in #1428
• Bump golangci/golangci-lint-action from 3.3.1 to 3.4.0 by @dependabot in #1426
• MDB-21194: [MongoDB] fix bug with purge garbage and change logic for retains by @Apelsin234 in #1429
• Implementation of the new binlog-server command by @Fizic in #1408
• Bump github.com/prometheus/client_golang from 1.7.1 to 1.11.1 in /internal/tools by @dependabot in #1433
• Bump golang.org/x/text from 0.3.6 to 0.3.8 in /internal/tools by @dependabot in #1437
• Implement retry download backup by @bularond in #1394
• adding push metrics as a defer in the upload method to make sure metrics are pushed even on failure by @francoispqt in #1423
• Revert #1423 by @usernamedt in #1442
• Implement database composer and partial fetch for Postgres by @Catmoonlight in #1434
• [MongoDB] fix retryable error from code to name by @Apelsin234 in #1444
• Bump actions/setup-go from 3 to 4 by @dependabot in #1446
• Adaptive timeout for mongodb restore by @Apelsin234 in #1449
• [Greenplum] Store metadata about backup restore points by @usernamedt in #1448
• Add Postgres partial restore by database names by @Catmoonlight in #1447
• Add tests postgres sentinel dto is incremental by @AlexUran in #1455
• Add tests for NewDtoSerializer by @denisSurkov in #1460
• feat: removing binlogs after using them by @Fizic in #1457
• Update golangci-lint in .github/workflows to 1.52.2 by @serprex in ...

This is a pre-release to test the aws iam roles config setting.

Release notes:
This release mainly consists of various bug fixes and improvements.

Common:

• Add arm(aarch64) build pipeline for ubuntu20.04 binaries (#1304)

PostgreSQL:

• Add warning output on backup-fetch retries (#1325)
• Remove redundant check for --without-files-metadata flag (#1321)
• PostgreSQL 15 Support (#1319)
• Fix of deleting WAL-E backups (#1292)

MySQL/MariaDB:

• Fix a regression in MariaDB backup-push handler (#1313)
• Binlog cache fixes (#1281)

Greenplum:

• Optimize delete handler (#1300, #1302)
• Optimize storage listing queries during the backup push (#1312)
• Use AO files metadata instead of backup references (#1308)
• Refactor AO/AOCS storage metadata queries (#1280, #1278)

Storages:

• Azure authentication using the default credential chain (#1282)
• Update aws-sdk-go to v1.44.7 (#1262)
• Add check read/write to storage tools (#1275)

Release notes:

This release contains a huge amount of new features, bug fixes, and performance improvements. It is worth noting that v2.0 Postgres delta backups are not backward-compatible with the pre-2.0 versions. It means that the v2.0 delta backup can't be restored with the v1.1 or earlier versions. Also, this release introduces the initial support for Greenplum physical backups.

WAL-G for MySQL now supports the parallel backup-push/backup-fetch and has an optimized binlog upload mechanism. WAL-G for SQLServer now supports the backup encryption and compression as well as the external storage backup import/export.

Common:

• Update lz4, brotli #1165, #1179
• Multiple libsodium fixes #1087, #1227
• Bash/zsh shell completions #1250, #1252
• Add a 'flags' subcommand #1033
• Disable CPU Limits in turbo mode #1121
• Metrics forwarding via statsd protocol #1240

SQLServer:

• Add SQLServerDBConcurrency setting to limit concurrent database operations #1067
• Add LATEST keyword handling #1110
• Add norecovery option for backup log command #1115
• Add SQLServer compression / encryption #1161
• Backup import and export to the external storage #1184

Postgres:

• Add CopyComposer for full backups optimization using remote copy API #1012
• Add streamed JSON sentinel serialization to decrease memory usage #1103
• Extract heavy files and tarfilesets from the backup sentinel #1114
• Introduce WALG_TAR_DISABLE_FSYNC to skip fsync during backup-fetch #1144
• Add reduced memory usage backup mode #1101
• Improvements and fixes for remote backup #1175
• Add pgBackRest backups restore support (beta) #1139, #1202
• Add 'wal-restore' command #977
• Add 'delete garbage' command #1217, #1218
• Return custom error code when WAL file does not exist #1195
• Add WALG_ALIVE_CHECK_INTERVAL to fail backup on Postgres process failure #1260
• Reformat LSN output #1259

MySQL:

• Add wal-g binlog-analyze #1085
• Check binlog GTIDs before pushing binlogs to S3 #1190
• Add --check-gtids to wal-g binlog-push #1085
• Add --until-binlog-last-modified-time option to wal-g-mysql binlog-replay #1154
• Improve MySQL restore performance #1131
• Allow backups on cascade replicas #1168
• Fix mysql backup of a lagging replica #1223
• Make mysql bilong-push status observable to 3rd party tools #1229

Greenplum:

• Initial physical backups support for GPDB 6.19.2+. API might change in the future versions.

Storages:

• Add get/put/rm/cat to storage tools #1069, #1164
• S3: Add download by range #1083
• S3: Add connection reset retries #1137
• S3: Add WALG_S3_MAX_RETRIES setting #1148
• S3: Fix S3_LOG_LEVEL setting #1158
• S3: AWS authentication fixes #1209
• S3: Yandex Cloud session token auth support #1213
• SSH: Ensure trailing delimiter in the storage prefix #1130
• Azure: Switch to the recommended azblob SDK #1135, #1171
• Azure: Fix ListFolder never returning error #1208
• Swift: Ignore files that were not found during ListFolder #1129

This is a pre-release to test S3 authentication with mechanisms other than key/secret.

This pre-release introduces the Greenplum physical backups support.