Releases: open-gpdb/gpdb
6.28.17
- Add lineage tracking to pgaudit
- New temp_tables_stat extension
- New hooks for better command center support* Fix tablespaces support in Yezzey
- Fix metadata index corruption in Yezzey
- Fix wrong query results with nextval on replicated tables
6.29.0
What's new in 6.29.0
Features
Added support of fast analyze for AO tables [backported from GP7]
It uses a new sampling algorithm for AO tables speeding up ANALYZE performance. Benchmarks show the average performance improvement of about two times. To use the new algorithm a new GUC gp_use_fastanalyze should be on.
Added lineage support to pgaudit
- New modes for
pgaudit.logGUC:ast_ctas,ast_sel,ast_modto enable logging ofCREATE TABLE AS SELECT,SELECT,INSERT,UPDATEandDELETE. One use case of logging AST would be to find out which columns in source tables were used to fill the target table. - Some fixes in places where GP wasn't fully compatible with pgaudit
skip_locked option for analyze
Usages: analyze skip_locked mytable; , analyze skip_locked rootpartition mytable;, analyze skip_locked, This option skips a table if it is already blocked. Allows for considerable regular maintenance speed-ups.
Depends on gp_aux_catalog extension of v1.1.
Prioritize least maintained tables in analyzedb
The analyzedb algorithm has been modified to prioritize tables that hasn't been analyzed lately. This helps to keep statistics up-to-date even when maintenance windows is too short to analyze the whole database.
Support diverged timelines in gprecoverseg
In a case when mirror has been promoted but the old primary keeps writing WAL segments before shutting down, we get diverged timelines. If the new mirror archives the necessary for pg_rewind WAL segments then incremental recovery is impossible and full recovery is required.
6.29.0 adds restore_command_hint GUC and backports --restore-target-wal argument for pg_rewind. Together these features allow pg_rewind to pull WAL segments from archive and do incremental recovery.
Proper memory accounting for zstd
Added a new memory context zstd_context so that zstd memory is accounted for by resource groups or resource queues. This prevents server crashes on OOM errors in zstd code.
To enable this feature gp_enable_zstd_memory_accounting should be turned on.
FTS maintenance mode
The FTS maintenance mode allows to completely turn off FTS as a temporary measure to mitigate global failures: for example, in case of network partitioning or when a data center goes down. Can be used either before failure or when recovering from it.
gp_fts_maintenance turns this mode ON (requires reload, gpstop -u). When this is ON, FTS stops switching segments and updating gp_segment_configuration. Meaning that:
- on
mirrorfailure we get a read-only cluster while themirroris down or maintenance turned off; - on
primaryfailure we get unavailable cluster while theprimaryis down or maintenance turned off; - on both
primaryandmirrorfailure we getdouble fault, but to recover from it it's enough to just get those segments running. Important restarting a cluster withdouble faultwill fails until at least primary is up and reachable.
The usefulness of this mode is in quick recovery after a disaster: we don't need to run gprecoverseg or manually editinggp_segment_configurationbecause it doesn't change in maintenance mode. When the problem is gone the cluster should be working and maintenance lifted.
ON MASTER syntax support for external file:// tables
Allows to CREATE EXTERNAL TABLE ... location ('file://...) ON MASTER... . Before it only worked on segments. A useful use-case example is reading logs with external file tables instead of unsafe external web tables.
Faster server startup when shared buffers is big [backport from GP7]
A small but important fix speeding up memory allocation for shared buffers on GP startup. Even with small buffers the difference is measurable. For example, with a 32GB nodes, 1 segment per host, 2 segment hosts:
| shared_buffers-> | 128MB | 1GB | 10GB |
|---|---|---|---|
| 6.29 start time | 6.304s | 6.614s | 6.792s |
| 6.28 start time | 6.867s | 10.788s | 43.011s |
Hence, for a bigger cluster with 200GB for shared_buffers and 200 segments we expect to lower startup times from tens of minutes to tens of seconds.
New hooks for better GPCC support
Allows to collect more performance metrics and to make those metrics more reliable.
- hook for interconnect metrics
- hook for EXPLAIN ANALYZE plan text
- hook to track utility queries
- missing
query_donehooks
Fixes
A critical bug in two-phase commit recovery
A failure to recover from an interrupted two-phase commit was leading to long downtimes and manual gp_segment_configuration editing and/or server restarts. The reason was a bug in prepared transaction recovery leading to failed startup of recovered segments.
Timeline collision preventing WAL archival
This happens when promoting a mirror. Greenplum picks the next timeline id which is already archived. This leads to archive collision and no WAL can be archived. The problem is solved by writing restore_command to recovery.conf upon recovery in pg_rewind and pg_basebackup. So that promote takes timeline id from remote archive and prevents conflicts.
Crash in TeardownTCPInterconnect
Fixes unhandled exceptions in TeardownTCPInterconnect that lead to stack overflow and crashes
Tablespaces support in yezzey
Before this fix yezzey_load_relation does not return the table to its original tablespace
Crash in ShareInputScan with MergeAppend
This happens in queries that union multiple plans and these plans have a ShareInputScan. For example, UNION ALL with a common CTE.
Crash on adding a new partition with DDL triggers
For example, this leads to crash before 6.29.0
create event trigger tr on ddl_command_end execute procedure tr_f();
create table tbl(r int) partition by range(r) (start(1) end(10) every(1));
alter table tbl add partition start(11) end(20);Wrong results for ITAS on replicated tables
leads to more rows being inserted. For example, before 6.29 the below query inserts as many rows as there are segments:
set optimizer=off;
create table dist_dst (x bigint) distributed by (x);
create table repl_src (y bigint) distributed replicated;
create sequence seq increment by 1 minvalue 1 maxvalue 42 start 1 no cycle;
insert into repl_src values (1);
insert into dist_dst (select nextval('seq') from repl_src);Что нового в версии 6.29
Основные доработки
Поддержка быстрого analyze для AO таблиц [бэкпорт из GP7]
Используется новый алгоритм сэмплинга AO таблиц, позволяющий увеличить скорость ANALYZE. Ряд бенчмарков показывает улучшение приблизительно в два раза.
Для использования нового алгоритма должен быть включен параметр конфигурации gp_use_fastanalyze
pgaudit улучшен для трекинга источников данных
- В список значений гука
pgaudit.logдобавленыast_ctas,ast_sel,ast_modдля включения логирования внутреннего представления запросов соотвественноCREATE TABLE AS SELECT,SELECT, а такжеINSERT,UPDATEиDELETE. Логирование дерева запроса может применяться, чтобы определять, из каких столбцов исходных таблиц была заполнена данными новая таблица. - Улучшена совместимость ГП с pgaudit
Поддержка skip locked в analyze
Добавлена опция skip_locked в движок СУБД. Примеры использования: analyze skip_locked mytable; , analyze skip_locked rootpartition mytable;, analyze skip_locked. Опция пропускает таблицу, если на ней уже кто-то удерживает блокировку. Позволяет значительно ускорить регулярное обслуживание БД.
Для использования необходимо расширение gp_aux_catalog версии 1.1.
Приоритезация старых таблиц в analyzedb
В сприпте analyzedb изменён алгоритм сортировки таблиц. Ранее таблицы сортировались по oid, теперь сортировка идёт в первую очередь по дате сбора статистики и только потом по oid. Это позволяет приоритезировать наименее обслуженные таблицы в выделенное окно обслуживания.
Улучшение инкрементального восстановления в gprecoverseg при расхождении таймлайна
В ситуации, когда mirror был запромоучен, а primary продолжил писать WALы перед завершением работы, то для восстановления старого primary требуется синхронизация разошедшихся WALов с применением pg_rewind. При наличии archive_command необходимые WAL-файлы с зеркала могут быть перемещены, например, в S3. В такой ситуации gprecoverseg из-за отсутствия нужных файлов не сможет произвести инкрементальное восстановление и необходима полная переналивка сегмента.
Данная доработка позволяет использовать новый гук restore_command_hint и бекпортированный в pg_rewind из PG --restore-target-wal чтобы подтянуть недостающие WAL файлы из архива (S3 в нашем случае), когда их нет на запромоученной реплике.
Учёт памяти zstd в ресурсных группах
Позволяет избегать выхода ZSTD за пределы памяти, доступной ресурсному менеджеру (группе или очереди), благодаря перемещению ZSTD в отдельный контекст памяти zstd_context. И, таким образом, снижает вероятность неконтролируемого падения кластера по нехватке памяти.
Включается через guc gp_enable_zstd_memory_accounting.
Maintenance режим для FTS
Добавлена возможность полного отключения FTS на кластере в качестве временной меры для митигации глобальных сбоев: отказ ДЦ, частичная или полная сетевая недоступность. Может быть использована как до отказа, так и при восстановлении из него.
Включается параметром конфигурации gp_fts_maintenance (необходим рестарт или релоад с gpstop -u). При включенном мейнтенанс FTS перестаёт переключать сегменты. Если в этом режиме у нас:
- отказывает
mirror: получаем read-only кластер до тех пор, пока mirror недоступен или мейнтенанс не снят (и fts починит кластер); - отказывает
primary: получаем кластер недоступный ни на запись, ни на чтение до тех пор, пока primary не будет восстановлен или пока мейнтенанс не будет снят и fts не починит кластер; - в случае отказа и
primary, иmirrorполучаем привычный `doub...
6.28.10
6.28.8
6.28.7
Bugs fixed
Fixed mirror promote failure in StartupXLOG + read_local_xlog_page, commit 8c06a0a
Fixed coredumps in MergeAppend node in share input mutator, commit c6f16f4
Fixed long startup for catalogs with huge pg_attribute table, commit eee15c5
Features provided
Enabled zstd memory resource group accounting, commit 4e500c4
Added completion for the DISTRIBUTED clause in CREATE TABLE, commit e200f86
Added pgaudit and sr_plan extensions build by default, commit bdd8559
Full Changelog: 6.28.3...6.28.7
6.28.3
What's Changed
- fixed a number of tests, stable tag for install on production environment
Full Changelog: 6.28.2...6.28.3
6.28.2
Основные изменения open-gpdb 6.27 -> 6.28
Множество фиксов, исправляющих ошибки в различных компонентах системы
Дальнейшее развитие yezzey и proxy
Статистика interconnect
Author: Maxim Smyatkin smyatkinmaxim@gmail.com
Date: Tue Feb 18 13:24:21 2025 +0300
Add views for getting cummulative IC stats
- gp_interconnect_stats_per_segment
- gp_interconnect_stats_per_segment_per_segment
- gp_interconnect_stats_per_segment_per_host
For now this feature is implemented as a contrib package to avoid
upgrading system catalogs in a minor release.
Also, it only supports UDPIFC protocol because that's what most
installations use.
Быстрый аналог TRY_CONVERT из MSSQL
Author: Vladimir Rachkin 33569237+robozmey@users.noreply.github.com
Date: Thu Feb 13 18:35:12 2025 +0300
Add TRY_CONVERT extension
Temp таблицы без bloat каталога
Author: reshke reshkekirill@gmail.com
Date: Wed Jan 15 19:00:14 2025 +0500
Fast Temporary tables for GP. (#7)
BASED on https://www.postgresql.org/message-id/20160729141552.4062e19b%40fujitsu by afiskon
Gpshrink - как gpexpand только наоборот
Author: Kirill asefru78@mail.ru
Date: Mon Nov 18 13:38:52 2024 +0300
Cherry-pick Enhancement: Add gpshrink to support elastic scaling. With some editions by me. (#32) (#16)
Co-authored-by: reshke <reshke@double.cloud>
Доп статистика в explain analyze
Author: Vladimir Rachkin 33569237+robozmey@users.noreply.github.com
Date: Mon Oct 14 16:26:24 2024 +0300
Rows out в EXPLAIN ANALYZE (#44)
Основные изменения pivotal 6.25.3 -> 6.27
Author: bhari hari-prasad.bachala@broadcom.com
Date: Tue May 21 14:14:31 2024 +0530
[Orca] Fallback to planner if a function in 'from' clause uses 'WITH ORDINALITY' (#17480)
Author: bhari hari-prasad.bachala@broadcom.com
Date: Tue May 21 14:10:52 2024 +0530
[6X] Fix orca preprocess step for query with Select-Project-NaryJoin pattern (#17476)
Author: Chris Hajas chajas@vmware.com
Date: Thu Feb 1 12:12:02 2024 -0800
Support Query Parameters in Orca
Author: wenru yan ywenru@vmware.com
Date: Tue May 7 02:24:46 2024 +0000
Improve refresh materialized view with "no data" option
Author: chaotian 108248800+charliettxx@users.noreply.github.com
Date: Tue May 7 16:24:45 2024 +0800
Fix mdqa bug as groupkey is constant (#17377)
Author: Soumyadeep Chakraborty soumyadeep2007@gmail.com
Date: Thu May 2 12:02:10 2024 -0700
Report dtx protocol command dispatch errors reliably
Author: Soumyadeep Chakraborty soumyadeep2007@gmail.com
Date: Wed May 1 11:05:54 2024 -0700
cdbdisp: Set numResults output variable
Author: Soumyadeep Chakraborty soumyadeep2007@gmail.com
Date: Thu May 2 11:29:53 2024 -0700
binswap: Ignore tables with RLE+ZSTD compression
Author: Chris Hajas chajas@vmware.com
Date: Mon Apr 22 13:50:18 2024 -0700
Use full precision for statistics values in Orca
Author: David Kimura dkimura@vmware.com
Date: Tue Jan 12 15:05:16 2021 -0800
Fix Orca stream operator<< precision (#11322)
Author: xuejing zhao 80750564+zxuejing@users.noreply.github.com
Date: Tue Apr 30 10:30:06 2024 +0800
fix panic when concurrent update with before row update trigger.
Author: Alvaro Herrera alvherre@alvh.no-ip.org
Date: Mon Apr 25 10:32:13 2022 +0200
Always pfree strings returned by GetDatabasePath
Author: interma interma@outlook.com
Date: Wed Apr 10 12:49:21 2024 +0800
Fix an obvious memory leak in _bitmap_xlog_insert_bitmapwords(), it
caused the memory usage of startup process to continue to grow.
Author: Soumyadeep Chakraborty soumyadeep2007@gmail.com
Date: Tue Apr 16 10:14:11 2024 -0700
Retire gp_reject_internal_tcp_connection
Author: Zhenghua Lyu kainwen@gmail.com
Date: Tue Apr 16 22:03:18 2024 +0800
Add warning and guard code for zero-column table.
Author: Juyi.lmz juyi.lmz@alibaba-inc.com
Date: Thu Apr 11 15:07:05 2024 +0800
Fix token for user id xxx doesn't exist (#17325)
Author: Soumyadeep Chakraborty soumyadeep2007@gmail.com
Date: Thu Apr 11 16:36:32 2024 -0700
Fix ZSTD for RLE when specified at table level
Author: Adam Lee adlee@vmware.com
Date: Tue Sep 6 15:56:57 2022 +0800
GlobalShellExecutor checks the whole output as the termination condition
Author: Noah Misch noah@leadboat.com
Date: Mon Aug 7 06:05:56 2023 -0700
Reject substituting extension schemas or owners matching ["$'\].
Author: Ashwin Agrawal aashwin@vmware.com
Date: Wed Apr 3 11:07:52 2024 -0700
Create INIT_FORK for AO aux relations during TRUNCATE operation
Author: Sunil S sunilfeb26@gmail.com
Date: Thu Dec 14 12:03:09 2023 +0530
Modified XLByteToSeg invocation as per definition
Author: Xing Guo higuoxing@gmail.com
Date: Tue Apr 2 15:37:28 2024 +0800
[6X] Add missing volatile qualifier. (#17287)
Author: NISHANT SHARMA 99574850+24nishant@users.noreply.github.com
Date: Fri Mar 29 11:17:21 2024 +0530
Handle Invalid Return Value For Opfamily Mdid. (#17229)
Author: Haolin Wang whaolin@vmware.com
Date: Mon Mar 25 14:37:57 2024 +0800
ignore type change of AORelHashEntryData for base 6.26.4
Author: Haolin Wang whaolin@vmware.com
Date: Tue Jan 9 09:52:04 2024 +0800
Fix TRUNCATE could remove shared AO hash entry in use by other transactions.
Author: dreamedcheng drec.wu@gmail.com
Date: Tue Mar 26 08:35:17 2024 +0800
[6X] Fix wrong result of numeric window sum/avg agg.
Author: wuchengwen wcw190496@alibaba-inc.com
Date: Tue Mar 19 14:15:42 2024 +0800
Check orphaned files functions exclude idle sessions when doing safety check
Author: NISHANT SHARMA 99574850+24nishant@users.noreply.github.com
Date: Wed Mar 20 16:20:33 2024 +0530
Prevent overwriting of worker in ORCA (#17096)
Author: Jimmy Yih jyih@vmware.com
Date: Wed Mar 13 14:53:03 2024 -0700
Ignore symbol ConfigureNamesBool_gp for base 6.26.4
Author: Soumyadeep Chakraborty soumyadeep2007@gmail.com
Date: Tue Mar 12 11:23:28 2024 -0700
CO: Support RLE with Zstandard
Author: Tom Lane tgl@sss.pgh.pa.us
Date: Mon Nov 6 10:56:43 2023 -0500
Detect integer overflow while computing new array dimensions.
Author: Andres Freund andres@anarazel.de
Date: Sun Oct 29 22:13:54 2017 -0700
Provide overflow safe integer math inline functions.
Author: David Kimura dkimura@vmware.com
Date: Wed Mar 6 11:34:40 2024 -0800
[ORCA] Fix ident to const predicate push down optimization
Author: Jingyu Wang wjingyu@vmware.com
Date: Thu Feb 8 17:49:27 2024 -0800
Add ORCA GUC's for dynamic index/bitmap scan
Author: Nathan Bossart nathan@postgresql.org
Date: Wed Feb 14 17:09:31 2024 -0600
Remove obsolete check in SIGTERM handler for the startup process.
Author: Chris Hajas chajas@vmware.com
Date: Wed Feb 28 10:40:44 2024 -0800
6X: Re-support DISTINCT-qualified Window Aggregate in Orca (#17127)
Author: QingMa qing.ma@broadcom.com
Date: Fri Feb 23 08:37:24 2024 +0000
Introduce GUC `gp_enable_mdqa_shared_scan` for three-stage aggregation
Author: chaotian 108248800+charliettxx@users.noreply.github.com
Date: Wed Feb 28 13:36:50 2024 +0800
Fix join condition lost after pull up sublink to join (#17114)
Author: Chris Hajas chajas@vmware.com
Date: Tue Feb 13 12:27:30 2024 -0800
Fix Orca to not generate unnecessary slices for plans with Subplans
Author: David Kimura dkimura@vmware.com
Date: Mon Feb 12 08:38:39 2024 -0800
[ORCA] Fix SIGSEGV using subquery exists on materialized view
Author: HouLei hlinbit@yeah.net
Date: Tue Feb 20 22:25:20 2024 +0800
Add multi-thread compression for gpfdist in gpdb6 (#16914)
Author: Soumyadeep Chakraborty soumyadeep2007@gmail.com
Date: Mon Feb 12 14:21:02 2024 -0800
Add a memory barrier in Own/DisownLatch
Author: Robert Haas rhaas@postgresql.org
Date: Tue Sep 9 17:45:20 2014 -0400
Change the spinlock primitives to function as compiler barriers.
Author: Soumyadeep Chakraborty soumyadeep2007@gmail.com
Date: Tue Feb 6 18:39:48 2024 -0800
Fix race between termination and resqueue wakeup
Author: Soumyadeep Chakraborty soumyadeep2007@gmail.com
Date: Tue Feb 6 18:39:47 2024 -0800
Fix a statement leak involving self-deadlocks
Author: Dev Chattopadhyay cdev@vmware.com
Date: Fri Feb 2 10:44:08 2024 -0800
Keep partition child lock until end of select
Author: THANATOSLAVA 99771015+THANATOSLAVA@users.noreply.github.com
Date: Tue Feb 6 07:31:02 2024 -0800
Fix query hang / fallback if involving CTE of replicated (Backport) (#17048)
Author: wenru yan ywenru@vmware.com
Date: Tue Jan 30 07:56:24 2024 +0000
[6X][backport] Fix FTS PROBE process memory leak
Author: rainmaple 1205022533@qq.com
Date: Tue Jan 30 10:07:24 2024 +0800
Fix the bug which leading to wrong answer of sql string_agg(orderby) & agg(distinct).
Author: wenru yan ywenru@vmware.com
Date: Thu Jan 25 08:12:15 2024 +0000
Fix query with DynamicBitmap scan crash when QueryFinishPending is true
Author: Chen Mulong chenmulong@gmail.com
Date: Sat Jan 20 12:02:19 2024 +0800
[6X_STABLE] Fix bug: PLPY function causes master process reset. (#17003)
Author: David Kimura dkimura@vmware.com
Date: Tue Jan 9 13:27:40 2024 -0800
[ORCA] Add GUC to disable right outer join (ROJ) (#16887)
Author: Thomas Munro tmunro@postgresql.org
Date: Mon Jun 8 13:57:24 2020 +1200
Fix locking bugs that could corrupt pg_control.
Author: Hongxu Ma <interma@o...
6.27.5
What's Changed
- Enahce tab completion for various commands by @reshke in #5
- Update nodeBitmapHeapscan.c by @reshke in #6
- Fix oversights of previuos commit by @reshke in #14
- Explain node summary by @robozmey in #3
- Cherry-pick Enhancement: Add gpshrink to support elastic scaling. With some editions by me. (#32) by @diPhantxm in #16
- Perform nowait lock before analyze so we could skip locked objects by @leborchuk in #15
New Contributors
- @visill made their first contribution in #4
- @leborchuk made their first contribution in #9
- @robozmey made their first contribution in #3
- @diPhantxm made their first contribution in #16
Full Changelog: 6.27.1...6.27.5
