luciano/pg_ivm
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
pg_ivm/pg_ivm.h

72 lines
2.4 KiB
C
Raw Normal View History

Split files to make it easier to follow the core code
2022-04-27 05:45:47 +00:00
/*-------------------------------------------------------------------------
*
* pg_ivm.h
* incremental view maintenance extension
*
* Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
* Portions Copyright (c) 2022, IVM Development Group
*
*-------------------------------------------------------------------------
*/
Initial release of pg_ivm 1.0
2022-03-31 12:48:53 +00:00
#ifndef _PG_IVM_H_
Fix a header gurad macro
2022-05-02 18:19:58 +00:00
#define _PG_IVM_H_
Initial release of pg_ivm 1.0
2022-03-31 12:48:53 +00:00
Split files to make it easier to follow the core code
2022-04-27 05:45:47 +00:00
#include "catalog/objectaddress.h"
#include "fmgr.h"
#include "nodes/params.h"
Add CTE support (#47) Simple CTEs which does not contain aggregates or DISTINCT are now supported similarly to simple sub-queries. Before a view is maintained, all CTEs are converted to corresponding subqueries to enable to treat CTEs as same as subqueries. For this end, codes of the static function inline_cte in the core (optimizer/plan/subselect.c) was imported. Prohibit Unreferenced CTE is prohibited. When a table in a unreferenced CTE is TRUNCATEd, the contents of the IMMV is not affected so it must not be truncated. For confirming it at the maintenance time, we have to check if the modified table used in a CTE is actually referenced. Although it would possible, we just disallow to create such IMMVs for now since such unreferenced CTE is useless unless it doesn't contain modifying commands, that is already prohibited.
2023-01-30 02:28:27 +00:00
#include "nodes/pathnodes.h"
Split files to make it easier to follow the core code
2022-04-27 05:45:47 +00:00
#include "parser/parse_node.h"
#include "tcop/dest.h"
#include "utils/queryenvironment.h"
Fix potential view inconsistency issues (#121) Previously, the view contents could become inconsistent with the base tables in the following scenarios: 1) A concurrent transaction modifies a base table and commits before the incremental view maintenance starts in the current transaction. 2) A concurrent transaction modifies a base table and commits before the create_immv or refresh_immv command generates data. 3) Concurrent transactions incrementally update a view with a self-join or modify multiple base tables simultaneously. Incremental updates of a view are generally performed sequentially using an exclusive lock. However, even if we are able to acquire the lock, a concurrent transaction may have already incrementally updated the view and been committed before we can acquire it. In REPEATABLE READ or SERIALIZABLE isolation levels, this could lead to an inconsistent view state, which is the cause of the first issue. To fix this, a new field, lastivmupdate, has been added to the pg_ivm_immv catalog to record the transaction ID of the most recent update to the view. Before performing view maintenance, the transaction ID is checked. If the transaction was still in progress at the start of the current transaction, an error is raised to prevent anomalies. To fix the second issue, the timing of CreateTrigger() has been moved to before data generation. This ensures that locks conflicting with table modifications have been acquired on all base tables. In addition, the latest snapshot is used in READ COMMITTED level during the data generation to reflect committed changes from concurrent transactions. Additionally, inconsistencies that cannot be avoided through locking are prevented by checking the transaction ID of the last view update, as done for the first issue. However, concurrent table modifications and create_immv execution still cannot be detected at the time of view creation. Therefore, create_immv raises a warning in REPEATABLE READ or SERIALIZABLE isolation levels, suggesting that the command be used in READ COMMITTED mode or that refresh_immv be executed afterward to ensure the view remains consistent. The third issue was caused by the snapshot used for checking tuple visibility in the table's pre-update state not being the latest one. To fix this, the latest snapshot is now used in READ COMMITTED mode. Isolation tests are also added. Issue #104
2025-03-10 09:26:54 +00:00
#define Natts_pg_ivm_immv 4
Initial release of pg_ivm 1.0
2022-03-31 12:48:53 +00:00
#define Anum_pg_ivm_immv_immvrelid 1
Add aggregates support Built-in count, sum, avg are supported. We have not supported min/max yet, but will support in future. When an IMMV with any aggregates are defined, additional columns are created for each aggregate function. Although such columns are "hidden" in pgsql-ivm version, they are always visible for users in the extension version.
2022-06-21 11:50:45 +00:00
#define Anum_pg_ivm_immv_viewdef 2
#define Anum_pg_ivm_immv_ispopulated 3
Fix potential view inconsistency issues (#121) Previously, the view contents could become inconsistent with the base tables in the following scenarios: 1) A concurrent transaction modifies a base table and commits before the incremental view maintenance starts in the current transaction. 2) A concurrent transaction modifies a base table and commits before the create_immv or refresh_immv command generates data. 3) Concurrent transactions incrementally update a view with a self-join or modify multiple base tables simultaneously. Incremental updates of a view are generally performed sequentially using an exclusive lock. However, even if we are able to acquire the lock, a concurrent transaction may have already incrementally updated the view and been committed before we can acquire it. In REPEATABLE READ or SERIALIZABLE isolation levels, this could lead to an inconsistent view state, which is the cause of the first issue. To fix this, a new field, lastivmupdate, has been added to the pg_ivm_immv catalog to record the transaction ID of the most recent update to the view. Before performing view maintenance, the transaction ID is checked. If the transaction was still in progress at the start of the current transaction, an error is raised to prevent anomalies. To fix the second issue, the timing of CreateTrigger() has been moved to before data generation. This ensures that locks conflicting with table modifications have been acquired on all base tables. In addition, the latest snapshot is used in READ COMMITTED level during the data generation to reflect committed changes from concurrent transactions. Additionally, inconsistencies that cannot be avoided through locking are prevented by checking the transaction ID of the last view update, as done for the first issue. However, concurrent table modifications and create_immv execution still cannot be detected at the time of view creation. Therefore, create_immv raises a warning in REPEATABLE READ or SERIALIZABLE isolation levels, suggesting that the command be used in READ COMMITTED mode or that refresh_immv be executed afterward to ensure the view remains consistent. The third issue was caused by the snapshot used for checking tuple visibility in the table's pre-update state not being the latest one. To fix this, the latest snapshot is now used in READ COMMITTED mode. Isolation tests are also added. Issue #104
2025-03-10 09:26:54 +00:00
#define Anum_pg_ivm_immv_lastivmupdate 4
Initial release of pg_ivm 1.0
2022-03-31 12:48:53 +00:00
Split files to make it easier to follow the core code
2022-04-27 05:45:47 +00:00
/* pg_ivm.c */
extern void CreateChangePreventTrigger(Oid matviewOid);
extern Oid PgIvmImmvRelationId(void);
extern Oid PgIvmImmvPrimaryKeyIndexId(void);
Fix not to create IMMV including other IMMV(#30) (#31) Previously, IMMV including IMMV in its definition can be created by create_immv(), but it should not be supported by IMMV because we cannot maintain it recursively for now. This patch prevents it by raising an error for such view definition on create_immv().
2022-10-18 06:51:41 +00:00
extern bool isImmv(Oid immv_oid);
Change the schema from pg_catalog to pgivm (#116) Previously, pg_upgrade failed due to the permission denied because the pg_ivm_immv catalog was in the pg_catalog catalog (Issue #79). To fix this, all objects created by pg_ivm are moved to theschema pgivm, which is also created by pg_ivm. pg_ivm is still not relocatable and this must be installed to the pgivm schema because the catalog and some internal functions are referred to unqualified by the schema name from the pg_ivm module. In future, this might be able to relocatable during installation, though. This commit affects compatibility with previous releases. To allow to access objects like create_immv function as previous, you need to qualify them with the schema name or setup search_path properly.
2025-02-17 03:07:21 +00:00
extern List *PgIvmFuncName(char *name);
Split files to make it easier to follow the core code
2022-04-27 05:45:47 +00:00
/* createas.c */
extern ObjectAddress ExecCreateImmv(ParseState *pstate, CreateTableAsStmt *stmt,
QueryCompletion *qc);
Add support for PostgreSQL 16 (#69) (#70) Build errors/warnings against PostgreSQL 16 are fixed. Also, adapted to the change of codes, including: - Get rid of the "new" and "old" entries in a view's rangetable. (Although, removed codes were dead codes because pg_ivm doesn't have any rules in pg_rewrite.) - Rework query relation permission checking - Require empty Bitmapsets to be represented as NULL - Fix some compiler warnings
2023-09-11 06:23:51 +00:00
extern void CreateIvmTriggersOnBaseTables(Query *qry, Oid matviewOid);
extern void CreateIndexOnIMMV(Query *query, Relation matviewRel);
Split files to make it easier to follow the core code
2022-04-27 05:45:47 +00:00
extern Query *rewriteQueryForIMMV(Query *query, List *colNames);
Add aggregates support Built-in count, sum, avg are supported. We have not supported min/max yet, but will support in future. When an IMMV with any aggregates are defined, additional columns are created for each aggregate function. Although such columns are "hidden" in pgsql-ivm version, they are always visible for users in the extension version.
2022-06-21 11:50:45 +00:00
extern void makeIvmAggColumn(ParseState *pstate, Aggref *aggref, char *resname, AttrNumber *next_resno, List **aggs);
Split files to make it easier to follow the core code
2022-04-27 05:45:47 +00:00
/* matview.c */
Add get_immv_def func (#23) get_immv_def reconstructs the underlying SELECT command for a IMMV. This is a decompiled reconstruction, not the original text of the command.
2022-09-30 09:59:51 +00:00
extern Query *get_immv_query(Relation matviewRel);
Improve refresh_immv behavior a bit - Allow to use qualified name - Confirm if executed by the owner of the IMMV - Improve the message when specified relation is not an IMMV - Create a unique index at refresh with no dat if possible This is actually required, but we want it behave as same as the pgsql-ivm version for now.
2022-06-23 02:33:06 +00:00
extern ObjectAddress ExecRefreshImmv(const RangeVar *relation, bool skipData,
const char *queryString, QueryCompletion *qc);
Fix potential view inconsistency issues (#121) Previously, the view contents could become inconsistent with the base tables in the following scenarios: 1) A concurrent transaction modifies a base table and commits before the incremental view maintenance starts in the current transaction. 2) A concurrent transaction modifies a base table and commits before the create_immv or refresh_immv command generates data. 3) Concurrent transactions incrementally update a view with a self-join or modify multiple base tables simultaneously. Incremental updates of a view are generally performed sequentially using an exclusive lock. However, even if we are able to acquire the lock, a concurrent transaction may have already incrementally updated the view and been committed before we can acquire it. In REPEATABLE READ or SERIALIZABLE isolation levels, this could lead to an inconsistent view state, which is the cause of the first issue. To fix this, a new field, lastivmupdate, has been added to the pg_ivm_immv catalog to record the transaction ID of the most recent update to the view. Before performing view maintenance, the transaction ID is checked. If the transaction was still in progress at the start of the current transaction, an error is raised to prevent anomalies. To fix the second issue, the timing of CreateTrigger() has been moved to before data generation. This ensures that locks conflicting with table modifications have been acquired on all base tables. In addition, the latest snapshot is used in READ COMMITTED level during the data generation to reflect committed changes from concurrent transactions. Additionally, inconsistencies that cannot be avoided through locking are prevented by checking the transaction ID of the last view update, as done for the first issue. However, concurrent table modifications and create_immv execution still cannot be detected at the time of view creation. Therefore, create_immv raises a warning in REPEATABLE READ or SERIALIZABLE isolation levels, suggesting that the command be used in READ COMMITTED mode or that refresh_immv be executed afterward to ensure the view remains consistent. The third issue was caused by the snapshot used for checking tuple visibility in the table's pre-update state not being the latest one. To fix this, the latest snapshot is now used in READ COMMITTED mode. Isolation tests are also added. Issue #104
2025-03-10 09:26:54 +00:00
extern ObjectAddress RefreshImmvByOid(Oid matviewOid, bool is_create, bool skipData,
Add support for PostgreSQL 17 (#92) Compilation errors and warning are fixed. The design of create_immv is also chaned as similar to PG17, that is, firstly a relation is created without data then it is populated by using the refresh logic. This commit contains the following changes: - Change functions to use a safe search_path during maintenance operations when used with PostgreSQL 17 This prevents maintenance operations (automatic maintenance of IMMVs and refresh_immv) from performing unsafe access. Functions used by IMMVs that need to reference non-default schemas must specify a search path during function creation. - refresh_immv can be executed by users with the MAINTAIN privilege when used with PostgreSQL 17 Issue #90
2024-07-31 03:37:43 +00:00
const char *queryString, QueryCompletion *qc);
Split files to make it easier to follow the core code
2022-04-27 05:45:47 +00:00
extern bool ImmvIncrementalMaintenanceIsEnabled(void);
extern Datum IVM_immediate_before(PG_FUNCTION_ARGS);
extern Datum IVM_immediate_maintenance(PG_FUNCTION_ARGS);
Support exists_subquery (#53) Add EXISTS clause support in IVM Correlated subqueries using EXISTS in WHERE clause are supported. An EXISTS subquery in WHERE clause is rewritten to LATERAL subquery in FROM clause, and IVM' process can handle this like as a normal join. Also, hidden columns "ivm_exists_count_X__" are added to check if EXISTS condition is satisfied. This column stores the count of how many rows in the subquery are correlated to (joined to) each row of the main query. When a base table contained in EXISTS clause is modified, this count value in IMMV is updated, and a row whose count becomes zero is deleted. restrictions : - EXISTS subqueries are allowed only in WHERE clause. - aggregate functions are not supported together with EXISTS. - EXISTS subqueries in a subquery are not supported. - EXISTS condition can use only with AND Expr
2023-08-31 02:51:36 +00:00
extern Query* rewrite_query_for_exists_subquery(Query *query);
Use snapshot to check tuple visibility in pre-update state (#28) When multiple tables are updated or the view contains a self-join, we need to calculate table states that was before it is modified during incremental view maintenance. For get the pre-update state, tuples inserted in a command must be removed when the table is scanned. Previously, we used xmin and cmin system columns for this purpose, but this way is problematic because after a tuple is frozen, its xmin no longer has any meaning. Actually, we will get inconsistent view results after XID wraparound. Also, we can see similar similar inconsistency when using sub-transaction because xmin values do not always monotonically increasing by command executions. To fix this, we use a snapshot that taken just before the table is modified for checking tuple visibility in pre-state table instead of using xmin and cmin system columns. A new function returning boolean, ivm_visible_in_prestate, is added, and this is called in WHERE clause of sub-queries to calculate pre-state table. This function check if a specified tuple in the post-update table is visible or not using the snapshot and return true if the tuple is visible.
2022-09-30 02:53:12 +00:00
extern Datum ivm_visible_in_prestate(PG_FUNCTION_ARGS);
Fix potential view inconsistency issues (#121) Previously, the view contents could become inconsistent with the base tables in the following scenarios: 1) A concurrent transaction modifies a base table and commits before the incremental view maintenance starts in the current transaction. 2) A concurrent transaction modifies a base table and commits before the create_immv or refresh_immv command generates data. 3) Concurrent transactions incrementally update a view with a self-join or modify multiple base tables simultaneously. Incremental updates of a view are generally performed sequentially using an exclusive lock. However, even if we are able to acquire the lock, a concurrent transaction may have already incrementally updated the view and been committed before we can acquire it. In REPEATABLE READ or SERIALIZABLE isolation levels, this could lead to an inconsistent view state, which is the cause of the first issue. To fix this, a new field, lastivmupdate, has been added to the pg_ivm_immv catalog to record the transaction ID of the most recent update to the view. Before performing view maintenance, the transaction ID is checked. If the transaction was still in progress at the start of the current transaction, an error is raised to prevent anomalies. To fix the second issue, the timing of CreateTrigger() has been moved to before data generation. This ensures that locks conflicting with table modifications have been acquired on all base tables. In addition, the latest snapshot is used in READ COMMITTED level during the data generation to reflect committed changes from concurrent transactions. Additionally, inconsistencies that cannot be avoided through locking are prevented by checking the transaction ID of the last view update, as done for the first issue. However, concurrent table modifications and create_immv execution still cannot be detected at the time of view creation. Therefore, create_immv raises a warning in REPEATABLE READ or SERIALIZABLE isolation levels, suggesting that the command be used in READ COMMITTED mode or that refresh_immv be executed afterward to ensure the view remains consistent. The third issue was caused by the snapshot used for checking tuple visibility in the table's pre-update state not being the latest one. To fix this, the latest snapshot is now used in READ COMMITTED mode. Isolation tests are also added. Issue #104
2025-03-10 09:26:54 +00:00
extern void AtAbort_IVM(SubTransactionId subtxid);
extern void AtPreCommit_IVM(void);
Support exists_subquery (#53) Add EXISTS clause support in IVM Correlated subqueries using EXISTS in WHERE clause are supported. An EXISTS subquery in WHERE clause is rewritten to LATERAL subquery in FROM clause, and IVM' process can handle this like as a normal join. Also, hidden columns "ivm_exists_count_X__" are added to check if EXISTS condition is satisfied. This column stores the count of how many rows in the subquery are correlated to (joined to) each row of the main query. When a base table contained in EXISTS clause is modified, this count value in IMMV is updated, and a row whose count becomes zero is deleted. restrictions : - EXISTS subqueries are allowed only in WHERE clause. - aggregate functions are not supported together with EXISTS. - EXISTS subqueries in a subquery are not supported. - EXISTS condition can use only with AND Expr
2023-08-31 02:51:36 +00:00
extern char *getColumnNameStartWith(RangeTblEntry *rte, char *str, int *attnum);
Split files to make it easier to follow the core code
2022-04-27 05:45:47 +00:00
extern bool isIvmName(const char *s);
Support min/max aggregates (#18) In order to re-calculate min/max values for groups where the min or max value is deleted, we need the view query definition in string form. However, pg_get_viewdef cannot be used for this purpose because IMMV's defenition is in pg_ivm_immv but not pg_rewrite. Therefore, we have to convert query definition in pg_ivm_immv to query definition string. We can use pg_get_querydef in PG15, but we cannot in PG14 or earlier, so we use codes in ruleutil.c copied from PG13 or PG14 depending versions.
2022-07-25 04:11:33 +00:00
/* ruleutils.c */
Fix to get view definition string with correct column names (#26) Previously, a query string returned from pg_ivm_get_querydef did not include column names specified when IMMV was defined by create_immv. This caused failures in maintenance of MIN/MAX aggregate views whose columns had alias names. It is fixed by rewriting the result column name in the parse tree using the view's tuple descriptor prior to calling pg_get_querydef for PG15 or higher, or specifying the tuple descriptor to get_query_def for PG14 or earlier.
2022-09-29 13:24:28 +00:00
extern char *pg_ivm_get_viewdef(Relation immvrel, bool pretty);
Support min/max aggregates (#18) In order to re-calculate min/max values for groups where the min or max value is deleted, we need the view query definition in string form. However, pg_get_viewdef cannot be used for this purpose because IMMV's defenition is in pg_ivm_immv but not pg_rewrite. Therefore, we have to convert query definition in pg_ivm_immv to query definition string. We can use pg_get_querydef in PG15, but we cannot in PG14 or earlier, so we use codes in ruleutil.c copied from PG13 or PG14 depending versions.
2022-07-25 04:11:33 +00:00
Add CTE support (#47) Simple CTEs which does not contain aggregates or DISTINCT are now supported similarly to simple sub-queries. Before a view is maintained, all CTEs are converted to corresponding subqueries to enable to treat CTEs as same as subqueries. For this end, codes of the static function inline_cte in the core (optimizer/plan/subselect.c) was imported. Prohibit Unreferenced CTE is prohibited. When a table in a unreferenced CTE is TRUNCATEd, the contents of the IMMV is not affected so it must not be truncated. For confirming it at the maintenance time, we have to check if the modified table used in a CTE is actually referenced. Although it would possible, we just disallow to create such IMMVs for now since such unreferenced CTE is useless unless it doesn't contain modifying commands, that is already prohibited.
2023-01-30 02:28:27 +00:00
/* subselect.c */
extern void inline_cte(PlannerInfo *root, CommonTableExpr *cte);
Initial release of pg_ivm 1.0
2022-03-31 12:48:53 +00:00
#endif
Reference in a new issue Copy permalink