EXISTS subquery is currently allowed only directly under WHERE clause
or in AND expression that is directly under WHERE. However, the check
was insufficient previously so that views using expressions other than
AND containing an EXISTS subquery could be created without an error
and it caused incorrect maintenance results.
To fix this check, add a new boolean member allow_context into
check_ivm_restriction_context. This member means whether EXISTS
subquery is allowed in the current node being examined in
check_ivm_restriction_walker. This should be set to true just before
calling check_ivm_restriction_walker for nodes directly under WHERE
or operands of AND expression direct under WHERE, and is reset to
false on every call of the function.
In passing, move the check for OR and NOT expression from
rewrite_exists_subquery_walker to check_ivm_restriction_context,
with some code cleaning.
---------
Co-authored-by: Yugo Nagata <nagata@sraoss.co.jp>
Build errors/warnings against PostgreSQL 16 are fixed.
Also, adapted to the change of codes, including:
- Get rid of the "new" and "old" entries in a view's rangetable.
(Although, removed codes were dead codes because pg_ivm doesn't
have any rules in pg_rewrite.)
- Rework query relation permission checking
- Require empty Bitmapsets to be represented as NULL
- Fix some compiler warnings
Currently, types that does not have an equality operator cannot
be used in the target list of the view definition, because all
of target list entries are used for comparison to identify rows
to be updated or deleted in the view. Previously, an error is
raised at the time such view is incrementally maintained, this
is fixed to check that at the view creation time.
This restriction may be relaxed in future after we can use an
index to identify rows in the view.
Issue #61
Add EXISTS clause support in IVM
Correlated subqueries using EXISTS in WHERE clause are supported.
An EXISTS subquery in WHERE clause is rewritten to LATERAL subquery
in FROM clause, and IVM' process can handle this like as a normal join.
Also, hidden columns "ivm_exists_count_X__" are added to check if
EXISTS condition is satisfied. This column stores the count of how many
rows in the subquery are correlated to (joined to) each row of the main
query. When a base table contained in EXISTS clause is modified, this
count value in IMMV is updated, and a row whose count becomes zero
is deleted.
restrictions :
- EXISTS subqueries are allowed only in WHERE clause.
- aggregate functions are not supported together with EXISTS.
- EXISTS subqueries in a subquery are not supported.
- EXISTS condition can use only with AND Expr
Previously, functions names in pg_catalog schema that were used during
view maintenance were not qualified. This is problematic because
functions in other schema could be referenced unintentionally. Moreover,
that could result in privilege escalation that if a nefarious user who
can create a function, arbitrary functions could be executed under IMMV
owner's privilege.
CVE-2023-23554
Simple CTEs which does not contain aggregates or DISTINCT are
now supported similarly to simple sub-queries.
Before a view is maintained, all CTEs are converted to corresponding
subqueries to enable to treat CTEs as same as subqueries. For this
end, codes of the static function inline_cte in the core
(optimizer/plan/subselect.c) was imported.
Prohibit Unreferenced CTE is prohibited.
When a table in a unreferenced CTE is TRUNCATEd, the contents
of the IMMV is not affected so it must not be truncated. For
confirming it at the maintenance time, we have to check if the
modified table used in a CTE is actually referenced. Although
it would possible, we just disallow to create such IMMVs for now
since such unreferenced CTE is useless unless it doesn't contain
modifying commands, that is already prohibited.
It was added by 3de95c09 for consistency with the patch version
proposed for PostgreSQL core, but the called function was not
one we intended. Fortunately, the behavior has not been affected.
Now, I decided to remove this funciton since I come to think it is
confusable.
Also, added some comments on an argument left for the similar
consistency.
Some conditions on view definition that should be prohibited,
for example,
- SELECT ... FROM func(..., (SELECT ... FROM ...), ..) ...;
- SELECT expr(SELECT ... FROM ...) FROM ...;
were not checked. Also, some error messages and the order of
tests are improved.
When using DELETE or UPDATE, we must use exclusive lock for now
because apply_old_delta(_with_count) could result in wrong results
with concurrent transactions. We would like to improve it in future,
but we prevent it by using the lock for now.
The names of additional columns for aggregate views are derived from column
names specified in the first parameter of create_immv, but when the number
was less than the length of the actual target list, segmentation fault occurred.
Furthermore, when the number of specified columns is more than the target list,
it overrode additional column names and it caused a failure of incremental
maintenance of an aggregate view.
To fix then, check the length of the specified column name list not to access
invalid area, and also prevent from overriding additional column names.
It is intended that a unique index is created only if all primary
keys of tables in FROM clause appear in the target list. For this
purpose, pull_varnos_of_level was used to extract relations in the
FROM clause, but it is incorrect and actually we should use
get_relids_in_jointree.
Due to this bug, an index could be created even even where there
is a pkey attribute from just one of relations in FROM clause.
Previously, IMMV including IMMV in its definition can be created by
create_immv(), but it should not be supported by IMMV because we
cannot maintain it recursively for now. This patch prevents it by raising
an error for such view definition on create_immv().
We usually try to create a unique index on an IMMV for both
performance and constraint reasons. However, we don't need
such index for an aggregate view without GROUP BY because
it has always only one row. Also, we don't need to raise
NOTICE message to suggest to create an index.
This change was mistakenly introduced in cfe9491e6b,
but it should have been a separate commit. In this commit,
a typo and tests are fixed.
In order to re-calculate min/max values for groups where the min
or max value is deleted, we need the view query definition in string
form. However, pg_get_viewdef cannot be used for this purpose because
IMMV's defenition is in pg_ivm_immv but not pg_rewrite. Therefore,
we have to convert query definition in pg_ivm_immv to query
definition string. We can use pg_get_querydef in PG15, but we cannot
in PG14 or earlier, so we use codes in ruleutil.c copied from PG13
or PG14 depending versions.
In PostgreSQL 14 or later, OIDs of aggregate functions are
described in fmrgoids.h, but that in PostgreSQL 13 doesn't
contain aggregate function OIDs. Therefore, we get the OID
by passing the function name and arg type to to_regprocedure().
When a base table is truncated, the view content will be empty if the
view definition query does not contain an aggregate without a GROUP clause.
Therefore, such views can be truncated.
Aggregate views without a GROUP clause always have one row. Therefore,
if a base table is truncated, the view will not be empty and will contain
a row with NULL value (or 0 for count()). So, in this case, we refresh the
view instead of truncating it.
Built-in count, sum, avg are supported. We have not supported min/max
yet, but will support in future.
When an IMMV with any aggregates are defined, additional columns
are created for each aggregate function. Although such columns are
"hidden" in pgsql-ivm version, they are always visible for users
in the extension version.
refresh_immv(immv_name, with_data) is a function to refresh IMMV like
REFRESH MATERIALIZED VIEW command. It has two argument.
immv_name is incrementally maintainable materialized view's name, and
with_data is an option that is corresponding to the WITH [NO] DATA option.
When with_data is set false, the IMMV gets unpopulated.
One of differences between IMMVs unpopulated by this function and
normal materialized views unpopulated by REFRESH ... WITH NO DATA
is that such IMMVs can be referenced by SELECT but return no rows,
while unpopulated materialized views are not scanable.
The behaviour may be changed in future to raise an error when unpopulated
an IMMV is scanned.
