luciano/honeysql
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #424 from seancorfield/issue-422-quoting

Browse source 
#423 #424 #425
This commit is contained in:
Sean Corfield 2022-09-01 22:57:04 -07:00 committed by GitHub
commit c913ffe155
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
9 changed files with 147 additions and 73 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
CHANGELOG.md
View file

@ -1,6 +1,9 @@
# Changes
* 2.3.next in progress
* Address [#425](https://github.com/seancorfield/honeysql/issues/425) by clarifying that `INTERVAL` as special syntax may be MySQL-specific and PostgreSQL uses difference syntax (because `INTERVAL` is a data type there).
* Address [#423](https://github.com/seancorfield/honeysql/issues/423) by supporting `DEFAULT` values and `DEFAULT` rows in `VALUES` clause -- NEEDS DOCUMENTATION!
* **WIP** Address [#422](https://github.com/seancorfield/honeysql/issues/422) by auto-quoting unusual entity names when `:quoted` (and `:dialect`) are not specified, making HoneySQL more secure by default.
* Address [#419](https://github.com/seancorfield/honeysql/issues/419) by adding `honey.sql.protocols` and `InlineValue` with a `sqlize` function.
* Address [#413](https://github.com/seancorfield/honeysql/issues/413) by flagging a lack of `WHERE` clause for `DELETE`, `DELETE FROM`, and `UPDATE` when `:checking :basic` (or `:checking :strict`).
* Fix [#392](https://github.com/seancorfield/honeysql/issues/392) by adding support for `WITH` / (`NOT`) `MATERIALIZED` -- via PR [#420](https://github.com/seancorfield/honeysql/issues/420) [@robhanlon22](https://github.com/robhanlon22).

8
README.md
View file

@ -530,6 +530,8 @@ vectors where the first element is either a keyword or a symbol:
=> ["SELECT * FROM foo WHERE date_created > DATE_ADD(NOW(), INTERVAL ? HOURS)" 24]
```
> Note: The above example may be specific to MySQL but the general principle of vectors for function calls applies to all dialects.
A shorthand syntax also exists for simple function calls:
keywords that begin with `%` are interpreted as SQL function calls:
@ -703,11 +705,11 @@ INSERT INTO sample
0.291 32.621 4325]
```
#### Identifiers
#### Entity Names
To quote identifiers, pass the `:quoted true` option to `format` and they will
To quote SQL entity names, pass the `:quoted true` option to `format` and they will
be quoted according to the selected dialect. If you override the dialect in a
`format` call, by passing the `:dialect` option, identifiers will be automatically
`format` call, by passing the `:dialect` option, SQL entity names will be automatically
quoted. You can override the dialect and turn off quoting by passing `:quoted false`.
Valid `:dialect` options are `:ansi` (the default, use this for PostgreSQL),
`:mysql`, `:oracle`, or `:sqlserver`:

4
doc/differences-from-1-x.md
View file

@ -106,7 +106,7 @@ You can now select a non-ANSI dialect of SQL using the new `honey.sql/set-dialec
The `:quoting <dialect>` option has superseded by the new dialect machinery and a new `:quoted` option that turns quoting on or off. You either use `:dialect <dialect>` instead or set a default dialect (via `set-dialect!`) and then use `:quoted true` in `format` calls where you want quoting.
Identifiers are automatically quoted if you specify a `:dialect` option to `format`, unless you also specify `:quoted false`.
SQL entity names are automatically quoted if you specify a `:dialect` option to `format`, unless you also specify `:quoted false`.
The following options are no longer supported:
* `:allow-dashed-names?` -- if you provide dashed-names in 2.x, they will be left as-is if quoting is enabled, else they will be converted to snake_case (so you will either get `"dashed-names"` with quoting or `dashed_names` without). If you want dashed-names to be converted to snake_case when `:quoted true`, you also need to specify `:quoted-snake true`.
@ -131,7 +131,7 @@ The following new syntax has been added:
* `:default` -- for `DEFAULT` values (in inserts) and for declaring column defaults in table definitions,
* `:escape` -- used to wrap a regular expression so that non-standard escape characters can be provided,
* `:inline` -- used as a function to replace the `sql/inline` / `#sql/inline` machinery,
* `:interval` -- used as a function to support `INTERVAL <n> <units>`, e.g., `[:interval 30 :days]`,
* `:interval` -- used as a function to support `INTERVAL <n> <units>`, e.g., `[:interval 30 :days]` for databases that support it (e.g., MySQL),
* `:lateral` -- used to wrap a statement or expression, to provide a `LATERAL` join,
* `:lift` -- used as a function to prevent interpretation of a Clojure data structure as DSL syntax (e.g., when passing a vector or hash map as a parameter value) -- this should mostly be a replacement for `honeysql.format/value`,
* `:nest` -- used as a function to add an extra level of nesting (parentheses) around an expression,

8
doc/extending-honeysql.md
View file

@ -179,10 +179,10 @@ of it and would call `sql/format-expr` on each argument:
_New in HoneySQL 2.3.x_
The built-in dialects that HoneySQL supports are:
* `:ansi` -- the default, that quotes identifiers with double-quotes, like `"this"`
* `:mysql` -- quotes identifiers with backticks, and changes the precedence of `SET` in `UPDATE`
* `:oracle` -- quotes identifiers like `:ansi`, and does not use `AS` in aliases
* `:sqlserver` -- quotes identifiers with brackets, like `[this]`
* `:ansi` -- the default, that quotes SQL entity names with double-quotes, like `"this"`
* `:mysql` -- quotes SQL entity names with backticks, and changes the precedence of `SET` in `UPDATE`
* `:oracle` -- quotes SQL entity names like `:ansi`, and does not use `AS` in aliases
* `:sqlserver` -- quotes SQL entity names with brackets, like `[this]`
A dialect spec is a hash map containing at least `:quote` but also optionally `:clause-order-fn` and/or `:as`:
* `:quote` -- a unary function that takes a string and returns the quoted version of it

7
doc/getting-started.md
View file

@ -298,6 +298,8 @@ Most databases use `"` for quoting (the `:ansi` and `:oracle` dialects).
The `:sqlserver` dialect uses `[`..`]` and the `:mysql` dialect uses
```..```. In addition, the `:oracle` dialect disables `AS` in aliases.
> Note: by default, quoting is **off** which produces cleaner-looking SQL and assumes you control all the symbols/keywords used as table, column, and function names -- the "SQL entities". If you are building any SQL or DDL where the table, column, or function names could be provided by an external source, **you should specify `:quoted true` to ensure all SQL entities are safely quoted**. As of 2.3.next, if you do _not_ specify `:quoted` as an option, HoneySQL will automatically quote any SQL entities that seem unusual, i.e., that contain any characters that are not alphanumeric or underscore. Purely alphanumeric entities will not be quoted (no entities were quoted by default prior to 2.3.next). You can prevent that auto-quoting by explicitly passing `:quoted false` into the `format` call but, from a security point of view, you should think very carefully before you do that: quoting entity names helps protect you from injection attacks!
Currently, the only dialect that has substantive differences from
the others is `:mysql` for which the `:set` clause
has a different precedence than ANSI SQL.
@ -309,10 +311,11 @@ before you call `format` for the first time.
You can change the dialect for a single `format` call by
specifying the `:dialect` option in that call.
SQL entities are not quoted by default but if you specify the
Alphanumeric SQL entities are not quoted by default but if you specify the
dialect in a `format` call, they will be quoted. If you don't
specify a dialect in the `format` call, you can specify
`:quoted true` to have SQL entities quoted.
`:quoted true` to have SQL entities quoted. You can also enable quoting
globally via the `set-dialect!` function.
<!-- Reminder to doc author:
Reset dialect to default so other blocks are not affected for test-doc-blocks -->

22
doc/options.md
View file

@ -20,8 +20,8 @@ All options may be omitted. The default behavior of each option is described in
* `:dialect` -- a keyword that identifies a dialect to be used for this specific call to `format`; the default is to use what was specified in `set-dialect!` or `:ansi` if no other dialect has been set,
* `:inline` -- a Boolean indicating whether or not to inline parameter values, rather than use `?` placeholders and a sequence of parameter values; the default is `false` -- values are not inlined,
* `:params` -- a hash map providing values for named parameters, identified by names (keywords or symbols) that start with `?` in the DSL; the default is that any such named parameters will have `nil` values,
* `:quoted` -- a Boolean indicating whether or not to quote (strop) identifiers (table and column names); the default is `false` -- identifiers are not quoted,
* `:quoted-snake` -- a Boolean indicating whether or not quoted and string identifiers should have `-` replaced by `_`; the default is `false` -- quoted and string identifiers are left exactly as-is,
* `:quoted` -- a Boolean indicating whether or not to quote (strop) SQL entity names (table and column names); the default is `nil` -- alphanumeric SQL entity names are not quoted but (as of 2.3.next) "unusual" SQL entity names are quoted; a `false` value turns off all quoting,
* `:quoted-snake` -- a Boolean indicating whether or not quoted and string SQL entity names should have `-` replaced by `_`; the default is `false` -- quoted and string SQL entity names are left exactly as-is,
* `:values-default-columns` -- a sequence of column names that should have `DEFAULT` values instead of `NULL` values if used in a `VALUES` clause with no associated matching value in the hash maps passed in; the default behavior is for such missing columns to be given `NULL` values.
See below for the interaction between `:dialect` and `:quoted`.
@ -104,12 +104,12 @@ to values for this call to `format`. For example:
## `:quoted`
If `:quoted true`, or `:dialect` is provided (and `:quoted` is not
specified as `false`), identifiers that represent
specified as `false`), SQL entity names that represent
tables and columns will be quoted (stropped) according to the
selected dialect.
If `:quoted false`, identifiers that represent tables and columns
will not be quoted. If those identifiers are reserved words in
If `:quoted false`, SQL entity names that represent tables and columns
will not be quoted. If those SQL entity names are reserved words in
SQL, the generated SQL will be invalid.
The quoting (stropping) is dialect-dependent:
@ -118,19 +118,23 @@ The quoting (stropping) is dialect-dependent:
* `:oracle` -- uses double quotes
* `:sqlserver` -- user square brackets
As of 2.3.next, if `:quoted` and `:dialect` are not provided, and no
default quoting strategy has been specified (via `set-dialect!`) then
alphanumeric SQL entity names will not be quoted but "unusual" SQL entity names will
## `:quoted-snake`
Where strings are used to identify table or column names, they are
treated as-is. If `:quoted true` (or a `:dialect` is specified),
those identifiers are quoted as-is.
those SQL entity names are quoted as-is.
Where keywords or symbols are used to identify table or column
names, and `:quoted true` is provided, those identifiers are
names, and `:quoted true` is provided, those SQL entity names are
quoted as-is.
If `:quoted-snake true` is provided, those identifiers are quoted
If `:quoted-snake true` is provided, those SQL entity names are quoted
but any `-` in them are replaced by `_` -- that replacement is the
default in unquoted identifiers.
default in unquoted SQL entity names.
This allows quoting to be used but still maintain the Clojure
(kebab case) to SQL (snake case) mappings.

2
doc/special-syntax.md
View file

@ -198,6 +198,8 @@ that represents a time unit. Produces an `INTERVAL` expression:
;;=> ["DATE_ADD(NOW(), INTERVAL ? DAYS)" 30]
```
> Note: PostgreSQL has an `INTERVAL` data type which is unrelated to this syntax. In PostgreSQL, the closet equivalent would be `[:cast "30 days" :interval]` which will lift `"30 days"` out as a parameter. In DDL, for PostgreSQL, you can use `:interval` to produce the `INTERVAL` data type (without wrapping it in a vector).
## lateral
Accepts a single argument that can be a (`SELECT`) clause or

133
src/honey/sql.cljc
View file

@ -210,13 +210,25 @@
(let [col-fn (if (or *quoted* (string? e))
(if *quoted-snake* name-_ name)
name-_)
quote-fn (if (or *quoted* (string? e)) (:quote *dialect*) identity)
col-e (col-fn e)
dialect-q (:quote *dialect* identity)
quote-fn (cond (or *quoted* (string? e))
dialect-q
;; #422: if default quoting and "unusual"
;; characters in entity, then quote it:
(nil? *quoted*)
(fn opt-quote [part]
(if (re-find #"^[A-Za-z0-9_]+$" part)
part
(dialect-q part)))
:else
identity)
parts (if-let [n (when-not (or drop-ns (string? e))
(namespace-_ e))]
[n (col-fn e)]
[n col-e]
(if aliased
[(col-fn e)]
(str/split (col-fn e) #"\.")))
[col-e]
(str/split col-e #"\.")))
entity (str/join "." (map #(cond-> % (not= "*" %) (quote-fn)) parts))
suspicious #";"]
(when-not *allow-suspicious-entities*
@ -687,59 +699,74 @@
(str " " (sql-kw nowait))))))]))
(defn- format-values [k xs]
(cond (sequential? (first xs))
;; [[1 2 3] [4 5 6]]
(let [n-1 (map count xs)
;; issue #291: ensure all value sequences are the same length
xs' (if (apply = n-1)
xs
(let [n-n (apply max n-1)]
(map (fn [x] (take n-n (concat x (repeat nil)))) xs)))
[sqls params]
(reduce (fn [[sql params] [sqls' params']]
[(conj sql (str "(" (str/join ", " sqls') ")"))
(into params params')])
[[] []]
(map #'format-expr-list xs'))]
(into [(str (sql-kw k) " " (str/join ", " sqls))] params))
(let [first-xs (when (sequential? xs) (first (drop-while ident? xs)))]
(cond (contains? #{:default 'default} xs)
[(str (sql-kw xs) " " (sql-kw k))]
(empty? xs)
[(str (sql-kw k) " ()")]
(sequential? first-xs)
;; [[1 2 3] [4 5 6]]
(let [n-1 (map count (filter sequential? xs))
;; issue #291: ensure all value sequences are the same length
xs' (if (apply = n-1)
xs
(let [n-n (when (seq n-1) (apply max n-1))]
(map (fn [x]
(if (sequential? x)
(take n-n (concat x (repeat nil)))
x))
xs)))
[sqls params]
(reduce (fn [[sql params] [sqls' params']]
[(conj sql
(if (sequential? sqls')
(str "(" (str/join ", " sqls') ")")
sqls'))
(into params params')])
[[] []]
(map #(if (sequential? %)
(format-expr-list %)
[(sql-kw %)])
xs'))]
(into [(str (sql-kw k) " " (str/join ", " sqls))] params))
(map? (first xs))
;; [{:a 1 :b 2 :c 3}]
(let [cols-1 (keys (first xs))
;; issue #291: check for all keys in all maps but still
;; use the keys from the first map if they match so that
;; users can rely on the key ordering if they want to,
;; e.g., see test that uses array-map for the first row
cols-n (into #{} (mapcat keys) xs)
cols (if (= (set cols-1) cols-n) cols-1 cols-n)
[sqls params]
(reduce (fn [[sql params] [sqls' params']]
[(conj sql (str "(" (str/join ", " sqls') ")"))
(if params' (into params params') params')])
[[] []]
(map (fn [m]
(format-expr-list
(map #(get m
%
(map? first-xs)
;; [{:a 1 :b 2 :c 3}]
(let [cols-1 (keys (first xs))
;; issue #291: check for all keys in all maps but still
;; use the keys from the first map if they match so that
;; users can rely on the key ordering if they want to,
;; e.g., see test that uses array-map for the first row
cols-n (into #{} (mapcat keys) xs)
cols (if (= (set cols-1) cols-n) cols-1 cols-n)
[sqls params]
(reduce (fn [[sql params] [sqls' params']]
[(conj sql (str "(" (str/join ", " sqls') ")"))
(if params' (into params params') params')])
[[] []]
(map (fn [m]
(format-expr-list
(map #(get m
%
;; issue #366: use NULL or DEFAULT
;; for missing column values:
(if (contains? *values-default-columns* %)
[:default]
nil))
cols)))
xs))]
(into [(str "("
(str/join ", "
(map #(format-entity % {:drop-ns true}) cols))
") "
(sql-kw k)
" "
(str/join ", " sqls))]
params))
(if (contains? *values-default-columns* %)
[:default]
nil))
cols)))
xs))]
(into [(str "("
(str/join ", "
(map #(format-entity % {:drop-ns true}) cols))
") "
(sql-kw k)
" "
(str/join ", " sqls))]
params))
:else
(throw (ex-info ":values expects sequences or maps"
{:first (first xs)}))))
:else
(throw (ex-info ":values expects sequences or maps"
{:first (first xs)})))))
(comment
(into #{} (mapcat keys) [{:a 1 :b 2} {:b 3 :c 4}])

33
test/honey/sql_test.cljc
View file

@ -737,7 +737,28 @@ ORDER BY id = ? DESC
:values [{:name name
:enabled enabled}]})))))
(deftest issue-425-default-values-test
(testing "default values"
(is (= ["INSERT INTO table (a, b, c) DEFAULT VALUES"]
(format {:insert-into [:table [:a :b :c]] :values :default}))))
(testing "values with default row"
(is (= ["INSERT INTO table (a, b, c) VALUES (1, 2, 3), DEFAULT, (4, 5, 6)"]
(format {:insert-into [:table [:a :b :c]]
:values [[1 2 3] :default [4 5 6]]}
{:inline true}))))
(testing "values with default column"
(is (= ["INSERT INTO table (a, b, c) VALUES (1, DEFAULT, 3), DEFAULT"]
(format {:insert-into [:table [:a :b :c]]
:values [[1 [:default] 3] :default]}
{:inline true}))))
(testing "empty values"
(is (= ["INSERT INTO table (a, b, c) VALUES ()"]
(format {:insert-into [:table [:a :b :c]]
:values []})))))
(deftest issue-316-test
;; this is a pretty naive test -- there are other tricks to perform injection
;; that are not detected by HoneySQL and you should generally use :quoted true
(testing "SQL injection via keyword is detected"
(let [sort-column "foo; select * from users"]
(try
@ -936,3 +957,15 @@ ORDER BY id = ? DESC
(is (= ["SELECT `A\"B`"] (sut/format {:select (keyword "A\"B")} {:dialect :mysql})))
(is (= ["SELECT `A``B`"] (sut/format {:select (keyword "A`B")} {:dialect :mysql})))
(is (= ["SELECT \"A\"\"B\""] (sut/format {:select (keyword "A\"B")} {:dialect :oracle}))))
(deftest issue-422-quoting
;; default quote if strange entity:
(is (= ["SELECT A, \"B C\""] (sut/format {:select [:A (keyword "B C")]})))
;; default don't quote normal entity:
(is (= ["SELECT A, B_C"] (sut/format {:select [:A (keyword "B_C")]})))
;; quote all entities when quoting enabled:
(is (= ["SELECT \"A\", \"B C\""] (sut/format {:select [:A (keyword "B C")]}
{:quoted true})))
;; don't quote if quoting disabled (illegal SQL):
(is (= ["SELECT A, B C"] (sut/format {:select [:A (keyword "B C")]}
{:quoted false}))))