honeysql/doc/differences-from-1-x.md

Differences Between 1.x and 2.x

The goal of HoneySQL 1.x and earlier was to provide a DSL for vendor-neutral SQL, with the assumption that other libraries would provide the vendor-specific extensions to HoneySQL. HoneySQL 1.x's extension mechanism required quite a bit of internal knowledge (clause priorities and multiple multimethod extension points). It also used a number of custom record types, protocols, and data readers to provide various "escape hatches" in the DSL for representing arrays, function calls (in some situations), inlined values, parameters, and raw SQL, which led to a number of inconsistencies over time, as well as making some things very hard to express while other similar things were easy to express. Addressing bugs caused by vendor-specific differences and by some quirks of how SQL was generated gradually became harder and harder.

The goal of HoneySQL 2.x is to provide an easily-extensible DSL for SQL, supporting vendor-specific differences and extensions, that is as consistent as possible. A secondary goal is to make maintenance much easier by streamlining the machinery and reducing the number of different ways to write and/or extend the DSL.

The DSL itself -- the data structures that both versions convert to SQL and parameters via the format function -- is almost exactly the same between the two versions so that migration is relatively painless. The primary API -- the format function -- is preserved in 2.x, although the options have changed between 1.x and 2.x. See the Option Changes section below for the differences in the options supported. format can accept its options as a single hash map or as named arguments (1.x only supported the latter). If you are using Clojure 1.11, you can invoke format with a mixture of named arguments and a trailing hash map of additional options, if you wish.

HoneySQL 1.x supported Clojure 1.7 and later. HoneySQL 2.7.y requires Clojure 1.10.3 or later. Earlier versions of HoneySQL 2.x support Clojure 1.9.0.

Group, Artifact, and Namespaces

HoneySQL 2.x uses the group ID com.github.seancorfield with the original artifact ID of honeysql, in line with the recommendations in Inside Clojure's post about the changes in the Clojure CLI: Deprecated unqualified lib names; also Clojars Verified Group Names policy.

In addition, HoneySQL 2.x contains different namespaces so you can have both versions on your classpath without introducing any conflicts. The primary API is now in honey.sql and the helpers are in honey.sql.helpers.

HoneySQL 1.x

In deps.edn:

honeysql {:mvn/version "1.0.461"}
;; or, more correctly:
honeysql/honeysql {:mvn/version "1.0.461"}

Required as:

(ns my.project
  (:require [honeysql.core :as sql]))

Or if in the REPL:

(require '[honeysql.core :as sq])

In use:

  (sql/format {:select [:*] :from [:table] :where [:= :id 1]})
  ;;=> ["SELECT * FROM table WHERE id = ?" 1]
  (sql/format {:select [:*] :from [:table] :where [:= :id 1]} :quoting :mysql)
  ;;=> ["SELECT * FROM `table` WHERE `id` = ?" 1]

The namespaces were:

• honeysql.core -- the primary API (format, etc),
• honeysql.format -- the logic for the formatting engine,
• honeysql.helpers -- helper functions to build the DSL,
• honeysql.types -- records, protocols, and data readers,
• honeysql.util -- internal utilities (macros).

Supported Clojure versions: 1.7 and later.

HoneySQL 2.x

In deps.edn:

com.github.seancorfield/honeysql {:mvn/version "2.7.1295"}

Required as:

(ns my.project
  (:require [honey.sql :as sql]))

Or if in the REPL:

(require '[honey.sql :as sql])

In use:

  (sql/format {:select [:*] :from [:table] :where [:= :id 1]})
  ;;=> ["SELECT * FROM table WHERE id = ?" 1]
  (sql/format {:select [:*] :from [:table] :where [:= :id 1]} {:dialect :mysql})
  ;;=> ["SELECT * FROM `table` WHERE `id` = ?" 1]

The new namespaces are:

• honey.sql -- the primary API (just format now),
• honey.sql.helpers -- helper functions to build the DSL.

Supported Clojure versions: 1.10.3 and later.

API Changes

The primary API is just honey.sql/format. The array, call, inline, param, and raw functions have all become standard syntax in the DSL as functions (and their tagged literal equivalents have also gone away because they are no longer needed). [As of 2.0.0-rc3, call has been reinstated as an undocumented function in honey.sql purely to aid migration from 1.x]

Other honeysql.core functions that no longer exist include: build, qualify, and quote-identifier. Many other public functions were essentially undocumented (neither mentioned in the README nor in the tests) and also no longer exist.

As of 2.4.1002, the functionality of qualify can be achieved through the :. dot-selection special syntax.

You can now select a non-ANSI dialect of SQL using the new honey.sql/set-dialect! function (which sets a default dialect for all format operations) or by passing the new :dialect option to the format function. :ansi is the default dialect (which will mostly incorporate PostgreSQL usage over time). Other dialects supported are :mysql (which has a different quoting strategy and uses a different ranking for the :set clause), :oracle (which is essentially the :ansi dialect but will control other things over time), and :sqlserver (which is essentially the :ansi dialect but with a different quoting strategy). Other dialects and changes may be added over time.

Note: in general, all clauses are available in all dialects in HoneySQL unless the syntax of the clauses conflict between dialects (currently, no such clauses exist). The :mysql dialect is the only one so far that changes the priority ordering of a few clauses.

Option Changes

The :quoting <dialect> option has been superseded by the new dialect machinery and a new :quoted option that turns quoting on or off. You either use :dialect <dialect> instead (which turns on quoting by default) or set a default dialect (via set-dialect!) and then use :quoted true in format calls where you want quoting.

SQL entity names are automatically quoted if you specify a :dialect option to format, unless you also specify :quoted false.

The following options are no longer supported:

• :allow-dashed-names? -- if you provide dashed-names in 2.x, they will be left as-is if quoting is enabled, else they will be converted to snake_case (so you will either get "dashed-names" with quoting or dashed_names without). If you want dashed-names to be converted to snake_case when :quoted true, you also need to specify :quoted-snake true.
• :allow-namespaced-names? -- this supported foo/bar column names in SQL which I'd like to discourage.
• :namespace-as-table? -- this is the default in 2.x: :foo/bar will be treated as foo.bar which is more in keeping with next.jdbc.
• :parameterizer -- this would add a lot of complexity to the formatting engine and I do not know how widely it was used (especially in its arbitrarily extensible form). [As of 2.4.962, the ability to generated SQL with numbered parameters, i.e., $1 instead of positional parameters, ?, has been added via the :numbered true option]
• :return-param-names -- this was added to 1.x back in 2013 without an associated issue or PR so I've no idea what use case this was intended to support.

Note: I expect some push back on those first three options and the associated behavior changes.

DSL Changes

The general intent is that the data structure behind the DSL is unchanged, for the most part. The main deliberate change is the removal of the reader literals (and their associated helper functions) in favor of standardized syntax, e.g., [:array [1 2 3]] instead of either #sql/array [1 2 3] or (sql/array [1 2 3]).

The following new syntax has been added:

• :array -- used as a function to replace the sql/array / #sql/array machinery,
• :between -- this is now explicit syntax rather than being a special case in expressions,
• :case -- this is now explicit syntax,
• :cast -- [:cast expr :type] => CAST( expr AS type ),
• :composite -- explicit syntax to produce a comma-separated list of expressions, wrapped in parentheses,
• :default -- for DEFAULT values (in inserts) and for declaring column defaults in table definitions,
• :escape -- used to wrap a regular expression so that non-standard escape characters can be provided,
• :inline -- used as a function to replace the sql/inline / #sql/inline machinery,
• :interval -- used as a function to support INTERVAL <n> <units>, e.g., [:interval 30 :days] for databases that support it (e.g., MySQL) and, as of 2.4.1026, for INTERVAL 'n units', e.g., [:interval "24 hours"] for ANSI/PostgreSQL.
• :lateral -- used to wrap a statement or expression, to provide a LATERAL join,
• :lift -- used as a function to prevent interpretation of a Clojure data structure as DSL syntax (e.g., when passing a vector or hash map as a parameter value) -- this should mostly be a replacement for honeysql.format/value,
• :nest -- used as a function to add an extra level of nesting (parentheses) around an expression,
• :not -- this is now explicit syntax,
• :over -- the function-like part of a T-SQL window clause,
• :param -- used as a function to replace the sql/param / #sql/param machinery,
• :raw -- used as a function to replace the sql/raw / #sql/raw machinery. Vector subexpressions inside a [:raw ..] expression are formatted to SQL and parameters. Other subexpressions are just turned into strings and concatenated. This is different to the 1.x behavior but should be more flexible, since you can now embed :inline, :param, and :lift inside a :raw expression.

Note 1: in 1.x, inlining a string "foo" produced foo but in 2.x it produces 'foo', i.e., string literals become SQL strings without needing internal quotes (1.x required "'foo'").

Several additional pieces of syntax have also been added to support column definitions in CREATE TABLE clauses, now that 2.x supports DDL statement construction:

• :constraint, :default, :foreign-key, :index, :primary-key, :references, :unique,
• :entity -- used to force an expression to be rendered as a SQL entity (instead of a SQL keyword).

select and function calls

You can now SELECT a function call more easily, using [[...]]. This was previously an error -- missing an alias -- but it was a commonly requested change, to avoid using (sql/call ...):

user=> (sql/format {:select [:a [:b :c] [[:d :e]] [[:f :g] :h]]})
;; select a (column), b (aliased to c), d (fn call), f (fn call, aliased to h):
["SELECT a, b AS c, D(e), F(g) AS h"]

On a related note, sql/call has been removed because it should never be needed now: [:foo ...] should always be treated as a function call, consistently, avoiding the special cases in 1.x that necessitated the explicit sql/call syntax.

select modifiers

HoneySQL 1.x provided a :modifiers clause (and a modifiers) helper as a way to "modify" a SELECT to be DISTINCT. The nilenso/honeysql-helpers library extended that to support :distinct-on a group of columns. In HoneySQL 2.x, you use :select-distinct and :select-distinct-on (and their associated helpers) for that instead. MS SQL Server's TOP modifier is also supported via :select-top and :select-distinct-top.

set vs sset, set0, set1

The :set clause is dialect-dependent. In :mysql, it is ranked just before the :where clause. In all other dialects, it is ranked just before the :from clause. Accordingly, the :set0 and :set1 clauses are no longer supported (because they were workarounds in 1.x for this conflict). The helper is now called set rather than sset, set0, and set1 (so be aware of the conflict with clojure.core/set).

exists

HoneySQL 1.x implemented :exists as part of the DSL, which was incorrect: it should have been a function, and in 2.x it is:

;; 1.x: EXISTS should never have been implemented as SQL syntax: it's an operator!
;; (sq/format {:exists {:select [:a] :from [:foo]}})
;; -> ["EXISTS (SELECT a FROM foo)"]

;; 2.x: select function call with an alias:
user=> (sql/format {:select [[[:exists {:select [:a] :from [:foo]}] :x]]})
["SELECT EXISTS (SELECT a FROM foo) AS x"]

ORDER BY with NULLS FIRST or NULLS LAST

In HoneySQL 1.x, if you wanted to generate SQL like

ORDER BY ... DESC NULLS LAST

you needed to pass :nulls-last as a separate keyword, after :asc or :desc:

{:order-by [[:my-column :desc :nulls-last]]}

In HoneySQL 2.x, the direction and the null ordering rule are now combined into a single keyword:

{:order-by [[:my-column :desc-nulls-last]]}

Extensibility

The protocols and multimethods in 1.x have all gone away. The primary extension point is honey.sql/register-clause! which lets you specify the new clause (keyword), the formatter function for it, and the existing clause that it should be ranked before (format processes the DSL in clause order).

You can also register new "functions" that can implement special syntax (such as :array, :inline, :raw etc above) via honey.sql/register-fn!. This accepts a "function" name as a keyword and a formatter which will generally be a function of two arguments: the function name (so formatters can be reused across different names) and a vector of the arguments the function should accept.

And, finally, you can register new operators that will be recognized in expressions via honey.sql/register-op!. This accepts an operator name as a keyword and an optional named parameter to indicate whether it should ignore operands that evaluate to nil (via :ignore-nil). That can make it easier to construct complex expressions programmatically without having to worry about conditionally removing "optional" (nil) values.

Note: because of the changes in the extension machinery between 1.x and 2.x, it is not possible to use the nilenso/honeysql-postgress library with HoneySQL 2.x but the goal is to incorporate all of the syntax from that library into the core of HoneySQL.

Helpers

The honey.sql.helpers namespace includes a helper function that corresponds to every supported piece of the data DSL understood by HoneySQL (1.x only had a limited set of helper functions). Unlike 1.x helpers which sometimes had both a regular helper and a merge- helper, 2.x helpers will all merge clauses by default (if that makes sense for the underlying DSL): use :dissoc if you want to force an overwrite.

The only helpers that have non-merging behavior are:

• The SQL set operations intersect, union, union-all, except, and except-all which always wrap around their arguments,
• The SQL clauses delete, fetch, for, limit, lock, offset, on-constraint, set, truncate, update, and values which overwrite, rather than merge,
• The DDL helpers drop-column, drop-index, rename-table, and with-data,
• The function helper composite which is a convenience for the :composite syntax mentioned above: (composite :a :b) is the same as [:composite :a :b] which produces (a, b).