Merge pull request #378 from seancorfield/issue-351

first pass of caching formatter

.github/workflows/test-and-snapshot.yml

@@ -39,7 +39,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        java: [ '8', '14', '15', '16', '17' ]
+        java: [ '8', '14', '17' ]
     steps:
       - uses: actions/checkout@v2
       - uses: actions/setup-java@v2

.github/workflows/test.yml

@@ -7,7 +7,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        java: [ '8', '11', '14', '15', '16', '17' ]
+        java: [ '8', '11', '14', '17' ]
     steps:
       - uses: actions/checkout@v2
       - uses: actions/setup-java@v2

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 * 2.2.next in progress
   * Address #377 by adding `honey.sql/map=` to convert a hash map into an equality condition (for a `WHERE` clause).
+  * Address #351 by adding a `:cache` option to `honey.sql/format` (for Clojure only, not ClojureScript).
   * Address #281 by adding support for `SELECT * EXCEPT ..` and `SELECT * REPLACE ..` and `ARRAY<>` and `STRUCT<>` column types -- see [SQL Clause Reference - SELECT](https://cljdoc.org/d/com.github.seancorfield/honeysql/CURRENT/doc/getting-started/sql-clause-reference#select-select-distinct) and [SQL Clause Reference - DDL](https://cljdoc.org/d/com.github.seancorfield/honeysql/CURRENT/doc/getting-started/sql-clause-reference#ddl-clauses) respectively for more details.
   * Update `build-clj` to v0.6.7.
 

README.md

@@ -39,6 +39,12 @@ HoneySQL 1.x will continue to get critical security fixes but otherwise should b
 
 ## Usage
 
+This section includes a number of usage examples but does not dive deep into the
+way the data structure acts as a DSL that can specify SQL statements (as hash maps)
+and SQL expressions and function calls (as vectors). It is recommended that you read the
+[**Getting Started**](https://cljdoc.org/d/com.github.seancorfield/honeysql/CURRENT/doc/getting-started)
+section of the documentation before trying to use HoneySQL to build your own queries!
+
 From Clojure:
 <!-- {:test-doc-blocks/reader-cond :clj} -->
 ```clojure
@@ -512,10 +518,20 @@ There are also helpers for each of those:
 => ["SELECT * FROM foo UNION SELECT * FROM bar"]
 ```
 
-
 ### Functions
 
-Keywords that begin with `%` are interpreted as SQL function calls:
+Function calls (and expressions with operators) can be specified as
+vectors where the first element is either a keyword or a symbol:
+
+```clojure
+(-> (select :*) (from :foo)
+    (where [:> :date_created [:date_add [:now] [:interval 24 :hours]]])
+    (sql/format))
+=> ["SELECT * FROM foo WHERE date_created > DATE_ADD(NOW(), INTERVAL ? HOURS)" 24]
+```
+
+A shorthand syntax also exists for simple function calls:
+keywords that begin with `%` are interpreted as SQL function calls:
 
 ```clojure
 (-> (select :%count.*) (from :foo) sql/format)
@@ -535,11 +551,27 @@ regular function calls in a select:
 => ["SELECT COUNT(*) FROM foo"]
 ```
 ```clojure
+(-> (select [:%count.*]) (from :foo) sql/format)
+=> ["SELECT COUNT(*) FROM foo"]
+;; or even:
+(-> (select :%count.*) (from :foo) sql/format)
+=> ["SELECT COUNT(*) FROM foo"]
+```
+```clojure
 (-> (select [[:max :id]]) (from :foo) sql/format)
 => ["SELECT MAX(id) FROM foo"]
 ;; the pure data DSL requires an extra level of brackets:
 (-> {:select [[[:max :id]]], :from [:foo]} sql/format)
 => ["SELECT MAX(id) FROM foo"]
+;; the shorthand makes this simpler:
+(-> {:select [[:%max.id]], :from [:foo]} sql/format)
+=> ["SELECT MAX(id) FROM foo"]
+;; or even:
+(-> {:select [:%max.id], :from [:foo]} sql/format)
+=> ["SELECT MAX(id) FROM foo"]
+;; or even:
+(-> {:select :%max.id, :from :foo} sql/format)
+=> ["SELECT MAX(id) FROM foo"]
 ```
 
 ### Bindable parameters

deps.edn

@@ -16,7 +16,8 @@
   :test
   {:extra-paths ["test"]
    :extra-deps  {io.github.cognitect-labs/test-runner
-                 {:git/tag "v0.5.0" :git/sha "48c3c67"}}
+                 {:git/tag "v0.5.0" :git/sha "48c3c67"}
+                 org.clojure/core.cache {:mvn/version "RELEASE"}}
    :exec-fn     cognitect.test-runner.api/test}
 
   ;; various "runners" for tests/CI:

doc/general-reference.md

@@ -122,6 +122,32 @@ are two possible approaches:
 1. Use named parameters (e.g., `[:param :myval]`) instead of having the values directly in the DSL structure and then pass `{:params {:myval some-json}}` as part of the options in the call to `format`, or
 2. Use `[:lift ..]` wrapped around any structured values which tells HoneySQL not to interpret the vector or hash map value as a DSL: `[:lift some-json]`.
 
+## Caching
+
+As of 2.2.next, `format` can cache the SQL and parameters produced from the data structure so that it does not need to be computed on every call. This functionality is available only in Clojure and depends on [`org.clojure/core.cache`](https://github.com/clojure/core.cache) being on your classpath. If you are repeatedly building the same complex SQL statements over and over again, this can be a good way to provide a performance boost but there are some caveats.
+
+* You need `core.cache` as a dependency: `org.clojure/core.cache {:mvn/version "1.0.225"}` was the latest as of January 20th, 2022,
+* You need to create one or more caches yourself, from the various factory functions in the [`clojure.core.cache.wrapped` namespace](http://clojure.github.io/core.cache/#clojure.core.cache.wrapped),
+* You should use named parameters in your SQL DSL data structure, e.g., `:?foo` or `'?foo`, and pass the actual parameter values via the `:params` option to `format`.
+
+You can then pass the (atom containing the) cache to `format` using the `:cache` option. The call to `format` then looks in that cache for a match for the data structure passed in, i.e., the entire data structure is used as a key into the cache, including any literal parameter values. If the cache contains a match, the corresponding vector of a SQL string and parameters is used, otherwise the data structure is parsed as usual and the SQL string (and parameters) generated from it (and stored in the cache for the next call). Finally, named parameters in the vector are replaced by their values from the `:params` option.
+
+The code that _builds_ the DSL data structure will be run in all cases, so any conditional logic and helper function calls will still happen, since that is how the data structure is created and then passed to `format`. If you want to avoid overhead, you'd need to take steps to build the data structure separately and store it somewhere for reuse in the call to `format`.
+
+Since the data structure is used as the key into the cache, literal parameter values will lead to different keys:
+
+<!-- :test-doc-blocks/skip -->
+```clojure
+;; these are two different cache entries:
+(sql/format {:select :* :from :table :where [:= :id 1]} {:cache my-cache})
+(sql/format {:select :* :from :table :where [:= :id 2]} {:cache my-cache})
+;; these are the same cache entry:
+(sql/format {:select :* :from :table :where [:= :id :?id]} {:cache my-cache :params {:id 1}})
+(sql/format {:select :* :from :table :where [:= :id :?id]} {:cache my-cache :params {:id 2}})
+```
+
+Since HoneySQL accepts any of the `clojure.core.cache.wrapped` caches and runs every data structure through the provided `:cache`, it's up to you to ensure that your cache is appropriate for that usage: a "basic" cache will keep every entry until the cache is explicitly emptied; a TTL cache will keep each entry for a specific period of time; and so on.
+
 ## Other Sections Will Be Added!
 
 As questions arise about the use of HoneySQL 2.x, I will add new sections here.

doc/getting-started.md

@@ -349,6 +349,9 @@ HoneySQL supports quite a few [PostgreSQL extensions](postgresql.md).
 
 In addition to the `:quoted` and `:dialect` options described above,
 `format` also accepts `:checking`, `:inline`, and `:params`.
+As of 2.2.next, `format` accepts a `:cache` option -- see the
+[**Caching** section of the **General Reference**](https://cljdoc.org/d/com.github.seancorfield/honeysql/CURRENT/doc/getting-started/general-reference#caching)
+for details.
 
 The `:params` option was mentioned above and is used to specify
 the values of named parameters in the DSL.

src/honey/sql.cljc

@@ -1,4 +1,4 @@
-;; copyright (c) 2020-2021 sean corfield, all rights reserved
+;; copyright (c) 2020-2022 sean corfield, all rights reserved
 
 (ns honey.sql
   "Primary API for HoneySQL 2.x.
@@ -1361,6 +1361,23 @@
                     {:valid-dialects (vec (sort (keys dialects)))})))
   dialect)
 
+(def through-opts
+  "If org.clojure/core.cache is available, resolves to a function that
+  calls core.cache.wrapped/lookup-or-miss, otherwise to a function that
+  throws an exception.
+
+  In ClojureScript, a resolves to a function that throws an exception
+  because core.cache relies on JVM machinery and is Clojure-only."
+  #?(:clj (try (require 'clojure.core.cache.wrapped)
+               (let [lookup-or-miss (deref (resolve 'clojure.core.cache.wrapped/lookup-or-miss))]
+                 (fn [_opts cache data f]
+                   (lookup-or-miss cache data f)))
+               (catch Throwable _
+                 (fn [opts _cache _data _f]
+                   (throw (ex-info "include core.cached on the classpath to use the :cache option" opts)))))
+     :cljs (fn [opts _cache _data _f]
+             (throw (ex-info "cached queries are not supported in ClojureScript" opts)))))
+
 (defn format
   "Turn the data DSL into a vector containing a SQL string followed by
   any parameter values that were encountered in the DSL structure.
@@ -1374,7 +1391,8 @@
   as named arguments followed by other options in a hash map."
   ([data] (format data {}))
   ([data opts]
-   (let [dialect? (contains? opts :dialect)
+   (let [cache    (:cache opts)
+         dialect? (contains? opts :dialect)
          dialect  (when dialect? (get dialects (check-dialect (:dialect opts))))]
      (binding [*dialect* (if dialect? dialect @default-dialect)
                *checking* (if (contains? opts :checking)
@@ -1397,7 +1415,10 @@
                                 (:quoted-snake opts))
                *params* (:params opts)
                *values-default-columns* (:values-default-columns opts)]
-       (mapv #(unwrap % opts) (format-dsl data opts)))))
+       (if cache
+         (->> (through-opts opts cache data (fn [_] (format-dsl data (dissoc opts :cache))))
+              (mapv #(unwrap % opts)))
+         (mapv #(unwrap % opts) (format-dsl data opts))))))
   ([data k v & {:as opts}] (format data (assoc opts k v))))
 
 (defn set-dialect!

test/honey/cache_test.clj

@@ -0,0 +1,92 @@
+;; copyright (c) 2022 sean corfield, all rights reserved
+
+(ns honey.cache-test
+  (:refer-clojure :exclude [format group-by])
+  (:require [clojure.core.cache.wrapped :as cache]
+            [clojure.test :refer [deftest is]]
+            [honey.sql :as sut]
+            [honey.sql.helpers
+             :refer [select-distinct from join left-join right-join where
+                     group-by having order-by limit offset]]))
+
+(def big-complicated-map
+  (-> (select-distinct :f.* :b.baz :c.quux [:b.bla "bla-bla"]
+                       [[:now]] [[:raw "@x := 10"]])
+      (from [:foo :f] [:baz :b])
+      (join :draq [:= :f.b :draq.x]
+            :eldr [:= :f.e :eldr.t])
+      (left-join [:clod :c] [:= :f.a :c.d])
+      (right-join :bock [:= :bock.z :c.e])
+      (where [:or
+              [:and [:= :f.a "bort"] [:not= :b.baz [:param :param1]]]
+              [:and [:< 1 2] [:< 2 3]]
+              [:in :f.e [1 [:param :param2] 3]]
+              [:between :f.e 10 20]])
+      (group-by :f.a :c.e)
+      (having [:< 0 :f.e])
+      (order-by [:b.baz :desc] :c.quux [:f.a :nulls-first])
+      (limit 50)
+      (offset 10)))
+
+(defn- cache-size [cache] (-> cache (deref) (keys) (count)))
+
+(deftest cache-tests
+  (let [cache (cache/basic-cache-factory {})]
+    (is (zero? (cache-size cache)))
+    (is (= ["SELECT * FROM table WHERE id = ?" 1]
+           (sut/format {:select [:*] :from [:table] :where [:= :id 1]}
+                       {:cache cache})
+           (sut/format {:select [:*] :from [:table] :where [:= :id 1]}
+                       {:cache cache})))
+    (is (= 1 (cache-size cache)))
+    (is (= (sut/format {:select [:*] :from [:table] :where [:= :id 2]})
+           (sut/format {:select [:*] :from [:table] :where [:= :id 2]}
+                       {:cache cache})))
+    (is (= 2 (cache-size cache)))
+    (is (= (sut/format big-complicated-map {:params {:param1 "gabba" :param2 2}})
+           (sut/format big-complicated-map {:cache cache :params {:param1 "gabba" :param2 2}})
+           (sut/format big-complicated-map {:cache cache :params {:param1 "gabba" :param2 2}})))
+    (is (= 3 (cache-size cache)))
+    (is (= (sut/format big-complicated-map {:params {:param1 "foo" :param2 42}})
+           (sut/format big-complicated-map {:cache cache :params {:param1 "foo" :param2 42}})
+           (sut/format big-complicated-map {:cache cache :params {:param1 "foo" :param2 42}})))
+    (is (= 3 (cache-size cache)))
+    (println "Uncached, simple, embedded")
+    (time (dotimes [_ 100000]
+            (sut/format {:select [:*] :from [:table] :where [:= :id (rand-int 10)]})))
+    (println "Cached, simple, embedded")
+    (time (dotimes [_ 100000]
+            (sut/format {:select [:*] :from [:table] :where [:= :id (rand-int 10)]} {:cache cache})))
+    (is (= 11 (cache-size cache)))
+    (println "Uncached, complex, mixed")
+    (time (dotimes [_ 10000]
+            (sut/format big-complicated-map {:params {:param1 "gabba" :param2 (rand-int 10)}})))
+    (println "Cached, complex, mixed")
+    (time (dotimes [_ 10000]
+            (sut/format big-complicated-map {:cache cache :params {:param1 "gabba" :param2 (rand-int 10)}})))
+    (is (= 11 (cache-size cache))))
+  (let [cache (cache/basic-cache-factory {})]
+    (is (zero? (cache-size cache)))
+    (is (= ["SELECT * FROM table WHERE id = ?" 1]
+           (sut/format {:select [:*] :from [:table] :where [:= :id :?id]}
+                       {:cache cache :params {:id 1}})
+           (sut/format {:select [:*] :from [:table] :where [:= :id :?id]}
+                       {:cache cache :params {:id 1}})))
+    (is (= 1 (cache-size cache)))
+    (is (= (sut/format {:select [:*] :from [:table] :where [:= :id :?id]}
+                       {:params {:id 2}})
+           (sut/format {:select [:*] :from [:table] :where [:= :id :?id]}
+                       {:cache cache :params {:id 2}})))
+    (is (= 1 (cache-size cache)))
+    ;; different parameter names create different cache entries:
+    (is (= (sut/format {:select [:*] :from [:table] :where [:= :id :?x]}
+                       {:cache cache :params {:x 2}})
+           (sut/format {:select [:*] :from [:table] :where [:= :id :?y]}
+                       {:cache cache :params {:y 2}})))
+    (is (= 3 (cache-size cache)))
+    ;; swapping parameter names creates different cache entries:
+    (is (= (sut/format {:select [:*] :from [:table] :where [:and [:= :id :?x] [:= :foo :?y]]}
+                       {:cache cache :params {:x 2 :y 3}})
+           (sut/format {:select [:*] :from [:table] :where [:and [:= :id :?y] [:= :foo :?x]]}
+                       {:cache cache :params {:x 3 :y 2}})))
+    (is (= 5 (cache-size cache)))))

test/honey/sql_test.cljc

@@ -1,4 +1,4 @@
-;; copyright (c) 2021 sean corfield, all rights reserved
+;; copyright (c) 2021-2022 sean corfield, all rights reserved
 
 (ns honey.sql-test
   (:refer-clojure :exclude [format])