Fix #157 by adding next.jdbc/execute-batch!

This breaks the circular dependency completely.
`next.jdbc.prepare/execute-batch!` is deprecated.

CHANGELOG.md

@@ -5,7 +5,7 @@ Only accretive/fixative changes will be made from now on.
 ## Stable Builds
 
 * 1.1.next in progress
-  * Address #157 by using a `volatile!` as a way to break the circular dependency (code that requires `next.jdbc.prepare` and uses `execute-batch!` without also requiring something that causes `next.jdbc.result-set` to be loaded will no longer return generated keys from `execute-batch!` but that's an almost impossible path since nearly all code that uses `execute-batch!` will have called `next.jdbc/prepare` to get the `PreparedStatement` in the first place). _[I'm leaning toward adding `execute-batch!` to `next.jdbc` and deprecating the one in `next.jdbc.prepare`]_
+  * Fix #157 by copying `next.jdbc.prepare/execute-batch!` to `next.jdbc/execute-batch!` (to avoid a circular dependency that previously relied on requiring `next.jdbc.result-set` at runtime -- which was problematic for GraalVM-based native compilation); **`next.jdbc.prepare/execute-batch!` is deprecated:** it will continue to exist and work, but is no longer documented. In addition, `next.jdbc.prepare/execute-batch!` now relies on a private `volatile!` in order to reference `next.jdbc.result-set/datafiable-result-set` so that it is GraalVM-friendly. Note: code that requires `next.jdbc.prepare` and uses `execute-batch!` without also requiring something that causes `next.jdbc.result-set` to be loaded will no longer return generated keys from `execute-batch!` but that's an almost impossible path since nearly all code that uses `execute-batch!` will have called `next.jdbc/prepare` to get the `PreparedStatement` in the first place.
 
 * 1.1.613 -- 2020-11-05
   * Fix #144 by ensuring `camel-snake-case` is properly required before use in an uberjar context.
@@ -179,7 +179,7 @@ Only accretive/fixative changes will be made from now on.
   * Fix #43 by adjusting the spec for `insert-multi!` to "require less" of the `cols` and `rows` arguments.
   * Fix #42 by adding specs for `execute-batch!` and `set-parameters` in `next.jdbc.prepare`.
   * Fix #41 by improving docstrings and documentation, especially around prepared statement handling.
-  * Fix #40 by adding `next.jdbc.prepare/execute-batch!`.
+  * Fix #40 by adding `next.jdbc/execute-batch!` (previously `next.jdbc.prepare/execute-batch!`).
   * Added `assert`s in `next.jdbc.sql` as more informative errors for cases that would generate SQL exceptions (from malformed SQL).
   * Added spec for `:order-by` to reflect what is actually permitted.
   * Expose `next.jdbc.connect/dbtypes` as a table of known database types and aliases, along with their class name(s), port, and other JDBC string components.

doc/all-the-options.md

@@ -87,7 +87,7 @@ Not all databases or drivers support all of these options, or all values for any
 
 > Note: If `plan`, `execute!`, or `execute-one!` are passed a `DataSource`, a "db spec" hash map, or a JDBC URL string, they will call `prepare` to create a `PreparedStatement`, so they will accept the above options in those cases.
 
-In addition to the above, `next.jdbc.prepare/execute-batch!` (which does **not** create a `PreparedStatement`) accepts an options hash map that can also contain the following:
+In addition to the above, `next.jdbc/execute-batch!` (which does **not** create a `PreparedStatement`) accepts an options hash map that can also contain the following:
 
 * `:batch-size` -- an integer that determines how to partition the parameter groups for submitting to the database in batches,
 * `:large` -- a Boolean flag that indicates whether the batch will produce large update counts (`long` rather than `int` values),

doc/friendly-sql-functions.md

@@ -60,7 +60,7 @@ database. Several databases have a limit of 1,000 parameter placeholders.
 Oracle does not support this form of multi-row insert, requiring a different
 syntax altogether.
 
-You should look at [`next.jdbc.prepare/execute-batch!`](https://cljdoc.org/d/seancorfield/next.jdbc/CURRENT/api/next.jdbc.prepare#execute-batch!) for an alternative approach.
+You should look at [`next.jdbc/execute-batch!`](https://cljdoc.org/d/seancorfield/next.jdbc/CURRENT/api/next.jdbc#execute-batch!) for an alternative approach.
 
 ## `query`
 

doc/prepared-statements.md

@@ -93,12 +93,12 @@ Here we set parameters and add them in batches to the prepared statement, then w
   (.executeBatch ps)) ; returns int[]
 ```
 
-Both of those are somewhat ugly and contain a fair bit of boilerplate and Java interop, so a helper function is provided in `next.jdbc.prepare` to automate the execution of batched parameters:
+Both of those are somewhat ugly and contain a fair bit of boilerplate and Java interop, so a helper function is provided in `next.jdbc` to automate the execution of batched parameters:
 
 ```clojure
 (with-open [con (jdbc/get-connection ds)
             ps  (jdbc/prepare con ["insert into status (id,name) values (?,?)"])]
-  (p/execute-batch! ps [[1 "Approved"] [2 "Rejected"] [3 "New"]]))
+  (jdbc/execute-batch! ps [[1 "Approved"] [2 "Rejected"] [3 "New"]]))
 ```
 
 By default, this adds all the parameter groups and executes one batched command. It returns a (Clojure) vector of update counts (rather than `int[]`). If you provide an options hash map, you can specify a `:batch-size` and the parameter groups will be partitioned and executed as multiple batched commands. This is intended to allow very large sequences of parameter groups to be executed without running into limitations that may apply to a single batched command. If you expect the update counts to be very large (more than `Integer/MAX_VALUE`), you can specify `:large true` so that `.executeLargeBatch` is called instead of `.executeBatch`.
@@ -114,8 +114,8 @@ If you want to get the generated keys from an `insert` done via `execute-batch!`
                               {:return-keys true})]
   ;; this will call .getGeneratedKeys for each batch and return them as a
   ;; vector of datafiable result sets (the keys in map are database-specific):
-  (p/execute-batch! ps [[1 "Approved"] [2 "Rejected"] [3 "New"]]
-                    {:return-generated-keys true}))
+  (jdbc/execute-batch! ps [[1 "Approved"] [2 "Rejected"] [3 "New"]]
+                       {:return-generated-keys true}))
 ```
 
 This calls `rs/datafiable-result-set` behind the scenes so you can also pass a `:builder-fn` option to `execute-batch!` if you want something other than qualified as-is hash maps.

doc/tips-and-tricks.md

@@ -101,7 +101,7 @@ It's also worth noting that column comparisons are case-insensitive so `WHERE fo
 
 ### Batch Statements
 
-Even when using `next.jdbc.prepare/execute-batch!`, MySQL will still send multiple statements to the database unless you specify `:rewriteBatchedStatements true` as part of the db-spec hash map or JDBC URL when the datasource is created.
+Even when using `next.jdbc/execute-batch!`, MySQL will still send multiple statements to the database unless you specify `:rewriteBatchedStatements true` as part of the db-spec hash map or JDBC URL when the datasource is created.
 
 ### Streaming Result Sets
 
@@ -125,7 +125,7 @@ What does this mean for your use of `next.jdbc`? In `plan`, `execute!`, and `exe
 
 ### Batch Statements
 
-Even when using `next.jdbc.prepare/execute-batch!`, PostgreSQL will still send multiple statements to the database unless you specify `:reWriteBatchedInserts true` as part of the db-spec hash map or JDBC URL when the datasource is created.
+Even when using `next.jdbc/execute-batch!`, PostgreSQL will still send multiple statements to the database unless you specify `:reWriteBatchedInserts true` as part of the db-spec hash map or JDBC URL when the datasource is created.
 
 ### Streaming Result Sets
 
@@ -168,7 +168,7 @@ create table example(
 ;; => #:example{:tags ["tag1" "tag2"]}
 ```
 
-> Note: PostgreSQL JDBC driver supports only 7 primitive array types, but not array types like `UUID[]` -  
+> Note: PostgreSQL JDBC driver supports only 7 primitive array types, but not array types like `UUID[]` -
 [PostgreSQL™ Extensions to the JDBC API](https://jdbc.postgresql.org/documentation/head/arrays.html).
 
 ### Working with Date and Time

src/next/jdbc.clj

@@ -26,6 +26,8 @@
       return a hash map representing that row; this can be datafied to allow
       navigation of foreign keys into other tables (either by convention or
       via a schema definition),
+  * `execute-batch!` -- given a `PreparedStatement` and groups of parameters,
+      execute the statement in batch mode (via `.executeBatch`).
   * `prepare` -- given a `Connection` and SQL + parameters, construct a new
       `PreparedStatement`; in general this should be used with `with-open`,
   * `transact` -- the functional implementation of `with-transaction`,
@@ -59,10 +61,11 @@
   * `:return-keys` -- either `true` or a vector of key names to return."
   (:require [next.jdbc.connection]
             [next.jdbc.default-options :as opts]
-            [next.jdbc.prepare]
+            [next.jdbc.prepare :as prepare]
             [next.jdbc.protocols :as p]
-            [next.jdbc.result-set]
-            [next.jdbc.transaction]))
+            [next.jdbc.result-set :as rs]
+            [next.jdbc.transaction])
+  (:import (java.sql PreparedStatement)))
 
 (set! *warn-on-reflection* true)
 
@@ -260,6 +263,59 @@
    (p/-execute-one connectable sql-params
                    (assoc opts :next.jdbc/sql-params sql-params))))
 
+(defn execute-batch!
+  "Given a `PreparedStatement` and a vector containing parameter groups,
+  i.e., a vector of vector of parameters, use `.addBatch` to add each group
+  of parameters to the prepared statement (via `set-parameters`) and then
+  call `.executeBatch`. A vector of update counts is returned.
+
+  An options hash map may also be provided, containing `:batch-size` which
+  determines how to partition the parameter groups for submission to the
+  database. If omitted, all groups will be submitted as a single command.
+  If you expect the update counts to be larger than `Integer/MAX_VALUE`,
+  you can specify `:large true` and `.executeLargeBatch` will be called
+  instead.
+
+  By default, returns a Clojure vector of update counts. Some databases
+  allow batch statements to also return generated keys and you can attempt that
+  if you ensure the `PreparedStatement` is created with `:return-keys true`
+  and you also provide `:return-generated-keys true` in the options passed
+  to `execute-batch!`. Some databases will only return one generated key
+  per batch, some return all the generated keys, some will throw an exception.
+  If that is supported, `execute-batch!` will return a vector of hash maps
+  containing the generated keys as fully-realized, datafiable result sets,
+  whose content is database-dependent.
+
+  May throw `java.sql.BatchUpdateException` if any part of the batch fails.
+  You may be able to call `.getUpdateCounts` on that exception object to
+  get more information about which parts succeeded and which failed.
+
+  For additional caveats and database-specific options you may need, see:
+  https://cljdoc.org/d/seancorfield/next.jdbc/CURRENT/doc/getting-started/prepared-statements#caveats
+
+  Not all databases support batch execution."
+  ([ps param-groups]
+   (execute-batch! ps param-groups {}))
+  ([^PreparedStatement ps param-groups opts]
+   (let [params (if-let [n (:batch-size opts)]
+                  (if (and (number? n) (pos? n))
+                    (partition-all n param-groups)
+                    (throw (IllegalArgumentException.
+                            ":batch-size must be positive")))
+                  [param-groups])]
+     (into []
+           (mapcat (fn [group]
+                     (run! #(.addBatch (prepare/set-parameters ps %)) group)
+                     (let [result (if (:large opts)
+                                    (.executeLargeBatch ps)
+                                    (.executeBatch ps))]
+                       (if (:return-generated-keys opts)
+                         (rs/datafiable-result-set (.getGeneratedKeys ps)
+                                                   (p/get-connection ps {})
+                                                   opts)
+                         result))))
+           params))))
+
 (defn transact
   "Given a transactable object and a function (taking a `Connection`),
   execute the function over the connection in a transactional manner.

src/next/jdbc/prepare.clj

@@ -7,9 +7,6 @@
   `set-parameters` is public and may be useful if you have a `PreparedStatement`
   that you wish to reuse and (re)set the parameters on it.
 
-  `execute-batch!` provides a way to add batches of parameters to a
-  `PreparedStatement` and then execute it in batch mode (via `.executeBatch`).
-
   Defines the `SettableParameter` protocol for converting Clojure values
   to database-specific values.
 
@@ -192,37 +189,8 @@
 
 (def ^:private d-r-s (volatile! nil))
 
-(defn execute-batch!
-  "Given a `PreparedStatement` and a vector containing parameter groups,
-  i.e., a vector of vector of parameters, use `.addBatch` to add each group
-  of parameters to the prepared statement (via `set-parameters`) and then
-  call `.executeBatch`. A vector of update counts is returned.
-
-  An options hash map may also be provided, containing `:batch-size` which
-  determines how to partition the parameter groups for submission to the
-  database. If omitted, all groups will be submitted as a single command.
-  If you expect the update counts to be larger than `Integer/MAX_VALUE`,
-  you can specify `:large true` and `.executeLargeBatch` will be called
-  instead.
-
-  By default, returns a Clojure vector of update counts. Some databases
-  allow batch statements to also return generated keys and you can attempt that
-  if you ensure the `PreparedStatement` is created with `:return-keys true`
-  and you also provide `:return-generated-keys true` in the options passed
-  to `execute-batch!`. Some databases will only return one generated key
-  per batch, some return all the generated keys, some will throw an exception.
-  If that is supported, `execute-batch!` will return a vector of hash maps
-  containing the generated keys as fully-realized, datafiable result sets,
-  whose content is database-dependent.
-
-  May throw `java.sql.BatchUpdateException` if any part of the batch fails.
-  You may be able to call `.getUpdateCounts` on that exception object to
-  get more information about which parts succeeded and which failed.
-
-  For additional caveats and database-specific options you may need, see:
-  https://cljdoc.org/d/seancorfield/next.jdbc/CURRENT/doc/getting-started/prepared-statements#caveats
-
-  Not all databases support batch execution."
+(defn ^:no-doc execute-batch!
+  "Deprecated in favor of `next.jdbc/execute-batch!`."
   ([ps param-groups]
    (execute-batch! ps param-groups {}))
   ([^PreparedStatement ps param-groups opts]

src/next/jdbc/specs.clj

@@ -128,6 +128,11 @@
                                  :sql-params (s/nilable ::sql-params)
                                  :opts (s/? ::opts-map))))
 
+(s/fdef jdbc/execute-batch!
+        :args (s/cat :ps ::prepared-statement
+                     :param-groups (s/coll-of ::params :kind sequential?)
+                     :opts (s/? ::batch-opts)))
+
 (s/fdef jdbc/transact
         :args (s/cat :transactable ::transactable
                      :f fn?
@@ -153,11 +158,6 @@
                      :db-spec ::db-spec-or-jdbc
                      :close-fn (s/? fn?)))
 
-(s/fdef prepare/execute-batch!
-        :args (s/cat :ps ::prepared-statement
-                     :param-groups (s/coll-of ::params :kind sequential?)
-                     :opts (s/? ::batch-opts)))
-
 (s/fdef prepare/set-parameters
         :args (s/cat :ps ::prepared-statement
                      :params ::params))
@@ -230,12 +230,12 @@
    `jdbc/plan
    `jdbc/execute!
    `jdbc/execute-one!
+   `jdbc/execute-batch!
    `jdbc/transact
    `jdbc/with-transaction
    `jdbc/with-options
    `connection/->pool
    `connection/component
-   `prepare/execute-batch!
    `prepare/set-parameters
    `prepare/statement
    `sql/insert!

test/next/jdbc/prepare_test.clj

@@ -4,8 +4,10 @@
   "Stub test namespace for PreparedStatement creation etc.
 
   Most of this functionality is core to all of the higher-level stuff
-  so it gets tested that way, but there are some specific tests for
-  `execute-batch!` here."
+  so it gets tested that way.
+
+  The tests for the deprecated version of `execute-batch!` are here
+  as a guard against regressions."
   (:require [clojure.test :refer [deftest is testing use-fixtures]]
             [next.jdbc :as jdbc]
             [next.jdbc.test-fixtures

test/next/jdbc_test.clj

@@ -483,11 +483,113 @@ VALUES ('Pear', 'green', 49, 47)
     (is (every? boolean? (map :is_it data)))
     (is (every? boolean? (map :twiddle data)))))
 
+
+(deftest execute-batch-tests
+  (testing "simple batch insert"
+    (is (= [1 1 1 1 1 1 1 1 1 13]
+           (jdbc/with-transaction [t (ds) {:rollback-only true}]
+             (with-open [ps (jdbc/prepare t ["
+INSERT INTO fruit (name, appearance) VALUES (?,?)
+"])]
+               (let [result (jdbc/execute-batch! ps [["fruit1" "one"]
+                                                     ["fruit2" "two"]
+                                                     ["fruit3" "three"]
+                                                     ["fruit4" "four"]
+                                                     ["fruit5" "five"]
+                                                     ["fruit6" "six"]
+                                                     ["fruit7" "seven"]
+                                                     ["fruit8" "eight"]
+                                                     ["fruit9" "nine"]])]
+                 (conj result (count (jdbc/execute! t ["select * from fruit"]))))))))
+    (is (= 4 (count (jdbc/execute! (ds) ["select * from fruit"])))))
+  (testing "small batch insert"
+    (is (= [1 1 1 1 1 1 1 1 1 13]
+           (jdbc/with-transaction [t (ds) {:rollback-only true}]
+             (with-open [ps (jdbc/prepare t ["
+INSERT INTO fruit (name, appearance) VALUES (?,?)
+"])]
+               (let [result (jdbc/execute-batch! ps [["fruit1" "one"]
+                                                     ["fruit2" "two"]
+                                                     ["fruit3" "three"]
+                                                     ["fruit4" "four"]
+                                                     ["fruit5" "five"]
+                                                     ["fruit6" "six"]
+                                                     ["fruit7" "seven"]
+                                                     ["fruit8" "eight"]
+                                                     ["fruit9" "nine"]]
+                                                 {:batch-size 3})]
+                 (conj result (count (jdbc/execute! t ["select * from fruit"]))))))))
+    (is (= 4 (count (jdbc/execute! (ds) ["select * from fruit"])))))
+  (testing "big batch insert"
+    (is (= [1 1 1 1 1 1 1 1 1 13]
+           (jdbc/with-transaction [t (ds) {:rollback-only true}]
+             (with-open [ps (jdbc/prepare t ["
+INSERT INTO fruit (name, appearance) VALUES (?,?)
+"])]
+               (let [result (jdbc/execute-batch! ps [["fruit1" "one"]
+                                                     ["fruit2" "two"]
+                                                     ["fruit3" "three"]
+                                                     ["fruit4" "four"]
+                                                     ["fruit5" "five"]
+                                                     ["fruit6" "six"]
+                                                     ["fruit7" "seven"]
+                                                     ["fruit8" "eight"]
+                                                     ["fruit9" "nine"]]
+                                                 {:batch-size 8})]
+                 (conj result (count (jdbc/execute! t ["select * from fruit"]))))))))
+    (is (= 4 (count (jdbc/execute! (ds) ["select * from fruit"])))))
+  (testing "large batch insert"
+    (when-not (or (jtds?) (sqlite?))
+      (is (= [1 1 1 1 1 1 1 1 1 13]
+             (jdbc/with-transaction [t (ds) {:rollback-only true}]
+               (with-open [ps (jdbc/prepare t ["
+INSERT INTO fruit (name, appearance) VALUES (?,?)
+"])]
+                 (let [result (jdbc/execute-batch! ps [["fruit1" "one"]
+                                                       ["fruit2" "two"]
+                                                       ["fruit3" "three"]
+                                                       ["fruit4" "four"]
+                                                       ["fruit5" "five"]
+                                                       ["fruit6" "six"]
+                                                       ["fruit7" "seven"]
+                                                       ["fruit8" "eight"]
+                                                       ["fruit9" "nine"]]
+                                                   {:batch-size 4
+                                                    :large true})]
+                   (conj result (count (jdbc/execute! t ["select * from fruit"]))))))))
+      (is (= 4 (count (jdbc/execute! (ds) ["select * from fruit"]))))))
+  (testing "return generated keys"
+    (when-not (mssql?)
+      (let [results
+            (jdbc/with-transaction [t (ds) {:rollback-only true}]
+              (with-open [ps (jdbc/prepare t ["
+INSERT INTO fruit (name, appearance) VALUES (?,?)
+"]
+                                           {:return-keys true})]
+                (let [result (jdbc/execute-batch! ps [["fruit1" "one"]
+                                                      ["fruit2" "two"]
+                                                      ["fruit3" "three"]
+                                                      ["fruit4" "four"]
+                                                      ["fruit5" "five"]
+                                                      ["fruit6" "six"]
+                                                      ["fruit7" "seven"]
+                                                      ["fruit8" "eight"]
+                                                      ["fruit9" "nine"]]
+                                                  {:batch-size 4
+                                                   :return-generated-keys true})]
+                  (conj result (count (jdbc/execute! t ["select * from fruit"]))))))]
+        (is (= 13 (last results)))
+        (is (every? map? (butlast results)))
+        ;; Derby and SQLite only return one generated key per batch so there
+        ;; are only three keys, plus the overall count here:
+        (is (< 3 (count results))))
+      (is (= 4 (count (jdbc/execute! (ds) ["select * from fruit"])))))))
+
 (deftest folding-test
   (jdbc/execute-one! (ds) ["delete from fruit"])
   (with-open [con (jdbc/get-connection (ds))
               ps  (jdbc/prepare con ["insert into fruit(name) values (?)"])]
-    (prep/execute-batch! ps (mapv #(vector (str "Fruit-" %)) (range 1 1001))))
+    (jdbc/execute-batch! ps (mapv #(vector (str "Fruit-" %)) (range 1 1001))))
   (testing "foldable result set"
     (testing "from a Connection"
       (let [result