luciano/next-jdbc
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fixes #40 by adding next.jdbc.prepare/execute-batch!

Browse source
This commit is contained in:
Sean Corfield 2019-07-09 16:00:10 -07:00
parent 7184ef996c
commit 29d46439e6
5 changed files with 143 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
CHANGELOG.md
View file

@ -6,6 +6,7 @@ Only accretive/fixative changes will be made from now on.
The following changes have been committed to the **master** branch since the 1.0.1 release:
* Fix #40 by adding `next.jdbc.prepare/execute-batch!`.
* Improved docstrings and documentation, especially around prepared statement handling.
## Stable Builds

2
README.md
View file

@ -8,7 +8,7 @@ The latest versions on Clojars and on cljdoc:
[![Clojars Project](https://clojars.org/seancorfield/next.jdbc/latest-version.svg)](https://clojars.org/seancorfield/next.jdbc) [![cljdoc badge](https://cljdoc.org/badge/seancorfield/next.jdbc)](https://cljdoc.org/d/seancorfield/next.jdbc/CURRENT)
This documentation is for the 1.0.1 release -- [see the CHANGELOG](CHANGELOG.md).
This documentation is for **master**, after the 1.0.1 release -- [see the CHANGELOG](CHANGELOG.md).
* [Getting Started](/doc/getting-started.md)
* [Migrating from `clojure.java.jdbc`](/doc/migration-from-clojure-java-jdbc.md)

21
doc/prepared-statements.md
View file

@ -41,9 +41,10 @@ If you need more specialized parameter handling than the protocol can provide, t
## Batched Parameters
By default, `next.jdbc` assumes that you are providing a single set of parameter values and then executing the prepared statement. If you want to run a single prepared statement with multiple sets of parameters, you might want to take advantage of the increased performance that may come from using JDBC's batching machinery.
By default, `next.jdbc` assumes that you are providing a single set of parameter values and then executing the prepared statement. If you want to run a single prepared statement with multiple groups of parameters, you might want to take advantage of the increased performance that may come from using JDBC's batching machinery.
```clojure
;; assuming require next.jdbc.prepare :as p
(with-open [con (jdbc/get-connection ds)
ps (jdbc/prepare con ["insert into status (id,name) values (?,?)"])]
(p/set-parameters ps [1 "Approved"])
@ -52,22 +53,32 @@ By default, `next.jdbc` assumes that you are providing a single set of parameter
(.addBatch ps)
(p/set-parameters ps [3 "New"])
(.addBatch ps)
(.executeBatch ps))
(.executeBatch ps)) ; returns int[]
```
Here we set parameters and add them in batches to the prepared statement, then we execute the prepared statement in batch mode. You could also do the above like this, assuming you have those sets of parameters in a sequence:
Here we set parameters and add them in batches to the prepared statement, then we execute the prepared statement in batch mode. You could also do the above like this, assuming you have those groups of parameters in a sequence:
```clojure
(with-open [con (jdbc/get-connection ds)
ps (jdbc/prepare con ["insert into status (id,name) values (?,?)"])]
(run! #(.addBatch (p/set-parameters ps %))
[[1 "Approved"] [2 "Rejected"] [3 "New"]])
(.executeBatch ps))
(.executeBatch ps)) ; returns int[]
```
A helper function is provided in `next.jdbc.prepare` to automate the execution of batched parameters:
```clojure
(with-open [con (jdbc/get-connection ds)
ps (jdbc/prepare con ["insert into status (id,name) values (?,?)"])]
(p/execute-batch! ps [[1 "Approved"] [2 "Rejected"] [3 "New"]]))
```
By default, this adds all the parameter groups and executes one batched command. It returns a (Clojure) vector of update counts (rather than `int[]`). If you provide an options hash map, you can specify a `:batch-size` and the parameter groups will be partitioned and executed as multiple batched commands. This is intended to allow very large sequences of parameter groups to be executed without running into limitations that may apply to a single batched command. If you expect the update counts to be very large (more than `Integer/MAX_VALUE`), you can specify `:large true` so that `.executeLargeBatch` is called instead of `.executeBatch`. Note: not all databases support `.executeLargeBatch`.
There are several caveats around using batched parameters. Some JDBC drivers need a "hint" in order to perform the batch operation as a single command for the database. In particular, PostgreSQL requires the `:reWriteBatchedInserts true` option and MySQL requires `:rewriteBatchedStatement true` (both non-standard JDBC options, of course!).
In addition, if the batch operation fails for one of the sets of parameters, it is database-specific whether the remaining sets of parameters are used, i.e., whether the operation is performed for any further sets of parameters after the one that failed. The result of calling `.executeBatch` is an array of integers (specifically a Java array `int[]`). Each element of the array is the number of rows affected by the operation for each set of parameters. `.executeBatch` may throw a `BatchUpdateException` and calling `.getUpdatedCounts` on the exception may return an array containing a mix of update counts and error values. Some databases don't always return an update count but instead a value indicating the number of rows is not known (but sometimes you can still get the update counts).
In addition, if the batch operation fails for a group of parameters, it is database-specific whether the remaining groups of parameters are used, i.e., whether the operation is performed for any further groups of parameters after the one that failed. The result of calling `execute-batch!` is a vector of integers. Each element of the vector is the number of rows affected by the operation for each group of parameters. `execute-batch!` may throw a `BatchUpdateException` and calling `.getUpdateCounts` (or `.getLargeUpdateCounts`) on the exception may return an array containing a mix of update counts and error values (a Java `int[]` or `long[]`). Some databases don't always return an update count but instead a value indicating the number of rows is not known (but sometimes you can still get the update counts).
Finally, some database drivers don't do batched operations at all -- they accept `.executeBatch` but they run the operation as separate commands for the database rather than a single batched command.

38
src/next/jdbc/prepare.clj
View file

@ -7,6 +7,9 @@
`set-parameters` is public and may be useful if you have a `PreparedStatement`
that you wish to reuse and (re)set the parameters on it.
`execute-batch!` provides a way to add batches of parameters to a
`PreparedStatement` and then execute it in batch mode (via `.executeBatch`).
Defines the `SettableParameter` protocol for converting Clojure values
to database-specific values."
(:require [next.jdbc.protocols :as p])
@ -133,3 +136,38 @@
java.sql.Connection
(prepare [this sql-params opts]
(create this (first sql-params) (rest sql-params) opts)))
(defn execute-batch!
"Given a `PreparedStatement` and a vector containing parameter groups,
i.e., a vector of vector of parameters, use `.addBatch` to add each group
of parameters to the prepared statement and then call `.executeBatch`.
A vector of update counts is returned.
An options hash map may also be provided, containing `:batch-size` which
determines how to partition the parameter groups for submission to the
database. If omitted, all groups will be submitted as a single command.
If you expect the update counts to be larger than `Integer/MAX_VALUE`,
you can specify `:large true` and `.executeLargeBatch` will be called
instead.
Returns a Clojure vector of update counts.
May throw `java.sql.BatchUpdateException` if any part of the batch fails.
You may be able to call `.getUpdateCounts` on that exception object to
get more information about which parts succeeded and which failed."
([ps param-groups]
(execute-batch! ps param-groups {}))
([^PreparedStatement ps param-groups opts]
(let [params (if-let [n (:batch-size opts)]
(if (and (number? n) (pos? n))
(partition-all n param-groups)
(throw (IllegalArgumentException.
":batch-size must be positive")))
[param-groups])]
(into []
(mapcat (fn [group]
(run! #(.addBatch (set-parameters ps %)) group)
(if (:large opts)
(.executeLargeBatch ps)
(.executeBatch ps))))
params))))

93
test/next/jdbc/prepare_test.clj
View file

@ -3,11 +3,92 @@
(ns next.jdbc.prepare-test
"Stub test namespace for PreparedStatement creation etc.
This functionality is core to all of the higher-level stuff so it mostly
gets tested that way, but there should be some dedicated tests in here
eventually that ensure all of the options specific to PreparedStatement
actually work they way they're supposed to!"
(:require [clojure.test :refer [deftest is testing]]
[next.jdbc.prepare :refer :all]))
Most of this functionality is core to all of the higher-level stuff
so it gets tested that way, but there are some specific tests for
`execute-batch!` here."
(:require [clojure.test :refer [deftest is testing use-fixtures]]
[next.jdbc :as jdbc]
[next.jdbc.test-fixtures :refer [with-test-db ds sqlite?]]
[next.jdbc.prepare :as prep]
[next.jdbc.specs :as specs]))
(set! *warn-on-reflection* true)
(use-fixtures :once with-test-db)
(specs/instrument)
(deftest execute-batch-tests
(testing "simple batch insert"
(is (= [1 1 1 1 1 1 1 1 1 13]
(jdbc/with-transaction [t (ds) {:rollback-only true}]
(with-open [ps (jdbc/prepare t ["
INSERT INTO fruit (name, appearance) VALUES (?,?)
"])]
(let [result (prep/execute-batch! ps [["fruit1" "one"]
["fruit2" "two"]
["fruit3" "three"]
["fruit4" "four"]
["fruit5" "five"]
["fruit6" "six"]
["fruit7" "seven"]
["fruit8" "eight"]
["fruit9" "nine"]])]
(conj result (count (jdbc/execute! t ["select * from fruit"]))))))))
(is (= 4 (count (jdbc/execute! (ds) ["select * from fruit"])))))
(testing "small batch insert"
(is (= [1 1 1 1 1 1 1 1 1 13]
(jdbc/with-transaction [t (ds) {:rollback-only true}]
(with-open [ps (jdbc/prepare t ["
INSERT INTO fruit (name, appearance) VALUES (?,?)
"])]
(let [result (prep/execute-batch! ps [["fruit1" "one"]
["fruit2" "two"]
["fruit3" "three"]
["fruit4" "four"]
["fruit5" "five"]
["fruit6" "six"]
["fruit7" "seven"]
["fruit8" "eight"]
["fruit9" "nine"]]
{:batch-size 3})]
(conj result (count (jdbc/execute! t ["select * from fruit"]))))))))
(is (= 4 (count (jdbc/execute! (ds) ["select * from fruit"])))))
(testing "big batch insert"
(is (= [1 1 1 1 1 1 1 1 1 13]
(jdbc/with-transaction [t (ds) {:rollback-only true}]
(with-open [ps (jdbc/prepare t ["
INSERT INTO fruit (name, appearance) VALUES (?,?)
"])]
(let [result (prep/execute-batch! ps [["fruit1" "one"]
["fruit2" "two"]
["fruit3" "three"]
["fruit4" "four"]
["fruit5" "five"]
["fruit6" "six"]
["fruit7" "seven"]
["fruit8" "eight"]
["fruit9" "nine"]]
{:batch-size 8})]
(conj result (count (jdbc/execute! t ["select * from fruit"]))))))))
(is (= 4 (count (jdbc/execute! (ds) ["select * from fruit"])))))
(testing "large batch insert"
(when-not (sqlite?)
(is (= [1 1 1 1 1 1 1 1 1 13]
(jdbc/with-transaction [t (ds) {:rollback-only true}]
(with-open [ps (jdbc/prepare t ["
INSERT INTO fruit (name, appearance) VALUES (?,?)
"])]
(let [result (prep/execute-batch! ps [["fruit1" "one"]
["fruit2" "two"]
["fruit3" "three"]
["fruit4" "four"]
["fruit5" "five"]
["fruit6" "six"]
["fruit7" "seven"]
["fruit8" "eight"]
["fruit9" "nine"]]
{:batch-size 4
:large true})]
(conj result (count (jdbc/execute! t ["select * from fruit"]))))))))
(is (= 4 (count (jdbc/execute! (ds) ["select * from fruit"])))))))