luciano/next-jdbc
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'master' of github.com:seancorfield/next-jdbc

Browse source 
# Conflicts:
#	src/next/jdbc/connection.clj
This commit is contained in:
Sean Corfield 2019-07-14 19:12:42 -07:00
commit e0e209aa1c
9 changed files with 167 additions and 33 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
CHANGELOG.md
View file

@ -6,10 +6,15 @@ Only accretive/fixative changes will be made from now on.
The following changes have been committed to the **master** branch since the 1.0.1 release:
* Fix #45 by adding TimesTen driver support.
* Fix #46 by allowing `:host` to be `:none` which tells `next.jdbc` to omit the host/port section of the JDBC URL, so that local databases can be used with `:dbtype`/`:classname` for database types that `next.jdbc` does not know. Also added `:dbname-separator` and `:host-prefix` to the "db-spec" to allow fine-grained control over how the JDBC URL is assembled.
* Fix #45 by adding [TimesTen](https://www.oracle.com/database/technologies/related/timesten.html) driver support.
* Fix #44 so that `insert-multi!` with an empty `rows` vector returns `[]`.
* Fix #43 by adjusting the spec for `insert-multi!` to "require less" of the `cols` and `rows` arguments.
* Fix #42 by adding specs for `execute-batch!` and `set-parameters` in `next.jdbc.prepare`.
* Fix #41 by improving docstrings and documentation, especially around prepared statement handling.
* Fix #40 by adding `next.jdbc.prepare/execute-batch!`.
* Added `assert`s in `next.jdbc.sql` as more informative errors for cases that would generate SQL exceptions (from malformed SQL).
* Added spec for `:order-by` to reflect what is actually permitted.
* Expose `next.jdbc.connect/dbtypes` as a table of known database types and aliases, along with their class name(s), port, and other JDBC string components.
## Stable Builds

2
doc/all-the-options.md
View file

@ -10,7 +10,9 @@ Although `get-datasource` does not accept options, the "db spec" hash map passed
* `:dbtype` -- a string that identifies the type of JDBC database being used,
* `:dbname` -- a string that identifies the name of the actual database being used,
* `:dbname-separator` -- an optional string that can be used to override the `/` or `:` that is normally placed in front of the database name in the JDBC URL,
* `:host` -- an optional string that identifies the IP address or hostname of the server on which the database is running; the default is `"127.0.0.1"`,
* `:host-prefix` -- an optional string that can be used to override the `//` that is normally placed in front of the IP address or hostname in the JDBC URL,
* `:port` -- an optional integer that identifies the port on which the database is running; for common database types, `next.jdbc` knows the default so this should only be needed for non-standard setups or "exotic" database types,
* `:classname` -- an optional string that identifies the name of the JDBC driver class to be used for the connection; for common database types, `next.jdbc` knows the default so this should only be needed for "exotic" database types,
* `:user` -- an optional string that identifies the database username to be used when authenticating,

13
doc/getting-started.md
View file

@ -33,6 +33,8 @@ For the examples in this documentation, we will use a local H2 database on disk,
com.h2database/h2 {:mvn/version "1.4.197"}}}
```
### Create & Populate a Database
In this REPL session, we'll define an H2 datasource, create a database with a simple table, and then add some data and query it:
```clojure
@ -59,10 +61,15 @@ user=> (jdbc/execute! ds ["select * from address"])
[#:ADDRESS{:ID 1, :NAME "Sean Corfield", :EMAIL "sean@corfield.org"}]
user=>
```
### The "db-spec" hash map
We described the database with just `:dbtype` and `:dbname` because it is created as a local file and needs no authentication. For most databases, you would need `:user` and `:password` for authentication, and if the database is running on a remote machine you would need `:host` and possibly `:port` (`next.jdbc` tries to guess the correct port based on the `:dbtype`).
> Note: You can see the full list of `:dbtype` values supported in [next.jdbc/get-datasource](https://cljdoc.org/d/seancorfield/next.jdbc/CURRENT/api/next.jdbc#get-datasource)'s docstring. If you need this programmatically, you can get it from the [next.jdbc.connection/dbtypes](https://cljdoc.org/d/seancorfield/next.jdbc/CURRENT/api/next.jdbc.connection#dbtypes) hash map. If those lists differ, the hash map is the definitive list (and I'll need to fix the docstring!). The docstring of that Var explains how to tell `next.jdbc` about additional databases.
### `execute!` & `execute-one!`
We used `execute!` to create the `address` table, to insert a new row into it, and to query it. In all three cases, `execute!` returns a vector of hash maps with namespace-qualified keys, representing the result set from the operation, if available. When no result set is produced, `next.jdbc` returns a "result set" containing the "update count" from the operation (which is usually the number of rows affected). By default, H2 uses uppercase names and `next.jdbc` returns these as-is.
If you only want a single row back -- the first row of any result set, generated keys, or update counts -- you can use `execute-one!` instead. Continuing the REPL session, we'll insert another address and ask for the generated keys to be returned, and then we'll query for a single row:
@ -80,6 +87,8 @@ user=>
Since we used `execute-one!`, we get just one row back (a hash map). This also shows how you provide parameters to SQL statements -- with `?` in the SQL and then the corresponding parameter values in the vector after the SQL string.
### `plan` & Reducing Result Sets
While these functions are fine for retrieving result sets as data, most of the time you want to process that data efficiently, so `next.jdbc` provides a SQL execution function that works with `reduce` and with transducers to consume the result set without the intermediate overhead of creating Clojure data structures for every row:
```clojure
@ -140,11 +149,11 @@ If `with-transaction` is given a datasource, it will create and close the connec
As you are developing with `next.jdbc`, it can be useful to have assistance from `clojure.spec` in checking calls to `next.jdbc`'s functions, to provide explicit argument checking and/or better error messages for some common mistakes, e.g., trying to pass a plain SQL string where a vector (containing a SQL string, and no parameters) is expected.
You can enable argument checking for functions in both `next.jdbc` and `next.jdbc.sql` by requiring the `next.jdbc.specs` namespace and instrumenting the functions. A convenience function is provided:
You can enable argument checking for functions in `next.jdbc`, `next.jdbc.sql`, and `next.jdbc.prepare` by requiring the `next.jdbc.specs` namespace and instrumenting the functions. A convenience function is provided:
```clojure
(require '[next.jdbc.specs :as specs])
(specs/instrument) ; instruments all next.jdbc functions
(specs/instrument) ; instruments all next.jdbc API functions
(jdbc/execute! ds "SELECT * FROM fruit")
Call to #'next.jdbc/execute! did not conform to spec.

8
src/next/jdbc.clj
View file

@ -67,6 +67,14 @@
* `:classname` -- if you need to override the default for the `:dbtype`
(or you want to use a database that next.jdbc does not know about!)
The following optional keys can be used to control how JDBC URLs are
assembled. This may be needed for `:dbtype` values that `next.jdbc`
does not recognize:
* `:dbname-separator` -- override the `/` or `:` that normally precedes
the database name in the JDBC URL
* `:host-prefix` -- override the `//` that normally precedes the IP
address or hostname in the JDBC URL
Any additional options provided will be passed to the JDBC driver's
`.getConnection` call as a `java.util.Properties` structure.

57
src/next/jdbc/connection.clj
View file

@ -71,7 +71,8 @@
"A map of all known database types (including aliases) to the class name(s)
and port that `next.jdbc` supports out of the box. Just for completeness,
this also includes the prefixes used in the JDBC string for the `:host`
and `:dbname` (which are `//` and `/` respectively for nearly all types).
and `:dbname` (which are `//` and either `/` or `:` respectively for
nearly all types).
For known database types, you can use `:dbtype` (and omit `:classname`).
@ -81,6 +82,24 @@
`{:dbtype \"acme\" :classname \"com.acme.JdbcDriver\" ...}`
The value of `:dbtype` should be the string that the driver is associated
with in the JDBC URL, i.e., the value that comes between the `jdbc:`
prefix and the `://<host>...` part. In the above example, the JDBC URL
that would be generated would be `jdbc:acme://<host>:<port>/<dbname>`.
If you want `next.jdbc` to omit the host/port part of the URL, specify
`:host :none`, which would produce a URL like: `jdbc:acme:<dbname>`,
which allows you to work with local databases (or drivers that do not
need host/port information).
The default prefix for the host name (or IP address) is `//`. You
can override this via the `:host-prefix` option.
The default separator between the host/port and the database name is `/`.
The default separator between the subprotocol and the database name,
for local databases with no host/port, is `:`. You can override this
via the `:dbname-separator` option.
JDBC drivers are not provided by `next.jdbc` -- you need to specify the
driver(s) you need as additional dependencies in your project. For
example:
@ -127,32 +146,44 @@
(defn- spec->url+etc
"Given a database spec, return a JDBC URL and a map of any additional options."
[{:keys [dbtype dbname host port classname] :as db-spec}]
[{:keys [dbtype dbname host port classname
dbname-separator host-prefix]
:as db-spec}]
(let [;; allow aliases for dbtype
subprotocol (aliases dbtype dbtype)
host (or host "127.0.0.1")
port (or port (ports subprotocol))
db-sep (dbname-separators dbtype "/")
url (cond (= "h2:mem" dbtype)
(str "jdbc:" subprotocol ":" dbname ";DB_CLOSE_DELAY=-1")
host (or host "127.0.0.1")
port (or port (ports subprotocol))
db-sep (or dbname-separator (dbname-separators dbtype "/"))
local-sep (or dbname-separator (dbname-separators dbtype ":"))
url (cond (#{"derby" "hsqldb" "sqlite"} subprotocol)
(str "jdbc:" subprotocol local-sep dbname)
(#{"h2"} subprotocol)
(str "jdbc:" subprotocol ":"
(str "jdbc:" subprotocol local-sep
(if (re-find #"^([A-Za-z]:)?[\./\\]" dbname)
;; DB name starts with relative or absolute path
dbname
;; otherwise make it local
(str "./" dbname)))
(#{"derby" "hsqldb" "sqlite"} subprotocol)
(str "jdbc:" subprotocol ":" dbname)
(#{"h2:mem"} subprotocol)
(str "jdbc:" subprotocol local-sep dbname ";DB_CLOSE_DELAY=-1")
(#{"timesten:client" "timesten:direct"} subprotocol)
(str "jdbc:" subprotocol db-sep dbname)
(str "jdbc:" subprotocol local-sep dbname)
(= :none host)
(str "jdbc:" subprotocol local-sep dbname)
:else
(str "jdbc:" subprotocol ":"
(host-prefixes subprotocol "//")
(or host-prefix (host-prefixes subprotocol "//"))
host
(when port (str ":" port))
db-sep dbname))
etc (dissoc db-spec :dbtype :dbname :host :port :classname)]
etc (dissoc db-spec
:dbtype :dbname :host :port :classname
:dbname-separator :host-prefix)]
;; verify the datasource is loadable
(if-let [class-name (or classname (classnames subprotocol))]
(swap! driver-cache update class-name

24
src/next/jdbc/specs.clj
View file

@ -26,15 +26,20 @@
(s/def ::dbtype string?)
(s/def ::dbname string?)
(s/def ::dbname-separator string?)
(s/def ::classname string?)
(s/def ::user string?)
(s/def ::password string?)
(s/def ::host string?)
(s/def ::host (s/or :name string?
:none #{:none}))
(s/def ::host-prefix string?)
(s/def ::port pos-int?)
(s/def ::db-spec-map (s/keys :req-un [::dbtype ::dbname]
:opt-un [::classname
::user ::password
::host ::port]))
::host ::port
::dbname-separator
::host-prefix]))
(s/def ::connection #(instance? Connection %))
(s/def ::datasource #(instance? DataSource %))
@ -47,7 +52,13 @@
(s/def ::connectable any?)
(s/def ::key-map (s/map-of keyword? any?))
(s/def ::example-map (s/map-of keyword? any? :min-count 1))
(s/def ::opts-map (s/map-of keyword? any?))
(s/def ::order-by-col (s/or :col keyword?
:dir (s/cat :col keyword?
:dir #{:asc :desc})))
(s/def ::order-by (s/coll-of ::order-by-col :kind vector? :min-count 1))
(s/def ::opts-map (s/and (s/map-of keyword? any?)
(s/keys :opt-un [::order-by])))
(s/def ::transactable any?)
@ -120,8 +131,11 @@
(s/fdef sql/insert-multi!
:args (s/and (s/cat :connectable ::connectable
:table keyword?
:cols (s/coll-of keyword? :kind vector?)
:rows (s/coll-of (s/coll-of any? :kind vector?) :kind vector?)
:cols (s/coll-of keyword?
:kind sequential?
:min-count 1)
:rows (s/coll-of (s/coll-of any? :kind sequential?)
:kind sequential?)
:opts (s/? ::opts-map))
#(apply = (count (:cols %))
(map count (:rows %)))))

25
src/next/jdbc/sql.clj
View file

@ -38,6 +38,7 @@
[(conj conds (str e " = ?")) (conj params v)])))
[[] []]
key-map)]
(assert (seq where) "key-map may not be empty")
(into [(str (str/upper-case (name clause)) " "
(str/join (if (= :where clause) " AND " ", ") where))]
params)))
@ -79,10 +80,11 @@
(defn- for-order
"Given an `:order-by` vector, return an `ORDER BY` clause."
[order-by opts]
(if (vector? order-by)
(str "ORDER BY "
(str/join ", " (map #(for-order-col % opts) order-by)))
(throw (IllegalArgumentException. ":order-by must be a vector"))))
(when-not (vector? order-by)
(throw (IllegalArgumentException. ":order-by must be a vector")))
(assert (seq order-by) ":order-by may not be empty")
(str "ORDER BY "
(str/join ", " (map #(for-order-col % opts) order-by))))
(defn- for-query
"Given a table name and either a hash map of column names and values or a
@ -147,6 +149,7 @@
(let [entity-fn (:table-fn opts identity)
params (as-keys key-map opts)
places (as-? key-map opts)]
(assert (seq key-map) "key-map may not be empty")
(into [(str "INSERT INTO " (entity-fn (name table))
" (" params ")"
" VALUES (" places ")")]
@ -159,7 +162,11 @@
Applies any `:table-fn` / `:column-fn` supplied in the options."
[table cols rows opts]
(assert (apply = (count cols) (map count rows)))
(assert (apply = (count cols) (map count rows))
"column counts are not consistent across cols and rows")
;; to avoid generating bad SQL
(assert (seq cols) "cols may not be empty")
(assert (seq rows) "rows may not be empty")
(let [table-fn (:table-fn opts identity)
column-fn (:column-fn opts identity)
params (str/join ", " (map (comp column-fn name) cols))
@ -199,9 +206,11 @@
([connectable table cols rows]
(insert-multi! connectable table cols rows {}))
([connectable table cols rows opts]
(execute! connectable
(for-insert-multi table cols rows opts)
(merge {:return-keys true} opts))))
(if (seq rows)
(execute! connectable
(for-insert-multi table cols rows opts)
(merge {:return-keys true} opts))
[])))
(defn query
"Syntactic sugar over `execute!` to provide a query alias.

23
test/next/jdbc/connection_test.clj
View file

@ -51,6 +51,29 @@
(is (= (#'c/spec->url+etc {:dbtype "sqlserver" :dbname db-name})
(#'c/spec->url+etc {:dbtype "sqlserver" :dbname db-name :port 1433})))))
(deftest custom-dbtypes
(is (= ["jdbc:acme:my-db" {}]
(#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
:dbname "my-db" :host :none})))
(is (= ["jdbc:acme://127.0.0.1/my-db" {}]
(#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
:dbname "my-db"})))
(is (= ["jdbc:acme://12.34.56.70:1234/my-db" {}]
(#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
:dbname "my-db" :host "12.34.56.70" :port 1234})))
(is (= ["jdbc:acme:dsn=my-db" {}]
(#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
:dbname "my-db" :host :none
:dbname-separator ":dsn="})))
(is (= ["jdbc:acme:(*)127.0.0.1/my-db" {}]
(#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
:dbname "my-db"
:host-prefix "(*)"})))
(is (= ["jdbc:acme:(*)12.34.56.70:1234/my-db" {}]
(#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
:dbname "my-db" :host "12.34.56.70" :port 1234
:host-prefix "(*)"}))))
;; these are the 'local' databases that we can always test against
(def test-db-type ["derby" "h2" "h2:mem" "hsqldb" "sqlite"])

41
test/next/jdbc/sql_test.clj
View file

@ -55,9 +55,9 @@
["DELETE FROM [user] WHERE id = ? and opt is null" 9]))))
(deftest test-for-update
(testing "empty example (SQL error)"
(is (= (#'sql/for-update :user {:status 42} {} {:table-fn sql-server :column-fn mysql})
["UPDATE [user] SET `status` = ? WHERE " 42])))
(testing "empty example (would be a SQL error)"
(is (thrown? AssertionError ; changed in #44
(#'sql/for-update :user {:status 42} {} {:table-fn sql-server :column-fn mysql}))))
(testing "by example"
(is (= (#'sql/for-update :user {:status 42} {:id 9} {:table-fn sql-server :column-fn mysql})
["UPDATE [user] SET `status` = ? WHERE `id` = ?" 42 9])))
@ -154,7 +154,30 @@
(is (= 6 (count (sql/query (ds) ["select * from fruit"]))))
(is (= {:next.jdbc/update-count 2}
(sql/delete! (ds) :fruit ["id > ?" 4])))
(is (= 4 (count (sql/query (ds) ["select * from fruit"]))))))
(is (= 4 (count (sql/query (ds) ["select * from fruit"])))))
(testing "multiple insert/delete with sequential cols/rows" ; per #43
(is (= (cond (derby?)
[{:1 nil}] ; WTF Apache Derby?
(sqlite?)
[{(keyword "last_insert_rowid()") 11}]
:else
[{:FRUIT/ID 9} {:FRUIT/ID 10} {:FRUIT/ID 11}])
(sql/insert-multi! (ds) :fruit
'(:name :appearance :cost :grade)
'(("Kiwi" "green & fuzzy" 100 99.9)
("Grape" "black" 10 50)
("Lemon" "yellow" 20 9.9)))))
(is (= 7 (count (sql/query (ds) ["select * from fruit"]))))
(is (= {:next.jdbc/update-count 1}
(sql/delete! (ds) :fruit {:id 9})))
(is (= 6 (count (sql/query (ds) ["select * from fruit"]))))
(is (= {:next.jdbc/update-count 2}
(sql/delete! (ds) :fruit ["id > ?" 4])))
(is (= 4 (count (sql/query (ds) ["select * from fruit"])))))
(testing "empty insert-multi!" ; per #44
(is (= [] (sql/insert-multi! (ds) :fruit
[:name :appearance :cost :grade]
[])))))
(deftest no-empty-example-maps
(is (thrown? clojure.lang.ExceptionInfo
@ -163,3 +186,13 @@
(sql/update! (ds) :fruit {} {})))
(is (thrown? clojure.lang.ExceptionInfo
(sql/delete! (ds) :fruit {}))))
(deftest no-empty-columns
(is (thrown? clojure.lang.ExceptionInfo
(sql/insert-multi! (ds) :fruit [] [[] [] []]))))
(deftest no-empty-order-by
(is (thrown? clojure.lang.ExceptionInfo
(sql/find-by-keys (ds) :fruit
{:name "Apple"}
{:order-by []}))))