Merge branch 'master' of github.com:seancorfield/next-jdbc

# Conflicts:
#	src/next/jdbc/connection.clj

CHANGELOG.md

@@ -6,10 +6,15 @@ Only accretive/fixative changes will be made from now on.
 
 The following changes have been committed to the **master** branch since the 1.0.1 release:
 
-* Fix #45 by adding TimesTen driver support.
+* Fix #46 by allowing `:host` to be `:none` which tells `next.jdbc` to omit the host/port section of the JDBC URL, so that local databases can be used with `:dbtype`/`:classname` for database types that `next.jdbc` does not know. Also added `:dbname-separator` and `:host-prefix` to the "db-spec" to allow fine-grained control over how the JDBC URL is assembled.
+* Fix #45 by adding [TimesTen](https://www.oracle.com/database/technologies/related/timesten.html) driver support.
+* Fix #44 so that `insert-multi!` with an empty `rows` vector returns `[]`.
+* Fix #43 by adjusting the spec for `insert-multi!` to "require less" of the `cols` and `rows` arguments.
 * Fix #42 by adding specs for `execute-batch!` and `set-parameters` in `next.jdbc.prepare`.
 * Fix #41 by improving docstrings and documentation, especially around prepared statement handling.
 * Fix #40 by adding `next.jdbc.prepare/execute-batch!`.
+* Added `assert`s in `next.jdbc.sql` as more informative errors for cases that would generate SQL exceptions (from malformed SQL).
+* Added spec for `:order-by` to reflect what is actually permitted.
 * Expose `next.jdbc.connect/dbtypes` as a table of known database types and aliases, along with their class name(s), port, and other JDBC string components.
 
 ## Stable Builds

doc/all-the-options.md

@@ -10,7 +10,9 @@ Although `get-datasource` does not accept options, the "db spec" hash map passed
 
 * `:dbtype` -- a string that identifies the type of JDBC database being used,
 * `:dbname` -- a string that identifies the name of the actual database being used,
+* `:dbname-separator` -- an optional string that can be used to override the `/` or `:` that is normally placed in front of the database name in the JDBC URL,
 * `:host` -- an optional string that identifies the IP address or hostname of the server on which the database is running; the default is `"127.0.0.1"`,
+* `:host-prefix` -- an optional string that can be used to override the `//` that is normally placed in front of the IP address or hostname in the JDBC URL,
 * `:port` -- an optional integer that identifies the port on which the database is running; for common database types, `next.jdbc` knows the default so this should only be needed for non-standard setups or "exotic" database types,
 * `:classname` -- an optional string that identifies the name of the JDBC driver class to be used for the connection; for common database types, `next.jdbc` knows the default so this should only be needed for "exotic" database types,
 * `:user` -- an optional string that identifies the database username to be used when authenticating,

doc/getting-started.md

@@ -33,6 +33,8 @@ For the examples in this documentation, we will use a local H2 database on disk,
         com.h2database/h2 {:mvn/version "1.4.197"}}}
 ```
 
+### Create & Populate a Database
+
 In this REPL session, we'll define an H2 datasource, create a database with a simple table, and then add some data and query it:
 
 ```clojure
@@ -59,10 +61,15 @@ user=> (jdbc/execute! ds ["select * from address"])
 [#:ADDRESS{:ID 1, :NAME "Sean Corfield", :EMAIL "sean@corfield.org"}]
 user=>
 ```
+
+### The "db-spec" hash map
+
 We described the database with just `:dbtype` and `:dbname` because it is created as a local file and needs no authentication. For most databases, you would need `:user` and `:password` for authentication, and if the database is running on a remote machine you would need `:host` and possibly `:port` (`next.jdbc` tries to guess the correct port based on the `:dbtype`).
 
 > Note: You can see the full list of `:dbtype` values supported in [next.jdbc/get-datasource](https://cljdoc.org/d/seancorfield/next.jdbc/CURRENT/api/next.jdbc#get-datasource)'s docstring. If you need this programmatically, you can get it from the [next.jdbc.connection/dbtypes](https://cljdoc.org/d/seancorfield/next.jdbc/CURRENT/api/next.jdbc.connection#dbtypes) hash map. If those lists differ, the hash map is the definitive list (and I'll need to fix the docstring!). The docstring of that Var explains how to tell `next.jdbc` about additional databases.
 
+### `execute!` & `execute-one!`
+
 We used `execute!` to create the `address` table, to insert a new row into it, and to query it. In all three cases, `execute!` returns a vector of hash maps with namespace-qualified keys, representing the result set from the operation, if available. When no result set is produced, `next.jdbc` returns a "result set" containing the "update count" from the operation (which is usually the number of rows affected). By default, H2 uses uppercase names and `next.jdbc` returns these as-is.
 
 If you only want a single row back -- the first row of any result set, generated keys, or update counts -- you can use `execute-one!` instead. Continuing the REPL session, we'll insert another address and ask for the generated keys to be returned, and then we'll query for a single row:
@@ -80,6 +87,8 @@ user=>
 
 Since we used `execute-one!`, we get just one row back (a hash map). This also shows how you provide parameters to SQL statements -- with `?` in the SQL and then the corresponding parameter values in the vector after the SQL string.
 
+### `plan` & Reducing Result Sets
+
 While these functions are fine for retrieving result sets as data, most of the time you want to process that data efficiently, so `next.jdbc` provides a SQL execution function that works with `reduce` and with transducers to consume the result set without the intermediate overhead of creating Clojure data structures for every row:
 
 ```clojure
@@ -140,11 +149,11 @@ If `with-transaction` is given a datasource, it will create and close the connec
 
 As you are developing with `next.jdbc`, it can be useful to have assistance from `clojure.spec` in checking calls to `next.jdbc`'s functions, to provide explicit argument checking and/or better error messages for some common mistakes, e.g., trying to pass a plain SQL string where a vector (containing a SQL string, and no parameters) is expected.
 
-You can enable argument checking for functions in both `next.jdbc` and `next.jdbc.sql` by requiring the `next.jdbc.specs` namespace and instrumenting the functions. A convenience function is provided:
+You can enable argument checking for functions in `next.jdbc`, `next.jdbc.sql`, and `next.jdbc.prepare` by requiring the `next.jdbc.specs` namespace and instrumenting the functions. A convenience function is provided:
 
 ```clojure
 (require '[next.jdbc.specs :as specs])
-(specs/instrument) ; instruments all next.jdbc functions
+(specs/instrument) ; instruments all next.jdbc API functions
 
 (jdbc/execute! ds "SELECT * FROM fruit")
 Call to #'next.jdbc/execute! did not conform to spec.

src/next/jdbc.clj

@@ -67,6 +67,14 @@
   * `:classname` -- if you need to override the default for the `:dbtype`
       (or you want to use a database that next.jdbc does not know about!)
 
+  The following optional keys can be used to control how JDBC URLs are
+  assembled. This may be needed for `:dbtype` values that `next.jdbc`
+  does not recognize:
+  * `:dbname-separator` -- override the `/` or `:` that normally precedes
+      the database name in the JDBC URL
+  * `:host-prefix` -- override the `//` that normally precedes the IP
+      address or hostname in the JDBC URL
+
   Any additional options provided will be passed to the JDBC driver's
   `.getConnection` call as a `java.util.Properties` structure.
 

src/next/jdbc/connection.clj

@@ -71,7 +71,8 @@
   "A map of all known database types (including aliases) to the class name(s)
   and port that `next.jdbc` supports out of the box. Just for completeness,
   this also includes the prefixes used in the JDBC string for the `:host`
-  and `:dbname` (which are `//` and `/` respectively for nearly all types).
+  and `:dbname` (which are `//` and either `/` or `:` respectively for
+  nearly all types).
 
   For known database types, you can use `:dbtype` (and omit `:classname`).
 
@@ -81,6 +82,24 @@
 
      `{:dbtype \"acme\" :classname \"com.acme.JdbcDriver\" ...}`
 
+  The value of `:dbtype` should be the string that the driver is associated
+  with in the JDBC URL, i.e., the value that comes between the `jdbc:`
+  prefix and the `://<host>...` part. In the above example, the JDBC URL
+  that would be generated would be `jdbc:acme://<host>:<port>/<dbname>`.
+
+  If you want `next.jdbc` to omit the host/port part of the URL, specify
+  `:host :none`, which would produce a URL like: `jdbc:acme:<dbname>`,
+  which allows you to work with local databases (or drivers that do not
+  need host/port information).
+
+  The default prefix for the host name (or IP address) is `//`. You
+  can override this via the `:host-prefix` option.
+
+  The default separator between the host/port and the database name is `/`.
+  The default separator between the subprotocol and the database name,
+  for local databases with no host/port, is `:`. You can override this
+  via the `:dbname-separator` option.
+
   JDBC drivers are not provided by `next.jdbc` -- you need to specify the
   driver(s) you need as additional dependencies in your project. For
   example:
@@ -127,32 +146,44 @@
 
 (defn- spec->url+etc
   "Given a database spec, return a JDBC URL and a map of any additional options."
-  [{:keys [dbtype dbname host port classname] :as db-spec}]
+  [{:keys [dbtype dbname host port classname
+           dbname-separator host-prefix]
+    :as db-spec}]
   (let [;; allow aliases for dbtype
         subprotocol (aliases dbtype dbtype)
-        host (or host "127.0.0.1")
+        host        (or host "127.0.0.1")
-        port (or port (ports subprotocol))
+        port        (or port (ports subprotocol))
-        db-sep (dbname-separators dbtype "/")
+        db-sep      (or dbname-separator (dbname-separators dbtype "/"))
-        url (cond (= "h2:mem" dbtype)
+        local-sep   (or dbname-separator (dbname-separators dbtype ":"))
-                  (str "jdbc:" subprotocol ":" dbname ";DB_CLOSE_DELAY=-1")
+        url (cond (#{"derby" "hsqldb" "sqlite"} subprotocol)
+                  (str "jdbc:" subprotocol local-sep dbname)
+
                   (#{"h2"} subprotocol)
-                  (str "jdbc:" subprotocol ":"
+                  (str "jdbc:" subprotocol local-sep
                        (if (re-find #"^([A-Za-z]:)?[\./\\]" dbname)
                          ;; DB name starts with relative or absolute path
                          dbname
                          ;; otherwise make it local
                          (str "./" dbname)))
-                  (#{"derby" "hsqldb" "sqlite"} subprotocol)
+
-                  (str "jdbc:" subprotocol ":" dbname)
+                  (#{"h2:mem"} subprotocol)
+                  (str "jdbc:" subprotocol local-sep dbname ";DB_CLOSE_DELAY=-1")
+
                   (#{"timesten:client" "timesten:direct"} subprotocol)
-                  (str "jdbc:" subprotocol db-sep dbname)
+                  (str "jdbc:" subprotocol local-sep dbname)
+
+                  (= :none host)
+                  (str "jdbc:" subprotocol local-sep dbname)
+
                   :else
                   (str "jdbc:" subprotocol ":"
-                       (host-prefixes subprotocol "//")
+                       (or host-prefix (host-prefixes subprotocol "//"))
                        host
                        (when port (str ":" port))
                        db-sep dbname))
-        etc (dissoc db-spec :dbtype :dbname :host :port :classname)]
+        etc (dissoc db-spec
+                    :dbtype :dbname :host :port :classname
+                    :dbname-separator :host-prefix)]
     ;; verify the datasource is loadable
     (if-let [class-name (or classname (classnames subprotocol))]
       (swap! driver-cache update class-name

src/next/jdbc/specs.clj

@@ -26,15 +26,20 @@
 
 (s/def ::dbtype string?)
 (s/def ::dbname string?)
+(s/def ::dbname-separator string?)
 (s/def ::classname string?)
 (s/def ::user string?)
 (s/def ::password string?)
-(s/def ::host string?)
+(s/def ::host (s/or :name string?
+                    :none #{:none}))
+(s/def ::host-prefix string?)
 (s/def ::port pos-int?)
 (s/def ::db-spec-map (s/keys :req-un [::dbtype ::dbname]
                              :opt-un [::classname
                                       ::user ::password
-                                      ::host ::port]))
+                                      ::host ::port
+                                      ::dbname-separator
+                                      ::host-prefix]))
 
 (s/def ::connection #(instance? Connection %))
 (s/def ::datasource #(instance? DataSource %))
@@ -47,7 +52,13 @@
 (s/def ::connectable any?)
 (s/def ::key-map (s/map-of keyword? any?))
 (s/def ::example-map (s/map-of keyword? any? :min-count 1))
-(s/def ::opts-map (s/map-of keyword? any?))
+
+(s/def ::order-by-col (s/or :col keyword?
+                            :dir (s/cat :col keyword?
+                                        :dir #{:asc :desc})))
+(s/def ::order-by (s/coll-of ::order-by-col :kind vector? :min-count 1))
+(s/def ::opts-map (s/and (s/map-of keyword? any?)
+                         (s/keys :opt-un [::order-by])))
 
 (s/def ::transactable any?)
 
@@ -120,8 +131,11 @@
 (s/fdef sql/insert-multi!
         :args (s/and (s/cat :connectable ::connectable
                             :table keyword?
-                            :cols (s/coll-of keyword? :kind vector?)
+                            :cols (s/coll-of keyword?
-                            :rows (s/coll-of (s/coll-of any? :kind vector?) :kind vector?)
+                                             :kind sequential?
+                                             :min-count 1)
+                            :rows (s/coll-of (s/coll-of any? :kind sequential?)
+                                             :kind sequential?)
                             :opts (s/? ::opts-map))
                      #(apply = (count (:cols %))
                         (map count (:rows %)))))

src/next/jdbc/sql.clj

@@ -38,6 +38,7 @@
                                         [(conj conds (str e " = ?")) (conj params v)])))
                                   [[] []]
                                   key-map)]
+    (assert (seq where) "key-map may not be empty")
     (into [(str (str/upper-case (name clause)) " "
                 (str/join (if (= :where clause) " AND " ", ") where))]
           params)))
@@ -79,10 +80,11 @@
 (defn- for-order
   "Given an `:order-by` vector, return an `ORDER BY` clause."
   [order-by opts]
-  (if (vector? order-by)
+  (when-not (vector? order-by)
-    (str "ORDER BY "
+    (throw (IllegalArgumentException. ":order-by must be a vector")))
-         (str/join ", " (map #(for-order-col % opts) order-by)))
+  (assert (seq order-by) ":order-by may not be empty")
-    (throw (IllegalArgumentException. ":order-by must be a vector"))))
+  (str "ORDER BY "
+       (str/join ", " (map #(for-order-col % opts) order-by))))
 
 (defn- for-query
   "Given a table name and either a hash map of column names and values or a
@@ -147,6 +149,7 @@
   (let [entity-fn (:table-fn opts identity)
         params    (as-keys key-map opts)
         places    (as-? key-map opts)]
+    (assert (seq key-map) "key-map may not be empty")
     (into [(str "INSERT INTO " (entity-fn (name table))
                 " (" params ")"
                 " VALUES (" places ")")]
@@ -159,7 +162,11 @@
 
   Applies any `:table-fn` / `:column-fn` supplied in the options."
   [table cols rows opts]
-  (assert (apply = (count cols) (map count rows)))
+  (assert (apply = (count cols) (map count rows))
+          "column counts are not consistent across cols and rows")
+  ;; to avoid generating bad SQL
+  (assert (seq cols) "cols may not be empty")
+  (assert (seq rows) "rows may not be empty")
   (let [table-fn  (:table-fn opts identity)
         column-fn (:column-fn opts identity)
         params    (str/join ", " (map (comp column-fn name) cols))
@@ -199,9 +206,11 @@
   ([connectable table cols rows]
    (insert-multi! connectable table cols rows {}))
   ([connectable table cols rows opts]
-   (execute! connectable
+   (if (seq rows)
-             (for-insert-multi table cols rows opts)
+     (execute! connectable
-             (merge {:return-keys true} opts))))
+               (for-insert-multi table cols rows opts)
+               (merge {:return-keys true} opts))
+     [])))
 
 (defn query
   "Syntactic sugar over `execute!` to provide a query alias.

test/next/jdbc/connection_test.clj

@@ -51,6 +51,29 @@
     (is (= (#'c/spec->url+etc {:dbtype "sqlserver" :dbname db-name})
            (#'c/spec->url+etc {:dbtype "sqlserver" :dbname db-name :port 1433})))))
 
+(deftest custom-dbtypes
+  (is (= ["jdbc:acme:my-db" {}]
+         (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
+                             :dbname "my-db" :host :none})))
+  (is (= ["jdbc:acme://127.0.0.1/my-db" {}]
+         (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
+                             :dbname "my-db"})))
+  (is (= ["jdbc:acme://12.34.56.70:1234/my-db" {}]
+         (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
+                             :dbname "my-db" :host "12.34.56.70" :port 1234})))
+  (is (= ["jdbc:acme:dsn=my-db" {}]
+         (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
+                             :dbname "my-db" :host :none
+                             :dbname-separator ":dsn="})))
+  (is (= ["jdbc:acme:(*)127.0.0.1/my-db" {}]
+         (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
+                             :dbname "my-db"
+                             :host-prefix "(*)"})))
+  (is (= ["jdbc:acme:(*)12.34.56.70:1234/my-db" {}]
+         (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String"
+                             :dbname "my-db" :host "12.34.56.70" :port 1234
+                             :host-prefix "(*)"}))))
+
 ;; these are the 'local' databases that we can always test against
 (def test-db-type ["derby" "h2" "h2:mem" "hsqldb" "sqlite"])
 

test/next/jdbc/sql_test.clj

@@ -55,9 +55,9 @@
            ["DELETE FROM [user] WHERE id = ? and opt is null" 9]))))
 
 (deftest test-for-update
-  (testing "empty example (SQL error)"
+  (testing "empty example (would be a SQL error)"
-    (is (= (#'sql/for-update :user {:status 42} {} {:table-fn sql-server :column-fn mysql})
+    (is (thrown? AssertionError ; changed in #44
-           ["UPDATE [user] SET `status` = ? WHERE " 42])))
+                 (#'sql/for-update :user {:status 42} {} {:table-fn sql-server :column-fn mysql}))))
   (testing "by example"
     (is (= (#'sql/for-update :user {:status 42} {:id 9} {:table-fn sql-server :column-fn mysql})
            ["UPDATE [user] SET `status` = ? WHERE `id` = ?" 42 9])))
@@ -154,7 +154,30 @@
     (is (= 6 (count (sql/query (ds) ["select * from fruit"]))))
     (is (= {:next.jdbc/update-count 2}
            (sql/delete! (ds) :fruit ["id > ?" 4])))
-    (is (= 4 (count (sql/query (ds) ["select * from fruit"]))))))
+    (is (= 4 (count (sql/query (ds) ["select * from fruit"])))))
+  (testing "multiple insert/delete with sequential cols/rows" ; per #43
+    (is (= (cond (derby?)
+                 [{:1 nil}] ; WTF Apache Derby?
+                 (sqlite?)
+                 [{(keyword "last_insert_rowid()") 11}]
+                 :else
+                 [{:FRUIT/ID 9} {:FRUIT/ID 10} {:FRUIT/ID 11}])
+           (sql/insert-multi! (ds) :fruit
+                              '(:name :appearance :cost :grade)
+                              '(("Kiwi" "green & fuzzy" 100 99.9)
+                                ("Grape" "black" 10 50)
+                                ("Lemon" "yellow" 20 9.9)))))
+    (is (= 7 (count (sql/query (ds) ["select * from fruit"]))))
+    (is (= {:next.jdbc/update-count 1}
+           (sql/delete! (ds) :fruit {:id 9})))
+    (is (= 6 (count (sql/query (ds) ["select * from fruit"]))))
+    (is (= {:next.jdbc/update-count 2}
+           (sql/delete! (ds) :fruit ["id > ?" 4])))
+    (is (= 4 (count (sql/query (ds) ["select * from fruit"])))))
+  (testing "empty insert-multi!" ; per #44
+    (is (= [] (sql/insert-multi! (ds) :fruit
+                                 [:name :appearance :cost :grade]
+                                 [])))))
 
 (deftest no-empty-example-maps
   (is (thrown? clojure.lang.ExceptionInfo
@@ -163,3 +186,13 @@
                (sql/update! (ds) :fruit {} {})))
   (is (thrown? clojure.lang.ExceptionInfo
                (sql/delete! (ds) :fruit {}))))
+
+(deftest no-empty-columns
+  (is (thrown? clojure.lang.ExceptionInfo
+               (sql/insert-multi! (ds) :fruit [] [[] [] []]))))
+
+(deftest no-empty-order-by
+  (is (thrown? clojure.lang.ExceptionInfo
+               (sql/find-by-keys (ds) :fruit
+                                 {:name "Apple"}
+                                 {:order-by []}))))