diff --git a/CHANGELOG.md b/CHANGELOG.md index cfdf8bf..88d46fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,10 +6,15 @@ Only accretive/fixative changes will be made from now on. The following changes have been committed to the **master** branch since the 1.0.1 release: -* Fix #45 by adding TimesTen driver support. +* Fix #46 by allowing `:host` to be `:none` which tells `next.jdbc` to omit the host/port section of the JDBC URL, so that local databases can be used with `:dbtype`/`:classname` for database types that `next.jdbc` does not know. Also added `:dbname-separator` and `:host-prefix` to the "db-spec" to allow fine-grained control over how the JDBC URL is assembled. +* Fix #45 by adding [TimesTen](https://www.oracle.com/database/technologies/related/timesten.html) driver support. +* Fix #44 so that `insert-multi!` with an empty `rows` vector returns `[]`. +* Fix #43 by adjusting the spec for `insert-multi!` to "require less" of the `cols` and `rows` arguments. * Fix #42 by adding specs for `execute-batch!` and `set-parameters` in `next.jdbc.prepare`. * Fix #41 by improving docstrings and documentation, especially around prepared statement handling. * Fix #40 by adding `next.jdbc.prepare/execute-batch!`. +* Added `assert`s in `next.jdbc.sql` as more informative errors for cases that would generate SQL exceptions (from malformed SQL). +* Added spec for `:order-by` to reflect what is actually permitted. * Expose `next.jdbc.connect/dbtypes` as a table of known database types and aliases, along with their class name(s), port, and other JDBC string components. ## Stable Builds diff --git a/doc/all-the-options.md b/doc/all-the-options.md index 34d6ea4..83eb335 100644 --- a/doc/all-the-options.md +++ b/doc/all-the-options.md @@ -10,7 +10,9 @@ Although `get-datasource` does not accept options, the "db spec" hash map passed * `:dbtype` -- a string that identifies the type of JDBC database being used, * `:dbname` -- a string that identifies the name of the actual database being used, +* `:dbname-separator` -- an optional string that can be used to override the `/` or `:` that is normally placed in front of the database name in the JDBC URL, * `:host` -- an optional string that identifies the IP address or hostname of the server on which the database is running; the default is `"127.0.0.1"`, +* `:host-prefix` -- an optional string that can be used to override the `//` that is normally placed in front of the IP address or hostname in the JDBC URL, * `:port` -- an optional integer that identifies the port on which the database is running; for common database types, `next.jdbc` knows the default so this should only be needed for non-standard setups or "exotic" database types, * `:classname` -- an optional string that identifies the name of the JDBC driver class to be used for the connection; for common database types, `next.jdbc` knows the default so this should only be needed for "exotic" database types, * `:user` -- an optional string that identifies the database username to be used when authenticating, diff --git a/doc/getting-started.md b/doc/getting-started.md index 6fd02ea..d834bef 100644 --- a/doc/getting-started.md +++ b/doc/getting-started.md @@ -33,6 +33,8 @@ For the examples in this documentation, we will use a local H2 database on disk, com.h2database/h2 {:mvn/version "1.4.197"}}} ``` +### Create & Populate a Database + In this REPL session, we'll define an H2 datasource, create a database with a simple table, and then add some data and query it: ```clojure @@ -59,10 +61,15 @@ user=> (jdbc/execute! ds ["select * from address"]) [#:ADDRESS{:ID 1, :NAME "Sean Corfield", :EMAIL "sean@corfield.org"}] user=> ``` + +### The "db-spec" hash map + We described the database with just `:dbtype` and `:dbname` because it is created as a local file and needs no authentication. For most databases, you would need `:user` and `:password` for authentication, and if the database is running on a remote machine you would need `:host` and possibly `:port` (`next.jdbc` tries to guess the correct port based on the `:dbtype`). > Note: You can see the full list of `:dbtype` values supported in [next.jdbc/get-datasource](https://cljdoc.org/d/seancorfield/next.jdbc/CURRENT/api/next.jdbc#get-datasource)'s docstring. If you need this programmatically, you can get it from the [next.jdbc.connection/dbtypes](https://cljdoc.org/d/seancorfield/next.jdbc/CURRENT/api/next.jdbc.connection#dbtypes) hash map. If those lists differ, the hash map is the definitive list (and I'll need to fix the docstring!). The docstring of that Var explains how to tell `next.jdbc` about additional databases. +### `execute!` & `execute-one!` + We used `execute!` to create the `address` table, to insert a new row into it, and to query it. In all three cases, `execute!` returns a vector of hash maps with namespace-qualified keys, representing the result set from the operation, if available. When no result set is produced, `next.jdbc` returns a "result set" containing the "update count" from the operation (which is usually the number of rows affected). By default, H2 uses uppercase names and `next.jdbc` returns these as-is. If you only want a single row back -- the first row of any result set, generated keys, or update counts -- you can use `execute-one!` instead. Continuing the REPL session, we'll insert another address and ask for the generated keys to be returned, and then we'll query for a single row: @@ -80,6 +87,8 @@ user=> Since we used `execute-one!`, we get just one row back (a hash map). This also shows how you provide parameters to SQL statements -- with `?` in the SQL and then the corresponding parameter values in the vector after the SQL string. +### `plan` & Reducing Result Sets + While these functions are fine for retrieving result sets as data, most of the time you want to process that data efficiently, so `next.jdbc` provides a SQL execution function that works with `reduce` and with transducers to consume the result set without the intermediate overhead of creating Clojure data structures for every row: ```clojure @@ -140,11 +149,11 @@ If `with-transaction` is given a datasource, it will create and close the connec As you are developing with `next.jdbc`, it can be useful to have assistance from `clojure.spec` in checking calls to `next.jdbc`'s functions, to provide explicit argument checking and/or better error messages for some common mistakes, e.g., trying to pass a plain SQL string where a vector (containing a SQL string, and no parameters) is expected. -You can enable argument checking for functions in both `next.jdbc` and `next.jdbc.sql` by requiring the `next.jdbc.specs` namespace and instrumenting the functions. A convenience function is provided: +You can enable argument checking for functions in `next.jdbc`, `next.jdbc.sql`, and `next.jdbc.prepare` by requiring the `next.jdbc.specs` namespace and instrumenting the functions. A convenience function is provided: ```clojure (require '[next.jdbc.specs :as specs]) -(specs/instrument) ; instruments all next.jdbc functions +(specs/instrument) ; instruments all next.jdbc API functions (jdbc/execute! ds "SELECT * FROM fruit") Call to #'next.jdbc/execute! did not conform to spec. diff --git a/src/next/jdbc.clj b/src/next/jdbc.clj index 75eba04..0fe7e16 100644 --- a/src/next/jdbc.clj +++ b/src/next/jdbc.clj @@ -67,6 +67,14 @@ * `:classname` -- if you need to override the default for the `:dbtype` (or you want to use a database that next.jdbc does not know about!) + The following optional keys can be used to control how JDBC URLs are + assembled. This may be needed for `:dbtype` values that `next.jdbc` + does not recognize: + * `:dbname-separator` -- override the `/` or `:` that normally precedes + the database name in the JDBC URL + * `:host-prefix` -- override the `//` that normally precedes the IP + address or hostname in the JDBC URL + Any additional options provided will be passed to the JDBC driver's `.getConnection` call as a `java.util.Properties` structure. diff --git a/src/next/jdbc/connection.clj b/src/next/jdbc/connection.clj index c2165b6..55700d4 100644 --- a/src/next/jdbc/connection.clj +++ b/src/next/jdbc/connection.clj @@ -71,7 +71,8 @@ "A map of all known database types (including aliases) to the class name(s) and port that `next.jdbc` supports out of the box. Just for completeness, this also includes the prefixes used in the JDBC string for the `:host` - and `:dbname` (which are `//` and `/` respectively for nearly all types). + and `:dbname` (which are `//` and either `/` or `:` respectively for + nearly all types). For known database types, you can use `:dbtype` (and omit `:classname`). @@ -81,6 +82,24 @@ `{:dbtype \"acme\" :classname \"com.acme.JdbcDriver\" ...}` + The value of `:dbtype` should be the string that the driver is associated + with in the JDBC URL, i.e., the value that comes between the `jdbc:` + prefix and the `://...` part. In the above example, the JDBC URL + that would be generated would be `jdbc:acme://:/`. + + If you want `next.jdbc` to omit the host/port part of the URL, specify + `:host :none`, which would produce a URL like: `jdbc:acme:`, + which allows you to work with local databases (or drivers that do not + need host/port information). + + The default prefix for the host name (or IP address) is `//`. You + can override this via the `:host-prefix` option. + + The default separator between the host/port and the database name is `/`. + The default separator between the subprotocol and the database name, + for local databases with no host/port, is `:`. You can override this + via the `:dbname-separator` option. + JDBC drivers are not provided by `next.jdbc` -- you need to specify the driver(s) you need as additional dependencies in your project. For example: @@ -127,32 +146,44 @@ (defn- spec->url+etc "Given a database spec, return a JDBC URL and a map of any additional options." - [{:keys [dbtype dbname host port classname] :as db-spec}] + [{:keys [dbtype dbname host port classname + dbname-separator host-prefix] + :as db-spec}] (let [;; allow aliases for dbtype subprotocol (aliases dbtype dbtype) - host (or host "127.0.0.1") - port (or port (ports subprotocol)) - db-sep (dbname-separators dbtype "/") - url (cond (= "h2:mem" dbtype) - (str "jdbc:" subprotocol ":" dbname ";DB_CLOSE_DELAY=-1") + host (or host "127.0.0.1") + port (or port (ports subprotocol)) + db-sep (or dbname-separator (dbname-separators dbtype "/")) + local-sep (or dbname-separator (dbname-separators dbtype ":")) + url (cond (#{"derby" "hsqldb" "sqlite"} subprotocol) + (str "jdbc:" subprotocol local-sep dbname) + (#{"h2"} subprotocol) - (str "jdbc:" subprotocol ":" + (str "jdbc:" subprotocol local-sep (if (re-find #"^([A-Za-z]:)?[\./\\]" dbname) ;; DB name starts with relative or absolute path dbname ;; otherwise make it local (str "./" dbname))) - (#{"derby" "hsqldb" "sqlite"} subprotocol) - (str "jdbc:" subprotocol ":" dbname) + + (#{"h2:mem"} subprotocol) + (str "jdbc:" subprotocol local-sep dbname ";DB_CLOSE_DELAY=-1") + (#{"timesten:client" "timesten:direct"} subprotocol) - (str "jdbc:" subprotocol db-sep dbname) + (str "jdbc:" subprotocol local-sep dbname) + + (= :none host) + (str "jdbc:" subprotocol local-sep dbname) + :else (str "jdbc:" subprotocol ":" - (host-prefixes subprotocol "//") + (or host-prefix (host-prefixes subprotocol "//")) host (when port (str ":" port)) db-sep dbname)) - etc (dissoc db-spec :dbtype :dbname :host :port :classname)] + etc (dissoc db-spec + :dbtype :dbname :host :port :classname + :dbname-separator :host-prefix)] ;; verify the datasource is loadable (if-let [class-name (or classname (classnames subprotocol))] (swap! driver-cache update class-name diff --git a/src/next/jdbc/specs.clj b/src/next/jdbc/specs.clj index 63cc8bc..dc7f24d 100644 --- a/src/next/jdbc/specs.clj +++ b/src/next/jdbc/specs.clj @@ -26,15 +26,20 @@ (s/def ::dbtype string?) (s/def ::dbname string?) +(s/def ::dbname-separator string?) (s/def ::classname string?) (s/def ::user string?) (s/def ::password string?) -(s/def ::host string?) +(s/def ::host (s/or :name string? + :none #{:none})) +(s/def ::host-prefix string?) (s/def ::port pos-int?) (s/def ::db-spec-map (s/keys :req-un [::dbtype ::dbname] :opt-un [::classname ::user ::password - ::host ::port])) + ::host ::port + ::dbname-separator + ::host-prefix])) (s/def ::connection #(instance? Connection %)) (s/def ::datasource #(instance? DataSource %)) @@ -47,7 +52,13 @@ (s/def ::connectable any?) (s/def ::key-map (s/map-of keyword? any?)) (s/def ::example-map (s/map-of keyword? any? :min-count 1)) -(s/def ::opts-map (s/map-of keyword? any?)) + +(s/def ::order-by-col (s/or :col keyword? + :dir (s/cat :col keyword? + :dir #{:asc :desc}))) +(s/def ::order-by (s/coll-of ::order-by-col :kind vector? :min-count 1)) +(s/def ::opts-map (s/and (s/map-of keyword? any?) + (s/keys :opt-un [::order-by]))) (s/def ::transactable any?) @@ -120,8 +131,11 @@ (s/fdef sql/insert-multi! :args (s/and (s/cat :connectable ::connectable :table keyword? - :cols (s/coll-of keyword? :kind vector?) - :rows (s/coll-of (s/coll-of any? :kind vector?) :kind vector?) + :cols (s/coll-of keyword? + :kind sequential? + :min-count 1) + :rows (s/coll-of (s/coll-of any? :kind sequential?) + :kind sequential?) :opts (s/? ::opts-map)) #(apply = (count (:cols %)) (map count (:rows %))))) diff --git a/src/next/jdbc/sql.clj b/src/next/jdbc/sql.clj index f976774..d58a867 100644 --- a/src/next/jdbc/sql.clj +++ b/src/next/jdbc/sql.clj @@ -38,6 +38,7 @@ [(conj conds (str e " = ?")) (conj params v)]))) [[] []] key-map)] + (assert (seq where) "key-map may not be empty") (into [(str (str/upper-case (name clause)) " " (str/join (if (= :where clause) " AND " ", ") where))] params))) @@ -79,10 +80,11 @@ (defn- for-order "Given an `:order-by` vector, return an `ORDER BY` clause." [order-by opts] - (if (vector? order-by) - (str "ORDER BY " - (str/join ", " (map #(for-order-col % opts) order-by))) - (throw (IllegalArgumentException. ":order-by must be a vector")))) + (when-not (vector? order-by) + (throw (IllegalArgumentException. ":order-by must be a vector"))) + (assert (seq order-by) ":order-by may not be empty") + (str "ORDER BY " + (str/join ", " (map #(for-order-col % opts) order-by)))) (defn- for-query "Given a table name and either a hash map of column names and values or a @@ -147,6 +149,7 @@ (let [entity-fn (:table-fn opts identity) params (as-keys key-map opts) places (as-? key-map opts)] + (assert (seq key-map) "key-map may not be empty") (into [(str "INSERT INTO " (entity-fn (name table)) " (" params ")" " VALUES (" places ")")] @@ -159,7 +162,11 @@ Applies any `:table-fn` / `:column-fn` supplied in the options." [table cols rows opts] - (assert (apply = (count cols) (map count rows))) + (assert (apply = (count cols) (map count rows)) + "column counts are not consistent across cols and rows") + ;; to avoid generating bad SQL + (assert (seq cols) "cols may not be empty") + (assert (seq rows) "rows may not be empty") (let [table-fn (:table-fn opts identity) column-fn (:column-fn opts identity) params (str/join ", " (map (comp column-fn name) cols)) @@ -199,9 +206,11 @@ ([connectable table cols rows] (insert-multi! connectable table cols rows {})) ([connectable table cols rows opts] - (execute! connectable - (for-insert-multi table cols rows opts) - (merge {:return-keys true} opts)))) + (if (seq rows) + (execute! connectable + (for-insert-multi table cols rows opts) + (merge {:return-keys true} opts)) + []))) (defn query "Syntactic sugar over `execute!` to provide a query alias. diff --git a/test/next/jdbc/connection_test.clj b/test/next/jdbc/connection_test.clj index f38f482..c9bfe2b 100644 --- a/test/next/jdbc/connection_test.clj +++ b/test/next/jdbc/connection_test.clj @@ -51,6 +51,29 @@ (is (= (#'c/spec->url+etc {:dbtype "sqlserver" :dbname db-name}) (#'c/spec->url+etc {:dbtype "sqlserver" :dbname db-name :port 1433}))))) +(deftest custom-dbtypes + (is (= ["jdbc:acme:my-db" {}] + (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String" + :dbname "my-db" :host :none}))) + (is (= ["jdbc:acme://127.0.0.1/my-db" {}] + (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String" + :dbname "my-db"}))) + (is (= ["jdbc:acme://12.34.56.70:1234/my-db" {}] + (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String" + :dbname "my-db" :host "12.34.56.70" :port 1234}))) + (is (= ["jdbc:acme:dsn=my-db" {}] + (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String" + :dbname "my-db" :host :none + :dbname-separator ":dsn="}))) + (is (= ["jdbc:acme:(*)127.0.0.1/my-db" {}] + (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String" + :dbname "my-db" + :host-prefix "(*)"}))) + (is (= ["jdbc:acme:(*)12.34.56.70:1234/my-db" {}] + (#'c/spec->url+etc {:dbtype "acme" :classname "java.lang.String" + :dbname "my-db" :host "12.34.56.70" :port 1234 + :host-prefix "(*)"})))) + ;; these are the 'local' databases that we can always test against (def test-db-type ["derby" "h2" "h2:mem" "hsqldb" "sqlite"]) diff --git a/test/next/jdbc/sql_test.clj b/test/next/jdbc/sql_test.clj index 31d8f99..ea89850 100644 --- a/test/next/jdbc/sql_test.clj +++ b/test/next/jdbc/sql_test.clj @@ -55,9 +55,9 @@ ["DELETE FROM [user] WHERE id = ? and opt is null" 9])))) (deftest test-for-update - (testing "empty example (SQL error)" - (is (= (#'sql/for-update :user {:status 42} {} {:table-fn sql-server :column-fn mysql}) - ["UPDATE [user] SET `status` = ? WHERE " 42]))) + (testing "empty example (would be a SQL error)" + (is (thrown? AssertionError ; changed in #44 + (#'sql/for-update :user {:status 42} {} {:table-fn sql-server :column-fn mysql})))) (testing "by example" (is (= (#'sql/for-update :user {:status 42} {:id 9} {:table-fn sql-server :column-fn mysql}) ["UPDATE [user] SET `status` = ? WHERE `id` = ?" 42 9]))) @@ -154,7 +154,30 @@ (is (= 6 (count (sql/query (ds) ["select * from fruit"])))) (is (= {:next.jdbc/update-count 2} (sql/delete! (ds) :fruit ["id > ?" 4]))) - (is (= 4 (count (sql/query (ds) ["select * from fruit"])))))) + (is (= 4 (count (sql/query (ds) ["select * from fruit"]))))) + (testing "multiple insert/delete with sequential cols/rows" ; per #43 + (is (= (cond (derby?) + [{:1 nil}] ; WTF Apache Derby? + (sqlite?) + [{(keyword "last_insert_rowid()") 11}] + :else + [{:FRUIT/ID 9} {:FRUIT/ID 10} {:FRUIT/ID 11}]) + (sql/insert-multi! (ds) :fruit + '(:name :appearance :cost :grade) + '(("Kiwi" "green & fuzzy" 100 99.9) + ("Grape" "black" 10 50) + ("Lemon" "yellow" 20 9.9))))) + (is (= 7 (count (sql/query (ds) ["select * from fruit"])))) + (is (= {:next.jdbc/update-count 1} + (sql/delete! (ds) :fruit {:id 9}))) + (is (= 6 (count (sql/query (ds) ["select * from fruit"])))) + (is (= {:next.jdbc/update-count 2} + (sql/delete! (ds) :fruit ["id > ?" 4]))) + (is (= 4 (count (sql/query (ds) ["select * from fruit"]))))) + (testing "empty insert-multi!" ; per #44 + (is (= [] (sql/insert-multi! (ds) :fruit + [:name :appearance :cost :grade] + []))))) (deftest no-empty-example-maps (is (thrown? clojure.lang.ExceptionInfo @@ -163,3 +186,13 @@ (sql/update! (ds) :fruit {} {}))) (is (thrown? clojure.lang.ExceptionInfo (sql/delete! (ds) :fruit {})))) + +(deftest no-empty-columns + (is (thrown? clojure.lang.ExceptionInfo + (sql/insert-multi! (ds) :fruit [] [[] [] []])))) + +(deftest no-empty-order-by + (is (thrown? clojure.lang.ExceptionInfo + (sql/find-by-keys (ds) :fruit + {:name "Apple"} + {:order-by []}))))