luciano/next-jdbc
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Address #16 by renaming reducible! to plan for Beta 1

Browse source
This commit is contained in:
Sean Corfield 2019-05-22 16:22:14 -07:00
parent 146539528a
commit a32c5b9e66
14 changed files with 34 additions and 38 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
CHANGELOG.md
View file

@ -11,5 +11,6 @@
The following changes have been committed to the **master** branch and will be in the next release (Beta 1):
* Set up CircleCI testing (just local DBs for now).
* Address #21 by adding `next.jdbc.specs` -- still need to update the docs!
* Address #21 by adding `next.jdbc.specs` and documenting basic usage.
* Fix #19 by caching loaded database driver classes.
* Address #16 by renaming `reducible!` to `plan` (**BREAKING CHANGE!**).

9
CONTRIBUTING.md
View file

@ -1,11 +1,6 @@
# Contributing to `next.jdbc`
The next generation of [`clojure.java.jdbc`](https://github.com/clojure/java.jdbc)
is *still not accepting contributions* because it I have not yet decided
whether it will become a Contrib project (like `clojure.java.jdbc`)
or whether it will continue to be a public, open source project.
Please do not send Pull Requests at this point!
Feel free to [open Issues](https://github.com/seancorfield/next-jdbc/issues) with opinions
and suggestions -- I'm happy to have discussions about any aspect of the library!
I welcome Pull Requests for source code changes *that are accompanied by tests*, and for documentation changes. For any substantial change, please open an issue for discussion first, or find me in the `#sql` stream on Clojurians Zulip or the `#sql` channel on Clojurians Slack to chat about it.

14
README.md
View file

@ -24,11 +24,11 @@ Why another JDBC library? Why a different API from `clojure.java.jdbc`?
Those are my three primary drivers. In addition, the `db-spec`-as-hash-map approach in `clojure.java.jdbc` has caused a lot of frustration and confusion in the past, especially with the wide range of conflicting options that are supported. `next.jdbc` is heavily protocol-based so it's easier to mix'n'match how you use it with direct Java JDBC code (and the protocol-based approach contributes to the improved performance overall). There's a much clearer path of `db-spec` -> `DataSource` -> `Connection` now, which should steer people toward more connection reuse and better performing apps.
I also wanted `datafy`/`nav` support baked right in (it was added to `clojure.java.jdbc` back in December 2018 as an undocumented, experimental API in a separate namespace). It is the default behavior for `execute!` and `execute!`. The protocol-based function `next.jdbc.result-set/datafiable-row` can be used with `reducible!` if you need to add `datafy`/`nav` support to rows you are creating in your reduction.
I also wanted `datafy`/`nav` support baked right in (it was added to `clojure.java.jdbc` back in December 2018 as an undocumented, experimental API in a separate namespace). It is the default behavior for `execute!` and `execute!`. The protocol-based function `next.jdbc.result-set/datafiable-row` can be used with `plan` if you need to add `datafy`/`nav` support to rows you are creating in your reduction.
At this point, I would consider the API to be fairly stable and [alpha builds are now available on Clojars](https://clojars.org/seancorfield/next.jdbc) (2019-04-20). The "syntactic sugar" SQL functions (`insert!`, `query`, `update!`, and `delete!`) go beyond what I wanted to include in the core API so they are in `next.jdbc.sql`. I know that their equivalents in `clojure.java.jdbc` are heavily used (based on the number of questions and JIRA issues I get).
As `next.jdbc` moves from alpha to beta, the last breaking change has been made (renaming `reducible!` to `plan`) and the API should be considered stable. Only accretive and fixative changes will be made from now on.
I am still [considering whether this should go into Contrib](https://github.com/seancorfield/next-jdbc/issues/3) as new namespaces in `clojure.java.jdbc` or whether it will continue to live standalone so I'm not accepting Pull Requests yet (but I can't disable them on GitHub!). There are pros and cons to both choices, in my mind.
[Alpha builds have been available on Clojars](https://clojars.org/seancorfield/next.jdbc) for over a month (as of 2019-05-22). The first beta build will be available shortly. The "syntactic sugar" SQL functions (`insert!`, `query`, `update!`, and `delete!`) go beyond what I wanted to include in the core API so they are in `next.jdbc.sql`. I know that their equivalents in `clojure.java.jdbc` are heavily used (based on the number of questions and JIRA issues I get).
## Usage
@ -37,11 +37,11 @@ The primary concepts behind `next.jdbc` are that you start by producing a `javax
From a `DataSource`, either you or `next.jdbc` can create a `java.sql.Connection` via the `get-connection` function. You can specify an options hash map to `get-connection` to modify the connection that is created: `:read-only`, `:auto-commit`.
The primary SQL execution API in `next.jdbc` is:
* `reducible!` -- yields an `IReduceInit` that, when reduced, executes the SQL statement and then reduces over the `ResultSet` with as little overhead as possible.
* `plan` -- yields an `IReduceInit` that, when reduced, executes the SQL statement and then reduces over the `ResultSet` with as little overhead as possible.
* `execute!` -- executes the SQL statement and produces a vector of realized hash maps, that use qualified keywords for the column names, of the form `:<table>/<column>`. If you join across multiple tables, the qualified keywords will reflect the originating tables for each of the columns. If the SQL produces named values that do not come from an associated table, a simple, unqualified keyword will be used. The realized hash maps returned by `execute!` are `Datafiable` and thus `Navigable` (see Clojure 1.10's `datafy` and `nav` functions, and tools like Cognitect's REBL). Alternatively, you can specify `{:builder-fn rs/as-arrays}` and produce a vector with column names followed by vectors of row values. `rs/as-maps` is the default for `:builder-fn` but there are also `rs/as-unqualified-maps` and `rs/as-unqualified-arrays` if you want unqualified `:<column>` column names (and there are also lower-case variants of all of these).
* `execute-one!` -- executes the SQL statement and produces a single realized hash map. The realized hash map returned by `execute-one!` is `Datafiable` and thus `Navigable`.
In addition, there are API functions to create `PreparedStatement`s (`prepare`) from `Connection`s, which can be passed to `reducible!`, `execute!`, or `execute-one!`, and to run code inside a transaction (the `transact` function and the `with-transaction` macro).
In addition, there are API functions to create `PreparedStatement`s (`prepare`) from `Connection`s, which can be passed to `plan`, `execute!`, or `execute-one!`, and to run code inside a transaction (the `transact` function and the `with-transaction` macro).
Since `next.jdbc` uses raw Java JDBC types, you can use `with-open` directly to reuse connections and ensure they are cleaned up correctly:
@ -49,7 +49,7 @@ Since `next.jdbc` uses raw Java JDBC types, you can use `with-open` directly to
(let [my-datasource (get-datasource {:dbtype "..." :dbname "..." ...})]
(with-open [connection (get-connection my-datasource)]
(execute! connection [...])
(reduce my-fn init-value (reducible! connection [...]))
(reduce my-fn init-value (plan connection [...]))
(execute! connection [...])
```
@ -58,7 +58,7 @@ Since `next.jdbc` uses raw Java JDBC types, you can use `with-open` directly to
There are three intended usage scenarios that may drive the API to change:
* Execute a SQL statement to obtain a single, fully-realized, `Datafiable` hash map that represents either the first row from a `ResultSet`, the first generated keys result (again, from a `ResultSet`), or the first result where neither of those are available (`next.jdbc` yields `{:next.jdbc/update-count N}` when it can only return an update count). This usage is currently supported by `execute-one!`.
* Execute a SQL statement to obtain a fully-realized, `Datafiable` result set -- a vector of hash maps. This usage is supported by `execute!`. You can also produce a vector of column names/row values (`next.jdbc.result-set/as-arrays`).
* Execute a SQL statement and process it in a single eager operation, which may allow for the results to be streamed from the database (how to persuade JDBC to do that is database-specific!), and which cleans up resources before returning the result -- even if the reduction is short-circuited via `reduced`. This usage is supported by `reducible!`.
* Execute a SQL statement and process it in a single eager operation, which may allow for the results to be streamed from the database (how to persuade JDBC to do that is database-specific!), and which cleans up resources before returning the result -- even if the reduction is short-circuited via `reduced`. This usage is supported by `plan`.
In addition, convenience functions -- "syntactic sugar" -- are provided to insert rows, run queries, update rows, and delete rows, using the same names as in `clojure.java.jdbc`. These are in `next.jdbc.sql` since they involve SQL creation -- they may move into a separate "sibling" library, since they are not part of the intended core API.

2
doc/all-the-options.md
View file

@ -36,7 +36,7 @@ The "friendly" SQL functions all accept the following options:
Any function that might realize a row or a result set will accept:
* `:builder-fn` -- a function that implements the `RowBuilder` and `ResultSetBuilder` protocols; strictly speaking, `reducible!` and `execute-one!` only need `RowBuilder` to be implemented (and `reducible!` only needs that if it actually has to realize a row) but most generation functions will implement both for ease of use.
* `:builder-fn` -- a function that implements the `RowBuilder` and `ResultSetBuilder` protocols; strictly speaking, `plan` and `execute-one!` only need `RowBuilder` to be implemented (and `plan` only needs that if it actually has to realize a row) but most generation functions will implement both for ease of use.
## Prepared Statements

2
doc/datafy-nav-and-schema.md
View file

@ -43,6 +43,6 @@ When called (`datafy` on a row), it adds metadata to the row with a key of `cloj
When that is called (`nav` on a row, column name, and column value), if a `:schema` entry exists for that column or it matches the default convention described above, then it will fetch row(s) using `next.jdbc`'s `Executable` functions `-execute-one` or `-execute-all`, passing in the connectable and options closed over.
The protocol `next.jdbc.result-set/DatafiableRow` has a default implementation of `datafiable-row` for `clojure.lang.IObj` that just adds the metadata to support `datafy`. There is also an implementation baked into the result set handling behind `reducible!` so that you can call `datafiable-row` directly during reduction and get a fully-realized row that can be `datafy`'d (and then `nav`igated).
The protocol `next.jdbc.result-set/DatafiableRow` has a default implementation of `datafiable-row` for `clojure.lang.IObj` that just adds the metadata to support `datafy`. There is also an implementation baked into the result set handling behind `plan` so that you can call `datafiable-row` directly during reduction and get a fully-realized row that can be `datafy`'d (and then `nav`igated).
[<: All The Options](/doc/all-the-options.md) | [Migration from `clojure.java.jdbc` :>](/doc/migration-from-clojure-java-jdbc.md)

2
doc/friendly-sql-functions.md
View file

@ -1,6 +1,6 @@
# Friendly SQL Functions
In [Getting Started](/doc/getting-started.md), we used `execute!` and `execute-one!` for all our SQL operations, except when we were reducing a result set. These functions (and `reducible!`) all expect a "connectable" and a vector containing a SQL string followed by any parameter values required.
In [Getting Started](/doc/getting-started.md), we used `execute!` and `execute-one!` for all our SQL operations, except when we were reducing a result set. These functions (and `plan`) all expect a "connectable" and a vector containing a SQL string followed by any parameter values required.
A "connectable" can be a `javax.sql.DataSource`, a `java.sql.Connection`, or something that can produce a datasource (when `get-datasource` is called on it). It can also be a `java.sql.PreparedStatement` but we'll cover that a bit later...

10
doc/getting-started.md
View file

@ -85,7 +85,7 @@ While these functions are fine for retrieving result sets as data, most of the t
```clojure
user=> (into #{}
(map :ADDRESS/NAME)
(jdbc/reducible! ds ["select * from address"]))
(jdbc/plan ds ["select * from address"]))
#{"Sean Corfield" "Someone Else"}
user=>
```
@ -95,7 +95,7 @@ This produces a set of all the unique names in the `address` table, directly fro
```clojure
user=> (into #{}
(map :name)
(jdbc/reducible! ds ["select * from address"]))
(jdbc/plan ds ["select * from address"]))
#{"Sean Corfield" "Someone Else"}
user=>
```
@ -112,7 +112,7 @@ If you want to run multiple SQL operations without that overhead each time, you
(with-open [con (jdbc/get-connection ds)]
(jdbc/execute! con ...)
(jdbc/execute! con ...)
(into [] (map :column) (jdbc/reducible! con ...)))
(into [] (map :column) (jdbc/plan con ...)))
```
If any of these operations throws an exception, the connection will still be closed but operations prior to the exception will have already been committed to the database. If you want to reuse a connection across multiple operations but have them all rollback if an exception occurs, you can use `next.jdbc/with-transaction`:
@ -121,7 +121,7 @@ If any of these operations throws an exception, the connection will still be clo
(jdbc/with-transaction [tx ds]
(jdbc/execute! tx ...)
(jdbc/execute! tx ...)
(into [] (map :column) (jdbc/reducible! tx ...)))
(into [] (map :column) (jdbc/plan tx ...)))
```
If `with-transaction` is given a datasource, it will create and close the connection for you. If you pass in an existing connection, `with-transaction` will set up a transaction on that connection and, after either committing or rolling back the transaction, will restore the state of the connection and leave it open:
@ -132,7 +132,7 @@ If `with-transaction` is given a datasource, it will create and close the connec
(jdbc/with-transaction [tx con] ; will commit or rollback this group:
(jdbc/execute! tx ...)
(jdbc/execute! tx ...)
(into [] (map :column) (jdbc/reducible! tx ...)))
(into [] (map :column) (jdbc/plan tx ...)))
(jdbc/execute! con ...)) ; committed
```

4
doc/migration-from-clojure-java-jdbc.md
View file

@ -21,7 +21,7 @@ If you used `:as-arrays? true`, you will need to use a `:builder-fn` option of `
* `get-datasource` -- has no equivalent in `clojure.java.jdbc` but is intended to emphasize `javax.sql.DataSource` as a starting point,
* `get-connection` -- overlaps with `clojure.java.jdbc` (and returns a `java.sql.Connection`) but accepts only a subset of the options (`:dbtype`/`:dbname` hash map, `String` JDBC URI); `clojure.java.jdbc/get-connection` accepts `{:datasource ds}` whereas `next.jdbc/get-connection` accepts the `javax.sql.DataSource` object directly,
* `prepare` -- somewhat similar to `clojure.java.jdbc/prepare-statement` but it accepts a vector of SQL and parameters (compared to just a raw SQL string),
* `reducible!` -- somewhat similar to `clojure.java.jdbc/reducible-query` but accepts arbitrary SQL statements for execution,
* `plan` -- somewhat similar to `clojure.java.jdbc/reducible-query` but accepts arbitrary SQL statements for execution,
* `execute!` -- has no direct equivalent in `clojure.java.jdbc` (but it can replace most uses of both `query` and `db-do-commands`),
* `execute-one!` -- has no equivalent in `clojure.java.jdbc` (but it can replace most uses of `query` that currently use `:result-set-fn first`),
* `transact` -- similar to `clojure.java.jdbc/db-transaction*`,
@ -45,7 +45,7 @@ The `next.jdbc.sql` namespace contains several functions with similarities to `c
If you are using `:identifiers` and/or `:entities`, you will need to change to appropriate `:builder-fn` and/or `:table-fn`/`:column-fn` options. For the latter, instead of the `quoted` function, there is the `next.jdbc.quoted` namespace which contains functions for the common quoting strategies.
If you are using `:result-set-fn` and/or `:row-fn`, you will need to change to explicit calls (to the result set function, or to `map` the row function), or to use the `reducible!` approach with `reduce` or various transducing functions. Note: this means that result sets are never exposed lazily in `next.jdbc` -- in `clojure.java.jdbc` you had to be careful that your `:result-set-fn` was eager, but in `next.jdbc` you either reduce the result set eagerly (via `reducible!`) or you get a fully-realized result set data structure back (from `execute!` and `execute-one!`). As with `clojure.java.jdbc` however, you can still stream result sets from the database and process them via reduction (was `reducible-query`, now `reducible!`). Remember that you can terminate a reduction early by using the `reduced` function to wrap the final value you produce.
If you are using `:result-set-fn` and/or `:row-fn`, you will need to change to explicit calls (to the result set function, or to `map` the row function), or to use the `plan` approach with `reduce` or various transducing functions. Note: this means that result sets are never exposed lazily in `next.jdbc` -- in `clojure.java.jdbc` you had to be careful that your `:result-set-fn` was eager, but in `next.jdbc` you either reduce the result set eagerly (via `plan`) or you get a fully-realized result set data structure back (from `execute!` and `execute-one!`). As with `clojure.java.jdbc` however, you can still stream result sets from the database and process them via reduction (was `reducible-query`, now `plan`). Remember that you can terminate a reduction early by using the `reduced` function to wrap the final value you produce.
## Further Minor differences

4
doc/prepared-statements.md
View file

@ -2,9 +2,9 @@
Under the hood, whenever you ask `next.jdbc` to execute some SQL it creates a `java.sql.PreparedStatement`, adds in the parameters you provide, and then calls `.execute` on it. Then it attempts to get a `ResultSet` from that and either return it or process it. If you asked for generated keys to be returned, that `ResultSet` will contain the those generated keys if your database supports it, otherwise it will be whatever the `.execute` function produces. If no `ResultSet` is available at all, `next.jdbc` will ask for the count of updated rows and return that as if it were a result set.
If you have a SQL operation that you intend to run multiple times on the same `java.sql.Connection`, it may be worth creating the prepared statement yourself and reusing it. `next.jdbc/prepare` accepts a connection and a vector of SQL and optional parameters and returns a `java.sql.PreparedStatement` which can be passed to `reducible!`, `execute!`, or `execute-one!` as the first argument. It is your responsibility to close the prepared statement after it has been used.
If you have a SQL operation that you intend to run multiple times on the same `java.sql.Connection`, it may be worth creating the prepared statement yourself and reusing it. `next.jdbc/prepare` accepts a connection and a vector of SQL and optional parameters and returns a `java.sql.PreparedStatement` which can be passed to `plan`, `execute!`, or `execute-one!` as the first argument. It is your responsibility to close the prepared statement after it has been used.
If you need to pass an option map to `reducible!`, `execute!`, or `execute-one!` when passing a prepared statement, you must pass `nil` or `[]` as the second argument:
If you need to pass an option map to `plan`, `execute!`, or `execute-one!` when passing a prepared statement, you must pass `nil` or `[]` as the second argument:
```clojure
(with-open [con (jdbc/get-connection ds)]

8
doc/result-set-builders.md
View file

@ -24,7 +24,7 @@ This protocol defines four functions and is used whenever `next.jdbc` needs to m
* `(with-column builder row i)` -- given the row so far, fetches column `i` from the current row of the `ResultSet`, converts it to a Clojure value, and adds it to the row (for `as-maps` this is a call to `.getObject`, a call to `read-column-by-index` -- see the `ReadableColumn` protocol below, and a call to `assoc!`),
* `(row! builder row)` -- completes the row (a `(persistent! row)` call by default).
`execute!` and `execute-one!` call these functions for each row they need to build. `reducible!` _may_ call these functions if the reducing function causes a row to be materialized.
`execute!` and `execute-one!` call these functions for each row they need to build. `plan` _may_ call these functions if the reducing function causes a row to be materialized.
## ResultSet Protocol
@ -34,7 +34,7 @@ This protocol defines three functions and is used whenever `next.jdbc` needs to
* `(with-row builder rs row)` -- given the result set so far and a new row, returns the updated result set (a `(conj! rs row)` call by default),
* `(rs! builder rs)` -- completes the result set (a `(persistent! rs)` call by default).
Only `execute!` expects this protocol to be implemented. `execute-one!` and `reducible!` do not call these functions.
Only `execute!` expects this protocol to be implemented. `execute-one!` and `plan` do not call these functions.
## Result Set Builder Functions
@ -42,13 +42,13 @@ The `as-*` functions described above are all implemented in terms of these proto
The options hash map for any `next.jdbc` function can contain a `:builder-fn` key and the value is used as the row/result set builder function. The tests for `next.jdbc.result-set` include a [record-based builder function](https://github.com/seancorfield/next-jdbc/blob/master/test/next/jdbc/result_set_test.clj#L148-L164) as an example of how you can extend this to satisfy your needs.
The options hash map passed to the builder function will contain a `:next.jdbc/sql-params` key, whose value is the SQL + parameters vector passed into the top-level `next.jdbc` functions (`reducible!`, `execute!`, and `execute-one!`).
The options hash map passed to the builder function will contain a `:next.jdbc/sql-params` key, whose value is the SQL + parameters vector passed into the top-level `next.jdbc` functions (`plan`, `execute!`, and `execute-one!`).
# ReadableColumn
As mentioned above, when `with-column` is called, the expectation is that the row builder will call `.getObject` on the current state of the `ResultSet` object with the column index and will then call `read-column-by-index`, passing the column value, the `ResultSetMetaData`, and the column index. That function is part of the `ReadableColumn` protocol that you can extend to handle conversion of arbitrary database-specific types to Clojure values.
In addition, inside `reducible!`, as each value is looked up by name in the current state of the `ResultSet` object, the `read-column-by-label` function is called, again passing the column value and the column label (the name used in the SQL to identify that column). This function is also part of the `ReadableColumn` protocol.
In addition, inside `plan`, as each value is looked up by name in the current state of the `ResultSet` object, the `read-column-by-label` function is called, again passing the column value and the column label (the name used in the SQL to identify that column). This function is also part of the `ReadableColumn` protocol.
The default implementation of this protocol is for these two functions to return `nil` as `nil`, a `Boolean` value as a canonical `true` or `false` value (unfortunately, JDBC drivers cannot be relied on to return unique values here!), and for all other objects to be returned as-is.

4
src/next/jdbc.clj
View file

@ -9,7 +9,7 @@
* `PreparedStatement` -- SQL and parameters combined, from a connection,
and the following functions and a macro:
* `reducible!` -- given a connectable and SQL + parameters or a statement,
* `plan` -- given a connectable and SQL + parameters or a statement,
return a reducible that, when reduced will execute the SQL and consume
the `ResultSet` produced,
* `execute!` -- given a connectable and SQL + parameters or a statement,
@ -128,7 +128,7 @@
([connection sql-params opts]
(p/prepare connection sql-params opts)))
(defn reducible!
(defn plan
"General SQL execution function.
Returns a reducible that, when reduced, runs the SQL and yields the result.

2
src/next/jdbc/result_set.clj
View file

@ -208,7 +208,7 @@
on, to lazily navigate through foreign key relationships into other tables.
If `datafiable-row` is called when reducing the result set produced by
`next.jdbc/reducible!`, the row is fully-realized from the `ResultSet`
`next.jdbc/plan`, the row is fully-realized from the `ResultSet`
first."
(datafiable-row [this connectable opts]
"Produce a datafiable representation of a row from a `ResultSet`."))

4
src/next/jdbc/specs.clj
View file

@ -59,7 +59,7 @@
:sql-params ::sql-params
:opts (s/? ::opts-map)))
(s/fdef jdbc/reducible!
(s/fdef jdbc/plan
:args (s/alt :prepared (s/cat :stmt ::prepared-statement)
:sql (s/cat :connectable ::connectable
:sql-params ::sql-params
@ -146,7 +146,7 @@
(st/instrument [`jdbc/get-datasource
`jdbc/get-connection
`jdbc/prepare
`jdbc/reducible!
`jdbc/plan
`jdbc/execute!
`jdbc/execute-one!
`jdbc/transact

4
test/next/jdbc_test.clj
View file

@ -15,11 +15,11 @@
(specs/instrument)
(deftest basic-tests
(testing "reducible!"
(testing "plan"
(is (= "Apple"
(reduce (fn [_ row] (reduced (:name row)))
nil
(jdbc/reducible!
(jdbc/plan
(ds)
["select * from fruit where appearance = ?" "red"])))))
(testing "execute-one!"