luciano/next-jdbc
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix #267 by adding :schema-opts

Browse source
This commit is contained in:
Sean Corfield 2023-12-06 21:32:50 -08:00
parent d6fe4c1577
commit 4910030ad3
5 changed files with 106 additions and 58 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
CHANGELOG.md
View file

@ -3,6 +3,7 @@
Only accretive/fixative changes will be made from now on.
* 1.3.next in progress
* Address [#267](https://github.com/seancorfield/next-jdbc/issues/267) by adding the `:schema-opts` option to override the default conventions for identifying foreign keys in columns.
* Address [#264](https://github.com/seancorfield/next-jdbc/issues/264) by letting `insert-multi!` accept empty rows (and producing an empty result vector). This improves compatibility with `clojure.javaj.jdbc`.
* Address [#258](https://github.com/seancorfield/next-jdbc/issues/258) by updating all the library (driver) versions in Getting Started to match the latest versions being tested (from `deps.edn`).
* Expand examples for calling `next.jdbc.sql/find-by-keys` to show `LIKE` and `IN` clauses.

11
doc/all-the-options.md
View file

@ -70,6 +70,17 @@ In addition, `execute!` accepts the `:multi-rs true` option to return multiple r
> Note: Subject to the caveats above about `:builder-fn`, that means that `plan`, `execute!`, `execute-one!`, and the "friendly" SQL functions will all accept these options for generating rows and result sets.
## Datafying & Navigating Rows and Result Sets
Any function that produces a result set will accept the following options
that modify the behavior of `datafy` and `nav` applied to the rows in that
result set:
* `:schema` -- override the conventions for identifying foreign keys and the related (primary) keys in the tables to which they refer, on a per table/column basis; can also be used to indicate a fk relationship is one-to-many or many-to-many rather than one-to-one or one-to-many,
* `:schema-opts` -- override the default conventions for identifying foreign keys and the related (primary) keys in the tables to which they refer, as a whole.
See [`datafy`, `nav`, and `:schema`](/doc/datafy-nav-schema.md) for more details.
## Statements & Prepared Statements
Any function that creates a `Statement` or a `PreparedStatement` will accept the following options (see below for additional options for `PreparedStatement`):

39
doc/datafy-nav-and-schema.md
View file

@ -1,6 +1,6 @@
# `datafy`, `nav`, and the `:schema` option
Clojure 1.10 introduced a new namespace, [`clojure.datafy`](http://clojure.github.io/clojure/clojure.datafy-api.html), and two new protocols (`Datafiable` and `Navigable`) that allow for generalized, lazy navigation around data structures. Cognitect also released [REBL](http://rebl.cognitect.com/) -- a graphical, interactive tool for browsing Clojure data structures, based on the new `datafy` and `nav` functions.
Clojure 1.10 introduced a new namespace, [`clojure.datafy`](http://clojure.github.io/clojure/clojure.datafy-api.html), and two new protocols (`Datafiable` and `Navigable`) that allow for generalized, lazy navigation around data structures. Cognitect also released REBL (now Nubank's [Morse](https://github.com/nubank/morse)) -- a graphical, interactive tool for browsing Clojure data structures, based on the new `datafy` and `nav` functions.
Shortly after REBL's release, I added experimental support to `clojure.java.jdbc` for `datafy` and `nav` that supported lazy navigation through result sets into foreign key relationships and connected rows and tables. `next.jdbc` bakes that support into result sets produced by `execute!` and `execute-one!`.
@ -13,8 +13,8 @@ Additional tools that understand `datafy` and `nav` include [Portal](https://git
Here's how the process works, for result sets produced by `next.jdbc`:
* `execute!` and `execute-one!` produce result sets containing rows that are `Datafiable`,
* Tools like Portal, Reveal, and REBL can call `datafy` on result sets to render them as "pure data" (which they already are, but this makes them also `Navigable`),
* Tools like Portal, Reveal, and REBL allow users to "drill down" into elements of rows in the "pure data" result set, using `nav`,
* Tools like Portal, Reveal, and Morse can call `datafy` on result sets to render them as "pure data" (which they already are, but this makes them also `Navigable`),
* Tools like Portal, Reveal, and Morse allow users to "drill down" into elements of rows in the "pure data" result set, using `nav`,
* If a column in a row represents a foreign key into another table, calling `nav` will fetch the related row(s),
* Those can in turn be `datafy`'d and `nav`'d to continue drilling down through connected data in the database.
@ -26,6 +26,18 @@ By default, `next.jdbc` assumes that a column named `<something>id` or `<somethi
You can override this default behavior for any column in any table by providing a `:schema` option that is a hash map whose keys are column names (usually the table-qualified keywords that `next.jdbc` produces by default) and whose values are table-qualified keywords, optionally wrapped in vectors, that identity the name of the table to which that column is a foreign key and the name of the key column within that table.
As of 1.3.next, you can also override this behavior via the `:schema-opts`
option. This is a hash map whose keys can be:
* `:fk-suffix` -- a string used instead of `"id"` to identify foreign keys,
* `:pk` -- a string used instead of `"id"` for the primary key column in the target table,
* `:pk-fn` -- a function that takes the table name and the value of `:pk` and returns the name of the primary key column in the target table, instead of just using the value of `:pk` (the default is effectively `(constantly <pk>)`).
For `:fk-suffix`, the `_` is still permitted and optional in the column name,
so if you specified `:schema-opts {:fk-suffix "fk"}` then `addressfk` and
`address_fk` would both be treated as foreign keys into the `address` table.
The `:pk-fn` can
The default behavior in the example above is equivalent to this `:schema` value:
```clojure
@ -35,6 +47,16 @@ The default behavior in the example above is equivalent to this `:schema` value:
{:schema {:contact/addressid :address/id}})
```
or these `:schema-opts` values:
```clojure
(jdbc/execute! ds
["select * from contact where city = ?" "San Francisco"]
;; a one-to-one or many-to-one relationship
{:schema-opts {:fk-suffix "id" :pk "id"
:pk-fn (constantly "id")}})
```
If you had a table to track the valid/bouncing status of email addresses over time, `:deliverability`, where `email` is the non-unique key, you could provide automatic navigation into that using:
```clojure
@ -45,6 +67,11 @@ If you had a table to track the valid/bouncing status of email addresses over ti
:address/email [:deliverability/email]}})
```
Since this relies on a foreign key that does not follow a standard suffix
pattern, there is no comparable `:schema-opts` version. In addition, the
`:schema-opts` approach cannot designate a one-to-many or many-to-many
relationship.
When you indicate a `*-to-many` relationship, by wrapping the foreign table/key in a vector, `next.jdbc`'s implementation of `nav` will fetch a multi-row result set from the target table.
If you use foreign key constraints in your database, you could probably generate this `:schema` data structure automatically from the metadata in your database. Similarly, if you use a library that depends on an entity relationship map (such as [seql](https://exoscale.github.io/seql/) or [walkable](https://walkable.gitlab.io/)), then you could probably generate this `:schema` data structure from that entity map.
@ -55,7 +82,11 @@ Making rows datafiable is implemented by adding metadata to each row with a key
When called (`datafy` on a row), it adds metadata to the row with a key of `clojure.core.protocols/nav` and another function as the value. That function also closes over the connectable and options passed in.
When that is called (`nav` on a row, column name, and column value), if a `:schema` entry exists for that column or it matches the default convention described above, then it will fetch row(s) using `next.jdbc`'s `Executable` functions `-execute-one` or `-execute-all`, passing in the connectable and options closed over.
When that is called (`nav` on a row, column name, and column value), if a
`:schema` entry exists for that column or it matches the convention described
above (either by default or via `:schema-opts`), then it will fetch row(s)
using `next.jdbc`'s `Executable` functions `-execute-one` or `-execute-all`,
passing in the connectable and options closed over.
The protocol `next.jdbc.result-set/DatafiableRow` has a default implementation of `datafiable-row` for `clojure.lang.IObj` that just adds the metadata to support `datafy`. There is also an implementation baked into the result set handling behind `plan` so that you can call `datafiable-row` directly during reduction and get a fully-realized row that can be `datafy`'d (and then `nav`igated).

3
doc/tips-and-tricks.md
View file

@ -551,3 +551,6 @@ If you are using `plan`, you'll most likely be accessing columns by just the lab
[]
(jdbc/plan ds ["select * from some_table"]))
```
See also [`datafy`, `nav`, and `:schema` > **SQLite**](/doc/datafy-nav-and-schema.md#sqlite)
for additional caveats on the `next.jdbc.datafy` namespace when using SQLite.

100
src/next/jdbc/result_set.clj
View file

@ -624,9 +624,7 @@
;; in reality, this is going to be over-optimistic and will like cause `nav`
;; to fail on attempts to navigate into result sets that are not hash maps
(datafiable-row [this connectable opts]
(vary-meta
this
assoc
(vary-meta this assoc
`core-p/datafy (navize-row connectable opts)
`core-p/nav (navable-row connectable opts))))
@ -1025,12 +1023,52 @@
(defn- default-schema
"The default schema lookup rule for column names.
If a column name ends with `_id` or `id`, it is assumed to be a foreign key
into the table identified by the first part of the column name."
[col]
(let [[_ table] (re-find #"(?i)^(.+?)_?id$" (name col))]
We have a foreign key column suffix convention of `<table><fk>` or
`<table>_<fk>`, which maps to a (primary) key in the `<table` called
`<pk>`.
By default, both `<fk>` and `<pk>` are assumed to be `id`. That can be
overridden by the `:schema-opts` hash map in the options:
* `:fk-suffix` -- the suffix for foreign key columns, default `id`
* `:pk` -- the (primary) key column name, default `id`
* `:pk-fn` -- a function to apply to the table name and the value of `:pk`
to get the (primary) key column name, default `(constantly <pk>)`."
[opts col]
(let [fk-suffix (get-in opts [:schema-opts :fk-suffix] "id")
pk (get-in opts [:schema-opts :pk] "id")
pk-fn (get-in opts [:schema-opts :pk-fn] (constantly (name pk)))
[_ table] (re-find (re-pattern (str "(?i)^(.+?)_?"
(name fk-suffix)
"$"))
(name col))]
(when table
[(keyword table) :id])))
[(keyword table) (keyword (pk-fn table pk))])))
(comment
(default-schema {} :userstatusid)
(default-schema {} :userstatus_id)
(default-schema {} :user_statusid)
(default-schema {:schema-opts {:fk-suffix "did"}} :user_id)
(default-schema {:schema-opts {:fk-suffix "did"}} :user_did)
(default-schema {:schema-opts {:fk-suffix "(did|id)"}} :user_id)
(default-schema {:schema-opts {:fk-suffix "(did|id)"}} :user_did)
(default-schema {:schema-opts {:fk-suffix "(did|id)"
:pk :did}} :user_did)
(default-schema {:schema-opts {:fk-suffix "(did|id)"
:pk :did
:pk-fn (fn [table pk]
(if (= "user" table)
"id"
pk))}}
:user_did)
(default-schema {:schema-opts {:fk-suffix "(did|id)"
:pk :did
:pk-fn (fn [table pk]
(if (= "user" table)
"id"
pk))}}
:book_did)
)
(defn- expand-schema
"Given a (possibly nil) schema entry, return it expanded to a triple of:
@ -1074,48 +1112,10 @@
"Given a connectable object, return a function that knows how to turn a row
into a `nav`igable object.
A `:schema` option can provide a map from qualified column names
(`:<table>/<column>`) to tuples that indicate for which table they are a
foreign key, the name of the key within that table, and (optionality) the
cardinality of that relationship (`:many`, `:one`).
If no `:schema` item is provided for a column, the convention of `<table>id` or
`<table>_id` is used, and the assumption is that such columns are foreign keys
in the `<table>` portion of their name, the key is called `id`, and the
cardinality is `:one`.
Rows are looked up using `-execute-all` or `-execute-one`, and the `:table-fn`
option, if provided, is applied to the assumed table name and `:column-fn` if
provided to the assumed foreign key column name."
See navable-row below for more details."
[connectable opts]
(fn [row]
(vary-meta
row
assoc
`core-p/nav (fn [_ k v]
(try
(let [[table fk cardinality]
(expand-schema k (or (get-in opts [:schema k])
(default-schema k)))]
(if (and fk connectable)
(let [table-fn (:table-fn opts identity)
column-fn (:column-fn opts identity)
exec-fn! (if (= :many cardinality)
p/-execute-all
p/-execute-one)]
(exec-fn! connectable
[(str "SELECT * FROM "
(table-fn (name table))
" WHERE "
(column-fn (name fk))
" = ?")
v]
opts))
v))
(catch Exception _
;; assume an exception means we just cannot
;; navigate anywhere, so return just the value
v))))))
(vary-meta row assoc `core-p/nav (navable-row connectable opts))))
(defn- navable-row
"Given a connectable object, return a function that knows how to `nav`
@ -1131,6 +1131,8 @@
in the `<table>` portion of their name, the key is called `id`, and the
cardinality is `:one`.
That convention can in turn be modified via the `:schema-opts` option.
Rows are looked up using `-execute-all` or `-execute-one`, and the `:table-fn`
option, if provided, is applied to the assumed table name and `:column-fn` if
provided to the assumed foreign key column name."
@ -1139,7 +1141,7 @@
(try
(let [[table fk cardinality]
(expand-schema k (or (get-in opts [:schema k])
(default-schema k)))]
(default-schema opts k)))]
(if (and fk connectable)
(let [table-fn (:table-fn opts identity)
column-fn (:column-fn opts identity)