diff --git a/CHANGELOG.md b/CHANGELOG.md index 527c5d1..10fcb5b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ Only accretive/fixative changes will be made from now on. * 1.3.next in progress + * Address [#267](https://github.com/seancorfield/next-jdbc/issues/267) by adding the `:schema-opts` option to override the default conventions for identifying foreign keys in columns. * Address [#264](https://github.com/seancorfield/next-jdbc/issues/264) by letting `insert-multi!` accept empty rows (and producing an empty result vector). This improves compatibility with `clojure.javaj.jdbc`. * Address [#258](https://github.com/seancorfield/next-jdbc/issues/258) by updating all the library (driver) versions in Getting Started to match the latest versions being tested (from `deps.edn`). * Expand examples for calling `next.jdbc.sql/find-by-keys` to show `LIKE` and `IN` clauses. diff --git a/doc/all-the-options.md b/doc/all-the-options.md index 9a31c3c..5eb0da7 100644 --- a/doc/all-the-options.md +++ b/doc/all-the-options.md @@ -70,6 +70,17 @@ In addition, `execute!` accepts the `:multi-rs true` option to return multiple r > Note: Subject to the caveats above about `:builder-fn`, that means that `plan`, `execute!`, `execute-one!`, and the "friendly" SQL functions will all accept these options for generating rows and result sets. +## Datafying & Navigating Rows and Result Sets + +Any function that produces a result set will accept the following options +that modify the behavior of `datafy` and `nav` applied to the rows in that +result set: + +* `:schema` -- override the conventions for identifying foreign keys and the related (primary) keys in the tables to which they refer, on a per table/column basis; can also be used to indicate a fk relationship is one-to-many or many-to-many rather than one-to-one or one-to-many, +* `:schema-opts` -- override the default conventions for identifying foreign keys and the related (primary) keys in the tables to which they refer, as a whole. + +See [`datafy`, `nav`, and `:schema`](/doc/datafy-nav-schema.md) for more details. + ## Statements & Prepared Statements Any function that creates a `Statement` or a `PreparedStatement` will accept the following options (see below for additional options for `PreparedStatement`): diff --git a/doc/datafy-nav-and-schema.md b/doc/datafy-nav-and-schema.md index eebc5d7..abbdb81 100644 --- a/doc/datafy-nav-and-schema.md +++ b/doc/datafy-nav-and-schema.md @@ -1,6 +1,6 @@ # `datafy`, `nav`, and the `:schema` option -Clojure 1.10 introduced a new namespace, [`clojure.datafy`](http://clojure.github.io/clojure/clojure.datafy-api.html), and two new protocols (`Datafiable` and `Navigable`) that allow for generalized, lazy navigation around data structures. Cognitect also released [REBL](http://rebl.cognitect.com/) -- a graphical, interactive tool for browsing Clojure data structures, based on the new `datafy` and `nav` functions. +Clojure 1.10 introduced a new namespace, [`clojure.datafy`](http://clojure.github.io/clojure/clojure.datafy-api.html), and two new protocols (`Datafiable` and `Navigable`) that allow for generalized, lazy navigation around data structures. Cognitect also released REBL (now Nubank's [Morse](https://github.com/nubank/morse)) -- a graphical, interactive tool for browsing Clojure data structures, based on the new `datafy` and `nav` functions. Shortly after REBL's release, I added experimental support to `clojure.java.jdbc` for `datafy` and `nav` that supported lazy navigation through result sets into foreign key relationships and connected rows and tables. `next.jdbc` bakes that support into result sets produced by `execute!` and `execute-one!`. @@ -13,8 +13,8 @@ Additional tools that understand `datafy` and `nav` include [Portal](https://git Here's how the process works, for result sets produced by `next.jdbc`: * `execute!` and `execute-one!` produce result sets containing rows that are `Datafiable`, -* Tools like Portal, Reveal, and REBL can call `datafy` on result sets to render them as "pure data" (which they already are, but this makes them also `Navigable`), -* Tools like Portal, Reveal, and REBL allow users to "drill down" into elements of rows in the "pure data" result set, using `nav`, +* Tools like Portal, Reveal, and Morse can call `datafy` on result sets to render them as "pure data" (which they already are, but this makes them also `Navigable`), +* Tools like Portal, Reveal, and Morse allow users to "drill down" into elements of rows in the "pure data" result set, using `nav`, * If a column in a row represents a foreign key into another table, calling `nav` will fetch the related row(s), * Those can in turn be `datafy`'d and `nav`'d to continue drilling down through connected data in the database. @@ -26,6 +26,18 @@ By default, `next.jdbc` assumes that a column named `id` or `)`). + +For `:fk-suffix`, the `_` is still permitted and optional in the column name, +so if you specified `:schema-opts {:fk-suffix "fk"}` then `addressfk` and +`address_fk` would both be treated as foreign keys into the `address` table. + +The `:pk-fn` can + The default behavior in the example above is equivalent to this `:schema` value: ```clojure @@ -35,6 +47,16 @@ The default behavior in the example above is equivalent to this `:schema` value: {:schema {:contact/addressid :address/id}}) ``` +or these `:schema-opts` values: + +```clojure +(jdbc/execute! ds + ["select * from contact where city = ?" "San Francisco"] + ;; a one-to-one or many-to-one relationship + {:schema-opts {:fk-suffix "id" :pk "id" + :pk-fn (constantly "id")}}) +``` + If you had a table to track the valid/bouncing status of email addresses over time, `:deliverability`, where `email` is the non-unique key, you could provide automatic navigation into that using: ```clojure @@ -45,6 +67,11 @@ If you had a table to track the valid/bouncing status of email addresses over ti :address/email [:deliverability/email]}}) ``` +Since this relies on a foreign key that does not follow a standard suffix +pattern, there is no comparable `:schema-opts` version. In addition, the +`:schema-opts` approach cannot designate a one-to-many or many-to-many +relationship. + When you indicate a `*-to-many` relationship, by wrapping the foreign table/key in a vector, `next.jdbc`'s implementation of `nav` will fetch a multi-row result set from the target table. If you use foreign key constraints in your database, you could probably generate this `:schema` data structure automatically from the metadata in your database. Similarly, if you use a library that depends on an entity relationship map (such as [seql](https://exoscale.github.io/seql/) or [walkable](https://walkable.gitlab.io/)), then you could probably generate this `:schema` data structure from that entity map. @@ -55,7 +82,11 @@ Making rows datafiable is implemented by adding metadata to each row with a key When called (`datafy` on a row), it adds metadata to the row with a key of `clojure.core.protocols/nav` and another function as the value. That function also closes over the connectable and options passed in. -When that is called (`nav` on a row, column name, and column value), if a `:schema` entry exists for that column or it matches the default convention described above, then it will fetch row(s) using `next.jdbc`'s `Executable` functions `-execute-one` or `-execute-all`, passing in the connectable and options closed over. +When that is called (`nav` on a row, column name, and column value), if a +`:schema` entry exists for that column or it matches the convention described +above (either by default or via `:schema-opts`), then it will fetch row(s) +using `next.jdbc`'s `Executable` functions `-execute-one` or `-execute-all`, +passing in the connectable and options closed over. The protocol `next.jdbc.result-set/DatafiableRow` has a default implementation of `datafiable-row` for `clojure.lang.IObj` that just adds the metadata to support `datafy`. There is also an implementation baked into the result set handling behind `plan` so that you can call `datafiable-row` directly during reduction and get a fully-realized row that can be `datafy`'d (and then `nav`igated). diff --git a/doc/tips-and-tricks.md b/doc/tips-and-tricks.md index 68b8e64..b2a8a02 100644 --- a/doc/tips-and-tricks.md +++ b/doc/tips-and-tricks.md @@ -551,3 +551,6 @@ If you are using `plan`, you'll most likely be accessing columns by just the lab [] (jdbc/plan ds ["select * from some_table"])) ``` + +See also [`datafy`, `nav`, and `:schema` > **SQLite**](/doc/datafy-nav-and-schema.md#sqlite) +for additional caveats on the `next.jdbc.datafy` namespace when using SQLite. diff --git a/src/next/jdbc/result_set.clj b/src/next/jdbc/result_set.clj index 4073c7e..0c83d9e 100644 --- a/src/next/jdbc/result_set.clj +++ b/src/next/jdbc/result_set.clj @@ -624,11 +624,9 @@ ;; in reality, this is going to be over-optimistic and will like cause `nav` ;; to fail on attempts to navigate into result sets that are not hash maps (datafiable-row [this connectable opts] - (vary-meta - this - assoc - `core-p/datafy (navize-row connectable opts) - `core-p/nav (navable-row connectable opts)))) + (vary-meta this assoc + `core-p/datafy (navize-row connectable opts) + `core-p/nav (navable-row connectable opts)))) (defn datafiable-result-set "Given a ResultSet, a connectable, and an options hash map, return a fully @@ -1025,12 +1023,52 @@ (defn- default-schema "The default schema lookup rule for column names. - If a column name ends with `_id` or `id`, it is assumed to be a foreign key - into the table identified by the first part of the column name." - [col] - (let [[_ table] (re-find #"(?i)^(.+?)_?id$" (name col))] + We have a foreign key column suffix convention of `` or + `
_`, which maps to a (primary) key in the ``. + + By default, both `` and `` are assumed to be `id`. That can be + overridden by the `:schema-opts` hash map in the options: + * `:fk-suffix` -- the suffix for foreign key columns, default `id` + * `:pk` -- the (primary) key column name, default `id` + * `:pk-fn` -- a function to apply to the table name and the value of `:pk` + to get the (primary) key column name, default `(constantly )`." + [opts col] + (let [fk-suffix (get-in opts [:schema-opts :fk-suffix] "id") + pk (get-in opts [:schema-opts :pk] "id") + pk-fn (get-in opts [:schema-opts :pk-fn] (constantly (name pk))) + [_ table] (re-find (re-pattern (str "(?i)^(.+?)_?" + (name fk-suffix) + "$")) + (name col))] (when table - [(keyword table) :id]))) + [(keyword table) (keyword (pk-fn table pk))]))) + +(comment + (default-schema {} :userstatusid) + (default-schema {} :userstatus_id) + (default-schema {} :user_statusid) + (default-schema {:schema-opts {:fk-suffix "did"}} :user_id) + (default-schema {:schema-opts {:fk-suffix "did"}} :user_did) + (default-schema {:schema-opts {:fk-suffix "(did|id)"}} :user_id) + (default-schema {:schema-opts {:fk-suffix "(did|id)"}} :user_did) + (default-schema {:schema-opts {:fk-suffix "(did|id)" + :pk :did}} :user_did) + (default-schema {:schema-opts {:fk-suffix "(did|id)" + :pk :did + :pk-fn (fn [table pk] + (if (= "user" table) + "id" + pk))}} + :user_did) + (default-schema {:schema-opts {:fk-suffix "(did|id)" + :pk :did + :pk-fn (fn [table pk] + (if (= "user" table) + "id" + pk))}} + :book_did) + ) (defn- expand-schema "Given a (possibly nil) schema entry, return it expanded to a triple of: @@ -1072,50 +1110,12 @@ (defn- navize-row "Given a connectable object, return a function that knows how to turn a row - into a `nav`igable object. + into a `nav`igable object. - A `:schema` option can provide a map from qualified column names - (`:
/`) to tuples that indicate for which table they are a - foreign key, the name of the key within that table, and (optionality) the - cardinality of that relationship (`:many`, `:one`). - - If no `:schema` item is provided for a column, the convention of `
id` or - `
_id` is used, and the assumption is that such columns are foreign keys - in the `
` portion of their name, the key is called `id`, and the - cardinality is `:one`. - - Rows are looked up using `-execute-all` or `-execute-one`, and the `:table-fn` - option, if provided, is applied to the assumed table name and `:column-fn` if - provided to the assumed foreign key column name." + See navable-row below for more details." [connectable opts] (fn [row] - (vary-meta - row - assoc - `core-p/nav (fn [_ k v] - (try - (let [[table fk cardinality] - (expand-schema k (or (get-in opts [:schema k]) - (default-schema k)))] - (if (and fk connectable) - (let [table-fn (:table-fn opts identity) - column-fn (:column-fn opts identity) - exec-fn! (if (= :many cardinality) - p/-execute-all - p/-execute-one)] - (exec-fn! connectable - [(str "SELECT * FROM " - (table-fn (name table)) - " WHERE " - (column-fn (name fk)) - " = ?") - v] - opts)) - v)) - (catch Exception _ - ;; assume an exception means we just cannot - ;; navigate anywhere, so return just the value - v)))))) + (vary-meta row assoc `core-p/nav (navable-row connectable opts)))) (defn- navable-row "Given a connectable object, return a function that knows how to `nav` @@ -1131,6 +1131,8 @@ in the `
` portion of their name, the key is called `id`, and the cardinality is `:one`. + That convention can in turn be modified via the `:schema-opts` option. + Rows are looked up using `-execute-all` or `-execute-one`, and the `:table-fn` option, if provided, is applied to the assumed table name and `:column-fn` if provided to the assumed foreign key column name." @@ -1139,7 +1141,7 @@ (try (let [[table fk cardinality] (expand-schema k (or (get-in opts [:schema k]) - (default-schema k)))] + (default-schema opts k)))] (if (and fk connectable) (let [table-fn (:table-fn opts identity) column-fn (:column-fn opts identity) @@ -1156,6 +1158,6 @@ opts)) v)) (catch Exception _ - ;; assume an exception means we just cannot - ;; navigate anywhere, so return just the value + ;; assume an exception means we just cannot + ;; navigate anywhere, so return just the value v))))