fix #267 by adding :schema-opts

CHANGELOG.md

@@ -3,6 +3,7 @@
 Only accretive/fixative changes will be made from now on.
 
 * 1.3.next in progress
+  * Address [#267](https://github.com/seancorfield/next-jdbc/issues/267) by adding the `:schema-opts` option to override the default conventions for identifying foreign keys in columns.
   * Address [#264](https://github.com/seancorfield/next-jdbc/issues/264) by letting `insert-multi!` accept empty rows (and producing an empty result vector). This improves compatibility with `clojure.javaj.jdbc`.
   * Address [#258](https://github.com/seancorfield/next-jdbc/issues/258) by updating all the library (driver) versions in Getting Started to match the latest versions being tested (from `deps.edn`).
   * Expand examples for calling `next.jdbc.sql/find-by-keys` to show `LIKE` and `IN` clauses.

doc/all-the-options.md

@@ -70,6 +70,17 @@ In addition, `execute!` accepts the `:multi-rs true` option to return multiple r
 
 > Note: Subject to the caveats above about `:builder-fn`, that means that `plan`, `execute!`, `execute-one!`, and the "friendly" SQL functions will all accept these options for generating rows and result sets.
 
+## Datafying & Navigating Rows and Result Sets
+
+Any function that produces a result set will accept the following options
+that modify the behavior of `datafy` and `nav` applied to the rows in that
+result set:
+
+* `:schema` -- override the conventions for identifying foreign keys and the related (primary) keys in the tables to which they refer, on a per table/column basis; can also be used to indicate a fk relationship is one-to-many or many-to-many rather than one-to-one or one-to-many,
+* `:schema-opts` -- override the default conventions for identifying foreign keys and the related (primary) keys in the tables to which they refer, as a whole.
+
+See [`datafy`, `nav`, and `:schema`](/doc/datafy-nav-schema.md) for more details.
+
 ## Statements & Prepared Statements
 
 Any function that creates a `Statement` or a `PreparedStatement` will accept the following options (see below for additional options for `PreparedStatement`):

doc/datafy-nav-and-schema.md

@@ -1,6 +1,6 @@
 # `datafy`, `nav`, and the `:schema` option
 
-Clojure 1.10 introduced a new namespace, [`clojure.datafy`](http://clojure.github.io/clojure/clojure.datafy-api.html), and two new protocols (`Datafiable` and `Navigable`) that allow for generalized, lazy navigation around data structures. Cognitect also released [REBL](http://rebl.cognitect.com/) -- a graphical, interactive tool for browsing Clojure data structures, based on the new `datafy` and `nav` functions.
+Clojure 1.10 introduced a new namespace, [`clojure.datafy`](http://clojure.github.io/clojure/clojure.datafy-api.html), and two new protocols (`Datafiable` and `Navigable`) that allow for generalized, lazy navigation around data structures. Cognitect also released REBL (now Nubank's [Morse](https://github.com/nubank/morse)) -- a graphical, interactive tool for browsing Clojure data structures, based on the new `datafy` and `nav` functions.
 
 Shortly after REBL's release, I added experimental support to `clojure.java.jdbc` for `datafy` and `nav` that supported lazy navigation through result sets into foreign key relationships and connected rows and tables. `next.jdbc` bakes that support into result sets produced by `execute!` and `execute-one!`.
 
@@ -13,8 +13,8 @@ Additional tools that understand `datafy` and `nav` include [Portal](https://git
 Here's how the process works, for result sets produced by `next.jdbc`:
 
 * `execute!` and `execute-one!` produce result sets containing rows that are `Datafiable`,
-* Tools like Portal, Reveal, and REBL can call `datafy` on result sets to render them as "pure data" (which they already are, but this makes them also `Navigable`),
+* Tools like Portal, Reveal, and Morse can call `datafy` on result sets to render them as "pure data" (which they already are, but this makes them also `Navigable`),
-* Tools like Portal, Reveal, and REBL allow users to "drill down" into elements of rows in the "pure data" result set, using `nav`,
+* Tools like Portal, Reveal, and Morse allow users to "drill down" into elements of rows in the "pure data" result set, using `nav`,
 * If a column in a row represents a foreign key into another table, calling `nav` will fetch the related row(s),
 * Those can in turn be `datafy`'d and `nav`'d to continue drilling down through connected data in the database.
 
@@ -26,6 +26,18 @@ By default, `next.jdbc` assumes that a column named `<something>id` or `<somethi
 
 You can override this default behavior for any column in any table by providing a `:schema` option that is a hash map whose keys are column names (usually the table-qualified keywords that `next.jdbc` produces by default) and whose values are table-qualified keywords, optionally wrapped in vectors, that identity the name of the table to which that column is a foreign key and the name of the key column within that table.
 
+As of 1.3.next, you can also override this behavior via the `:schema-opts`
+option. This is a hash map whose keys can be:
+* `:fk-suffix` -- a string used instead of `"id"` to identify foreign keys,
+* `:pk` -- a string used instead of `"id"` for the primary key column in the target table,
+* `:pk-fn` -- a function that takes the table name and the value of `:pk` and returns the name of the primary key column in the target table, instead of just using the value of `:pk` (the default is effectively `(constantly <pk>)`).
+
+For `:fk-suffix`, the `_` is still permitted and optional in the column name,
+so if you specified `:schema-opts {:fk-suffix "fk"}` then `addressfk` and
+`address_fk` would both be treated as foreign keys into the `address` table.
+
+The `:pk-fn` can
+
 The default behavior in the example above is equivalent to this `:schema` value:
 
 ```clojure
@@ -35,6 +47,16 @@ The default behavior in the example above is equivalent to this `:schema` value:
                {:schema {:contact/addressid :address/id}})
 ```
 
+or these `:schema-opts` values:
+
+```clojure
+(jdbc/execute! ds
+               ["select * from contact where city = ?" "San Francisco"]
+               ;; a one-to-one or many-to-one relationship
+               {:schema-opts {:fk-suffix "id" :pk "id"
+                              :pk-fn (constantly "id")}})
+```
+
 If you had a table to track the valid/bouncing status of email addresses over time, `:deliverability`, where `email` is the non-unique key, you could provide automatic navigation into that using:
 
 ```clojure
@@ -45,6 +67,11 @@ If you had a table to track the valid/bouncing status of email addresses over ti
                          :address/email [:deliverability/email]}})
 ```
 
+Since this relies on a foreign key that does not follow a standard suffix
+pattern, there is no comparable `:schema-opts` version. In addition, the
+`:schema-opts` approach cannot designate a one-to-many or many-to-many
+relationship.
+
 When you indicate a `*-to-many` relationship, by wrapping the foreign table/key in a vector, `next.jdbc`'s implementation of `nav` will fetch a multi-row result set from the target table.
 
 If you use foreign key constraints in your database, you could probably generate this `:schema` data structure automatically from the metadata in your database. Similarly, if you use a library that depends on an entity relationship map (such as [seql](https://exoscale.github.io/seql/) or [walkable](https://walkable.gitlab.io/)), then you could probably generate this `:schema` data structure from that entity map.
@@ -55,7 +82,11 @@ Making rows datafiable is implemented by adding metadata to each row with a key
 
 When called (`datafy` on a row), it adds metadata to the row with a key of `clojure.core.protocols/nav` and another function as the value. That function also closes over the connectable and options passed in.
 
-When that is called (`nav` on a row, column name, and column value), if a `:schema` entry exists for that column or it matches the default convention described above, then it will fetch row(s) using `next.jdbc`'s `Executable` functions `-execute-one` or `-execute-all`, passing in the connectable and options closed over.
+When that is called (`nav` on a row, column name, and column value), if a
+`:schema` entry exists for that column or it matches the convention described
+above (either by default or via `:schema-opts`), then it will fetch row(s)
+using `next.jdbc`'s `Executable` functions `-execute-one` or `-execute-all`,
+passing in the connectable and options closed over.
 
 The protocol `next.jdbc.result-set/DatafiableRow` has a default implementation of `datafiable-row` for `clojure.lang.IObj` that just adds the metadata to support `datafy`. There is also an implementation baked into the result set handling behind `plan` so that you can call `datafiable-row` directly during reduction and get a fully-realized row that can be `datafy`'d (and then `nav`igated).
 

doc/tips-and-tricks.md

@@ -551,3 +551,6 @@ If you are using `plan`, you'll most likely be accessing columns by just the lab
         []
         (jdbc/plan ds ["select * from some_table"]))
 ```
+
+See also [`datafy`, `nav`, and `:schema` > **SQLite**](/doc/datafy-nav-and-schema.md#sqlite)
+for additional caveats on the `next.jdbc.datafy` namespace when using SQLite.

src/next/jdbc/result_set.clj

@@ -624,11 +624,9 @@
   ;; in reality, this is going to be over-optimistic and will like cause `nav`
   ;; to fail on attempts to navigate into result sets that are not hash maps
   (datafiable-row [this connectable opts]
-                  (vary-meta
+                  (vary-meta this assoc
-                   this
+                             `core-p/datafy (navize-row  connectable opts)
-                   assoc
+                             `core-p/nav    (navable-row connectable opts))))
-                   `core-p/datafy (navize-row  connectable opts)
-                   `core-p/nav    (navable-row connectable opts))))
 
 (defn datafiable-result-set
   "Given a ResultSet, a connectable, and an options hash map, return a fully
@@ -1025,12 +1023,52 @@
 (defn- default-schema
   "The default schema lookup rule for column names.
 
-  If a column name ends with `_id` or `id`, it is assumed to be a foreign key
+   We have a foreign key column suffix convention of `<table><fk>` or
-  into the table identified by the first part of the column name."
+   `<table>_<fk>`, which maps to a (primary) key in the `<table` called
-  [col]
+   `<pk>`.
-  (let [[_ table] (re-find #"(?i)^(.+?)_?id$" (name col))]
+
+   By default, both `<fk>` and `<pk>` are assumed to be `id`. That can be
+   overridden by the `:schema-opts` hash map in the options:
+   * `:fk-suffix` -- the suffix for foreign key columns, default `id`
+   * `:pk` -- the (primary) key column name, default `id`
+   * `:pk-fn` -- a function to apply to the table name and the value of `:pk`
+     to get the (primary) key column name, default `(constantly <pk>)`."
+  [opts col]
+  (let [fk-suffix (get-in opts [:schema-opts :fk-suffix] "id")
+        pk        (get-in opts [:schema-opts :pk]        "id")
+        pk-fn     (get-in opts [:schema-opts :pk-fn]     (constantly (name pk)))
+        [_ table] (re-find (re-pattern (str "(?i)^(.+?)_?"
+                                            (name fk-suffix)
+                                            "$"))
+                           (name col))]
     (when table
-      [(keyword table) :id])))
+      [(keyword table) (keyword (pk-fn table pk))])))
+
+(comment
+  (default-schema {} :userstatusid)
+  (default-schema {} :userstatus_id)
+  (default-schema {} :user_statusid)
+  (default-schema {:schema-opts {:fk-suffix "did"}} :user_id)
+  (default-schema {:schema-opts {:fk-suffix "did"}} :user_did)
+  (default-schema {:schema-opts {:fk-suffix "(did|id)"}} :user_id)
+  (default-schema {:schema-opts {:fk-suffix "(did|id)"}} :user_did)
+  (default-schema {:schema-opts {:fk-suffix "(did|id)"
+                                 :pk :did}} :user_did)
+  (default-schema {:schema-opts {:fk-suffix "(did|id)"
+                                 :pk :did
+                                 :pk-fn (fn [table pk]
+                                          (if (= "user" table)
+                                            "id"
+                                            pk))}}
+                  :user_did)
+  (default-schema {:schema-opts {:fk-suffix "(did|id)"
+                                 :pk :did
+                                 :pk-fn (fn [table pk]
+                                          (if (= "user" table)
+                                            "id"
+                                            pk))}}
+                  :book_did)
+  )
 
 (defn- expand-schema
   "Given a (possibly nil) schema entry, return it expanded to a triple of:
@@ -1072,50 +1110,12 @@
 
 (defn- navize-row
   "Given a connectable object, return a function that knows how to turn a row
-  into a `nav`igable object.
+   into a `nav`igable object.
 
-  A `:schema` option can provide a map from qualified column names
+   See navable-row below for more details."
-  (`:<table>/<column>`) to tuples that indicate for which table they are a
-  foreign key, the name of the key within that table, and (optionality) the
-  cardinality of that relationship (`:many`, `:one`).
-
-  If no `:schema` item is provided for a column, the convention of `<table>id` or
-  `<table>_id` is used, and the assumption is that such columns are foreign keys
-  in the `<table>` portion of their name, the key is called `id`, and the
-  cardinality is `:one`.
-
-  Rows are looked up using `-execute-all` or `-execute-one`, and the `:table-fn`
-  option, if provided, is applied to the assumed table name and `:column-fn` if
-  provided to the assumed foreign key column name."
   [connectable opts]
   (fn [row]
-    (vary-meta
+    (vary-meta row assoc `core-p/nav (navable-row connectable opts))))
-     row
-     assoc
-     `core-p/nav (fn [_ k v]
-                   (try
-                     (let [[table fk cardinality]
-                           (expand-schema k (or (get-in opts [:schema k])
-                                                (default-schema k)))]
-                       (if (and fk connectable)
-                         (let [table-fn  (:table-fn opts identity)
-                               column-fn (:column-fn opts identity)
-                               exec-fn!  (if (= :many cardinality)
-                                           p/-execute-all
-                                           p/-execute-one)]
-                           (exec-fn! connectable
-                                     [(str "SELECT * FROM "
-                                           (table-fn (name table))
-                                           " WHERE "
-                                           (column-fn (name fk))
-                                           " = ?")
-                                      v]
-                                     opts))
-                         v))
-                     (catch Exception _
-                       ;; assume an exception means we just cannot
-                       ;; navigate anywhere, so return just the value
-                       v))))))
 
 (defn- navable-row
   "Given a connectable object, return a function that knows how to `nav`
@@ -1131,6 +1131,8 @@
   in the `<table>` portion of their name, the key is called `id`, and the
   cardinality is `:one`.
 
+  That convention can in turn be modified via the `:schema-opts` option.
+
   Rows are looked up using `-execute-all` or `-execute-one`, and the `:table-fn`
   option, if provided, is applied to the assumed table name and `:column-fn` if
   provided to the assumed foreign key column name."
@@ -1139,7 +1141,7 @@
     (try
       (let [[table fk cardinality]
             (expand-schema k (or (get-in opts [:schema k])
-                                 (default-schema k)))]
+                                 (default-schema opts k)))]
         (if (and fk connectable)
           (let [table-fn  (:table-fn opts identity)
                 column-fn (:column-fn opts identity)
@@ -1156,6 +1158,6 @@
                       opts))
           v))
       (catch Exception _
-                       ;; assume an exception means we just cannot
+        ;; assume an exception means we just cannot
-                       ;; navigate anywhere, so return just the value
+        ;; navigate anywhere, so return just the value
         v))))