luciano/specter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

merge

Browse source
This commit is contained in:
Nathan Marz 2016-06-08 13:58:15 -04:00
commit 9045e13386
8 changed files with 347 additions and 41 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
CHANGES.md
View file

@ -4,9 +4,11 @@
* Added `select-any` operation which selects a single element navigated to by the path. Which element returned is undefined. If no elements are navigated to, returns `com.rpl.specter/NONE`. This is the fastest selection operation.
* Added `selected-any?` operation that returns true if any element is navigated to.
* Huge performance improvements to `select`, `select-one`, `select-first`, and `select-one!`
* Added META navigator (thanks @aengelberg)
## 0.11.1 (unreleased)
## 0.11.1
* More efficient inline caching for Clojure version, benchmarks show inline caching within 5% of manually precompiled code for all cases
* Added navigators for transients in com.rpl.specter.transient namespace (thanks @aengelberg)
* Huge performance improvement for ALL transform on maps and vectors
* Significant performance improvements for FIRST/LAST for vectors
* Huge performance improvements for `if-path`, `cond-path`, `selected?`, and `not-selected?`, especially for condition path containing only static functions
@ -17,7 +19,7 @@
* Added :notpath metadata to signify pathedfn arguments that should be treated as regular arguments during inline factoring. If one of these arguments is not a static var reference or non-collection value, the path will not factor.
* Bug fix: `transformed` transform-fn no longer factors into `pred` when an anonymous function during inline factoring
* Bug fix: Fixed nil->val to not replace the val on `false`
* Bug fix: Eliminate reflection when using primitive paramaters in an inline cached path
* Bug fix: Eliminate reflection when using primitive parameters in an inline cached path
## 0.11.0
* New `path` macro does intelligent inline caching of the provided path. The path is factored into a static portion and into params which may change on each usage of the path (e.g. local parameters). The static part is factored and compiled on the first run-through, and then re-used for all subsequent invocations. As an example, `[ALL (keypath k)]` is factored into `[ALL keypath]`, which is compiled and cached, and `[k]`, which is provided on each execution. If it is not possible to precompile the path (e.g. [ALL some-local-variable]), nothing is cached and the path will be compiled on each run-through.

136
README.md
View file

@ -1,10 +1,103 @@
# Specter [![Build Status](https://travis-ci.org/nathanmarz/specter.svg?branch=master)](http://travis-ci.org/nathanmarz/specter)
Specter is library for Clojure and ClojureScript for querying and manipulating arbitrarily complicated data structures very concisely. Its use cases range from transforming the values of a map to manipulating deeply nested data structures to performing sophisticated recursive tree transformations. Without Specter, writing these manipulations in Clojure manually is cumbersome and prone to error.
Specter is a Clojure and ClojureScript library that, because of its far-ranging applicability, is hard to describe in just a few sentences. At its core, Specter is a library for "composable navigation". Most commonly it is used for querying and transforming nested data structures, but the concept generalizes far beyond that. Its effect is to enable you to write programs much more rapidly in a much more maintainable way.
Specter is fully extensible. At its core, it's just a protocol for how to navigate within a data structure. By extending this protocol, you can use Specter to navigate any data structure or object you have.
Here are three areas where Specter greatly improves Clojure programming:
**Specter makes common tasks concise instead of cumbersome and simple instead of complex**
Example 1: Append a sequence of elements to a nested vector
```clojure
(def data {:a [1 2 3]})
;; Manual Clojure
(update data :a (fn [v] (reduce conj v [4 5])))
;; Specter
(setval [:a END] [4 5] data)
```
Example 2: Increment every even number nested within map of vector of maps
```clojure
(def data {:a [{:aa 1 :bb 2}
{:cc 3}]
:b [{:dd 4}]})
;; Manual Clojure
(defn map-vals [m afn]
(->> m (map (fn [[k v]] [k (afn v)])) (into {})))
(map-vals data
(fn [v]
(mapv
(fn [m]
(map-vals
m
(fn [v] (if (even? v) (inc v) v))))
v)))
;; Specter
(transform [MAP-VALS ALL MAP-VALS even?] inc data)
```
**Specter is much faster than Clojure's limited built-in alternatives**
Example 1: Specter's `select` is 27% faster than `get-in`:
```clojure
(time
(dotimes [_ 10000000]
(get-in {:a {:b {:c 1}}} [:a :b :c])))
"Elapsed time: 640.666 msecs"
(time
(dotimes [_ 10000000]
(select [:a :b :c] {:a {:b {:c 1}}})))
"Elapsed time: 470.167 msecs"
```
Example 2: Specter's `transform` is 6x faster than `update-in`:
```clojure
(time
(dotimes [_ 10000000]
(update-in {:a {:b {:c 1}}} [:a :b :c] inc)))
"Elapsed time: 10662.014 msecs"
(time
(dotimes [_ 10000000]
(transform [:a :b :c] inc {:a {:b {:c 1}}})))
"Elapsed time: 1699.016 msecs"
```
**Specter makes sophisticated tasks – that are difficult to program manually – easy**
Example 1: Reverse the order of even numbers in a tree (with order based on depth first search):
```clojure
(transform (subselect (walker number?) even?)
reverse
[1 [[[2]] 3] 5 [6 [7 8]] 10])
;; => [1 [[[10]] 3] 5 [8 [7 6]] 2]
```
Example 2: Replace every continuous sequence of odd numbers with its sum:
```clojure
(transform (continuous-subseqs odd?)
(fn [aseq] [(reduce + aseq)])
[1 3 6 8 9 11 15 16]
)
;; => [4 6 8 35 16]
```
This is just the tip of the iceberg. Because Specter is completely extensible, it can be used to navigate any data structure or object you have. All the navigators that come with Specter are built upon [very simple abstractions](https://github.com/nathanmarz/specter/blob/0.11.1/src/clj/com/rpl/specter/protocols.cljx).
Even though Specter is so generic and flexible, its performance rivals hand-optimized code. Under the hood, Specter uses [advanced dynamic techniques](https://github.com/nathanmarz/specter/wiki/Specter-0.11.0:-Performance-without-the-tradeoffs) to strip away the overhead of composition. Additionally, the built-in navigators use the most efficient means possible of accessing data structures. For example, `ALL` uses `mapv` on vectors, `reduce-kv` on small maps, and `reduce-kv` in conjunction with transients on larger maps. You get the best of both worlds of elegance and performance.
Even though Specter is so generic and flexible, [its performance](https://github.com/nathanmarz/specter/wiki/Specter-0.11.0:-Performance-without-the-tradeoffs) rivals hand-optimized code. The only comparable functions in Clojure's core library are `get-in` and `update-in`. The equivalent Specter code is effectively identical (just different order of arguments), but Specter runs 30% faster than `get-in` and 5x faster than `update-in`.
# Latest Version
@ -23,7 +116,8 @@ Specter's API is contained in three files:
- [macros.clj](https://github.com/nathanmarz/specter/blob/master/src/clj/com/rpl/specter/macros.clj): This contains the core `select/transform/etc.` operations as well as macros for defining new navigators.
- [specter.cljx](https://github.com/nathanmarz/specter/blob/master/src/clj/com/rpl/specter.cljx): This contains the build-in navigators and functional versions of `select/transform/etc.`
- [zippers.cljx](https://github.com/nathanmarz/specter/blob/master/src/clj/com/rpl/specter/zipper.cljx): This integrates zipper-based navigation into Specter.
- [transient.cljx](https://github.com/nathanmarz/specter/blob/master/src/clj/com/rpl/specter/transient.cljx): This contains navigators for transient collections.
- [zipper.cljx](https://github.com/nathanmarz/specter/blob/master/src/clj/com/rpl/specter/zipper.cljx): This integrates zipper-based navigation into Specter.
# Questions?
@ -33,27 +127,10 @@ You can also find help in the #specter channel on [Clojurians](http://clojurians
# Examples
Increment all the values in a map:
Increment all the values in maps of maps:
```clojure
user> (use 'com.rpl.specter)
user> (use 'com.rpl.specter.macros)
user> (transform [ALL LAST]
inc
{:a 1 :b 2 :c 3})
{:a 2, :b 3, :c 4}
```
Increment all the values in maps of maps:
```clojure
user> (transform [ALL LAST ALL LAST]
inc
{:a {:aa 1} :b {:ba -1 :bb 2}})
{:a {:aa 2}, :b {:ba 0, :bb 3}}
```
Do the previous example more concisely:
```clojure
user> (def MAP-VALS (comp-paths ALL LAST))
user> (transform [MAP-VALS MAP-VALS]
inc
{:a {:aa 1} :b {:ba -1 :bb 2}})
@ -92,7 +169,7 @@ user> (transform [(srange 1 4) ALL odd?] inc [0 1 2 3 4 5 6 7])
[0 2 2 4 4 5 6 7]
```
Replace the subsequence from index 2 to 4 with [:a :b :c :d :e]:
Replace the subsequence from indices 2 to 4 with [:a :b :c :d :e]:
```clojure
user> (setval (srange 2 4) [:a :b :c :d :e] [0 1 2 3 4 5 6 7 8 9])
@ -106,7 +183,7 @@ user> (setval [ALL END] [:a :b] [[1] '(1 2) [:c]])
[[1 :a :b] (1 2 :a :b) [:c :a :b]]
```
Get all the numbers out of a map, no matter how they're nested:
Get all the numbers out of a data structure, no matter how they're nested:
```clojure
user> (select (walker number?)
@ -131,15 +208,6 @@ user> (transform [(srange 4 11) (filterer even?)]
[0 1 2 3 10 5 8 7 6 9 4 11 12 13 14 15]
```
Decrement every value in a map:
```clojure
user> (transform [ALL LAST]
dec
{:a 1 :b 3})
{:b 2 :a 0}
```
Append [:c :d] to every subsequence that has at least two even numbers:
```clojure
user> (setval [ALL
@ -226,7 +294,7 @@ The next examples demonstrate recursive navigation. Here's how to double all the
;; => [:a 1 [4 [[[3]]] :e] [8 5 [12 7]]]
```
Here's how to reverse the positions of all even numbers in a tree (with order based on a depth first search). This example uses conditional navigation instead of protocol paths to do the walk:
Here's how to reverse the positions of all even numbers in a tree (with order based on a depth first search). This example uses conditional navigation instead of protocol paths to do the walk and is much more efficient than using `walker`:
```clojure
(declarepath TreeValues)

3
project.clj
View file

@ -15,7 +15,8 @@
:namespaces [com.rpl.specter
com.rpl.specter.macros
com.rpl.specter.zipper
com.rpl.specter.protocols]
com.rpl.specter.protocols
com.rpl.specter.transient]
:source-uri
{#"target/classes" "https://github.com/nathanmarz/specter/tree/{version}/src/clj/{classpath}x#L{line}"
#".*" "https://github.com/nathanmarz/specter/tree/{version}/src/clj/{classpath}#L{line}"

80
scripts/benchmarks.clj
View file

@ -1,6 +1,7 @@
(ns com.rpl.specter.benchmarks
(:use [com.rpl.specter]
[com.rpl.specter macros]
[com.rpl.specter.transient]
[com.rpl.specter.impl :only [benchmark]])
(:require [clojure.walk :as walk]))
@ -32,7 +33,7 @@
(time-ms amt-per-iter afn)))))
(defn compare-benchmark [amt-per-iter afn-map]
(let [results (transform [ALL LAST]
(let [results (transform MAP-VALS
(fn [afn]
(average-time-ms 8 amt-per-iter afn))
afn-map)
@ -53,7 +54,7 @@
(let [data {:a {:b {:c 1}}}
p (comp-paths :a :b :c)]
(run-benchmark "get value in nested map" 10000000
(run-benchmark "get value in nested map" 5000000
(select-any [:a :b :c] data)
(select-one [:a :b :c] data)
(select-first [:a :b :c] data)
@ -61,6 +62,7 @@
(compiled-select-any p data)
(get-in data [:a :b :c])
(-> data :a :b :c)
(select-any [(keypath :a) (keypath :b) (keypath :c)] data)
))
@ -76,7 +78,7 @@
(my-update m3 :c afn))))))
(let [data {:a {:b {:c 1}}}]
(run-benchmark "update value in nested map" 1000000
(run-benchmark "update value in nested map" 500000
(update-in data [:a :b :c] inc)
(transform [:a :b :c] inc data)
(manual-transform data inc)
@ -163,7 +165,7 @@
(let [data [1 2 [[3]] [4 6 [7 [8]] 10]]]
(run-benchmark "update every value in a tree (represented with vectors)"
100000
50000
(walk/postwalk (fn [e] (if (and (number? e) (even? e)) (inc e) e)) data)
(transform [(walker number?) even?] inc data)
(transform [TreeValues even?] inc data)
@ -171,3 +173,73 @@
(tree-value-transform (fn [e] (if (even? e) (inc e) e)) data)
))
(let [toappend (range 1000)]
(run-benchmark "transient comparison: building up vectors"
10000
(reduce (fn [v i] (conj v i)) [] toappend)
(reduce (fn [v i] (conj! v i)) (transient []) toappend)
(setval END toappend [])
(setval END! toappend (transient []))))
(let [toappend (range 1000)]
(run-benchmark "transient comparison: building up vectors one at a time"
10000
(reduce (fn [v i] (conj v i)) [] toappend)
(reduce (fn [v i] (conj! v i)) (transient []) toappend)
(reduce (fn [v i] (setval END [i] v)) [] toappend)
(reduce (fn [v i] (setval END! [i] v)) (transient []) toappend)
))
(let [data (vec (range 1000))
tdata (transient data)
tdata2 (transient data)
idx 600]
(run-benchmark "transient comparison: assoc'ing in vectors"
2500000
(assoc data idx 0)
(assoc! tdata idx 0)
(setval (keypath idx) 0 data)
(setval (keypath! idx) 0 tdata2)))
(let [data (into {} (for [k (range 1000)]
[k (rand)]))
tdata (transient data)
tdata2 (transient data)
idx 600]
(run-benchmark "transient comparison: assoc'ing in maps"
2500000
(assoc data idx 0)
(assoc! tdata idx 0)
(setval (keypath idx) 0 data)
(setval (keypath! idx) 0 tdata2)))
(defn modify-submap
[m]
(assoc m 0 1 458 89))
(let [data (into {} (for [k (range 1000)]
[k (rand)]))
tdata (transient data)]
(run-benchmark "transient comparison: submap"
300000
(transform (submap [600 700]) modify-submap data)
(transform (submap! [600 700]) modify-submap tdata)))
(let [data {:x 1}
meta-map {:my :metadata}]
(run-benchmark "set metadata"
2000000
(with-meta data meta-map)
(setval META meta-map data)))
(let [data (with-meta {:x 1} {:my :metadata})]
(run-benchmark "get metadata"
20000000
(meta data)
(select-any META data)))
(let [data (with-meta {:x 1} {:my :metadata})]
(run-benchmark "vary metadata"
2000000
(vary-meta data assoc :y 2)
(setval [META :y] 2 data)))

9
src/clj/com/rpl/specter.cljx
View file

@ -522,6 +522,15 @@
NIL->VECTOR
(nil->val []))
(defnav ^{:doc "Navigates to the metadata of the structure, or nil if
the structure has no metadata or may not contain metadata."}
META
[]
(select* [this structure next-fn]
(next-fn (meta structure)))
(transform* [this structure next-fn]
(with-meta structure (next-fn (meta structure)))))
(defpathedfn
^{:doc "Adds the result of running select with the given path on the
current value to the collected vals."}

21
src/clj/com/rpl/specter/impl.cljx
View file

@ -523,6 +523,14 @@
(defn vec-count [v]
(count v))
#+clj
(defn transient-vec-count [^clojure.lang.ITransientVector v]
(.count v))
#+cljs
(defn transient-vec-count [v]
(count v))
(extend-protocol UpdateExtremes
#+clj clojure.lang.PersistentVector #+cljs cljs.core/PersistentVector
(update-first [v afn]
@ -562,6 +570,9 @@
#+clj clojure.lang.IPersistentVector #+cljs cljs.core/PersistentVector
(fast-empty? [v]
(= 0 (vec-count v)))
#+clj clojure.lang.ITransientVector #+cljs cljs.core/TransientVector
(fast-empty? [v]
(= 0 (transient-vec-count v)))
#+clj Object #+cljs default
(fast-empty? [s]
(empty? s))
@ -951,6 +962,16 @@
(next-fn structure)
))
(deftype TransientEndNavigator [])
(extend-protocol p/Navigator
TransientEndNavigator
(select* [this structure next-fn]
(next-fn []))
(transform* [this structure next-fn]
(let [res (next-fn [])]
(reduce conj! structure res))))
(defn extract-basic-filter-fn [path]
(cond (fn? path)
path

91
src/clj/com/rpl/specter/transient.cljx Normal file
View file

@ -0,0 +1,91 @@
(ns com.rpl.specter.transient
#+cljs
(:require-macros [com.rpl.specter.macros
:refer
[defnav
defpathedfn]])
(:use #+clj
[com.rpl.specter.macros :only
[defnav
defpathedfn]])
(:require [com.rpl.specter.impl :as i]
[com.rpl.specter :refer [subselect selected?]]))
(defnav
^{:doc "Navigates to the specified key of a transient collection,
navigating to nil if it doesn't exist."}
keypath!
[key]
(select* [this structure next-fn]
(next-fn (get structure key)))
(transform* [this structure next-fn]
(assoc! structure key (next-fn (get structure key)))))
(def END!
"Navigates to an empty (persistent) vector at the end of a transient vector."
(i/comp-paths* (i/->TransientEndNavigator)))
(defn- t-get-first
[tv]
(nth tv 0))
(defn- t-get-last
[tv]
(nth tv (dec (i/transient-vec-count tv))))
(defn- t-update-first
[tv next-fn]
(assoc! tv 0 (next-fn (nth tv 0))))
(defn- t-update-last
[tv next-fn]
(let [i (dec (i/transient-vec-count tv))]
(assoc! tv i (next-fn (nth tv i)))))
(def FIRST!
"Navigates to the first element of a transient vector."
(i/->PosNavigator t-get-first t-update-first))
(def LAST!
"Navigates to the last element of a transient vector."
(i/->PosNavigator t-get-last t-update-last))
#+clj
(defn- select-keys-from-transient-map
"Selects keys from transient map, because built-in select-keys uses
`find` which is unsupported."
[m m-keys]
(loop [result {}
m-keys m-keys]
(if-not (seq m-keys)
result
(let [k (first m-keys)
;; support Clojure 1.6 where contains? is broken on transients
item (get m k ::not-found)]
(recur (if-not (identical? item ::not-found)
(assoc result k item)
result)
(rest m-keys))))))
#+cljs
(defn- select-keys-from-transient-map
"Uses select-keys on a transient map."
[m m-keys]
(select-keys m m-keys))
(defnav
^{:doc "Navigates to the specified persistent submap of a transient map."}
submap!
[m-keys]
(select* [this structure next-fn]
(next-fn (select-keys-from-transient-map structure m-keys)))
(transform* [this structure next-fn]
(let [selected (select-keys-from-transient-map structure m-keys)
res (next-fn selected)]
(as-> structure %
(reduce (fn [m k]
(dissoc! m k))
% m-keys)
(reduce-kv (fn [m k v]
(assoc! m k v))
% res)))))

42
test/com/rpl/specter/core_test.cljx
View file

@ -27,6 +27,7 @@
#+cljs [cljs.test.check.generators :as gen]
#+cljs [cljs.test.check.properties :as prop :include-macros true]
[com.rpl.specter :as s]
[com.rpl.specter.transient :as t]
[clojure.set :as set]))
;;TODO:
@ -1149,3 +1150,44 @@
(is (= 2 (select-any (s/if-path odd? s/STOP s/STAY) 2)))
(is (= s/NONE (select-any [(s/if-path odd? s/STOP s/STAY) odd?] 2)))
)
(defspec transient-vector-test
(for-all+
[v (gen/vector (limit-size 5 gen/int))]
(every? identity
(for [[path transient-path f]
[[s/FIRST t/FIRST! (fnil inc 0)] ;; fnil in case vector is empty
[s/LAST t/LAST! (fnil inc 0)]
[(s/keypath 0) (t/keypath! 0) (fnil inc 0)]
[s/END t/END! #(conj % 1 2 3)]]]
(and (= (s/transform* path f v)
(persistent! (s/transform* transient-path f (transient v))))
(= (s/select* path v)
(s/select* transient-path (transient v))))))))
(defspec transient-map-test
(for-all+
[m (limit-size 5 (gen/not-empty (gen/map gen/keyword gen/int)))
new-key gen/keyword]
(let [existing-key (first (keys m))]
(every? identity
(for [[path transient-path f]
[[(s/keypath existing-key) (t/keypath! existing-key) inc]
[(s/keypath new-key) (t/keypath! new-key) (constantly 3)]
[(s/submap [existing-key new-key])
(t/submap! [existing-key new-key])
(constantly {new-key 1234})]]]
(and (= (s/transform* path f m)
(persistent! (s/transform* transient-path f (transient m))))
(= (s/select* path m)
(s/select* transient-path (transient m)))))))))
(defspec meta-test
(for-all+
[v (gen/vector gen/int)
meta-map (limit-size 5 (gen/map gen/keyword gen/int))]
(= meta-map
(meta (setval s/META meta-map v))
(first (select s/META (with-meta v meta-map)))
(first (select s/META (setval s/META meta-map v))))))