luciano/nippy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'v2.10'

Browse source
This commit is contained in:
Peter Taoussanis 2015-09-30 12:18:32 +07:00
commit df9eef0f2b
8 changed files with 327 additions and 277 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

17
CHANGELOG.md
View file

@ -1,5 +1,20 @@
> This project uses [Break Versioning](https://github.com/ptaoussanis/encore/blob/master/BREAK-VERSIONING.md) as of **Aug 16, 2014**. > This project uses [Break Versioning](https://github.com/ptaoussanis/encore/blob/master/BREAK-VERSIONING.md) as of **Aug 16, 2014**.
## v2.10.0 / 2015 Sep 30
> This is a major feature/performance release that **drops support for Clojure 1.4** but is otherwise non-breaking
* **BREAKING**: drop support for Clojure 1.4 (**now requires Clojure 1.5+**)
* **Performance**: various small performance improvements
* **New**: dynamic `*default-freeze-compressor-selector*`, `set-default-freeze-compressor-selector!` util
* **New**: dynamic `*custom-readers*`, `swap-custom-readers!` util
* **New**: edn writes now override dynamic `*print-level*`, `*print-length*` for safety
```clojure
[com.taoensso/nippy "2.10.0"]
```
## v2.9.1 / 2015 Sep 14 ## v2.9.1 / 2015 Sep 14
> This is a hotfix release with an **important fix** for Nippy encryption users > This is a hotfix release with an **important fix** for Nippy encryption users
@ -13,7 +28,7 @@
## v2.9.0 / 2015 Jun 1 ## v2.9.0 / 2015 Jun 1
> This is a major, **non-breaking** release that improves performance and makes thawing more resilient to certain failures. Identical to **v2.9.0-RC3**. > This is a major **non-breaking** release that improves performance and makes thawing more resilient to certain failures. Identical to **v2.9.0-RC3**.
* **Robustness**: improve error handling for unthawable records * **Robustness**: improve error handling for unthawable records
* **Performance**: switch `doseq` -> (faster) `run!` calls * **Performance**: switch `doseq` -> (faster) `run!` calls

6
README.md
View file

@ -1,7 +1,7 @@
**[API docs][]** | **[CHANGELOG][]** | [other Clojure libs][] | [Twitter][] | [contact/contrib](#contact--contributing) | current [Break Version][]: **[API docs][]** | **[CHANGELOG][]** | [other Clojure libs][] | [Twitter][] | [contact/contrib](#contact--contributing) | current [Break Version][]:
```clojure ```clojure
[com.taoensso/nippy "2.9.1"] ; Stable, see CHANGELOG for details [com.taoensso/nippy "2.10.0"] ; Stable, see CHANGELOG for details
``` ```
# Nippy, a Clojure serialization library # Nippy, a Clojure serialization library
@ -29,7 +29,7 @@ Nippy is an attempt to provide a reliable, high-performance **drop-in alternativ
Add the necessary dependency to your [Leiningen][] `project.clj` and `require` the library in your ns: Add the necessary dependency to your [Leiningen][] `project.clj` and `require` the library in your ns:
```clojure ```clojure
[com.taoensso/nippy "2.9.1"] ; project.clj [com.taoensso/nippy "2.10.0"] ; project.clj
(ns my-app (:require [taoensso.nippy :as nippy])) ; ns (ns my-app (:require [taoensso.nippy :as nippy])) ; ns
``` ```
@ -153,7 +153,7 @@ Otherwise reach me (Peter Taoussanis) at [taoensso.com][] or on [Twitter][]. Che
## License ## License
Copyright © 2012-2014 Peter Taoussanis. Distributed under the [Eclipse Public License][], the same as Clojure. Copyright © 2012-2015 Peter Taoussanis. Distributed under the [Eclipse Public License][], the same as Clojure.
[API docs]: http://ptaoussanis.github.io/nippy/ [API docs]: http://ptaoussanis.github.io/nippy/

BIN
benchmarks.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 17 KiB

21
project.clj
View file

@ -1,4 +1,4 @@
(defproject com.taoensso/nippy "2.9.1" (defproject com.taoensso/nippy "2.10.0"
:author "Peter Taoussanis <https://www.taoensso.com>" :author "Peter Taoussanis <https://www.taoensso.com>"
:description "Clojure serialization library" :description "Clojure serialization library"
:url "https://github.com/ptaoussanis/nippy" :url "https://github.com/ptaoussanis/nippy"
@ -12,10 +12,10 @@
*unchecked-math* :warn-on-boxed} *unchecked-math* :warn-on-boxed}
:dependencies :dependencies
[[org.clojure/clojure "1.4.0"] [[org.clojure/clojure "1.5.1"]
[org.clojure/tools.reader "0.9.2"] [org.clojure/tools.reader "0.9.2"]
[com.taoensso/encore "1.32.0"] [com.taoensso/encore "2.18.0"]
[org.iq80.snappy/snappy "0.3"] [org.iq80.snappy/snappy "0.4"]
[org.tukaani/xz "1.5"] [org.tukaani/xz "1.5"]
[net.jpountz.lz4/lz4 "1.3"]] [net.jpountz.lz4/lz4 "1.3"]]
@ -24,13 +24,13 @@
:server-jvm {:jvm-opts ^:replace ["-server" "-Xms1024m" "-Xmx2048m"]} :server-jvm {:jvm-opts ^:replace ["-server" "-Xms1024m" "-Xmx2048m"]}
:1.5 {:dependencies [[org.clojure/clojure "1.5.1"]]} :1.5 {:dependencies [[org.clojure/clojure "1.5.1"]]}
:1.6 {:dependencies [[org.clojure/clojure "1.6.0"]]} :1.6 {:dependencies [[org.clojure/clojure "1.6.0"]]}
:1.7 {:dependencies [[org.clojure/clojure "1.7.0-beta1"]]} :1.7 {:dependencies [[org.clojure/clojure "1.7.0"]]}
:1.8 {:dependencies [[org.clojure/clojure "1.8.0-alpha5"]]}
:test {:jvm-opts ["-Xms1024m" "-Xmx2048m"] :test {:jvm-opts ["-Xms1024m" "-Xmx2048m"]
:dependencies [[expectations "2.1.1"] :dependencies [[expectations "2.1.1"]
[org.clojure/test.check "0.7.0"] [org.clojure/test.check "0.8.2"]
;; [com.cemerick/double-check "0.6.1"] [org.clojure/data.fressian "0.2.1"]
[org.clojure/data.fressian "0.2.0"] [org.xerial.snappy/snappy-java "1.1.2"]]}
[org.xerial.snappy/snappy-java "1.1.1.7"]]}
:dev [:1.7 :test :dev [:1.7 :test
{:plugins {:plugins
[[lein-pprint "1.1.1"] [[lein-pprint "1.1.1"]
@ -42,8 +42,7 @@
:test-paths ["test" "src"] :test-paths ["test" "src"]
:aliases :aliases
{"test-all" ["with-profile" "default:+1.5:+1.6:+1.7" "expectations"] {"test-all" ["with-profile" "+1.5:+1.6:+1.7:+1.8" "expectations"]
;; "test-all" ["with-profile" "default:+1.6" "expectations"]
"test-auto" ["with-profile" "+test" "autoexpect"] "test-auto" ["with-profile" "+test" "autoexpect"]
"deploy-lib" ["do" "deploy" "clojars," "install"] "deploy-lib" ["do" "deploy" "clojars," "install"]
"start-dev" ["with-profile" "+server-jvm" "repl" ":headless"]} "start-dev" ["with-profile" "+server-jvm" "repl" ":headless"]}

253
src/taoensso/nippy.clj
View file

@ -1,9 +1,8 @@
(ns taoensso.nippy (ns taoensso.nippy
"High-performance JVM Clojure serialization library. Originally adapted from "High-performance JVM Clojure serialization library. Originally adapted from
Deep-Freeze." Deep-Freeze (https://goo.gl/OePPGr)."
{:author "Peter Taoussanis"} {:author "Peter Taoussanis (@ptaoussanis)"}
(:require [clojure.tools.reader.edn :as edn] (:require [taoensso.encore :as encore]
[taoensso.encore :as encore]
[taoensso.nippy [taoensso.nippy
(utils :as utils) (utils :as utils)
(compression :as compression) (compression :as compression)
@ -19,17 +18,9 @@
PersistentQueue PersistentTreeMap PersistentTreeSet PersistentList ; LazySeq PersistentQueue PersistentTreeMap PersistentTreeSet PersistentList ; LazySeq
IRecord ISeq])) IRecord ISeq]))
;;;; Encore version check (if (vector? taoensso.encore/encore-version)
(encore/assert-min-encore-version [2 16 0])
(let [min-encore-version 1.28] ; For `backport-run!` support (encore/assert-min-encore-version 2.16))
(if-let [assert! (ns-resolve 'taoensso.encore 'assert-min-encore-version)]
(assert! min-encore-version)
(throw
(ex-info
(format
"Insufficient com.taoensso/encore version (< %s). You may have a Leiningen dependency conflict (see http://goo.gl/qBbLvC for solution)."
min-encore-version)
{:min-version min-encore-version}))))
;;;; Nippy data format ;;;; Nippy data format
;; * 4-byte header (Nippy v2.x+) (may be disabled but incl. by default) [1]. ;; * 4-byte header (Nippy v2.x+) (may be disabled but incl. by default) [1].
@ -75,22 +66,22 @@
;; ** Negative ids reserved for user-defined types ** ;; ** Negative ids reserved for user-defined types **
;; ;;
(def ^:const id-reserved (int 0)) (def ^:const id-reserved (int 0))
;; 1 ;; 1 ; Deprecated
(def ^:const id-bytes (int 2)) (def ^:const id-bytes (int 2))
(def ^:const id-nil (int 3)) (def ^:const id-nil (int 3))
(def ^:const id-boolean (int 4)) (def ^:const id-boolean (int 4))
(def ^:const id-reader (int 5)) ; Fallback #2: pr-str output (def ^:const id-reader (int 5)) ; Fallback #2
(def ^:const id-serializable (int 6)) ; Fallback #1 (def ^:const id-serializable (int 6)) ; Fallback #1
(def ^:const id-char (int 10)) (def ^:const id-char (int 10))
;; 11 ;; 11 ; Deprecated
;; 12 ;; 12 ; Deprecated
(def ^:const id-string (int 13)) (def ^:const id-string (int 13))
(def ^:const id-keyword (int 14)) (def ^:const id-keyword (int 14))
(def ^:const id-list (int 20)) (def ^:const id-list (int 20))
(def ^:const id-vector (int 21)) (def ^:const id-vector (int 21))
;; 22 ;; 22 ; Deprecated
(def ^:const id-set (int 23)) (def ^:const id-set (int 23))
(def ^:const id-seq (int 24)) (def ^:const id-seq (int 24))
(def ^:const id-meta (int 25)) (def ^:const id-meta (int 25))
@ -133,10 +124,10 @@
;; (def ^:const id-map-small (int 112)) ; '' ;; (def ^:const id-map-small (int 112)) ; ''
;;; DEPRECATED (old types will be supported only for thawing) ;;; DEPRECATED (old types will be supported only for thawing)
(def ^:const id-old-reader (int 1)) ; as of 0.9.2, for +64k support (def ^:const id-reader-depr1 (int 1)) ; v0.9.2+ for +64k support
(def ^:const id-old-string (int 11)) ; as of 0.9.2, for +64k support (def ^:const id-string-depr1 (int 11)) ; v0.9.2+ for +64k support
(def ^:const id-old-map (int 22)) ; as of 0.9.0, for more efficient thaw (def ^:const id-map-depr1 (int 22)) ; v0.9.0+ for more efficient thaw
(def ^:const id-old-keyword (int 12)) ; as of 2.0.0-alpha5, for str consistecy (def ^:const id-keyword-depr1 (int 12)) ; v2.0.0-alpha5+ for str consistecy
) )
;;;; Ns imports (mostly for convenience of lib consumers) ;;;; Ns imports (mostly for convenience of lib consumers)
@ -164,13 +155,15 @@
(defmacro write-bytes [out ba & [small?]] (defmacro write-bytes [out ba & [small?]]
(let [out (with-meta out {:tag 'java.io.DataOutput}) (let [out (with-meta out {:tag 'java.io.DataOutput})
ba (with-meta ba {:tag 'bytes})] ba (with-meta ba {:tag 'bytes})]
(if small? ; Optimization, must be known before id's written
`(let [out# ~out, ba# ~ba `(let [out# ~out, ba# ~ba
size# (alength ba#)] size# (alength ba#)]
(if ~small? ; Optimization, must be known before id's written (.writeByte out# (byte size#))
(.writeByte out# (byte size#)) ; `byte` to throw on range error (.write out# ba# 0 size#))
(.writeInt out# (int size#)) ; `int` '' `(let [out# ~out, ba# ~ba
) size# (alength ba#)]
(.write out# ba# 0 size#)))) (.writeInt out# (int size#))
(.write out# ba# 0 size#)))))
(defmacro write-biginteger [out x] (defmacro write-biginteger [out x]
(let [x (with-meta x {:tag 'java.math.BigInteger})] (let [x (with-meta x {:tag 'java.math.BigInteger})]
@ -209,9 +202,8 @@
(println (format "DEBUG - freezer-coll: %s for %s" ~type (type ~'x))))) (println (format "DEBUG - freezer-coll: %s for %s" ~type (type ~'x)))))
(if (counted? ~'x) (if (counted? ~'x)
(do (.writeInt ~'out (count ~'x)) (do (.writeInt ~'out (count ~'x))
;; (doseq [i# ~'x] (freeze-to-out ~'out i#)) (encore/run!* (fn [i#] (freeze-to-out ~'out i#)) ~'x))
(encore/backport-run! (fn [i#] (freeze-to-out ~'out i#)) ~'x)) (let [bas# (ByteArrayOutputStream. 64)
(let [bas# (ByteArrayOutputStream.)
sout# (DataOutputStream. bas#) sout# (DataOutputStream. bas#)
cnt# (reduce (fn [^long cnt# i#] cnt# (reduce (fn [^long cnt# i#]
(freeze-to-out sout# i#) (freeze-to-out sout# i#)
@ -223,14 +215,11 @@
(defmacro ^:private freezer-kvs [type id & body] (defmacro ^:private freezer-kvs [type id & body]
`(freezer ~type ~id `(freezer ~type ~id
(.writeInt ~'out (* 2 (count ~'x))) (.writeInt ~'out (* 2 (count ~'x))) ; *2 here is vestigial
;; (doseq [kv# ~'x] (encore/run-kv!
;; (freeze-to-out ~'out (key kv#)) (fn [k# v#]
;; (freeze-to-out ~'out (val kv#))) (freeze-to-out ~'out k#)
(encore/backport-run! (freeze-to-out ~'out v#))
(fn [kv#]
(freeze-to-out ~'out (key kv#))
(freeze-to-out ~'out (val kv#)))
~'x))) ~'x)))
(freezer (Class/forName "[B") id-bytes (write-bytes out ^bytes x)) (freezer (Class/forName "[B") id-bytes (write-bytes out ^bytes x))
@ -291,17 +280,25 @@
(extend-type Long ; Optimized common-case type (extend-type Long ; Optimized common-case type
Freezable Freezable
(freeze-to-out* [x ^DataOutput out] (freeze-to-out* [x ^DataOutput out]
(let [^long x x]
(cond (cond
(<= Byte/MIN_VALUE x Byte/MAX_VALUE) (and (<= x #_Byte/MAX_VALUE 127)
(do (write-id out id-byte-as-long) (.writeByte out x)) (<= #_Byte/MIN_VALUE -128 x))
(do (write-id out id-byte-as-long)
(.writeByte out x))
(<= Short/MIN_VALUE x Short/MAX_VALUE) (and (<= x #_Short/MAX_VALUE 32767)
(do (write-id out id-short-as-long) (.writeShort out x)) (<= #_Short/MIN_VALUE -32768 x))
(do (write-id out id-short-as-long)
(.writeShort out x))
(<= Integer/MIN_VALUE x Integer/MAX_VALUE) (and (<= x #_Integer/MAX_VALUE 2147483647)
(do (write-id out id-int-as-long) (.writeInt out x)) (<= #_Integer/MIN_VALUE -2147483648 x))
(do (write-id out id-int-as-long)
(.writeInt out x))
:else (do (write-id out id-long) (.writeLong out x))))) :else (do (write-id out id-long)
(.writeLong out x))))))
;; ;;
@ -325,7 +322,7 @@
(def ^:dynamic *final-freeze-fallback* "Alpha - subject to change." nil) (def ^:dynamic *final-freeze-fallback* "Alpha - subject to change." nil)
(defn freeze-fallback-as-str "Alpha-subject to change." [x out] (defn freeze-fallback-as-str "Alpha-subject to change." [x out]
(freeze-to-out* {:nippy/unfreezable (pr-str x) :type (type x)} out)) (freeze-to-out* {:nippy/unfreezable (encore/pr-edn x) :type (type x)} out))
(comment (comment
(require '[clojure.core.async :as async]) (require '[clojure.core.async :as async])
@ -349,13 +346,13 @@
(do (when-debug-mode (do (when-debug-mode
(println (format "DEBUG - Reader fallback: %s" (type x)))) (println (format "DEBUG - Reader fallback: %s" (type x))))
(write-id out id-reader) (write-id out id-reader)
(write-utf8 out (pr-str x))) (write-utf8 out (encore/pr-edn x)))
:else ; Fallback #3: *final-freeze-fallback* :else ; Fallback #3: *final-freeze-fallback*
(if-let [ffb *final-freeze-fallback*] (ffb x out) (if-let [ffb *final-freeze-fallback*] (ffb x out)
(throw (ex-info (format "Unfreezable type: %s %s" (type x) (str x)) (throw (ex-info (format "Unfreezable type: %s %s" (type x) (str x))
{:type (type x) {:type (type x)
:as-str (pr-str x)})))))) :as-str (encore/pr-edn x)}))))))
(def ^:private head-meta-id (reduce-kv #(assoc %1 %3 %2) {} head-meta)) (def ^:private head-meta-id (reduce-kv #(assoc %1 %3 %2) {} head-meta))
(def ^:private get-head-ba (def ^:private get-head-ba
@ -366,6 +363,7 @@
(defn- wrap-header [data-ba head-meta] (defn- wrap-header [data-ba head-meta]
(if-let [head-ba (get-head-ba head-meta)] (if-let [head-ba (get-head-ba head-meta)]
;; TODO Would be nice if we could avoid the array copy here:
(encore/ba-concat head-ba data-ba) (encore/ba-concat head-ba data-ba)
(throw (ex-info (format "Unrecognized header meta: %s" head-meta) (throw (ex-info (format "Unrecognized header meta: %s" head-meta)
{:head-meta head-meta})))) {:head-meta head-meta}))))
@ -375,7 +373,7 @@
(defn freeze-to-out! (defn freeze-to-out!
"Low-level API. Serializes arg (any Clojure data type) to a DataOutput." "Low-level API. Serializes arg (any Clojure data type) to a DataOutput."
[^DataOutput data-output x & _] [^DataOutput data-output x]
(freeze-to-out data-output x)) (freeze-to-out data-output x))
(defn default-freeze-compressor-selector (defn default-freeze-compressor-selector
@ -386,29 +384,33 @@
[^bytes ba] [^bytes ba]
(let [ba-len (alength ba)] (let [ba-len (alength ba)]
(cond (cond
;; (> ba-len 1024) lzma2-compressor ;; (> ba-len 4098) lzma2-compressor
;; (> ba-len 512) lz4hc-compressor ;; (> ba-len 2048) lz4hc-compressor
(> ba-len 128) lz4-compressor (> ba-len 1024) lz4-compressor
:else nil))) :else nil)))
(encore/defonce* default-freeze-compressor-selector_ (encore/defonce* ^:dynamic *default-freeze-compressor-selector*
"EXPERIMENTAL. "(fn selector [^bytes ba])->compressor used by `(freeze <x> {:compressor :auto})."
Determines the global default default compressor selector default-freeze-compressor-selector)
(fn [^bytes ba])->compressor used by `(freeze <x> {:compressor :auto <...>})."
(atom default-freeze-compressor-selector)) (defn set-default-freeze-compressor-selector!
"Sets root binding of `*default-freeze-compressor-selector*`."
[selector]
(alter-var-root #'*default-freeze-compressor-selector* (constantly selector)))
(defn freeze (defn freeze
"Serializes arg (any Clojure data type) to a byte array. To freeze custom "Serializes arg (any Clojure data type) to a byte array. To freeze custom
types, extend the Clojure reader or see `extend-freeze`." types, extend the Clojure reader or see `extend-freeze`."
^bytes [x & [{:keys [compressor encryptor password skip-header?] (^bytes [x] (freeze x nil))
(^bytes [x {:keys [compressor encryptor password skip-header?]
:or {compressor :auto :or {compressor :auto
encryptor aes128-encryptor} encryptor aes128-encryptor}
:as opts}]] :as opts}]
(let [legacy-mode? (:legacy-mode opts) ; DEPRECATED Nippy v1-compatible freeze (let [legacy-mode? (:legacy-mode opts) ; DEPRECATED Nippy v1-compatible freeze
compressor (if-not legacy-mode? compressor snappy-compressor) compressor (if legacy-mode? snappy-compressor compressor)
encryptor (when password (if-not legacy-mode? encryptor nil)) encryptor (when password (if-not legacy-mode? encryptor nil))
skip-header? (or skip-header? legacy-mode?) skip-header? (or skip-header? legacy-mode?)
baos (ByteArrayOutputStream.) baos (ByteArrayOutputStream. 64)
dos (DataOutputStream. baos)] dos (DataOutputStream. baos)]
(freeze-to-out! dos x) (freeze-to-out! dos x)
(let [ba (.toByteArray baos) (let [ba (.toByteArray baos)
@ -417,14 +419,14 @@
(if (identical? compressor :auto) (if (identical? compressor :auto)
(if skip-header? (if skip-header?
lz4-compressor lz4-compressor
(@default-freeze-compressor-selector_ ba)) (*default-freeze-compressor-selector* ba))
(if (fn? compressor) (if (fn? compressor)
(compressor ba) ; Assume compressor selector fn (compressor ba) ; Assume compressor selector fn
compressor ; Assume compressor compressor ; Assume compressor
)) ))
ba (if-not compressor ba (compress compressor ba)) ba (if compressor (compress compressor ba) ba)
ba (if-not encryptor ba (encrypt encryptor password ba))] ba (if encryptor (encrypt encryptor password ba) ba)]
(if skip-header? ba (if skip-header? ba
(wrap-header ba (wrap-header ba
@ -433,19 +435,24 @@
(compression/header-id c)) :else)) (compression/header-id c)) :else))
:encryptor-id (when-let [e encryptor] :encryptor-id (when-let [e encryptor]
(or (encryption/standard-header-ids (or (encryption/standard-header-ids
(encryption/header-id e)) :else))}))))) (encryption/header-id e)) :else))}))))))
;;;; Thawing ;;;; Thawing
(declare thaw-from-in) (declare thaw-from-in)
(defmacro read-bytes [in & [small?]] (defmacro read-bytes [in & [small?]]
(if small? ; Optimization, must be known before id's written
`(let [in# ~in `(let [in# ~in
size# (if ~small? ; Optimization, must be known before id's written size# (.readByte in#)
(.readByte in#)
(.readInt in#))
ba# (byte-array size#)] ba# (byte-array size#)]
(.readFully in# ba# 0 size#) ba#)) (.readFully in# ba# 0 size#)
ba#)
`(let [in# ~in
size# (.readInt in#)
ba# (byte-array size#)]
(.readFully in# ba# 0 size#)
ba#)))
(defmacro read-biginteger [in] `(BigInteger. (read-bytes ~in))) (defmacro read-biginteger [in] `(BigInteger. (read-bytes ~in)))
(defmacro read-utf8 [in & [small?]] (defmacro read-utf8 [in & [small?]]
@ -454,18 +461,21 @@
(defmacro read-compact-long [in] `(long (BigInteger. (read-bytes ~in :small)))) (defmacro read-compact-long [in] `(long (BigInteger. (read-bytes ~in :small))))
(defmacro ^:private read-coll [in coll] (defmacro ^:private read-coll [in coll]
`(let [in# ~in] (encore/repeatedly-into* ~coll (.readInt in#) (thaw-from-in in#)))) `(let [in# ~in] (encore/repeatedly-into ~coll (.readInt in#)
(fn [] (thaw-from-in in#)))))
(defmacro ^:private read-kvs [in coll] (defmacro ^:private read-kvs [in coll]
`(let [in# ~in] `(let [in# ~in]
(encore/repeatedly-into* ~coll (quot (.readInt in#) 2) (encore/repeatedly-into ~coll (quot (.readInt in#) 2) ; /2 here is vestigial
[(thaw-from-in in#) (thaw-from-in in#)]))) (fn [] [(thaw-from-in in#) (thaw-from-in in#)]))))
(def ^:private class-method-sig (into-array Class [IPersistentMap])) (def ^:private class-method-sig (into-array Class [IPersistentMap]))
(declare ^:private custom-readers) (def ^:dynamic *custom-readers* "{<hash-or-byte-id> (fn [data-input])}" nil)
(defn swap-custom-readers! [f] (alter-var-root #'*custom-readers* f))
(defn- read-custom! [type-id in] (defn- read-custom! [type-id in]
(if-let [custom-reader (get @custom-readers type-id)] (if-let [custom-reader (get *custom-readers* type-id)]
(try (try
(custom-reader in) (custom-reader in)
(catch Exception e (catch Exception e
@ -491,7 +501,7 @@
id-reader id-reader
(let [edn (read-utf8 in)] (let [edn (read-utf8 in)]
(try (try
(edn/read-string {:readers *data-readers*} edn) (encore/read-edn {:readers *data-readers*} edn)
(catch Exception e (catch Exception e
{:type :reader {:type :reader
:throwable e :throwable e
@ -534,8 +544,8 @@
id-keyword (keyword (read-utf8 in)) id-keyword (keyword (read-utf8 in))
;;; Optimized, common-case types (v2.6+) ;;; Optimized, common-case types (v2.6+)
id-string-small (String. (read-bytes in :small) "UTF-8") id-string-small (read-utf8 in :small)
id-keyword-small (keyword (String. (read-bytes in :small) "UTF-8")) id-keyword-small (keyword (read-utf8 in :small))
id-queue (read-coll in (PersistentQueue/EMPTY)) id-queue (read-coll in (PersistentQueue/EMPTY))
id-sorted-set (read-coll in (sorted-set)) id-sorted-set (read-coll in (sorted-set))
@ -569,18 +579,22 @@
id-double (.readDouble in) id-double (.readDouble in)
id-bigdec (BigDecimal. (read-biginteger in) (.readInt in)) id-bigdec (BigDecimal. (read-biginteger in) (.readInt in))
id-ratio (/ (bigint (read-biginteger in)) ;; id-ratio (/ (bigint (read-biginteger in))
(bigint (read-biginteger in))) ;; (bigint (read-biginteger in)))
id-ratio (clojure.lang.Ratio.
(read-biginteger in)
(read-biginteger in))
id-date (Date. (.readLong in)) id-date (Date. (.readLong in))
id-uuid (UUID. (.readLong in) (.readLong in)) id-uuid (UUID. (.readLong in) (.readLong in))
;;; DEPRECATED ;;; DEPRECATED
id-old-reader (edn/read-string (.readUTF in)) id-reader-depr1 (encore/read-edn (.readUTF in))
id-old-string (.readUTF in) id-string-depr1 (.readUTF in)
id-old-map (apply hash-map (encore/repeatedly-into* [] id-map-depr1 (apply hash-map (encore/repeatedly-into [] (* 2 (.readInt in))
(* 2 (.readInt in)) (thaw-from-in in))) (fn [] (thaw-from-in in))))
id-old-keyword (keyword (.readUTF in)) id-keyword-depr1 (keyword (.readUTF in))
id-prefixed-custom ; Prefixed custom type id-prefixed-custom ; Prefixed custom type
(let [hash-id (.readShort in)] (let [hash-id (.readShort in)]
@ -596,7 +610,7 @@
(defn thaw-from-in! (defn thaw-from-in!
"Low-level API. Deserializes a frozen object from given DataInput to its "Low-level API. Deserializes a frozen object from given DataInput to its
original Clojure data type." original Clojure data type."
[data-input & _] [data-input]
(thaw-from-in data-input)) (thaw-from-in data-input))
(defn- try-parse-header [ba] (defn- try-parse-header [ba]
@ -633,16 +647,17 @@
Options include: Options include:
:compressor - An ICompressor, :auto (requires Nippy header), or nil. :compressor - An ICompressor, :auto (requires Nippy header), or nil.
:encryptor - An IEncryptor, :auto (requires Nippy header), or nil." :encryptor - An IEncryptor, :auto (requires Nippy header), or nil."
[^bytes ba
& [{:keys [compressor encryptor password v1-compatibility?]
:or {compressor :auto
encryptor :auto
v1-compatibility? true ; Recommend disabling when possible
}
:as opts}]]
(assert (not (contains? opts :headerless-meta)) ([ba] (thaw ba nil))
":headerless-meta `thaw` option removed as of Nippy v2.7.") ([^bytes ba
{:keys [v1-compatibility? compressor encryptor password]
:or {v1-compatibility? true ; Recommend disabling when possible
compressor :auto
encryptor :auto}
:as opts}]
(assert (not (:headerless-meta opts))
":headerless-meta `thaw` opt removed in Nippy v2.7+")
(let [ex (fn [msg & [e]] (throw (ex-info (format "Thaw failed: %s" msg) (let [ex (fn [msg & [e]] (throw (ex-info (format "Thaw failed: %s" msg)
{:opts (merge opts {:opts (merge opts
@ -651,18 +666,20 @@
e))) e)))
thaw-data thaw-data
(fn [data-ba compressor-id encryptor-id] (fn [data-ba compressor-id encryptor-id]
(let [compressor (if-not (identical? compressor :auto) compressor (let [compressor (if (identical? compressor :auto)
(get-auto-compressor compressor-id)) (get-auto-compressor compressor-id)
encryptor (if-not (identical? encryptor :auto) encryptor compressor)
(get-auto-encryptor encryptor-id))] encryptor (if (identical? encryptor :auto)
(get-auto-encryptor encryptor-id)
encryptor)]
(when (and encryptor (not password)) (when (and encryptor (not password))
(ex "Password required for decryption.")) (ex "Password required for decryption."))
(try (try
(let [ba data-ba (let [ba data-ba
ba (if-not encryptor ba (decrypt encryptor password ba)) ba (if encryptor (decrypt encryptor password ba) ba)
ba (if-not compressor ba (decompress compressor ba)) ba (if compressor (decompress compressor ba) ba)
dis (DataInputStream. (ByteArrayInputStream. ba))] dis (DataInputStream. (ByteArrayInputStream. ba))]
(thaw-from-in! dis)) (thaw-from-in! dis))
@ -697,7 +714,7 @@
;; Well-formed header definitely not present ;; Well-formed header definitely not present
(try (thaw-nippy-v1-data ba) (try (thaw-nippy-v1-data ba)
(catch Exception _ (catch Exception _
(thaw-data ba :no-header :no-header)))))) (thaw-data ba :no-header :no-header)))))))
(comment (thaw (freeze "hello")) (comment (thaw (freeze "hello"))
(thaw (freeze "hello" {:compressor nil})) (thaw (freeze "hello" {:compressor nil}))
@ -753,7 +770,6 @@
(.writeShort ~out ~(coerce-custom-type-id custom-type-id)))) (.writeShort ~out ~(coerce-custom-type-id custom-type-id))))
~@body))) ~@body)))
(defonce custom-readers (atom {})) ; {<hash-or-byte-id> (fn [data-input]) ...}
(defmacro extend-thaw (defmacro extend-thaw
"Extends Nippy to support thawing of a custom type with given id: "Extends Nippy to support thawing of a custom type with given id:
(extend-thaw :foo/my-type [data-input] ; Keyword id (extend-thaw :foo/my-type [data-input] ; Keyword id
@ -763,15 +779,20 @@
(->MyType (.readUTF data-input)))" (->MyType (.readUTF data-input)))"
[custom-type-id [in] & body] [custom-type-id [in] & body]
(assert-custom-type-id custom-type-id) (assert-custom-type-id custom-type-id)
(when (contains? @custom-readers (coerce-custom-type-id custom-type-id)) `(do
(when (contains? *custom-readers* ~(coerce-custom-type-id custom-type-id))
(println (format "Warning: resetting Nippy thaw for custom type with id: %s" (println (format "Warning: resetting Nippy thaw for custom type with id: %s"
custom-type-id))) ~custom-type-id)))
`(swap! custom-readers assoc (swap-custom-readers!
(fn [m#]
(assoc m#
~(coerce-custom-type-id custom-type-id) ~(coerce-custom-type-id custom-type-id)
(fn [~(with-meta in {:tag 'java.io.DataInput})] (fn [~(with-meta in {:tag 'java.io.DataInput})]
~@body))) ~@body))))))
(comment (defrecord MyType [data]) (comment
*custom-readers*
(defrecord MyType [data])
(extend-freeze MyType 1 [x out] (.writeUTF out (:data x))) (extend-freeze MyType 1 [x out] (.writeUTF out (:data x)))
(extend-thaw 1 [in] (->MyType (.readUTF in))) (extend-thaw 1 [in] (->MyType (.readUTF in)))
(thaw (freeze (->MyType "Joe")))) (thaw (freeze (->MyType "Joe"))))
@ -785,9 +806,9 @@
ba-len (alength ba) ba-len (alength ba)
compress? (> ba-len 1024)] compress? (> ba-len 1024)]
(.writeBoolean out compress?) (.writeBoolean out compress?)
(if-not compress? (write-bytes out ba) (if compress?
(let [ba* (compress lzma2-compressor ba)] (write-bytes out (compress lzma2-compressor ba))
(write-bytes out ba*))))) (write-bytes out ba))))
(extend-thaw 128 [in] (extend-thaw 128 [in]
(let [compressed? (.readBoolean in) (let [compressed? (.readBoolean in)
@ -905,11 +926,3 @@
(comment (inspect-ba (freeze "hello")) (comment (inspect-ba (freeze "hello"))
(seq (:data-ba (inspect-ba (freeze "hello"))))) (seq (:data-ba (inspect-ba (freeze "hello")))))
;;;; Deprecated API
(def freeze-to-stream! "DEPRECATED: Use `freeze-to-out!` instead."
freeze-to-out!)
(def thaw-from-stream! "DEPRECATED: Use `thaw-from-in!` instead."
thaw-from-in!)

30
src/taoensso/nippy/benchmarks.clj
View file

@ -1,7 +1,6 @@
(ns taoensso.nippy.benchmarks (ns taoensso.nippy.benchmarks
{:author "Peter Taoussanis"} {:author "Peter Taoussanis"}
(:require [clojure.tools.reader.edn :as edn] (:require [clojure.data.fressian :as fressian]
[clojure.data.fressian :as fressian]
[taoensso.encore :as encore] [taoensso.encore :as encore]
[taoensso.nippy :as nippy :refer (freeze thaw)])) [taoensso.nippy :as nippy :refer (freeze thaw)]))
@ -37,7 +36,7 @@
(println (str "\nLap " (inc l) "/" laps "...")) (println (str "\nLap " (inc l) "/" laps "..."))
(when reader? ; Slow (when reader? ; Slow
(println {:reader (bench1 #(pr-str %) #(edn/read-string %) (println {:reader (bench1 encore/pr-edn encore/read-edn
#(count (.getBytes ^String % "UTF-8")))})) #(count (.getBytes ^String % "UTF-8")))}))
(println {:default (bench1 #(freeze % {}) (println {:default (bench1 #(freeze % {})
@ -59,12 +58,35 @@
(println "\nDone! (Time for cake?)") (println "\nDone! (Time for cake?)")
true) true)
(comment (edn/read-string (pr-str data)) (comment (encore/read-edn (encore/pr-edn data))
(bench1 fressian-freeze fressian-thaw)) (bench1 fressian-freeze fressian-thaw))
(comment (comment
(set! *unchecked-math* false)
;; (bench {:reader? true :lzma2? true :fressian? true :laps 3}) ;; (bench {:reader? true :lzma2? true :fressian? true :laps 3})
;; (bench {:laps 4}) ;; (bench {:laps 4})
;; (bench {:laps 1 :lzma2? true})
;;; 2015 Sep 29, various micro optimizations (incl. &arg elimination)
{:reader {:round 63547, :freeze 19374, :thaw 44173, :size 27717}}
{:lzma2 {:round 51724, :freeze 33502, :thaw 18222, :size 11248}}
{:fressian {:round 8813, :freeze 6460, :thaw 2353, :size 16985}}
{:encrypted {:round 6005, :freeze 3768, :thaw 2237, :size 16164}}
{:default {:round 5417, :freeze 3354, :thaw 2063, :size 16145}}
{:fast {:round 4659, :freeze 2712, :thaw 1947, :size 17026}}
;;; 2015 Sep 15 - v2.10.0-alpha6, Clojure 1.7.0
{:reader {:round 94901, :freeze 25781, :thaw 69120, :size 27686}}
{:lzma2 {:round 65127, :freeze 43150, :thaw 21977, :size 11244}}
{:encrypted {:round 12590, :freeze 7565, :thaw 5025, :size 16148}}
{:fressian {:round 12085, :freeze 9168, :thaw 2917, :size 16972}}
{:default {:round 6974, :freeze 4582, :thaw 2392, :size 16123}}
{:fast {:round 6255, :freeze 3724, :thaw 2531, :size 17013}}
;;; 2015 Sep 14 - v2.10.0-alpha5, Clojure 1.7.0-RC1
{:default {:round 6870, :freeze 4376, :thaw 2494, :size 16227}}
{:fast {:round 6104, :freeze 3743, :thaw 2361, :size 17013}}
{:encrypted {:round 12155, :freeze 6908, :thaw 5247, :size 16244}}
;;; 2015 June 4 - v2.9.0, Clojure 1.7.0-RC1 ;;; 2015 June 4 - v2.9.0, Clojure 1.7.0-RC1
{:reader {:round 155353, :freeze 44192, :thaw 111161, :size 27693}} {:reader {:round 155353, :freeze 44192, :thaw 111161, :size 27693}}

3
src/taoensso/nippy/utils.clj
View file

@ -1,7 +1,6 @@
(ns taoensso.nippy.utils (ns taoensso.nippy.utils
{:author "Peter Taoussanis"} {:author "Peter Taoussanis"}
(:require [clojure.string :as str] (:require [clojure.string :as str]
[clojure.tools.reader.edn :as edn]
[taoensso.encore :as encore]) [taoensso.encore :as encore])
(:import [java.io ByteArrayInputStream ByteArrayOutputStream Serializable (:import [java.io ByteArrayInputStream ByteArrayOutputStream Serializable
ObjectOutputStream ObjectInputStream])) ObjectOutputStream ObjectInputStream]))
@ -34,7 +33,7 @@
(cast class object) (cast class object)
true))))) true)))))
(def readable? (memoize-type-test (fn [x] (-> x pr-str (edn/read-string)) true))) (def readable? (memoize-type-test (fn [x] (-> x encore/pr-edn encore/read-edn) true)))
(comment (comment
(serializable? "Hello world") (serializable? "Hello world")

30
test/taoensso/nippy/tests/main.clj
View file

@ -14,6 +14,8 @@
;;;; Core ;;;; Core
(expect (do (println (str "Clojure version: " *clojure-version*)) true))
(expect test-data ((comp thaw freeze) test-data)) (expect test-data ((comp thaw freeze) test-data))
(expect test-data ((comp #(thaw % {}) (expect test-data ((comp #(thaw % {})
#(freeze % {:legacy-mode true})) #(freeze % {:legacy-mode true}))
@ -38,7 +40,7 @@
(check-props/for-all [val check-gen/any] (check-props/for-all [val check-gen/any]
(= val (thaw (freeze val))))))) (= val (thaw (freeze val)))))))
;;; These can sometimes crash the JVM ;;; Trying to decrypt random (invalid) data can actually crash JVM
;; (expect Exception (thaw (freeze test-data {:password "malformed"}))) ;; (expect Exception (thaw (freeze test-data {:password "malformed"})))
;; (expect Exception (thaw (freeze test-data {:password [:salted "p"]}))) ;; (expect Exception (thaw (freeze test-data {:password [:salted "p"]})))
;; (expect Exception (thaw (freeze test-data {:password [:salted "p"]}) ;; (expect Exception (thaw (freeze test-data {:password [:salted "p"]})
@ -75,25 +77,26 @@
(nippy/extend-thaw :nippy-tests/MyType [s] (->MyType (.readUTF s))) (nippy/extend-thaw :nippy-tests/MyType [s] (->MyType (.readUTF s)))
(let [type (->MyType "val")] (= type (thaw (freeze type)))))) (let [type (->MyType "val")] (= type (thaw (freeze type))))))
;;;; Stable binary representation of vals ; EXPERIMENTAL ;;;; Stable binary representation of vals
(expect (seq (freeze test-data)) (expect (seq (freeze test-data))
(seq (freeze test-data))) ; f(x)=f(y) | x=y (seq (freeze test-data))) ; f(x)=f(y) | x=y
;;; As above, but try multiple times (catch protocol interface races): ;; As above, but try multiple times to catch possible protocol interface races:
(expect #(every? true? %) (expect #(every? true? %)
(repeatedly 1000 (fn [] (= (seq (freeze test-data)) (repeatedly 1000 (fn [] (= (seq (freeze test-data))
(seq (freeze test-data)))))) (seq (freeze test-data))))))
(expect (seq (-> test-data freeze)) ; f(x)=f(f-1(f(x))) ;; NB abandoning - no way to do this reliably w/o appropriate contracts from
(seq (-> test-data freeze thaw freeze))) ;; (seq <unordered-coll>):
;;
;;; As above, but with repeated refreeze (catch protocol interface races): ;; (expect (seq (-> test-data freeze)) ; f(x)=f(f-1(f(x)))
(expect (= (seq (freeze test-data)) ;; (seq (-> test-data freeze thaw freeze)))
(seq (reduce (fn [frozen _] (freeze (thaw frozen))) ;;
(freeze test-data) (range 1000))))) ;; As above, but with repeated refreeze to catch possible protocol interface races:
;; (expect (= (seq (freeze test-data))
;;; ;; (seq (reduce (fn [frozen _] (freeze (thaw frozen)))
;; (freeze test-data) (range 1000)))))
(defn qc-prop-bijection [& [n]] (defn qc-prop-bijection [& [n]]
(let [bin->val (atom {}) (let [bin->val (atom {})
@ -123,7 +126,6 @@
(let [{:keys [result bin->val val->bin]} (qc-prop-bijection 10)] (let [{:keys [result bin->val val->bin]} (qc-prop-bijection 10)]
[result (vals bin->val)])) [result (vals bin->val)]))
;; (expect #(:result %) (qc-prop-bijection 120)) ; Time is n-non-linear
(expect #(:result %) (qc-prop-bijection 80)) (expect #(:result %) (qc-prop-bijection 80))
;;;; Thread safety ;;;; Thread safety
@ -164,4 +166,4 @@
;;;; Benchmarks ;;;; Benchmarks
;; (expect (benchmarks/bench {})) ; Also tests :cached passwords (expect (benchmarks/bench {})) ; Also tests :cached passwords