luciano/nippy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'dev'

Browse source
This commit is contained in:
Peter Taoussanis 2014-08-27 19:25:19 +07:00
commit 38aa3344ae
11 changed files with 485 additions and 371 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

23
CHANGELOG.md
View file

@ -1,3 +1,26 @@
> This project uses [Break Versioning](https://github.com/ptaoussanis/encore/blob/master/BREAK-VERSIONING.md) as of **Aug 16, 2014**.
## v2.7.0-RC1 / 2014 Aug 27
> **Major release** with significant performance improvements, a new default compression type ([LZ4](http://blog.jpountz.net/post/28092106032/wow-lz4-is-fast)), and better support for a variety of compression/encryption tools.
>
> The data format is fully **backwards-compatible**, the API is backwards compatible **unless** you are using the `:headerless-meta` thaw option.
### Changes
* A number of internal performance improvements.
* Added [LZ4](http://blog.jpountz.net/post/28092106032/wow-lz4-is-fast) compressor, **replacing Snappy as the default** (often ~10+% faster with similar compression ratios). **Thanks to [mpenet](https://github.com/mpenet) for his work on this**!
* **BREAKING**: the `thaw` `:headerless-meta` option has been dropped. Its purpose was to provide Nippy v1 compatibility, which is now done automatically. To prevent any surprises, `thaw` calls with this option will now **throw an assertion error**.
* **IMPORTANT**: the `thaw` API has been improved (simplified). The default `:encryptor` and `:compressor` values are now both `:auto`, which'll choose intelligently based on data now included with the Nippy header. Behaviour remains the same for data written without a header: you must specify the correct `:compressor` and `:encryptor` values manually.
* Promoted from Alpha status: `taoensso.nippy.compression` ns, `taoensso.nippy.encryption` ns, `taoensso.nippy.tools` ns, `extend-freeze`, `extend-thaw`.
* All Nippy exceptions are now `ex-info`s.
* `extend-thaw` now prints a warning when replacing a pre-existing type id.
### NEW
* #50: `extend-freeze`, `extend-thaw` can now take arbitrary keyword type ids (see docstrings for more info).
## v2.6.3 / 2014 Apr 29 ## v2.6.3 / 2014 Apr 29
* Fix #48: broken freeze/thaw identity for empty lazy seqs (@vgeshel). * Fix #48: broken freeze/thaw identity for empty lazy seqs (@vgeshel).

20
README.md
View file

@ -1,10 +1,11 @@
**[API docs][]** | **[CHANGELOG][]** | [other Clojure libs][] | [Twitter][] | [contact/contributing](#contact--contributing) | current ([semantic][]) version: **[API docs][]** | **[CHANGELOG][]** | [other Clojure libs][] | [Twitter][] | [contact/contrib](#contact--contributing) | current [Break Version][]:
```clojure ```clojure
[com.taoensso/nippy "2.6.3"] ; Stable (please upgrade from v2.6.0 ASAP) [com.taoensso/nippy "2.6.3"] ; Stable
[com.taoensso/nippy "2.7.0-RC1"] ; Development
``` ```
v2.6 is a **major, backwards-compatible release** with: improved performance (incl. frozen data size), a new low-level DataInput/DataOuput API, improved support for headerless freezing, and 1-to-1 binary-value representation guarantees. See the [CHANGELOG][] for details. v2.7 is a major, **mostly backwards-compatible** release focused on improved performance and a new default compression scheme (LZ4). See the [CHANGELOG][] for details. Thanks to [mpenet](https://github.com/mpenet) for his work on the LZ4 support!
# Nippy, a Clojure serialization library # Nippy, a Clojure serialization library
@ -115,9 +116,9 @@ Couldn't be simpler!
See also the lower-level `freeze-to-out!` and `thaw-from-in!` fns for operating on `DataOutput` and `DataInput` types directly. See also the lower-level `freeze-to-out!` and `thaw-from-in!` fns for operating on `DataOutput` and `DataInput` types directly.
### Encryption (currently in **ALPHA**) ### Encryption (v2+)
Nippy v2+ also gives you **dead simple data encryption**. Add a single option to your usual freeze/thaw calls like so: Nippy also gives you **dead simple data encryption**. Add a single option to your usual freeze/thaw calls like so:
```clojure ```clojure
(nippy/freeze nippy/stress-data {:password [:salted "my-password"]}) ; Encrypt (nippy/freeze nippy/stress-data {:password [:salted "my-password"]}) ; Encrypt
@ -126,16 +127,16 @@ Nippy v2+ also gives you **dead simple data encryption**. Add a single option to
There's two default forms of encryption on offer: `:salted` and `:cached`. Each of these makes carefully-chosen trade-offs and is suited to one of two common use cases. See the `aes128-encryptor` [docstring](http://ptaoussanis.github.io/nippy/taoensso.nippy.encryption.html) for a detailed explanation of why/when you'd want one or the other. There's two default forms of encryption on offer: `:salted` and `:cached`. Each of these makes carefully-chosen trade-offs and is suited to one of two common use cases. See the `aes128-encryptor` [docstring](http://ptaoussanis.github.io/nippy/taoensso.nippy.encryption.html) for a detailed explanation of why/when you'd want one or the other.
### Custom types (v2.1+, ALPHA - subject to change) ### Custom types (v2.1+)
```clojure ```clojure
(defrecord MyType [data]) (defrecord MyType [data])
(nippy/extend-freeze MyType 1 ; A unique type id ∈[1, 128] (nippy/extend-freeze MyType :my-type/foo ; A unique (namespaced) type identifier
[x data-output] [x data-output]
(.writeUTF data-output (:data x))) (.writeUTF data-output (:data x)))
(nippy/extend-thaw 1 ; Same type id (nippy/extend-thaw :my-type/foo ; Same type id
[data-input] [data-input]
(->MyType (.readUTF data-input))) (->MyType (.readUTF data-input)))
@ -166,7 +167,8 @@ Copyright © 2012-2014 Peter Taoussanis. Distributed under the [Eclipse Publ
[CHANGELOG]: <https://github.com/ptaoussanis/nippy/releases> [CHANGELOG]: <https://github.com/ptaoussanis/nippy/releases>
[other Clojure libs]: <https://www.taoensso.com/clojure-libraries> [other Clojure libs]: <https://www.taoensso.com/clojure-libraries>
[Twitter]: <https://twitter.com/ptaoussanis> [Twitter]: <https://twitter.com/ptaoussanis>
[semantic]: <http://semver.org/> [SemVer]: <http://semver.org/>
[Break Version]: <https://github.com/ptaoussanis/encore/blob/master/BREAK-VERSIONING.md>
[Leiningen]: <http://leiningen.org/> [Leiningen]: <http://leiningen.org/>
[CDS]: <http://clojure-doc.org/> [CDS]: <http://clojure-doc.org/>
[ClojureWerkz]: <http://clojurewerkz.org/> [ClojureWerkz]: <http://clojurewerkz.org/>

BIN
benchmarks.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 23 KiB

After

Width:  |  Height:  |  Size: 23 KiB

44
project.clj
View file

@ -1,4 +1,4 @@
(defproject com.taoensso/nippy "2.6.3" (defproject com.taoensso/nippy "2.7.0-SNAPSHOT"
:author "Peter Taoussanis <https://www.taoensso.com>" :author "Peter Taoussanis <https://www.taoensso.com>"
:description "Clojure serialization library" :description "Clojure serialization library"
:url "https://github.com/ptaoussanis/nippy" :url "https://github.com/ptaoussanis/nippy"
@ -9,44 +9,42 @@
:min-lein-version "2.3.3" :min-lein-version "2.3.3"
:global-vars {*warn-on-reflection* true :global-vars {*warn-on-reflection* true
*assert* true} *assert* true}
:dependencies :dependencies
[[org.clojure/clojure "1.4.0"] [[org.clojure/clojure "1.4.0"]
[org.clojure/tools.reader "0.8.3"] [org.clojure/tools.reader "0.8.7"]
[com.taoensso/encore "1.3.1"] [com.taoensso/encore "1.7.1"]
[org.iq80.snappy/snappy "0.3"] [org.iq80.snappy/snappy "0.3"]
[org.tukaani/xz "1.5"]] [org.tukaani/xz "1.5"]
[net.jpountz.lz4/lz4 "1.2.0"]]
:test-paths ["test" "src"]
:profiles :profiles
{;; :default [:base :system :user :provided :dev] {;; :default [:base :system :user :provided :dev]
:server-jvm {:jvm-opts ^:replace ["-server" "-Xms1024m" "-Xmx2048m"]}
:1.5 {:dependencies [[org.clojure/clojure "1.5.1"]]} :1.5 {:dependencies [[org.clojure/clojure "1.5.1"]]}
:1.6 {:dependencies [[org.clojure/clojure "1.6.0"]]} :1.6 {:dependencies [[org.clojure/clojure "1.6.0"]]}
:test {:jvm-opts ["-Xms1024m" "-Xmx2048m"] :test {:jvm-opts ["-Xms1024m" "-Xmx2048m"]
:dependencies [[expectations "1.4.56"] :dependencies [[expectations "2.0.9"]
[org.clojure/test.check "0.5.7"] [org.clojure/test.check "0.5.9"]
;; [com.cemerick/double-check "0.5.7"]
[org.clojure/data.fressian "0.2.0"] [org.clojure/data.fressian "0.2.0"]
[org.xerial.snappy/snappy-java "1.1.1-M1"]] [org.xerial.snappy/snappy-java "1.1.1.3"]]}
:plugins [[lein-expectations "0.0.8"] :dev [:1.6 :test
[lein-autoexpect "1.2.2"]]} {:plugins
:dev* [:dev {:jvm-opts ^:replace ["-server"] [[lein-pprint "1.1.1"]
;; :hooks [cljx.hooks leiningen.cljsbuild] ; cljx [lein-ancient "0.5.5"]
}] [lein-expectations "0.0.8"]
:dev [lein-autoexpect "1.2.2"]
[:1.6 :test [codox "0.8.10"]]}]}
{:jvm-opts ^:replace ["-server" "-Xms1024m" "-Xmx2048m"]
:dependencies [] :test-paths ["test" "src"]
:plugins [[lein-ancient "0.5.4"]
[codox "0.6.7"]]}]}
;; :codox {:sources ["target/classes"]} ; cljx
:aliases :aliases
{"test-all" ["with-profile" "default:+1.5:+1.6" "expectations"] {"test-all" ["with-profile" "default:+1.5:+1.6" "expectations"]
;; "test-all" ["with-profile" "default:+1.6" "expectations"] ;; "test-all" ["with-profile" "default:+1.6" "expectations"]
"test-auto" ["with-profile" "+test" "autoexpect"] "test-auto" ["with-profile" "+test" "autoexpect"]
;; "build-once" ["do" "cljx" "once," "cljsbuild" "once"] ; cljx
;; "deploy-lib" ["do" "build-once," "deploy" "clojars," "install"] ; cljx
"deploy-lib" ["do" "deploy" "clojars," "install"] "deploy-lib" ["do" "deploy" "clojars," "install"]
"start-dev" ["with-profile" "+dev*" "repl" ":headless"]} "start-dev" ["with-profile" "+server-jvm" "repl" ":headless"]}
:repositories :repositories
{"sonatype" {"sonatype"

392
src/taoensso/nippy.clj
View file

@ -6,8 +6,8 @@
[taoensso.encore :as encore] [taoensso.encore :as encore]
[taoensso.nippy [taoensso.nippy
(utils :as utils) (utils :as utils)
(compression :as compression :refer (snappy-compressor)) (compression :as compression)
(encryption :as encryption :refer (aes128-encryptor))]) (encryption :as encryption)])
(:import [java.io ByteArrayInputStream ByteArrayOutputStream DataInputStream (:import [java.io ByteArrayInputStream ByteArrayOutputStream DataInputStream
DataOutputStream Serializable ObjectOutputStream ObjectInputStream DataOutputStream Serializable ObjectOutputStream ObjectInputStream
DataOutput DataInput] DataOutput DataInput]
@ -19,26 +19,46 @@
PersistentQueue PersistentTreeMap PersistentTreeSet PersistentList ; LazySeq PersistentQueue PersistentTreeMap PersistentTreeSet PersistentList ; LazySeq
IRecord ISeq])) IRecord ISeq]))
;;;; Nippy 2.x+ header spec (4 bytes) ;;;; Nippy data format
;; Header is optional but recommended + enabled by default. Uses: ;; * 4-byte header (Nippy v2.x+) (may be disabled but incl. by default) [1].
;; * Sanity check (data appears to be Nippy data). ;; { * 1-byte type id.
;; * Arb-length payload. } ...
;;
;; [1] Inclusion of header is strongly recommended. Purpose:
;; * Sanity check (confirm that data appears to be Nippy data).
;; * Nippy version check (=> supports changes to data schema over time). ;; * Nippy version check (=> supports changes to data schema over time).
;; * Encrypted &/or compressed data identification. ;; * Supports :auto thaw compressor, encryptor.
;; ;;
(def ^:private ^:const head-version 1) (def ^:private ^:const head-version 1)
(def ^:private head-sig (.getBytes "NPY" "UTF-8")) (def ^:private head-sig (.getBytes "NPY" "UTF-8"))
(def ^:private ^:const head-meta "Final byte stores version-dependent metadata." (def ^:private ^:const head-meta "Final byte stores version-dependent metadata."
{(byte 0) {:version 1 :compressed? false :encrypted? false} {(byte 0) {:version 1 :compressor-id nil :encryptor-id nil}
(byte 1) {:version 1 :compressed? true :encrypted? false} (byte 4) {:version 1 :compressor-id nil :encryptor-id :else}
(byte 2) {:version 1 :compressed? false :encrypted? true} (byte 5) {:version 1 :compressor-id :else :encryptor-id nil}
(byte 3) {:version 1 :compressed? true :encrypted? true}}) (byte 6) {:version 1 :compressor-id :else :encryptor-id :else}
;;
(byte 2) {:version 1 :compressor-id nil :encryptor-id :aes128-sha512}
;;
(byte 1) {:version 1 :compressor-id :snappy :encryptor-id nil}
(byte 3) {:version 1 :compressor-id :snappy :encryptor-id :aes128-sha512}
(byte 7) {:version 1 :compressor-id :snappy :encryptor-id :else}
;;
(byte 8) {:version 1 :compressor-id :lz4 :encryptor-id nil}
(byte 9) {:version 1 :compressor-id :lz4 :encryptor-id :aes128-sha512}
(byte 10) {:version 1 :compressor-id :lz4 :encryptor-id :else}
;;
(byte 11) {:version 1 :compressor-id :lzma2 :encryptor-id nil}
(byte 12) {:version 1 :compressor-id :lzma2 :encryptor-id :aes128-sha512}
(byte 13) {:version 1 :compressor-id :lzma2 :encryptor-id :else}})
(defmacro when-debug-mode [& body] (when #_true false `(do ~@body))) (defmacro when-debug-mode [& body] (when #_true false `(do ~@body)))
;;;; Data type IDs ;;;; Data type IDs
;; **Negative ids reserved for user-defined types**
(do ; Just for easier IDE collapsing (do ; Just for easier IDE collapsing
;; ** Negative ids reserved for user-defined types **
;;
(def ^:const id-reserved (int 0)) (def ^:const id-reserved (int 0))
;; 1 ;; 1
(def ^:const id-bytes (int 2)) (def ^:const id-bytes (int 2))
@ -78,7 +98,8 @@
(def ^:const id-ratio (int 70)) (def ^:const id-ratio (int 70))
(def ^:const id-record (int 80)) (def ^:const id-record (int 80))
;; (def ^:const id-type (int 81)) ; TODO ;; (def ^:const id-type (int 81)) ; TODO?
(def ^:const id-prefixed-custom (int 82))
(def ^:const id-date (int 90)) (def ^:const id-date (int 90))
(def ^:const id-uuid (int 91)) (def ^:const id-uuid (int 91))
@ -103,6 +124,21 @@
(def ^:const id-old-keyword (int 12)) ; as of 2.0.0-alpha5, for str consistecy (def ^:const id-old-keyword (int 12)) ; as of 2.0.0-alpha5, for str consistecy
) )
;;;; Ns imports (mostly for convenience of lib consumers)
(encore/defalias compress compression/compress)
(encore/defalias decompress compression/decompress)
(encore/defalias snappy-compressor compression/snappy-compressor)
(encore/defalias lzma2-compressor compression/lzma2-compressor)
(encore/defalias lz4-compressor compression/lz4-compressor)
(encore/defalias lz4hc-compressor compression/lz4hc-compressor)
(encore/defalias encrypt encryption/encrypt)
(encore/defalias decrypt encryption/decrypt)
(encore/defalias aes128-encryptor encryption/aes128-encryptor)
(encore/defalias freezable? utils/freezable?)
;;;; Freezing ;;;; Freezing
(defprotocol Freezable (defprotocol Freezable
@ -129,7 +165,7 @@
(let [x (with-meta x {:tag 'String})] (let [x (with-meta x {:tag 'String})]
`(write-bytes ~out (.getBytes ~x "UTF-8") ~small?))) `(write-bytes ~out (.getBytes ~x "UTF-8") ~small?)))
(defmacro write-compact-long "EXPERIMENTAL! Uses 2->9 bytes." [out x] (defmacro write-compact-long "Uses 2->9 bytes." [out x]
`(write-bytes ~out (.toByteArray (java.math.BigInteger/valueOf (long ~x))) `(write-bytes ~out (.toByteArray (java.math.BigInteger/valueOf (long ~x)))
:small)) :small))
@ -296,21 +332,25 @@
:else ; Fallback #3: *final-freeze-fallback* :else ; Fallback #3: *final-freeze-fallback*
(if-let [ffb *final-freeze-fallback*] (ffb x out) (if-let [ffb *final-freeze-fallback*] (ffb x out)
(throw (Exception. (format "Unfreezable type: %s %s" (throw (ex-info (format "Unfreezable type: %s %s" (type x) (str x))
(type x) (str x)))))))) {:type (type x)
:as-str (pr-str x)}))))))
(def ^:private head-meta-id (reduce-kv #(assoc %1 %3 %2) {} head-meta)) (def ^:private head-meta-id (reduce-kv #(assoc %1 %3 %2) {} head-meta))
(def ^:private get-head-ba
(memoize
(fn [head-meta]
(when-let [meta-id (get head-meta-id (assoc head-meta :version head-version))]
(encore/ba-concat head-sig (byte-array [meta-id]))))))
(defn- wrap-header [data-ba metadata] (defn- wrap-header [data-ba head-meta]
(if-let [meta-id (head-meta-id (assoc metadata :version head-version))] (if-let [head-ba (get-head-ba head-meta)]
(let [head-ba (encore/ba-concat head-sig (byte-array [meta-id]))] (encore/ba-concat head-ba data-ba)
(encore/ba-concat head-ba data-ba)) (throw (ex-info (format "Unrecognized header meta: %s" head-meta)
(throw (Exception. (str "Unrecognized header metadata: " metadata))))) {:head-meta head-meta}))))
(comment (wrap-header (.getBytes "foo") {:compressed? true (comment (wrap-header (.getBytes "foo") {:compressor-id :lz4
:encrypted? false})) :encryptor-id nil}))
(declare assert-legacy-args) ; Deprecated
(defn freeze-to-out! (defn freeze-to-out!
"Low-level API. Serializes arg (any Clojure data type) to a DataOutput." "Low-level API. Serializes arg (any Clojure data type) to a DataOutput."
@ -318,24 +358,30 @@
(freeze-to-out data-output x)) (freeze-to-out data-output x))
(defn freeze (defn freeze
"Serializes arg (any Clojure data type) to a byte array. For custom types "Serializes arg (any Clojure data type) to a byte array. To freeze custom
extend the Clojure reader or see `extend-freeze`." types, extend the Clojure reader or see `extend-freeze`."
^bytes [x & [{:keys [password compressor encryptor skip-header?] ^bytes [x & [{:keys [compressor encryptor password skip-header?]
:or {compressor snappy-compressor :or {compressor lz4-compressor
encryptor aes128-encryptor} encryptor aes128-encryptor}
:as opts}]] :as opts}]]
(when (:legacy-mode opts) ; Deprecated (let [legacy-mode? (:legacy-mode opts) ; DEPRECATED Nippy v1-compatible freeze
(assert-legacy-args compressor password)) compressor (if-not legacy-mode? compressor snappy-compressor)
(let [skip-header? (or skip-header? (:legacy-mode opts)) ; Deprecated encryptor (when password (if-not legacy-mode? encryptor nil))
bas (ByteArrayOutputStream.) skip-header? (or skip-header? legacy-mode?)
sout (DataOutputStream. bas)] baos (ByteArrayOutputStream.)
(freeze-to-out! sout x) dos (DataOutputStream. baos)]
(let [ba (.toByteArray bas) (freeze-to-out! dos x)
ba (if compressor (compression/compress compressor ba) ba) (let [ba (.toByteArray baos)
ba (if password (encryption/encrypt encryptor password ba) ba)] ba (if-not compressor ba (compress compressor ba))
ba (if-not encryptor ba (encrypt encryptor password ba))]
(if skip-header? ba (if skip-header? ba
(wrap-header ba {:compressed? (boolean compressor) (wrap-header ba
:encrypted? (boolean password)}))))) {:compressor-id (when-let [c compressor]
(or (compression/standard-header-ids
(compression/header-id c)) :else))
:encryptor-id (when-let [e encryptor]
(or (encryption/standard-header-ids
(encryption/header-id e)) :else))})))))
;;;; Thawing ;;;; Thawing
@ -353,8 +399,7 @@
(defmacro read-utf8 [in & [small?]] (defmacro read-utf8 [in & [small?]]
`(String. (read-bytes ~in ~small?) "UTF-8")) `(String. (read-bytes ~in ~small?) "UTF-8"))
(defmacro read-compact-long "EXPERIMENTAL!" [in] (defmacro read-compact-long [in] `(long (BigInteger. (read-bytes ~in :small))))
`(long (BigInteger. (read-bytes ~in :small))))
(defmacro ^:private read-coll [in coll] (defmacro ^:private read-coll [in coll]
`(let [in# ~in] (encore/repeatedly-into* ~coll (.readInt in#) (thaw-from-in in#)))) `(let [in# ~in] (encore/repeatedly-into* ~coll (.readInt in#) (thaw-from-in in#))))
@ -364,6 +409,20 @@
[(thaw-from-in in#) (thaw-from-in in#)]))) [(thaw-from-in in#) (thaw-from-in in#)])))
(declare ^:private custom-readers) (declare ^:private custom-readers)
(defn- read-custom! [type-id in]
(if-let [custom-reader (get @custom-readers type-id)]
(try
(custom-reader in)
(catch Exception e
(throw
(ex-info
(format "Reader exception for custom type with internal id: %s"
type-id) {:internal-type-id type-id} e))))
(throw
(ex-info
(format "No reader provided for custom type with internal id: %s"
type-id)
{:internal-type-id type-id}))))
(defn- thaw-from-in (defn- thaw-from-in
[^DataInput in] [^DataInput in]
@ -453,20 +512,16 @@
(* 2 (.readInt in)) (thaw-from-in in))) (* 2 (.readInt in)) (thaw-from-in in)))
id-old-keyword (keyword (.readUTF in)) id-old-keyword (keyword (.readUTF in))
(if-not (neg? type-id) id-prefixed-custom ; Prefixed custom type
(throw (Exception. (str "Unknown type ID: " type-id))) (let [hash-id (.readShort in)]
(read-custom! hash-id in))
;; Custom types (read-custom! type-id in) ; Unprefixed custom type (catchall)
(if-let [reader (get @custom-readers type-id)] )
(try (reader in)
(catch Exception e
(throw (Exception. (str "Reader exception for custom type ID: "
(- type-id)) e))))
(throw (Exception. (str "No reader provided for custom type ID: "
(- type-id)))))))
(catch Exception e (catch Exception e
(throw (Exception. (format "Thaw failed against type-id: %s" type-id) e)))))) (throw (ex-info (format "Thaw failed against type-id: %s" type-id)
{:type-id type-id} e))))))
(defn thaw-from-in! (defn thaw-from-in!
"Low-level API. Deserializes a frozen object from given DataInput to its "Low-level API. Deserializes a frozen object from given DataInput to its
@ -477,119 +532,164 @@
(defn- try-parse-header [ba] (defn- try-parse-header [ba]
(when-let [[head-ba data-ba] (encore/ba-split ba 4)] (when-let [[head-ba data-ba] (encore/ba-split ba 4)]
(let [[head-sig* [meta-id]] (encore/ba-split head-ba 3)] (let [[head-sig* [meta-id]] (encore/ba-split head-ba 3)]
(when (encore/ba= head-sig* head-sig) ; Appears to be well-formed (when (encore/ba= head-sig* head-sig) ; Header appears to be well-formed
[data-ba (head-meta meta-id {:unrecognized-meta? true})])))) [data-ba (get head-meta meta-id {:unrecognized-meta? true})]))))
(defn- get-auto-compressor [compressor-id]
(case compressor-id
nil nil
:snappy snappy-compressor
:lzma2 lzma2-compressor
:lz4 lz4-compressor
:no-header (throw (ex-info ":auto not supported on headerless data." {}))
:else (throw (ex-info ":auto not supported for non-standard compressors." {}))
(throw (ex-info (format "Unrecognized :auto compressor id: %s" compressor-id)
{:compressor-id compressor-id}))))
(defn- get-auto-encryptor [encryptor-id]
(case encryptor-id
nil nil
:aes128-sha512 aes128-encryptor
:no-header (throw (ex-info ":auto not supported on headerless data." {}))
:else (throw (ex-info ":auto not supported for non-standard encryptors."))
(throw (ex-info (format "Unrecognized :auto encryptor id: %s" encryptor-id)
{:encryptor-id encryptor-id}))))
(defn thaw (defn thaw
"Deserializes a frozen object from given byte array to its original Clojure "Deserializes a frozen object from given byte array to its original Clojure
data type. By default[1] supports data frozen with current and all previous data type. Supports data frozen with current and all previous versions of
versions of Nippy. For custom types extend the Clojure reader or see Nippy. To thaw custom types, extend the Clojure reader or see `extend-thaw`.
`extend-thaw`.
[1] :headerless-meta provides a fallback facility for data frozen without a Options include:
standard Nippy header (notably all Nippy v1 data). A default is provided for :compressor - An ICompressor, :auto (requires Nippy header), or nil.
Nippy v1 thaw compatibility, but it's recommended that you _disable_ this :encryptor - An IEncryptor, :auto (requires Nippy header), or nil."
fallback (`{:headerless-meta nil}`) if you're certain you won't be thawing [^bytes ba & [{:keys [compressor encryptor password]
headerless data." :or {compressor :auto
[^bytes ba & [{:keys [password compressor encryptor headerless-meta] encryptor :auto}
:or {compressor snappy-compressor
encryptor aes128-encryptor
headerless-meta ; Recommend set to nil when possible
{:version 1
:compressed? true
:encrypted? false}}
:as opts}]] :as opts}]]
(let [headerless-meta (merge headerless-meta (:legacy-opts opts)) ; Deprecated (assert (not (contains? opts :headerless-meta))
_ (assert (or (nil? headerless-meta) ":headerless-meta `thaw` option removed as of Nippy v2.7.")
(head-meta-id headerless-meta))
"Bad :headerless-meta (should be nil or a valid `head-meta` value)") (let [ex (fn [msg & [e]] (throw (ex-info (format "Thaw failed: %s" msg)
{:opts (merge opts
{:compressor compressor
:encryptor encryptor})}
e)))
thaw-data
(fn [data-ba compressor-id encryptor-id]
(let [compressor (if-not (identical? compressor :auto) compressor
(get-auto-compressor compressor-id))
encryptor (if-not (identical? encryptor :auto) encryptor
(get-auto-encryptor encryptor-id))]
(when (and encryptor (not password))
(ex "Password required for decryption."))
ex (fn [msg & [e]] (throw (Exception. (str "Thaw failed: " msg) e)))
try-thaw-data
(fn [data-ba {:keys [compressed? encrypted?] :as _head-or-headerless-meta}]
(let [password (when encrypted? password)
compressor (when compressed? compressor)]
(try (try
(let [ba data-ba (let [ba data-ba
ba (if password (encryption/decrypt encryptor password ba) ba) ba (if-not encryptor ba (decrypt encryptor password ba))
ba (if compressor (compression/decompress compressor ba) ba) ba (if-not compressor ba (decompress compressor ba))
sin (DataInputStream. (ByteArrayInputStream. ba))] dis (DataInputStream. (ByteArrayInputStream. ba))]
(thaw-from-in! sin)) (thaw-from-in! dis))
(catch Exception e (catch Exception e
(cond (ex "Decryption/decompression failure, or data unfrozen/damaged.")))))
password (if head-meta (ex "Wrong password/encryptor?" e)
(ex "Unencrypted data?" e))
compressor (if head-meta (ex "Encrypted data or wrong compressor?" e)
(ex "Uncompressed data?" e))
:else (if head-meta (ex "Corrupt data?" e)
(ex "Data may be unfrozen, corrupt, compressed &/or encrypted.")))))))]
(if-let [[data-ba {:keys [unrecognized-meta? compressed? encrypted?] thaw-nippy-v1-data ; A little hackish, but necessary
(fn [data-ba]
(try (thaw-data data-ba :snappy nil)
(catch Exception _
(thaw-data data-ba nil nil))))]
(if-let [[data-ba {:keys [compressor-id encryptor-id unrecognized-meta?]
:as head-meta}] (try-parse-header ba)] :as head-meta}] (try-parse-header ba)]
(cond ; A well-formed header _appears_ to be present ;; A well-formed header _appears_ to be present (it's possible though
(and (not headerless-meta) ; Cautious. It's unlikely but possible the ;; unlikely that this is a fluke and data is actually headerless):
; header sig match was a fluke and not an (try (thaw-data data-ba compressor-id encryptor-id)
; indication of a real, well-formed header.
; May really be headerless.
unrecognized-meta?)
(ex "Unrecognized (but apparently well-formed) header. Data frozen with newer Nippy version?")
;;; It's still possible below that the header match was a fluke, but it's
;;; _very_ unlikely. Therefore _not_ going to incl.
;;; `(not headerless-meta)` conditions below.
(and compressed? (not compressor))
(ex "Compressed data? Try again with compressor.")
(and encrypted? (not password))
(if (::tools-thaw? opts) ::need-password
(ex "Encrypted data? Try again with password."))
:else (try (try-thaw-data data-ba head-meta)
(catch Exception e (catch Exception e
(if headerless-meta (try (thaw-nippy-v1-data)
(try (try-thaw-data ba headerless-meta)
(catch Exception _ (catch Exception _
(throw e))) (if unrecognized-meta?
(throw e))))) (ex "Unrecognized (but apparently well-formed) header. Data frozen with newer Nippy version?"
e)
(throw e))))))
;; Well-formed header definitely not present ;; Well-formed header definitely not present
(if headerless-meta (try (thaw-nippy-v1-data ba)
(try-thaw-data ba headerless-meta) (catch Exception _
(ex "Data may be unfrozen, corrupt, compressed &/or encrypted."))))) (thaw-data ba :no-header :no-header))))))
(comment (thaw (freeze "hello")) (comment (thaw (freeze "hello"))
(thaw (freeze "hello" {:compressor nil})) (thaw (freeze "hello" {:compressor nil}))
(thaw (freeze "hello" {:password [:salted "p"]})) ; ex (thaw (freeze "hello" {:password [:salted "p"]})) ; ex: no pwd
(thaw (freeze "hello") {:password [:salted "p"]})) (thaw (freeze "hello") {:password [:salted "p"]}))
;;;; Custom types ;;;; Custom types
(defn- assert-custom-type-id [custom-type-id]
(assert (or (keyword? custom-type-id)
(and (integer? custom-type-id) (<= 1 custom-type-id 128)))))
(defn- coerce-custom-type-id
"* +ive byte id -> -ive byte id (for unprefixed custom types).
* Keyword id -> Short hash id (for prefixed custom types)."
[custom-type-id]
(assert-custom-type-id custom-type-id)
(if-not (keyword? custom-type-id)
(int (- custom-type-id))
(let [hash-id (hash custom-type-id)
short-hash-id (if (pos? hash-id)
(mod hash-id Short/MAX_VALUE)
(mod hash-id Short/MIN_VALUE))]
;; Make sure hash ids can't collide with byte ids (unlikely anyway):
(assert (not (<= -128 short-hash-id -1))
"Custom type id hash collision; please choose a different id")
(int short-hash-id))))
(comment (coerce-custom-type-id 77)
(coerce-custom-type-id :foo/bar))
(defmacro extend-freeze (defmacro extend-freeze
"Alpha - subject to change. "Extends Nippy to support freezing of a custom type (ideally concrete) with
Extends Nippy to support freezing of a custom type (ideally concrete) with given id of form:
id ∈[1, 128]: * Keyword - 2 byte overhead, resistent to id collisions.
* Byte ∈[1, 128] - no overhead, subject to id collisions.
(defrecord MyType [data]) (defrecord MyType [data])
(extend-freeze MyType 1 [x data-output] (extend-freeze MyType :foo/my-type [x data-output] ; Keyword id
(.writeUTF [data-output] (:data x)))
;; or
(extend-freeze MyType 1 [x data-output] ; Byte id
(.writeUTF [data-output] (:data x)))" (.writeUTF [data-output] (:data x)))"
[type custom-type-id [x out] & body] [type custom-type-id [x out] & body]
(assert (and (>= custom-type-id 1) (<= custom-type-id 128))) (assert-custom-type-id custom-type-id)
`(extend-type ~type `(extend-type ~type Freezable
Freezable
(~'freeze-to-out* [~x ~(with-meta out {:tag 'java.io.DataOutput})] (~'freeze-to-out* [~x ~(with-meta out {:tag 'java.io.DataOutput})]
(write-id ~out ~(int (- custom-type-id))) (if-not ~(keyword? custom-type-id)
;; Unprefixed [cust byte id][payload]:
(write-id ~out ~(coerce-custom-type-id custom-type-id))
;; Prefixed [const byte id][cust hash id][payload]:
(do (write-id ~out id-prefixed-custom)
(.writeShort ~out ~(coerce-custom-type-id custom-type-id))))
~@body))) ~@body)))
(defonce custom-readers (atom {})) ; {<custom-type-id> (fn [data-input]) ...} (defonce custom-readers (atom {})) ; {<hash-or-byte-id> (fn [data-input]) ...}
(defmacro extend-thaw (defmacro extend-thaw
"Alpha - subject to change. "Extends Nippy to support thawing of a custom type with given id:
Extends Nippy to support thawing of a custom type with id ∈[1, 128]: (extend-thaw :foo/my-type [data-input] ; Keyword id
(extend-thaw 1 [data-input] (->MyType (.readUTF data-input)))
;; or
(extend-thaw 1 [data-input] ; Byte id
(->MyType (.readUTF data-input)))" (->MyType (.readUTF data-input)))"
[custom-type-id [in] & body] [custom-type-id [in] & body]
(assert (and (>= custom-type-id 1) (<= custom-type-id 128))) (assert-custom-type-id custom-type-id)
`(swap! custom-readers assoc ~(int (- custom-type-id)) (when (contains? @custom-readers (coerce-custom-type-id custom-type-id))
(println (format "Warning: resetting Nippy thaw for custom type with id: %s"
custom-type-id)))
`(swap! custom-readers assoc
~(coerce-custom-type-id custom-type-id)
(fn [~(with-meta in {:tag 'java.io.DataInput})] (fn [~(with-meta in {:tag 'java.io.DataInput})]
~@body))) ~@body)))
@ -607,16 +707,14 @@
compress? (> ba-len 1024)] compress? (> ba-len 1024)]
(.writeBoolean out compress?) (.writeBoolean out compress?)
(if-not compress? (write-bytes out ba) (if-not compress? (write-bytes out ba)
(let [ba* (compression/compress compression/lzma2-compressor ba)] (let [ba* (compress lzma2-compressor ba)]
(write-bytes out ba*))))) (write-bytes out ba*)))))
(extend-thaw 128 [in] (extend-thaw 128 [in]
(let [compressed? (.readBoolean in) (let [compressed? (.readBoolean in)
ba (read-bytes in)] ba (read-bytes in)]
(thaw ba {:compressor compression/lzma2-compressor (thaw ba {:compressor (when compressed? lzma2-compressor)
:headerless-meta {:version 1 :encryptor nil})))
:compressed? compressed?
:encrypted? false}})))
(comment (comment
(->> (apply str (repeatedly 1000 rand)) (->> (apply str (repeatedly 1000 rand))
@ -703,8 +801,6 @@
;;;; Tools ;;;; Tools
(encore/defalias freezable? utils/freezable?)
(defn inspect-ba "Alpha - subject to change." (defn inspect-ba "Alpha - subject to change."
[ba & [thaw-opts]] [ba & [thaw-opts]]
(if-not (encore/bytes? ba) :not-ba (if-not (encore/bytes? ba) :not-ba
@ -719,7 +815,7 @@
[unwrapped-ba :no-header])] [unwrapped-ba :no-header])]
{:known-wrapper known-wrapper {:known-wrapper known-wrapper
:nippy2-header nippy-header ; Nippy v1.x didn't have a header :nippy-v2-header nippy-header ; Nippy v1.x didn't have a header
:thawable? (try (thaw unwrapped-ba thaw-opts) true :thawable? (try (thaw unwrapped-ba thaw-opts) true
(catch Exception _ false)) (catch Exception _ false))
:unwrapped-ba unwrapped-ba :unwrapped-ba unwrapped-ba
@ -738,23 +834,3 @@
(def thaw-from-stream! "DEPRECATED: Use `thaw-from-in!` instead." (def thaw-from-stream! "DEPRECATED: Use `thaw-from-in!` instead."
thaw-from-in!) thaw-from-in!)
(defn- assert-legacy-args [compressor password]
(when password
(throw (AssertionError. "Encryption not supported in legacy mode.")))
(when (and compressor (not= compressor snappy-compressor))
(throw (AssertionError. "Only Snappy compressor supported in legacy mode."))))
(defn freeze-to-bytes "DEPRECATED: Use `freeze` instead."
^bytes [x & {:keys [compress?]
:or {compress? true}}]
(freeze x {:skip-header? true
:compressor (when compress? snappy-compressor)
:password nil}))
(defn thaw-from-bytes "DEPRECATED: Use `thaw` instead."
[ba & {:keys [compressed?]
:or {compressed? true}}]
(thaw ba {:headerless-opts {:compressed? compressed?}
:compressor snappy-compressor
:password nil}))

111
src/taoensso/nippy/benchmarks.clj
View file

@ -3,8 +3,7 @@
(:require [clojure.tools.reader.edn :as edn] (:require [clojure.tools.reader.edn :as edn]
[clojure.data.fressian :as fressian] [clojure.data.fressian :as fressian]
[taoensso.encore :as encore] [taoensso.encore :as encore]
[taoensso.nippy :as nippy :refer (freeze thaw)] [taoensso.nippy :as nippy :refer (freeze thaw)]))
[taoensso.nippy.compression :as compression]))
(def data nippy/stress-data-benchable) (def data nippy/stress-data-benchable)
@ -43,14 +42,16 @@
(println {:default (bench1 #(freeze % {}) (println {:default (bench1 #(freeze % {})
#(thaw % {}))}) #(thaw % {}))})
(println {:fast (bench1 #(freeze % {:compressor nil}) (println {:fast (bench1 #(freeze % {:compressor nil
#(thaw % {:compressor nil}))}) :skip-header? true})
#(thaw % {:compressor nil
:encryptor nil}))})
(println {:encrypted (bench1 #(freeze % {:password [:cached "p"]}) (println {:encrypted (bench1 #(freeze % {:password [:cached "p"]})
#(thaw % {:password [:cached "p"]}))}) #(thaw % {:password [:cached "p"]}))})
(when lzma2? ; Slow as molasses (when lzma2? ; Slow as molasses
(println {:lzma2 (bench1 #(freeze % {:compressor compression/lzma2-compressor}) (println {:lzma2 (bench1 #(freeze % {:compressor nippy/lzma2-compressor})
#(thaw % {:compressor compression/lzma2-compressor}))})) #(thaw % {:compressor nippy/lzma2-compressor}))}))
(when fressian? (when fressian?
(println {:fressian (bench1 fressian-freeze fressian-thaw)}))) (println {:fressian (bench1 fressian-freeze fressian-thaw)})))
@ -65,83 +66,27 @@
;; (bench {:reader? true :lzma2? true :fressian? true :laps 1}) ;; (bench {:reader? true :lzma2? true :fressian? true :laps 1})
;; (bench {:laps 2}) ;; (bench {:laps 2})
;;; 2014 Apr 7 w/ some additional implementation tuning
{:default {:round 6533, :freeze 3618, :thaw 2915, :size 16139}}
{:fast {:round 6250, :freeze 3376, :thaw 2874, :size 16992}}
{:encrypted {:round 10583, :freeze 5581, :thaw 5002, :size 16164}}
;;; 2014 Apr 5 w/ headerless :fast, LZ4 replacing Snappy as default compressor
{:default {:round 7039, :freeze 3865, :thaw 3174, :size 16123}}
{:fast {:round 6394, :freeze 3379, :thaw 3015, :size 16992}}
{:encrypted {:round 11035, :freeze 5860, :thaw 5175, :size 16148}}
;;; 2014 Jan 22: with common-type size optimizations, enlarged stress-data ;;; 2014 Jan 22: with common-type size optimizations, enlarged stress-data
;; {:reader {:round 109544, :freeze 39523, :thaw 70021, :size 27681}} {:reader {:round 109544, :freeze 39523, :thaw 70021, :size 27681}}
;; {:default {:round 9234, :freeze 5128, :thaw 4106, :size 15989}} {:default {:round 9234, :freeze 5128, :thaw 4106, :size 15989}}
;; {:fast {:round 7402, :freeze 4021, :thaw 3381, :size 16957}} {:fast {:round 7402, :freeze 4021, :thaw 3381, :size 16957}}
;; {:encrypted {:round 12594, :freeze 6884, :thaw 5710, :size 16020}} {:encrypted {:round 12594, :freeze 6884, :thaw 5710, :size 16020}}
;; {:lzma2 {:round 66759, :freeze 44246, :thaw 22513, :size 11208}} {:lzma2 {:round 66759, :freeze 44246, :thaw 22513, :size 11208}}
;; {:fressian {:round 13052, :freeze 8694, :thaw 4358, :size 16942}} {:fressian {:round 13052, :freeze 8694, :thaw 4358, :size 16942}}
;;; 19 Oct 2013: Nippy v2.3.0, with lzma2 & (nb!) round=freeze+thaw ;;; 19 Oct 2013: Nippy v2.3.0, with lzma2 & (nb!) round=freeze+thaw
;; {:reader {:round 67798, :freeze 23202, :thaw 44596, :size 22971}} {:reader {:round 67798, :freeze 23202, :thaw 44596, :size 22971}}
;; {:default {:round 3632, :freeze 2349, :thaw 1283, :size 12369}} {:default {:round 3632, :freeze 2349, :thaw 1283, :size 12369}}
;; {:encrypted {:round 6970, :freeze 4073, :thaw 2897, :size 12388}} {:encrypted {:round 6970, :freeze 4073, :thaw 2897, :size 12388}}
;; {:fast {:round 3294, :freeze 2109, :thaw 1185, :size 13277}} {:fast {:round 3294, :freeze 2109, :thaw 1185, :size 13277}}
;; {:lzma2 {:round 44590, :freeze 29567, :thaw 15023, :size 9076}} {:lzma2 {:round 44590, :freeze 29567, :thaw 15023, :size 9076}})
;;; 11 Oct 2013: Nippy v2.2.0, with both ztellman mods
;; {:defaults {:round 4319, :freeze 2950, :thaw 1446, :data-size 12369}}
;; {:encrypted {:round 7675, :freeze 4479, :thaw 3160, :data-size 12388}}
;; {:fast {:round 3928, :freeze 2530, :thaw 1269, :data-size 13277}}
;; {:defaults-delta {:round 0.84 :freeze 0.79 :thaw 1.14}} ; vs 2.2.0
;;; 11 Oct 2013: Nippy v2.2.0, with first ztellman mod
;; {:defaults {:round 4059, :freeze 2578, :thaw 1351, :data-size 12342}}
;; {:encrypted {:round 7248, :freeze 4058, :thaw 3041, :data-size 12372}}
;; {:fast {:round 3430, :freeze 2085, :thaw 1229, :data-size 13277}}
;; {:defaults-delta {:round 0.79 :freeze 0.69 :thaw 1.07}} ; vs 2.2.0
;;; 11 Oct 2013: Nippy v2.2.0
;; {:defaults {:round 5135, :freeze 3711, :thaw 1266, :data-size 12393}}
;; {:encrypted {:round 8655, :freeze 5323, :thaw 3036, :data-size 12420}}
;; {:fast {:round 4670, :freeze 3282, :thaw 1294, :data-size 13277}}
;;; 7 Auguest 2013: Nippy v2.2.0-RC1
;; {:reader {:round 71582, :freeze 13656, :thaw 56730, :data-size 22964}}
;; {:defaults {:round 5619, :freeze 3710, :thaw 1783, :data-size 12368}}
;; {:encrypted {:round 9113, :freeze 5324, :thaw 3500, :data-size 12388}}
;; {:fast {:round 5130, :freeze 3286, :thaw 1667, :data-size 13325}}
;;; 17 June 2013: Clojure 1.5.1, JVM 7 Nippy 2.0.0-alpha6 w/fast io-streams
;; {:reader {:round 49819, :freeze 23601, :thaw 26247, :data-size 22966}}
;; {:defaults {:round 5670, :freeze 3536, :thaw 1919, :data-size 12396}}
;; {:encrypted {:round 9038, :freeze 5111, :thaw 3582, :data-size 12420}}
;; {:fast {:round 5182, :freeze 3177, :thaw 1820, :data-size 13342}}
;;; 16 June 2013: Clojure 1.5.1, Nippy 2.0.0-alpha6
;; {:reader {:freeze 23601, :thaw 26247, :round 49819, :data-size 22966}}
;; {:defaults {:freeze 3554, :thaw 2002, :round 5831, :data-size 12394}}
;; {:encrypted {:freeze 5117, :thaw 3600, :round 9006, :data-size 12420}}
;; {:fast {:freeze 3247, :thaw 1914, :round 5329, :data-size 13342}}
;;; 13 June 2013: Clojure 1.5.1, Nippy 2.0.0-alpha1
;; {:reader {:freeze 23124, :thaw 26469, :round 47674, :data-size 22923}}
;; {:defaults {:freeze 4007, :thaw 2520, :round 6038, :data-size 12387}}
;; {:encrypted {:freeze 5560, :thaw 3867, :round 9157, :data-size 12405}}
;; {:fast {:freeze 3429, :thaw 2078, :round 5577, :data-size 13237}}
;;; 11 June 2013: Clojure 1.5.1, Nippy 1.3.0-alpha1
;; {:reader {:freeze 17042, :thaw 31579, :round 48379, :data-size 22954}}
;; {:fast {:freeze 3078, :thaw 4684, :round 8117, :data-size 13274}}
;; {:defaults {:freeze 3810, :thaw 5295, :round 9052, :data-size 12394}}
;; {:encrypted {:freeze 5800, :thaw 6862, :round 12317, :data-size 12416}}
;;; Clojure 1.5.1, Nippy 1.2.1 (+ sorted-set, sorted-map)
;; (def data (dissoc data :sorted-set :sorted-map))
;; {:reader {:freeze 15037, :thaw 27885, :round 43945},
;; :nippy {:freeze 3194, :thaw 4734, :round 8380}}
;; {:reader-size 22975, :defaults-size 12400, :encrypted-size 12400}
;;; Clojure 1.4.0, Nippy 1.0.0 (+ tagged-uuid, tagged-date)
;; {:reader {:freeze 22595, :thaw 31148, :round 54059}
;; :nippy {:freeze 3324, :thaw 3725, :round 6918}}
;;; Clojure 1.3.0, Nippy 0.9.2
;; {:reader {:freeze 28505, :thaw 36451, :round 59545},
;; :nippy {:freeze 3751, :thaw 4184, :round 7769}}
(println (bench* (roundtrip data))) ; Snappy implementations
;; {:no-snappy [6163 6064 6042 6176] :JNI [6489 6446 6542 6412]
;; :native-array-copy [6569 6419 6414 6590]}
)

110
src/taoensso/nippy/compression.clj
View file

@ -1,19 +1,23 @@
(ns taoensso.nippy.compression (ns taoensso.nippy.compression
"Alpha - subject to change."
{:author "Peter Taoussanis"} {:author "Peter Taoussanis"}
(:require [taoensso.encore :as encore])
(:import [java.io ByteArrayInputStream ByteArrayOutputStream DataInputStream (:import [java.io ByteArrayInputStream ByteArrayOutputStream DataInputStream
DataOutputStream])) DataOutputStream]))
;;;; Interface ;;;; Interface
(defprotocol ICompressor (defprotocol ICompressor
(header-id [compressor])
(compress ^bytes [compressor ba]) (compress ^bytes [compressor ba])
(decompress ^bytes [compressor ba])) (decompress ^bytes [compressor ba]))
;;;; Default implementations ;;;; Default implementations
(def standard-header-ids "These'll support :auto thaw." #{:snappy :lzma2 :lz4})
(deftype SnappyCompressor [] (deftype SnappyCompressor []
ICompressor ICompressor
(header-id [_] :snappy)
(compress [_ ba] (org.iq80.snappy.Snappy/compress ba)) (compress [_ ba] (org.iq80.snappy.Snappy/compress ba))
(decompress [_ ba] (org.iq80.snappy.Snappy/uncompress ba 0 (alength ^bytes ba)))) (decompress [_ ba] (org.iq80.snappy.Snappy/uncompress ba 0 (alength ^bytes ba))))
@ -23,39 +27,113 @@
Write speed: very high. Write speed: very high.
Read speed: very high. Read speed: very high.
A good general-purpose compressor for Redis." A good general-purpose compressor."
(->SnappyCompressor)) (->SnappyCompressor))
(deftype LZMA2Compressor [compression-level] (deftype LZMA2Compressor [compression-level]
;; Compression level ∈ℕ[0,9] (low->high) with 6 LZMA2 default (we use 0) ;; Compression level ∈ℕ[0,9] (low->high) with 6 LZMA2 default (we use 0)
ICompressor ICompressor
(header-id [_] :lzma2)
(compress [_ ba] (compress [_ ba]
(let [ba-len (alength ^bytes ba) (let [baos (ByteArrayOutputStream.)
ba-os (ByteArrayOutputStream.) dos (DataOutputStream. baos)
;;
len-decomp (alength ^bytes ba)
;; Prefix with uncompressed length: ;; Prefix with uncompressed length:
_ (.writeInt (DataOutputStream. ba-os) ba-len) _ (.writeInt dos len-decomp)
xzs (org.tukaani.xz.XZOutputStream. ba-os xzs (org.tukaani.xz.XZOutputStream. baos
(org.tukaani.xz.LZMA2Options. compression-level))] (org.tukaani.xz.LZMA2Options. compression-level))]
(.write xzs ^bytes ba) (.write xzs ^bytes ba)
(.close xzs) (.close xzs)
(.toByteArray ba-os))) (.toByteArray baos)))
(decompress [_ ba] (decompress [_ ba]
(let [ba-is (ByteArrayInputStream. ba) (let [bais (ByteArrayInputStream. ba)
ba-len (.readInt (DataInputStream. ba-is)) dis (DataInputStream. bais)
ba (byte-array ba-len) ;;
xzs (org.tukaani.xz.XZInputStream. ba-is)] len-decomp (.readInt dis)
(.read xzs ba 0 ba-len) ba (byte-array len-decomp)
xzs (org.tukaani.xz.XZInputStream. bais)]
(.read xzs ba 0 len-decomp)
(when (not= -1 (.read xzs)) ; Good practice as extra safety measure (when (not= -1 (.read xzs)) ; Good practice as extra safety measure
(throw (Exception. "LZMA2 Decompress failed: corrupt data?"))) (throw (ex-info "LZMA2 Decompress failed: corrupt data?" {:ba ba})))
ba))) ba)))
(def lzma2-compressor (def lzma2-compressor
"Alpha - subject to change. "Default org.tukaani.xz.LZMA2 compressor:
Default org.tukaani.xz.LZMA2 compressor:
Ratio: high. Ratio: high.
Write speed: _very_ slow (also currently single-threaded). Write speed: _very_ slow (also currently single-threaded).
Read speed: slow. Read speed: slow.
A specialized compressor for large, low-write data." A specialized compressor for large, low-write data in space-sensitive
environments."
(->LZMA2Compressor 0)) (->LZMA2Compressor 0))
(deftype LZ4Compressor [^net.jpountz.lz4.LZ4Compressor compressor
^net.jpountz.lz4.LZ4Decompressor decompressor]
ICompressor
(header-id [_] :lz4)
(compress [_ ba]
(let [len-decomp (alength ^bytes ba)
max-len-comp (.maxCompressedLength compressor len-decomp)
ba-comp* (byte-array max-len-comp) ; Over-sized
len-comp (.compress compressor ba 0 len-decomp ba-comp* 0 max-len-comp)
;;
baos (ByteArrayOutputStream. (+ len-comp 4))
dos (DataOutputStream. baos)]
(.writeInt dos len-decomp) ; Prefix with uncompressed length
(.write dos ba-comp* 0 len-comp)
(.toByteArray baos)))
(decompress [_ ba]
(let [bais (ByteArrayInputStream. ba)
dis (DataInputStream. bais)
;;
len-decomp (.readInt dis)
len-comp (- (alength ^bytes ba) 4)
;; ba-comp (byte-array len-comp)
;; _ (.readFully dis ba-comp 0 len-comp)
ba-decomp (byte-array len-decomp)
_ (.decompress decompressor ba 4 ba-decomp 0 len-decomp)]
ba-decomp)))
(def ^:private ^net.jpountz.lz4.LZ4Factory lz4-factory
(net.jpountz.lz4.LZ4Factory/fastestInstance))
(def lz4-compressor
"Default net.jpountz.lz4 compressor:
Ratio: low.
Write speed: very high.
Read speed: very high.
A good general-purpose compressor, competitive with Snappy.
Thanks to Max Penet (@mpenet) for our first implementation,
Ref. https://github.com/mpenet/nippy-lz4"
(->LZ4Compressor (.fastCompressor lz4-factory)
(.fastDecompressor lz4-factory)))
(def lz4hc-compressor
"Like `lz4-compressor` but trades some write speed for ratio."
(->LZ4Compressor (.highCompressor lz4-factory)
(.fastDecompressor lz4-factory)))
(comment
(def ba-bench (.getBytes (apply str (repeatedly 1000 rand)) "UTF-8"))
(defn bench1 [compressor]
{:time (encore/bench 10000 {:nlaps-warmup 10000}
(->> ba-bench (compress compressor) (decompress compressor)))
:ratio (encore/round2 (/ (count (compress compressor ba-bench))
(count ba-bench)))})
(println
{:snappy (bench1 snappy-compressor)
;:lzma2 (bench1 lzma2-compressor) ; Slow!
:lz4 (bench1 lz4-compressor)
:lz4hc (bench1 lz4hc-compressor)})
;;; 2014 April 7
{:snappy {:time 2251, :ratio 0.852},
:lzma2 {:time 46684 :ratio 0.494}
:lz4 {:time 1184, :ratio 0.819},
:lz4hc {:time 5422, :ratio 0.761}})

47
src/taoensso/nippy/encryption.clj
View file

@ -1,13 +1,15 @@
(ns taoensso.nippy.encryption (ns taoensso.nippy.encryption
"Alpha - subject to change. "Simple no-nonsense crypto with reasonable defaults. Because your Clojure data
Simple no-nonsense crypto with reasonable defaults. Because your Clojure data
deserves some privacy." deserves some privacy."
{:author "Peter Taoussanis"} {:author "Peter Taoussanis"}
(:require [taoensso.encore :as encore])) (:require [taoensso.encore :as encore]))
;;;; Interface ;;;; Interface
(def standard-header-ids "These'll support :auto thaw." #{:aes128-sha512})
(defprotocol IEncryptor (defprotocol IEncryptor
(header-id [encryptor])
(encrypt ^bytes [encryptor pwd ba]) (encrypt ^bytes [encryptor pwd ba])
(decrypt ^bytes [encryptor pwd ba])) (decrypt ^bytes [encryptor pwd ba]))
@ -25,7 +27,7 @@
(defn- rand-bytes [size] (let [seed (byte-array size)] (.nextBytes prng seed) seed)) (defn- rand-bytes [size] (let [seed (byte-array size)] (.nextBytes prng seed) seed))
;;;; Default keygen ;;;; Default key-gen
(defn- sha512-key (defn- sha512-key
"SHA512-based key generator. Good JVM availability without extra dependencies "SHA512-based key generator. Good JVM availability without extra dependencies
@ -38,6 +40,7 @@
(recur (.digest sha512-md ba) (dec n)) (recur (.digest sha512-md ba) (dec n))
(-> ba (java.util.Arrays/copyOf aes128-block-size) (-> ba (java.util.Arrays/copyOf aes128-block-size)
(javax.crypto.spec.SecretKeySpec. "AES"))))) (javax.crypto.spec.SecretKeySpec. "AES")))))
(comment (comment
(time (sha512-key nil "hi" 1)) ; ~40ms per hash (fast) (time (sha512-key nil "hi" 1)) ; ~40ms per hash (fast)
(time (sha512-key nil "hi" 5)) ; ~180ms (default) (time (sha512-key nil "hi" 5)) ; ~180ms (default)
@ -47,54 +50,52 @@
;;;; Default implementations ;;;; Default implementations
(defn- destructure-typed-pwd (defn- destructure-typed-pwd [typed-password]
[typed-password] (let [throw-ex
(letfn [(throw-ex [] (fn [] (throw (ex-info
(throw (AssertionError.
(str "Expected password form: " (str "Expected password form: "
"[<#{:salted :cached}> <password-string>].\n " "[<#{:salted :cached}> <password-string>].\n "
"See `default-aes128-encryptor` docstring for details!"))))] "See `default-aes128-encryptor` docstring for details!")
(if-not (vector? typed-password) {:typed-password typed-password})))]
(throw-ex) (if-not (vector? typed-password) (throw-ex)
(let [[type password] typed-password] (let [[type password] typed-password]
(if-not (#{:salted :cached} type) (if-not (#{:salted :cached} type) (throw-ex)
(throw-ex)
[type password]))))) [type password])))))
(comment (destructure-typed-pwd [:salted "foo"])) (comment (destructure-typed-pwd [:salted "foo"]))
(defrecord AES128Encryptor [key-cache] (defrecord AES128Encryptor [key-gen key-cache]
IEncryptor IEncryptor
(encrypt [this typed-pwd data-ba] (header-id [_] (if (= key-gen sha512-key) :aes128-sha512 :aes128-other))
(encrypt [_ typed-pwd data-ba]
(let [[type pwd] (destructure-typed-pwd typed-pwd) (let [[type pwd] (destructure-typed-pwd typed-pwd)
salt? (= type :salted) salt? (identical? type :salted)
iv-ba (rand-bytes aes128-block-size) iv-ba (rand-bytes aes128-block-size)
salt-ba (when salt? (rand-bytes salt-size)) salt-ba (when salt? (rand-bytes salt-size))
prefix-ba (if-not salt? iv-ba (encore/ba-concat iv-ba salt-ba)) prefix-ba (if-not salt? iv-ba (encore/ba-concat iv-ba salt-ba))
key (encore/memoized (when-not salt? (:key-cache this)) key (encore/memoized (when-not salt? key-cache)
sha512-key salt-ba pwd) key-gen salt-ba pwd)
iv (javax.crypto.spec.IvParameterSpec. iv-ba)] iv (javax.crypto.spec.IvParameterSpec. iv-ba)]
(.init aes128-cipher javax.crypto.Cipher/ENCRYPT_MODE (.init aes128-cipher javax.crypto.Cipher/ENCRYPT_MODE
^javax.crypto.spec.SecretKeySpec key iv) ^javax.crypto.spec.SecretKeySpec key iv)
(encore/ba-concat prefix-ba (.doFinal aes128-cipher data-ba)))) (encore/ba-concat prefix-ba (.doFinal aes128-cipher data-ba))))
(decrypt [this typed-pwd ba] (decrypt [_ typed-pwd ba]
(let [[type pwd] (destructure-typed-pwd typed-pwd) (let [[type pwd] (destructure-typed-pwd typed-pwd)
salt? (= type :salted) salt? (= type :salted)
prefix-size (+ aes128-block-size (if salt? salt-size 0)) prefix-size (+ aes128-block-size (if salt? salt-size 0))
[prefix-ba data-ba] (encore/ba-split ba prefix-size) [prefix-ba data-ba] (encore/ba-split ba prefix-size)
[iv-ba salt-ba] (if-not salt? [prefix-ba nil] [iv-ba salt-ba] (if-not salt? [prefix-ba nil]
(encore/ba-split prefix-ba aes128-block-size)) (encore/ba-split prefix-ba aes128-block-size))
key (encore/memoized (when-not salt? (:key-cache this)) key (encore/memoized (when-not salt? key-cache)
sha512-key salt-ba pwd) key-gen salt-ba pwd)
iv (javax.crypto.spec.IvParameterSpec. iv-ba)] iv (javax.crypto.spec.IvParameterSpec. iv-ba)]
(.init aes128-cipher javax.crypto.Cipher/DECRYPT_MODE (.init aes128-cipher javax.crypto.Cipher/DECRYPT_MODE
^javax.crypto.spec.SecretKeySpec key iv) ^javax.crypto.spec.SecretKeySpec key iv)
(.doFinal aes128-cipher data-ba)))) (.doFinal aes128-cipher data-ba))))
(def aes128-encryptor (def aes128-encryptor
"Alpha - subject to change. "Default 128bit AES encryptor with multi-round SHA-512 key-gen.
Default 128bit AES encryptor with multi-round SHA-512 keygen.
Password form [:salted \"my-password\"] Password form [:salted \"my-password\"]
--------------------------------------- ---------------------------------------
@ -128,7 +129,7 @@
Faster than `aes128-salted`, and harder to attack any particular key - but Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised." increased danger if a key is somehow compromised."
(->AES128Encryptor (atom {}))) (->AES128Encryptor sha512-key (atom {})))
;;;; Default implementation ;;;; Default implementation

23
src/taoensso/nippy/tools.clj
View file

@ -1,7 +1,6 @@
(ns taoensso.nippy.tools (ns taoensso.nippy.tools
"Alpha - subject to change. "Utilities for third-party tools that want to add fully-user-configurable
Utilities for third-party tools that want to add fully-user-configurable Nippy Nippy support. Used by Carmine and Faraday."
support. Used by Carmine and Faraday."
{:author "Peter Taoussanis"} {:author "Peter Taoussanis"}
(:require [taoensso.nippy :as nippy])) (:require [taoensso.nippy :as nippy]))
@ -23,26 +22,12 @@
(comment (freeze (wrap-for-freezing "wrapped")) (comment (freeze (wrap-for-freezing "wrapped"))
(freeze "unwrapped")) (freeze "unwrapped"))
(defrecord EncryptedFrozen [ba])
(defn encrypted-frozen? [x] (instance? EncryptedFrozen x))
(def ^:dynamic *thaw-opts* nil) (def ^:dynamic *thaw-opts* nil)
(defmacro with-thaw-opts (defmacro with-thaw-opts
"Evaluates body using given options for any automatic deserialization in "Evaluates body using given options for any automatic deserialization in
context." context."
[opts & body] `(binding [*thaw-opts* ~opts] ~@body)) [opts & body] `(binding [*thaw-opts* ~opts] ~@body))
(defn thaw (defn thaw "Like `nippy/thaw` but takes options from *thaw-opts* binding."
"Like `nippy/thaw` but takes options from *thaw-opts* binding, and wraps
encrypted bytes for easy identification when no password has been provided
for decryption."
[ba & [{:keys [default-opts]}]] [ba & [{:keys [default-opts]}]]
(let [result (nippy/thaw ba (merge default-opts *thaw-opts* (nippy/thaw ba (merge default-opts *thaw-opts*)))
{:taoensso.nippy/tools-thaw? true}))]
(if (= result :taoensso.nippy/need-password)
(EncryptedFrozen. ba)
result)))
(comment (thaw (nippy/freeze "c" {:password [:cached "p"]}))
(with-thaw-opts {:password [:cached "p"]}
(thaw (nippy/freeze "c" {:password [:cached "p"]}))))

18
src/taoensso/nippy/utils.clj
View file

@ -1,7 +1,8 @@
(ns taoensso.nippy.utils (ns taoensso.nippy.utils
{:author "Peter Taoussanis"} {:author "Peter Taoussanis"}
(:require [clojure.string :as str] (:require [clojure.string :as str]
[clojure.tools.reader.edn :as edn]) [clojure.tools.reader.edn :as edn]
[taoensso.encore :as encore])
(:import [java.io ByteArrayInputStream ByteArrayOutputStream Serializable (:import [java.io ByteArrayInputStream ByteArrayOutputStream Serializable
ObjectOutputStream ObjectInputStream])) ObjectOutputStream ObjectInputStream]))
@ -17,12 +18,7 @@
cacheable? (not (re-find #"__\d+" (str t))) ; gensym form cacheable? (not (re-find #"__\d+" (str t))) ; gensym form
test (fn [] (try (f-test x) (catch Exception _ false)))] test (fn [] (try (f-test x) (catch Exception _ false)))]
(if-not cacheable? (test) (if-not cacheable? (test)
(if-let [dv (@cache t)] @dv @(encore/swap-val! cache t #(if % % (delay (test)))))))))
(locking cache ; For thread racing
(if-let [dv (@cache t)] @dv ; Retry after lock acquisition
(let [dv (delay (test))]
(swap! cache assoc t dv)
@dv)))))))))
(def serializable? (def serializable?
(memoize-type-test (memoize-type-test
@ -46,10 +42,10 @@
(readable? "Hello world") (readable? "Hello world")
(readable? (fn [])) (readable? (fn []))
(time (dotimes [_ 10000] (serializable? "Hello world"))) (time (dotimes [_ 10000] (serializable? "Hello world"))) ; Cacheable
(time (dotimes [_ 10000] (serializable? (fn [])))) (time (dotimes [_ 10000] (serializable? (fn [])))) ; Uncacheable
(time (dotimes [_ 10000] (readable? "Hello world"))) (time (dotimes [_ 10000] (readable? "Hello world"))) ; Cacheable
(time (dotimes [_ 10000] (readable? (fn []))))) (time (dotimes [_ 10000] (readable? (fn []))))) ; Uncacheable
;;;; ;;;;

32
test/taoensso/nippy/tests/main.clj
View file

@ -4,7 +4,6 @@
[clojure.test.check.properties :as check-props] [clojure.test.check.properties :as check-props]
[expectations :as test :refer :all] [expectations :as test :refer :all]
[taoensso.nippy :as nippy :refer (freeze thaw)] [taoensso.nippy :as nippy :refer (freeze thaw)]
[taoensso.nippy.compression :as compression]
[taoensso.nippy.benchmarks :as benchmarks])) [taoensso.nippy.benchmarks :as benchmarks]))
(comment (test/run-tests '[taoensso.nippy.tests.main])) (comment (test/run-tests '[taoensso.nippy.tests.main]))
@ -16,25 +15,30 @@
;;;; Core ;;;; Core
(expect test-data ((comp thaw freeze) test-data)) (expect test-data ((comp thaw freeze) test-data))
(expect test-data ((comp thaw #(freeze % {:legacy-mode true})) test-data)) (expect test-data ((comp #(thaw % {})
#(freeze % {:legacy-mode true}))
test-data))
(expect test-data ((comp #(thaw % {:password [:salted "p"]}) (expect test-data ((comp #(thaw % {:password [:salted "p"]})
#(freeze % {:password [:salted "p"]})) #(freeze % {:password [:salted "p"]}))
test-data)) test-data))
(expect test-data ((comp #(thaw % {:compressor compression/lzma2-compressor}) (expect test-data ((comp #(thaw % {:compressor nippy/lzma2-compressor})
#(freeze % {:compressor compression/lzma2-compressor})) #(freeze % {:compressor nippy/lzma2-compressor}))
test-data)) test-data))
(expect test-data ((comp #(thaw % {:compressor compression/lzma2-compressor (expect test-data ((comp #(thaw % {:compressor nippy/lzma2-compressor
:password [:salted "p"]}) :password [:salted "p"]})
#(freeze % {:compressor compression/lzma2-compressor #(freeze % {:compressor nippy/lzma2-compressor
:password [:salted "p"]})) :password [:salted "p"]}))
test-data)) test-data))
(expect test-data ((comp #(thaw % {:compressor nippy/lz4-compressor})
#(freeze % {:compressor nippy/lz4hc-compressor}))
test-data))
(expect ; Try roundtrip anything that simple-check can dream up (expect ; Try roundtrip anything that simple-check can dream up
(:result (check/quick-check 80 ; Time is n-non-linear (:result (check/quick-check 80 ; Time is n-non-linear
(check-props/for-all [val check-gen/any] (check-props/for-all [val check-gen/any]
(= val (nippy/thaw (nippy/freeze val))))))) (= val (nippy/thaw (nippy/freeze val)))))))
(expect AssertionError (thaw (freeze test-data {:password "malformed"}))) (expect Exception (thaw (freeze test-data {:password "malformed"})))
(expect Exception (thaw (freeze test-data {:password [:salted "p"]}))) (expect Exception (thaw (freeze test-data {:password [:salted "p"]})))
(expect Exception (thaw (freeze test-data {:password [:salted "p"]}) (expect Exception (thaw (freeze test-data {:password [:salted "p"]})
{:compressor nil})) {:compressor nil}))
@ -53,16 +57,22 @@
;;; Extend to custom Type ;;; Extend to custom Type
(defrecord MyType [data]) (defrecord MyType [data])
(nippy/extend-freeze MyType 1 [x s] (.writeUTF s (:data x))) (expect Exception (do (nippy/extend-freeze MyType 1 [x s] (.writeUTF s (:data x)))
(expect Exception (thaw (freeze (->MyType "val")))) (thaw (freeze (->MyType "val")))))
(expect (do (nippy/extend-thaw 1 [s] (->MyType (.readUTF s))) (expect (do (nippy/extend-thaw 1 [s] (->MyType (.readUTF s)))
(let [type (->MyType "val")] (= type (thaw (freeze type)))))) (let [type (->MyType "val")] (= type (thaw (freeze type))))))
;;; Extend to custom Record ;;; Extend to custom Record
(defrecord MyRec [data]) (defrecord MyRec [data])
(expect (do (nippy/extend-freeze MyRec 2 [x s] (.writeUTF s (str "fast-" (:data x)))) (expect (do (nippy/extend-freeze MyRec 2 [x s] (.writeUTF s (str "foo-" (:data x))))
(nippy/extend-thaw 2 [s] (->MyRec (.readUTF s))) (nippy/extend-thaw 2 [s] (->MyRec (.readUTF s)))
(= (->MyRec "fast-val") (thaw (freeze (->MyRec "val")))))) (= (->MyRec "foo-val") (thaw (freeze (->MyRec "val"))))))
;;; Keyword (prefixed) extensions
(expect
(do (nippy/extend-freeze MyType :nippy-tests/MyType [x s] (.writeUTF s (:data x)))
(nippy/extend-thaw :nippy-tests/MyType [s] (->MyType (.readUTF s)))
(let [type (->MyType "val")] (= type (thaw (freeze type))))))
;;;; Stable binary representation of vals ; EXPERIMENTAL ;;;; Stable binary representation of vals ; EXPERIMENTAL