luciano/nippy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'v2.0.0'

Browse source
This commit is contained in:
Peter Taoussanis 2013-06-13 16:42:14 +07:00
commit d44dc44399
10 changed files with 439 additions and 312 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

29
README.md
View file

@ -2,14 +2,16 @@ Current [semantic](http://semver.org/) version:
```clojure ```clojure
[com.taoensso/nippy "1.2.1"] ; Stable [com.taoensso/nippy "1.2.1"] ; Stable
[com.taoensso/nippy "1.3.0-alpha3"] ; Development (adds crypto support!) [com.taoensso/nippy "2.0.0-alpha1"] ; Development (see notes below)
``` ```
2.x adds pluggable compression, crypto support (also pluggable), an improved API (including much better error messages), and hugely improved performance. It **is backwards compatible**, but please note that the `freeze-to-bytes`/`thaw-from-bytes` API has been **deprecated** in favor of `freeze`/`thaw`.
# Nippy, a Clojure serialization library # Nippy, a Clojure serialization library
Clojure's [rich data types](http://clojure.org/datatypes) are *awesome*. And its [reader](http://clojure.org/reader) allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database). Clojure's [rich data types](http://clojure.org/datatypes) are *awesome*. And its [reader](http://clojure.org/reader) allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).
Nippy is an attempt to provide a drop-in, high-performance alternative to the reader. It's a fork of [Deep-Freeze](https://github.com/halgari/deep-freeze) and is used as the [Carmine Redis client](https://github.com/ptaoussanis/carmine) serializer. Nippy is an attempt to provide a reliable, high-performance **drop-in alternative to the reader**. It's used, among others, as the [Carmine Redis client](https://github.com/ptaoussanis/carmine) and [Faraday DynamoDB client]https://github.com/ptaoussanis/faraday) serializer.
## What's in the box™? ## What's in the box™?
* Small, uncomplicated **all-Clojure** library. * Small, uncomplicated **all-Clojure** library.
@ -17,8 +19,8 @@ Nippy is an attempt to provide a drop-in, high-performance alternative to the re
* Comprehesive, extensible **support for all major data types**. * Comprehesive, extensible **support for all major data types**.
* **Reader-fallback** for difficult/future types (including Clojure 1.4+ tagged literals). * **Reader-fallback** for difficult/future types (including Clojure 1.4+ tagged literals).
* **Full test coverage** for every supported type. * **Full test coverage** for every supported type.
* [Snappy](http://code.google.com/p/snappy/) **integrated de/compression** for efficient storage and network transfer. * Fully pluggable **compression**, including built-in high-performance [Snappy](http://code.google.com/p/snappy/) compressor.
* Enable **high-strength encryption** with a single `:password [:salted "my-password"]` option. (1.3.0+) * Fully pluggable **encryption**, including built-in high-strength AES128 enabled with a single `:password [:salted "my-password"]` option. (2.0.0+)
## Getting started ## Getting started
@ -76,24 +78,21 @@ nippy/stress-data
:bigdec (bigdec 3.1415926535897932384626433832795) :bigdec (bigdec 3.1415926535897932384626433832795)
:ratio 22/7 :ratio 22/7
:tagged-uuid (java.util.UUID/randomUUID)
;; Clojure 1.4+ :tagged-date (java.util.Date.)}
;; :tagged-uuid (java.util.UUID/randomUUID)
;; :tagged-date (java.util.Date.)
}
``` ```
Serialize it: Serialize it:
```clojure ```clojure
(def frozen-stress-data (nippy/freeze-to-bytes nippy/stress-data)) (def frozen-stress-data (nippy/freeze nippy/stress-data))
=> #<byte[] [B@3253bcf3> => #<byte[] [B@3253bcf3>
``` ```
Deserialize it: Deserialize it:
```clojure ```clojure
(nippy/thaw-from-bytes frozen-stress-data) (nippy/thaw frozen-stress-data)
=> {:bytes (byte-array [(byte 1) (byte 2) (byte 3)]) => {:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
:nil nil :nil nil
:boolean true :boolean true
@ -104,14 +103,14 @@ Couldn't be simpler!
### Encryption (currently in **ALPHA**) ### Encryption (currently in **ALPHA**)
As of 1.3.0, Nippy also gives you **dead simple data encryption**. Add a single option to your usual freeze/thaw calls like so: As of 2.0.0, Nippy also gives you **dead simple data encryption**. Add a single option to your usual freeze/thaw calls like so:
```clojure ```clojure
(nippy/freeze-to-bytes nippy/stress-data :password [:salted "my-password"]) ; Encrypt (nippy/freeze nippy/stress-data {:password [:salted "my-password"]}) ; Encrypt
(nippy/thaw-from-bytes <encrypted-data> :password [:salted "my-password"]) ; Decrypt (nippy/thaw <encrypted-data> {:password [:salted "my-password"]}) ; Decrypt
``` ```
There's two forms of encryption on offer: `:salted` and `:cached`. Each of these makes carefully-chosen trade-offs and is suited to one of two common use cases. See the `aes128-salted` and `aes128-cached` [docstrings](http://ptaoussanis.github.io/nippy/taoensso.nippy.crypto.html) for a detailed explanation of why/when you'd want one or the other. There's two default forms of encryption on offer: `:salted` and `:cached`. Each of these makes carefully-chosen trade-offs and is suited to one of two common use cases. See the `default-aes128-encryptor` [docstring](http://ptaoussanis.github.io/nippy/taoensso.nippy.encryption.html) for a detailed explanation of why/when you'd want one or the other.
## Performance ## Performance

BIN
benchmarks/chart.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 18 KiB

21
project.clj
View file

@ -1,16 +1,23 @@
(defproject com.taoensso/nippy "1.3.0-alpha3" (defproject com.taoensso/nippy "2.0.0-alpha1"
:description "Clojure serialization library" :description "Clojure serialization library"
:url "https://github.com/ptaoussanis/nippy" :url "https://github.com/ptaoussanis/nippy"
:license {:name "Eclipse Public License" :license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"} :url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.3.0"] :dependencies [[org.clojure/clojure "1.4.0"]
[expectations "1.4.43"]
[org.iq80.snappy/snappy "0.3"]] [org.iq80.snappy/snappy "0.3"]]
:profiles {:1.3 {:dependencies [[org.clojure/clojure "1.3.0"]]} :profiles {:1.4 {:dependencies [[org.clojure/clojure "1.4.0"]]}
:1.4 {:dependencies [[org.clojure/clojure "1.4.0"]]}
:1.5 {:dependencies [[org.clojure/clojure "1.5.1"]]} :1.5 {:dependencies [[org.clojure/clojure "1.5.1"]]}
:dev {:dependencies []} :dev {:dependencies []}
:test {:dependencies [[org.xerial.snappy/snappy-java "1.0.5-M3"]]}} :test {:dependencies [[org.xerial.snappy/snappy-java "1.0.5-M3"]]}
:aliases {"test-all" ["with-profile" "test,1.3:test,1.4:test,1.5" "test"]} :bench {:dependencies []
:plugins [[codox "0.6.4"]] :jvm-opts ["-server" "-XX:+UseCompressedOops"]}}
:aliases {"test-all" ["with-profile" "test,1.4:test,1.5" "expectations"]
"test-auto" ["with-profile" "test" "autoexpect"]
"start-dev" ["with-profile" "dev,test,bench" "repl" ":headless"]
"start-bench" ["trampoline" "start-dev"]}
:plugins [[lein-expectations "0.0.7"]
[lein-autoexpect "0.2.5"]
[codox "0.6.4"]]
:min-lein-version "2.0.0" :min-lein-version "2.0.0"
:warn-on-reflection true) :warn-on-reflection true)

321
src/taoensso/nippy.clj
View file

@ -1,16 +1,30 @@
(ns taoensso.nippy (ns taoensso.nippy
"Simple, high-performance Clojure serialization library. Adapted from "Simple, high-performance Clojure serialization library. Originally adapted
Deep-Freeze." from Deep-Freeze."
{:author "Peter Taoussanis"} {:author "Peter Taoussanis"}
(:require [taoensso.nippy.utils :as utils] (:require [taoensso.nippy
[taoensso.nippy.crypto :as crypto]) (utils :as utils)
(compression :as compression)
(encryption :as encryption)])
(:import [java.io DataInputStream DataOutputStream ByteArrayOutputStream (:import [java.io DataInputStream DataOutputStream ByteArrayOutputStream
ByteArrayInputStream] ByteArrayInputStream]
[clojure.lang Keyword BigInt Ratio PersistentQueue PersistentTreeMap [clojure.lang Keyword BigInt Ratio PersistentQueue PersistentTreeMap
PersistentTreeSet IPersistentList IPersistentVector IPersistentMap PersistentTreeSet IPersistentList IPersistentVector IPersistentMap
IPersistentSet IPersistentCollection])) IPersistentSet IPersistentCollection]))
;;;; Define type IDs ;; TODO Allow ba or wrapped-ba input?
;; TODO Provide ToFreeze, Frozen, Encrypted, etc. tooling helpers
;;;; Header IDs
;; Nippy 2.x+ prefixes frozen data with a 5-byte header:
(def ^:const id-nippy-magic-prefix (byte 17))
(def ^:const id-nippy-header-ver (byte 0))
;; * Compressor id (0 if no compressor)
;; * Encryptor id (0 if no encryptor)
(def ^:const id-nippy-reserved (byte 0))
;;;; Data type IDs
;; 1 ;; 1
(def ^:const id-bytes (int 2)) (def ^:const id-bytes (int 2))
@ -53,74 +67,80 @@
;;;; Shared low-level stream stuff ;;;; Shared low-level stream stuff
(defn- write-id! [^DataOutputStream stream ^Integer id] (.writeByte stream id)) (defn- write-id [^DataOutputStream stream ^Integer id] (.writeByte stream id))
(defn- write-bytes! (defn- write-bytes
"Writes arbitrary byte data, preceded by its length." "Writes arbitrary byte data, preceded by its length."
[^DataOutputStream stream ^bytes ba] [^DataOutputStream stream ^bytes ba]
(let [size (alength ba)] (let [size (alength ba)]
(.writeInt stream size) ; Encode size of byte array (.writeInt stream size) ; Encode size of byte array
(.write stream ba 0 size))) (.write stream ba 0 size)))
(defn- write-biginteger! (defn- write-biginteger
"Wrapper around `write-bytes!` for common case of writing a BigInteger." "Wrapper around `write-bytes` for common case of writing a BigInteger."
[^DataOutputStream stream ^BigInteger x] [^DataOutputStream stream ^BigInteger x]
(write-bytes! stream (.toByteArray x))) (write-bytes stream (.toByteArray x)))
(defn- read-bytes! (defn- read-bytes
"Reads arbitrary byte data, preceded by its length." "Reads arbitrary byte data, preceded by its length."
^bytes [^DataInputStream stream] ^bytes [^DataInputStream stream]
(let [size (.readInt stream) (let [size (.readInt stream)
ba (byte-array size)] ba (byte-array size)]
(.read stream ba 0 size) ba)) (.read stream ba 0 size) ba))
(defn- read-biginteger! (defn- read-biginteger
"Wrapper around `read-bytes!` for common case of reading a BigInteger. "Wrapper around `read-bytes` for common case of reading a BigInteger.
Note that as of Clojure 1.3, java.math.BigInteger ≠ clojure.lang.BigInt." Note that as of Clojure 1.3, java.math.BigInteger ≠ clojure.lang.BigInt."
^BigInteger [^DataInputStream stream] ^BigInteger [^DataInputStream stream]
(BigInteger. (read-bytes! stream))) (BigInteger. (read-bytes stream)))
;;;; Freezing ;;;; Freezing
(defprotocol Freezable (freeze [this stream])) (defprotocol Freezable (freeze-to-stream* [this stream]))
(defmacro freezer (defn- freeze-to-stream
"Like `freeze-to-stream*` but with metadata support."
[x ^DataOutputStream s]
(if-let [m (meta x)]
(do (write-id s id-meta)
(freeze-to-stream m s)))
(freeze-to-stream* x s))
(defmacro ^:private freezer
"Helper to extend Freezable protocol." "Helper to extend Freezable protocol."
[type id & body] [type id & body]
`(extend-type ~type `(extend-type ~type
~'Freezable ~'Freezable
(~'freeze [~'x ~(with-meta 's {:tag 'DataOutputStream})] (~'freeze-to-stream* [~'x ~(with-meta 's {:tag 'DataOutputStream})]
(write-id! ~'s ~id) (write-id ~'s ~id)
~@body))) ~@body)))
(defmacro coll-freezer (defmacro ^:private coll-freezer
"Extends Freezable to simple collection types." "Extends Freezable to simple collection types."
[type id & body] [type id & body]
`(freezer ~type ~id `(freezer ~type ~id
(.writeInt ~'s (count ~'x)) ; Encode collection length (.writeInt ~'s (count ~'x))
(doseq [i# ~'x] (freeze-to-stream!* ~'s i#)))) (doseq [i# ~'x] (freeze-to-stream i# ~'s))))
(defmacro kv-freezer (defmacro ^:private kv-freezer
"Extends Freezable to key-value collection types." "Extends Freezable to key-value collection types."
[type id & body] [type id & body]
`(freezer ~type ~id `(freezer ~type ~id
(.writeInt ~'s (* 2 (count ~'x))) ; Encode num kvs (.writeInt ~'s (* 2 (count ~'x)))
(doseq [[k# v#] ~'x] (doseq [[k# v#] ~'x]
(freeze-to-stream!* ~'s k#) (freeze-to-stream k# ~'s)
(freeze-to-stream!* ~'s v#)))) (freeze-to-stream v# ~'s))))
(freezer (Class/forName "[B") id-bytes (write-bytes! s x)) (freezer (Class/forName "[B") id-bytes (write-bytes s x))
(freezer nil id-nil) (freezer nil id-nil)
(freezer Boolean id-boolean (.writeBoolean s x)) (freezer Boolean id-boolean (.writeBoolean s x))
(freezer Character id-char (.writeChar s (int x))) (freezer Character id-char (.writeChar s (int x)))
(freezer String id-string (write-bytes! s (.getBytes x "UTF-8"))) (freezer String id-string (write-bytes s (.getBytes x "UTF-8")))
(freezer Keyword id-keyword (.writeUTF s (if-let [ns (namespace x)] (freezer Keyword id-keyword (.writeUTF s (if-let [ns (namespace x)]
(str ns "/" (name x)) (str ns "/" (name x))
(name x)))) (name x))))
(declare freeze-to-stream!*)
(coll-freezer PersistentQueue id-queue) (coll-freezer PersistentQueue id-queue)
(coll-freezer PersistentTreeSet id-sorted-set) (coll-freezer PersistentTreeSet id-sorted-set)
(kv-freezer PersistentTreeMap id-sorted-map) (kv-freezer PersistentTreeMap id-sorted-map)
@ -135,154 +155,188 @@
(freezer Short id-short (.writeShort s x)) (freezer Short id-short (.writeShort s x))
(freezer Integer id-integer (.writeInt s x)) (freezer Integer id-integer (.writeInt s x))
(freezer Long id-long (.writeLong s x)) (freezer Long id-long (.writeLong s x))
(freezer BigInt id-bigint (write-biginteger! s (.toBigInteger x))) (freezer BigInt id-bigint (write-biginteger s (.toBigInteger x)))
(freezer BigInteger id-bigint (write-biginteger! s x)) (freezer BigInteger id-bigint (write-biginteger s x))
(freezer Float id-float (.writeFloat s x)) (freezer Float id-float (.writeFloat s x))
(freezer Double id-double (.writeDouble s x)) (freezer Double id-double (.writeDouble s x))
(freezer BigDecimal id-bigdec (freezer BigDecimal id-bigdec
(write-biginteger! s (.unscaledValue x)) (write-biginteger s (.unscaledValue x))
(.writeInt s (.scale x))) (.writeInt s (.scale x)))
(freezer Ratio id-ratio (freezer Ratio id-ratio
(write-biginteger! s (.numerator x)) (write-biginteger s (.numerator x))
(write-biginteger! s (.denominator x))) (write-biginteger s (.denominator x)))
;; Use Clojure's own reader as final fallback ;; Use Clojure's own reader as final fallback
(freezer Object id-reader (write-bytes! s (.getBytes (pr-str x) "UTF-8"))) (freezer Object id-reader (write-bytes s (.getBytes (pr-str x) "UTF-8")))
(defn- freeze-to-stream!* [^DataOutputStream s x] (defn- wrap-nippy-header [data-ba compressor encryptor password]
(if-let [m (meta x)] (let [header-ba (byte-array
(do (write-id! s id-meta) [id-nippy-magic-prefix
(freeze-to-stream!* s m))) id-nippy-header-ver
(freeze x s)) (byte (if compressor (compression/header-id compressor) 0))
(byte (if password (encryption/header-id encryptor) 0))
id-nippy-reserved])]
(utils/ba-concat header-ba data-ba)))
(defn freeze-to-stream! (defn freeze
"Serializes x to given output stream." "Serializes arg (any Clojure data type) to a byte array. Enable
([data-output-stream x] ; For <= 1.0.1 compatibility `:legacy-mode?` flag to produce bytes readable by Nippy < 2.x."
(freeze-to-stream! data-output-stream x true)) ^bytes [x & [{:keys [print-dup? password compressor encryptor legacy-mode?]
([data-output-stream x print-dup?] :or {print-dup? true
(binding [*print-dup* print-dup?] ; For `pr-str` compressor compression/default-snappy-compressor
(freeze-to-stream!* data-output-stream x)))) encryptor encryption/default-aes128-encryptor}}]]
(defn freeze-to-bytes
"Serializes x to a byte array and returns the array."
^bytes [x & {:keys [compress? print-dup? password]
:or {compress? true
print-dup? true}}]
(let [ba (ByteArrayOutputStream.) (let [ba (ByteArrayOutputStream.)
stream (DataOutputStream. ba)] stream (DataOutputStream. ba)]
(freeze-to-stream! stream x print-dup?) (binding [*print-dup* print-dup?] (freeze-to-stream x stream))
(let [ba (.toByteArray ba) (let [ba (.toByteArray ba)
ba (if compress? (utils/compress-bytes ba) ba) ba (if compressor (compression/compress compressor ba) ba)
ba (if password (crypto/encrypt-aes128 password ba) ba)] ba (if password (encryption/encrypt encryptor password ba) ba)]
ba))) (if legacy-mode? ba (wrap-nippy-header ba compressor encryptor password)))))
;;;; Thawing ;;;; Thawing
(declare thaw-from-stream!*) (declare thaw-from-stream)
(defn coll-thaw! (defn coll-thaw
"Thaws simple collection types." "Thaws simple collection types."
[^DataInputStream s] [coll ^DataInputStream s]
(repeatedly (.readInt s) #(thaw-from-stream!* s))) (utils/repeatedly-into coll (.readInt s) #(thaw-from-stream s)))
(defn coll-thaw-kvs! (defn coll-thaw-kvs
"Thaws key-value collection types." "Thaws key-value collection types."
[^DataInputStream s] [coll ^DataInputStream s]
(repeatedly (/ (.readInt s) 2) (utils/repeatedly-into coll (/ (.readInt s) 2)
(fn [] [(thaw-from-stream!* s) (thaw-from-stream!* s)]))) (fn [] [(thaw-from-stream s) (thaw-from-stream s)])))
(defn- thaw-from-stream!* (defn- thaw-from-stream
[^DataInputStream s] [^DataInputStream s]
(let [type-id (.readByte s)] (let [type-id (.readByte s)]
(utils/case-eval (utils/case-eval
type-id type-id
id-reader (read-string (String. (read-bytes! s) "UTF-8")) id-reader (read-string (String. (read-bytes s) "UTF-8"))
id-bytes (read-bytes! s) id-bytes (read-bytes s)
id-nil nil id-nil nil
id-boolean (.readBoolean s) id-boolean (.readBoolean s)
id-char (.readChar s) id-char (.readChar s)
id-string (String. (read-bytes! s) "UTF-8") id-string (String. (read-bytes s) "UTF-8")
id-keyword (keyword (.readUTF s)) id-keyword (keyword (.readUTF s))
id-queue (into (PersistentQueue/EMPTY) (coll-thaw! s)) id-queue (coll-thaw (PersistentQueue/EMPTY) s)
id-sorted-set (into (sorted-set) (coll-thaw! s)) id-sorted-set (coll-thaw (sorted-set) s)
id-sorted-map (into (sorted-map) (coll-thaw-kvs! s)) id-sorted-map (coll-thaw-kvs (sorted-map) s)
id-list (into '() (reverse (coll-thaw! s))) id-list (into '() (rseq (coll-thaw [] s)))
id-vector (into [] (coll-thaw! s)) id-vector (coll-thaw [] s)
id-set (into #{} (coll-thaw! s)) id-set (coll-thaw #{} s)
id-map (into {} (coll-thaw-kvs! s)) id-map (coll-thaw-kvs {} s)
id-coll (doall (coll-thaw! s)) id-coll (seq (coll-thaw [] s))
id-meta (let [m (thaw-from-stream!* s)] (with-meta (thaw-from-stream!* s) m)) id-meta (let [m (thaw-from-stream s)] (with-meta (thaw-from-stream s) m))
id-byte (.readByte s) id-byte (.readByte s)
id-short (.readShort s) id-short (.readShort s)
id-integer (.readInt s) id-integer (.readInt s)
id-long (.readLong s) id-long (.readLong s)
id-bigint (bigint (read-biginteger! s)) id-bigint (bigint (read-biginteger s))
id-float (.readFloat s) id-float (.readFloat s)
id-double (.readDouble s) id-double (.readDouble s)
id-bigdec (BigDecimal. (read-biginteger! s) (.readInt s)) id-bigdec (BigDecimal. (read-biginteger s) (.readInt s))
id-ratio (/ (bigint (read-biginteger! s)) id-ratio (/ (bigint (read-biginteger s))
(bigint (read-biginteger! s))) (bigint (read-biginteger s)))
;;; DEPRECATED ;;; DEPRECATED
id-old-reader (read-string (.readUTF s)) id-old-reader (read-string (.readUTF s))
id-old-string (.readUTF s) id-old-string (.readUTF s)
id-old-map (apply hash-map (repeatedly (* 2 (.readInt s)) id-old-map (apply hash-map (utils/repeatedly-into [] (* 2 (.readInt s))
#(thaw-from-stream!* s))) #(thaw-from-stream s)))
(throw (Exception. (str "Failed to thaw unknown type ID: " type-id)))))) (throw (Exception. (str "Failed to thaw unknown type ID: " type-id))))))
(defn thaw-from-stream! (defn thaw
"Deserializes an object from given input stream." "Deserializes frozen bytes to their original Clojure data type. Enable
[data-input-stream read-eval?] `:legacy-mode?` to read bytes written by Nippy < 2.x.
(binding [*read-eval* read-eval?]
(let [;; Support older versions of Nippy that wrote a version header
maybe-schema-header (thaw-from-stream!* data-input-stream)]
(if (and (string? maybe-schema-header)
(.startsWith ^String maybe-schema-header "\u0000~"))
(thaw-from-stream!* data-input-stream)
maybe-schema-header))))
(defn thaw-from-bytes WARNING: Enabling `:read-eval?` can lead to security vulnerabilities unless
"Deserializes an object from given byte array." you are sure you know what you're doing."
[ba & {:keys [compressed? read-eval? password] [^bytes ba & [{:keys [read-eval? password compressor encryptor legacy-mode?
:or {compressed? true strict?]
read-eval? false ; For `read-string` injection safety - NB!!! :or {compressor compression/default-snappy-compressor
}}] encryptor encryption/default-aes128-encryptor}}]]
(try
(-> (let [ba (if password (crypto/decrypt-aes128 password ba) ba) (let [ex (fn [msg & [e]] (throw (Exception. (str "Thaw failed. " msg) e)))
ba (if compressed? (utils/uncompress-bytes ba) ba)] thaw-data (fn [data-ba compressor password]
ba) (let [ba data-ba
(ByteArrayInputStream.) ba (if password (encryption/decrypt encryptor password ba) ba)
(DataInputStream.) ba (if compressor (compression/decompress compressor ba) ba)
(thaw-from-stream! read-eval?)) stream (DataInputStream. (ByteArrayInputStream. ba))]
(binding [*read-eval* read-eval?] (thaw-from-stream stream))))]
(if legacy-mode? ; Nippy < 2.x
(try (thaw-data ba compressor password)
(catch Exception e (catch Exception e
(throw (Exception. (cond password (ex "Unencrypted data or wrong password?" e)
(cond password "Thaw failed. Unencrypted data or bad password?" compressor (ex "Encrypted or uncompressed data?" e)
compressed? "Thaw failed. Encrypted or uncompressed data?" :else (ex "Encrypted and/or compressed data?" e))))
:else "Thaw failed. Encrypted and/or compressed data?")
e)))))
(comment ;; Nippy >= 2.x, we have a header!
(-> (freeze-to-bytes "my data" :password [:salted "password"]) (let [[[id-magic* id-header* id-comp* id-enc* _] data-ba]
(thaw-from-bytes)) (utils/ba-split ba 5)
(-> (freeze-to-bytes "my data" :compress? true)
(thaw-from-bytes :compressed? false)))
(def stress-data compressed? (not (zero? id-comp*))
"Reference data used for tests & benchmarks." encrypted? (not (zero? id-enc*))]
(let [support-tagged-literals?
(utils/version-sufficient? (clojure-version) "1.4.0")]
(cond
(not= id-magic* id-nippy-magic-prefix)
(ex (str "Not Nippy data, data frozen with Nippy < 2.x, "
"or data may be corrupt?\n"
"Enable `:legacy-mode?` option for data frozen with Nippy < 2.x."))
(> id-header* id-nippy-header-ver)
(ex "Data frozen with newer Nippy version. Please upgrade.")
(and strict? (not encrypted?) password)
(ex (str "Data is not encrypted. Try again w/o password.\n"
"Disable `:strict?` option to ignore this error. "))
(and strict? (not compressed?) compressor)
(ex (str "Data is not compressed. Try again w/o compressor.\n"
"Disable `:strict?` option to ignore this error."))
(and encrypted? (not password))
(ex "Data is encrypted. Please try again with a password.")
(and encrypted? password
(not= id-enc* (encryption/header-id encryptor)))
(ex "Data encrypted with a different Encrypter.")
(and compressed? compressor
(not= id-comp* (compression/header-id compressor)))
(ex "Data compressed with a different Compressor.")
:else
(try (thaw-data data-ba (when compressed? compressor)
(when encrypted? password))
(catch Exception e
(if (and encrypted? password)
(ex "Wrong password, or data may be corrupt?" e)
(ex "Data may be corrupt?" e)))))))))
(comment (thaw (freeze "hello"))
(thaw (freeze "hello" {:compressor nil}))
(thaw (freeze "hello" {:compressor nil}) {:strict? true}) ; ex
(thaw (freeze "hello" {:password [:salted "p"]})) ; ex
(thaw (freeze "hello") {:password [:salted "p"]}))
;;;; Stress data
(def stress-data "Reference data used for tests & benchmarks."
(let []
{:bytes (byte-array [(byte 1) (byte 2) (byte 3)]) {:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
:nil nil :nil nil
:boolean true :boolean true
@ -323,6 +377,25 @@
:ratio 22/7 :ratio 22/7
;; Clojure 1.4+ ;; Clojure 1.4+ tagged literals
:tagged-uuid (when support-tagged-literals? (java.util.UUID/randomUUID)) :tagged-uuid (java.util.UUID/randomUUID)
:tagged-date (when support-tagged-literals? (java.util.Date.))})) :tagged-date (java.util.Date.)}))
;;;; Deprecated API
(defn freeze-to-bytes "DEPRECATED: Use `freeze` instead."
^bytes [x & {:keys [print-dup? compress? password]
:or {print-dup? true
compress? true}}]
(freeze x {:print-dup? print-dup?
:compressor (when compress? compression/default-snappy-compressor)
:password password
:legacy-mode? true}))
(defn thaw-from-bytes "DEPRECATED: Use `thaw` instead."
[ba & {:keys [read-eval? compressed? password]
:or {compressed? true}}]
(thaw ba {:read-eval? read-eval?
:compressor (when compressed? compression/default-snappy-compressor)
:password password
:legacy-mode? true}))

71
src/taoensso/nippy/benchmarks.clj
View file

@ -1,26 +1,31 @@
(ns taoensso.nippy.benchmarks (ns taoensso.nippy.benchmarks
{:author "Peter Taoussanis"} {:author "Peter Taoussanis"}
(:use [taoensso.nippy :as nippy :only (freeze-to-bytes thaw-from-bytes)]) (:require [taoensso.nippy :as nippy :refer (freeze thaw)]
(:require [taoensso.nippy.utils :as utils] [taoensso.nippy.utils :as utils]))
[taoensso.nippy.crypto :as crypto]))
;; Remove stuff from stress-data that breaks reader ;; Remove stuff from stress-data that breaks reader
(def data (dissoc nippy/stress-data :queue :queue-empty :bytes)) (def data (dissoc nippy/stress-data :queue :queue-empty :bytes))
(defmacro bench [& body] `(utils/bench 10000 (do ~@body) :warmup-laps 1000)) (defmacro bench [& body] `(utils/bench 10000 (do ~@body) :warmup-laps 2000))
(defn reader-freeze [x] (binding [*print-dup* false] (pr-str x))) (defn freeze-reader [x] (binding [*print-dup* false] (pr-str x)))
(defn reader-thaw [x] (binding [*read-eval* false] (read-string x))) (defn thaw-reader [x] (binding [*read-eval* false] (read-string x)))
(def reader-roundtrip (comp reader-thaw reader-freeze)) (def roundtrip-reader (comp thaw-reader freeze-reader))
(def roundtrip-defaults (comp nippy/thaw-from-bytes nippy/freeze-to-bytes)) (def roundtrip-defaults (comp thaw freeze))
(def roundtrip-encrypted (comp #(nippy/thaw-from-bytes % :password [:cached "p"]) (def roundtrip-encrypted (comp #(thaw % {:password [:cached "p"]})
#(nippy/freeze-to-bytes % :password [:cached "p"]))) #(freeze % {:password [:cached "p"]})))
(def roundtrip-fast (comp #(nippy/thaw-from-bytes % :compressed? false) (def roundtrip-fast (comp #(thaw % {})
#(nippy/freeze-to-bytes % :compress? false))) #(freeze % {:compressor nil})))
(defn autobench [] (bench (roundtrip-defaults data) (defn autobench []
(roundtrip-encrypted data))) (println "Benchmarking roundtrips")
(println "-----------------------")
(let [results {:defaults (bench (roundtrip-defaults data))
:encrypted (bench (roundtrip-encrypted data))
:fast (bench (roundtrip-fast data))}]
(println results)
results))
(comment (comment
@ -30,38 +35,42 @@
(println (println
{:reader {:reader
{:freeze (bench (reader-freeze data)) {:freeze (bench (freeze-reader data))
:thaw (let [frozen (reader-freeze data)] :thaw (let [frozen (freeze-reader data)] (bench (thaw-reader frozen)))
(bench (reader-thaw frozen))) :round (bench (roundtrip-reader data))
:round (bench (reader-roundtrip data)) :data-size (count (.getBytes ^String (freeze-reader data) "UTF-8"))}})
:data-size (count (.getBytes ^String (reader-freeze data) "UTF-8"))}})
(println (println
{:defaults {:defaults
{:freeze (bench (freeze-to-bytes data)) {:freeze (bench (freeze data))
:thaw (let [frozen (freeze-to-bytes data)] :thaw (let [frozen (freeze data)] (bench (thaw frozen)))
(bench (thaw-from-bytes frozen)))
:round (bench (roundtrip-defaults data)) :round (bench (roundtrip-defaults data))
:data-size (count (freeze-to-bytes data))}}) :data-size (count (freeze data))}})
(println (println
{:encrypted {:encrypted
{:freeze (bench (freeze-to-bytes data :password [:cached "p"])) {:freeze (bench (freeze data {:password [:cached "p"]}))
:thaw (let [frozen (freeze-to-bytes data :password [:cached "p"])] :thaw (let [frozen (freeze data {:password [:cached "p"]})]
(bench (thaw-from-bytes frozen :password [:cached "p"]))) (bench (thaw frozen {:password [:cached "p"]})))
:round (bench (roundtrip-encrypted data)) :round (bench (roundtrip-encrypted data))
:data-size (count (freeze-to-bytes data :password [:cached "p"]))}}) :data-size (count (freeze data {:password [:cached "p"]}))}})
(println (println
{:fast {:fast
{:freeze (bench (freeze-to-bytes data :compress? false)) {:freeze (bench (freeze data {:compressor nil}))
:thaw (let [frozen (freeze-to-bytes data :compress? false)] :thaw (let [frozen (freeze data {:compressor nil})]
(bench (thaw-from-bytes frozen :compressed? false))) (bench (thaw frozen)))
:round (bench (roundtrip-fast data)) :round (bench (roundtrip-fast data))
:data-size (count (freeze-to-bytes data :compress? false))}}) :data-size (count (freeze data {:compressor nil}))}})
(println "Done! (Time for cake?)")) (println "Done! (Time for cake?)"))
;;; 13 June 2013: Clojure 1.5.1, Nippy 2.0.0-alpha1
;; {:reader {:freeze 23124, :thaw 26469, :round 47674, :data-size 22923}}
;; {:defaults {:freeze 4007, :thaw 2520, :round 6038, :data-size 12387}}
;; {:encrypted {:freeze 5560, :thaw 3867, :round 9157, :data-size 12405}}
;; {:fast {:freeze 3429, :thaw 2078, :round 5577, :data-size 13237}}
;;; 11 June 2013: Clojure 1.5.1, Nippy 1.3.0-alpha1 ;;; 11 June 2013: Clojure 1.5.1, Nippy 1.3.0-alpha1
;; {:reader {:freeze 17042, :thaw 31579, :round 48379, :data-size 22954}} ;; {:reader {:freeze 17042, :thaw 31579, :round 48379, :data-size 22954}}
;; {:fast {:freeze 3078, :thaw 4684, :round 8117, :data-size 13274}} ;; {:fast {:freeze 3078, :thaw 4684, :round 8117, :data-size 13274}}

23
src/taoensso/nippy/compression.clj Normal file
View file

@ -0,0 +1,23 @@
(ns taoensso.nippy.compression
"Alpha - subject to change."
{:author "Peter Taoussanis"}
(:require [taoensso.nippy.utils :as utils]))
;;;; Interface
(defprotocol ICompressor
(header-id [compressor]) ; Unique, >0, <= 128
(compress ^bytes [compressor ba])
(decompress ^bytes [compressor ba]))
;;;; Default implementations
(deftype DefaultSnappyCompressor []
ICompressor
(header-id [_] 1)
(compress [_ ba] (org.iq80.snappy.Snappy/compress ba))
(decompress [_ ba] (org.iq80.snappy.Snappy/uncompress ba 0 (alength ^bytes ba))))
(def default-snappy-compressor
"Default org.iq80.snappy.Snappy compressor."
(DefaultSnappyCompressor.))

130
src/taoensso/nippy/crypto.clj → src/taoensso/nippy/encryption.clj
View file

@ -1,24 +1,19 @@
(ns taoensso.nippy.crypto (ns taoensso.nippy.encryption
"Alpha - subject to change. "Alpha - subject to change.
Simple no-nonsense crypto with reasonable defaults. Because your Clojure data Simple no-nonsense crypto with reasonable defaults. Because your Clojure data
deserves some privacy." deserves some privacy."
{:author "Peter Taoussanis"} {:author "Peter Taoussanis"}
(:require [clojure.string :as str] (:require [taoensso.nippy.utils :as utils]))
[taoensso.nippy.utils :as utils]))
;;;; Interface ;;;; Interface
(defprotocol IEncrypter (defprotocol IEncryptor
(gen-key ^javax.crypto.spec.SecretKeySpec [encrypter salt-ba pwd]) (header-id [encryptor]) ; Unique, >0, <= 128
(encrypt ^bytes [encrypter pwd ba]) (encrypt ^bytes [encryptor pwd ba])
(decrypt ^bytes [encrypter pwd ba])) (decrypt ^bytes [encryptor pwd ba]))
(defrecord AES128Encrypter [key-work-factor key-cache]) ;;;; Default digests, ciphers, etc.
;;;; Digests, ciphers, etc.
;; 128bit keys have good JVM availability and are
;; entirely sufficient, Ref. http://goo.gl/2YRQG
(def ^:private ^:const aes128-block-size (int 16)) (def ^:private ^:const aes128-block-size (int 16))
(def ^:private ^:const salt-size (int 16)) (def ^:private ^:const salt-size (int 16))
@ -36,10 +31,10 @@
(defn- sha512-key (defn- sha512-key
"SHA512-based key generator. Good JVM availability without extra dependencies "SHA512-based key generator. Good JVM availability without extra dependencies
(PBKDF2, bcrypt, scrypt, etc.). Decent security with multiple rounds." (PBKDF2, bcrypt, scrypt, etc.). Decent security with multiple rounds."
[salt-ba ^String pwd key-work-factor] [salt-ba ^String pwd]
(loop [^bytes ba (let [pwd-ba (.getBytes pwd "UTF-8")] (loop [^bytes ba (let [pwd-ba (.getBytes pwd "UTF-8")]
(if salt-ba (utils/ba-concat salt-ba pwd-ba) pwd-ba)) (if salt-ba (utils/ba-concat salt-ba pwd-ba) pwd-ba))
n (* (int Short/MAX_VALUE) key-work-factor)] n (* (int Short/MAX_VALUE) (if salt-ba 5 64))]
(if-not (zero? n) (if-not (zero? n)
(recur (.digest sha512-md ba) (dec n)) (recur (.digest sha512-md ba) (dec n))
(-> ba (java.util.Arrays/copyOf aes128-block-size) (-> ba (java.util.Arrays/copyOf aes128-block-size)
@ -52,37 +47,62 @@
(time (sha512-key nil "hi" 128)) ; ~4500ms (paranoid) (time (sha512-key nil "hi" 128)) ; ~4500ms (paranoid)
) )
;;;; Default implementation ;;;; Default implementations
(extend-type AES128Encrypter (defn- destructure-typed-pwd
IEncrypter [typed-password]
(gen-key [{:keys [key-work-factor key-cache]} salt-ba pwd] (letfn [(throw-ex []
;; Trade-off: salt-ba and key-cache mutually exclusive (throw (Exception.
(utils/memoized key-cache sha512-key salt-ba pwd key-work-factor)) (str "Expected password form: "
"[<#{:salted :cached}> <password-string>].\n "
"See `default-aes128-encryptor` docstring for details!"))))]
(if-not (vector? typed-password)
(throw-ex)
(let [[type password] typed-password]
(if-not (#{:salted :cached} type)
(throw-ex)
[type password])))))
(encrypt [{:keys [key-cache] :as this} pwd data-ba] (comment (destructure-typed-pwd [:salted "foo"]))
(let [salt? (not key-cache)
(defrecord DefaultAES128Encryptor [key-cache]
IEncryptor
(header-id [_] 1)
(encrypt [this typed-pwd data-ba]
(let [[type pwd] (destructure-typed-pwd typed-pwd)
salt? (= type :salted)
iv-ba (rand-bytes aes128-block-size) iv-ba (rand-bytes aes128-block-size)
salt-ba (when salt? (rand-bytes salt-size)) salt-ba (when salt? (rand-bytes salt-size))
prefix-ba (if-not salt? iv-ba (utils/ba-concat iv-ba salt-ba)) prefix-ba (if-not salt? iv-ba (utils/ba-concat iv-ba salt-ba))
key (gen-key this salt-ba pwd) key (utils/memoized (when-not salt? (:key-cache this))
sha512-key salt-ba pwd)
iv (javax.crypto.spec.IvParameterSpec. iv-ba)] iv (javax.crypto.spec.IvParameterSpec. iv-ba)]
(.init aes128-cipher javax.crypto.Cipher/ENCRYPT_MODE key iv) (.init aes128-cipher javax.crypto.Cipher/ENCRYPT_MODE
^javax.crypto.spec.SecretKeySpec key iv)
(utils/ba-concat prefix-ba (.doFinal aes128-cipher data-ba)))) (utils/ba-concat prefix-ba (.doFinal aes128-cipher data-ba))))
(decrypt [{:keys [key-cache] :as this} pwd ba] (decrypt [this typed-pwd ba]
(let [salt? (not key-cache) (let [[type pwd] (destructure-typed-pwd typed-pwd)
salt? (= type :salted)
prefix-size (+ aes128-block-size (if salt? salt-size 0)) prefix-size (+ aes128-block-size (if salt? salt-size 0))
[prefix-ba data-ba] (utils/ba-split ba prefix-size) [prefix-ba data-ba] (utils/ba-split ba prefix-size)
[iv-ba salt-ba] (if-not salt? [prefix-ba nil] [iv-ba salt-ba] (if-not salt? [prefix-ba nil]
(utils/ba-split prefix-ba aes128-block-size)) (utils/ba-split prefix-ba aes128-block-size))
key (gen-key this salt-ba pwd) key (utils/memoized (when-not salt? (:key-cache this))
sha512-key salt-ba pwd)
iv (javax.crypto.spec.IvParameterSpec. iv-ba)] iv (javax.crypto.spec.IvParameterSpec. iv-ba)]
(.init aes128-cipher javax.crypto.Cipher/DECRYPT_MODE key iv) (.init aes128-cipher javax.crypto.Cipher/DECRYPT_MODE
^javax.crypto.spec.SecretKeySpec key iv)
(.doFinal aes128-cipher data-ba)))) (.doFinal aes128-cipher data-ba))))
(def aes128-salted (def default-aes128-encryptor
"USE CASE: You want more than a small, finite number of passwords (e.g. each "Alpha - subject to change.
Default 128bit AES encryptor with multi-round SHA-512 keygen.
Password form [:salted \"my-password\"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
item encrypted will use a unique user-provided password). item encrypted will use a unique user-provided password).
IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
@ -94,11 +114,11 @@
particular key. particular key.
Slower than `aes128-cached`, and easier to attack any particular key - but Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent." keys are independent.
(AES128Encrypter. 5 nil))
(def aes128-cached Password form [:cached \"my-password\"]
"USE CASE: You want only a small, finite number of passwords (e.g. a limited ---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
number of staff/admins, or you'll be using a single password to number of staff/admins, or you'll be using a single password to
encrypt many items). encrypt many items).
@ -112,37 +132,19 @@
Faster than `aes128-salted`, and harder to attack any particular key - but Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised." increased danger if a key is somehow compromised."
(AES128Encrypter. 64 (atom {}))) (DefaultAES128Encryptor. (atom {})))
(defn- destructure-typed-password ;;;; Default implementation
"[<type> <password>] -> [Encrypter <password>]"
[typed-password]
(letfn [(throw-ex []
(throw (Exception.
(str "Expected password form: "
"[<#{:salted :cached}> <password-string>].\n "
"See `aes128-salted`, `aes128-cached` for details."))))]
(if-not (vector? typed-password)
(throw-ex)
(let [[type password] typed-password]
[(case type :salted aes128-salted :cached aes128-cached (throw-ex))
password]))))
(defn encrypt-aes128 [typed-password ba]
(let [[encrypter password] (destructure-typed-password typed-password)]
(encrypt encrypter password ba)))
(defn decrypt-aes128 [typed-password ba]
(let [[encrypter password] (destructure-typed-password typed-password)]
(decrypt encrypter password ba)))
(comment (comment
(encrypt-aes128 "my-password" (.getBytes "Secret message")) ; Malformed (def dae default-aes128-encryptor)
(time (gen-key aes128-salted nil "my-password")) (def secret-ba (.getBytes "Secret message" "UTF-8"))
(time (gen-key aes128-cached nil "my-password")) (encrypt dae "p" secret-ba) ; Malformed
(time (->> (.getBytes "Secret message" "UTF-8") (time (encrypt dae [:salted "p"] secret-ba))
(encrypt-aes128 [:salted "p"]) (time (encrypt dae [:cached "p"] secret-ba))
(encrypt-aes128 [:cached "p"]) (time (->> secret-ba
(decrypt-aes128 [:cached "p"]) (encrypt dae [:salted "p"])
(decrypt-aes128 [:salted "p"]) (encrypt dae [:cached "p"])
(decrypt dae [:cached "p"])
(decrypt dae [:salted "p"])
(String.)))) (String.))))

47
src/taoensso/nippy/utils.clj
View file

@ -1,7 +1,6 @@
(ns taoensso.nippy.utils (ns taoensso.nippy.utils
{:author "Peter Taoussanis"} {:author "Peter Taoussanis"}
(:require [clojure.string :as str]) (:require [clojure.string :as str]))
(:import org.iq80.snappy.Snappy))
(defmacro case-eval (defmacro case-eval
"Like `case` but evaluates test constants for their compile-time value." "Like `case` but evaluates test constants for their compile-time value."
@ -14,25 +13,25 @@
clauses) clauses)
~(when default default)))) ~(when default default))))
(defn pairs (defn repeatedly-into
"Like (partition 2 coll) but faster and returns lazy seq of vector pairs." "Like `repeatedly` but faster and `conj`s items into given collection."
[coll] [coll n f]
(lazy-seq (if-not (instance? clojure.lang.IEditableCollection coll)
(when-let [s (seq coll)] (loop [v coll idx 0]
(let [n (next s)] (if (>= idx n)
(cons [(first s) (first n)] (pairs (next n))))))) v
(recur (conj v (f)) (inc idx))))
(loop [v (transient coll) idx 0]
(if (>= idx n)
(persistent! v)
(recur (conj! v (f)) (inc idx))))))
(defmacro time-ns (defmacro time-ns "Returns number of nanoseconds it takes to execute body."
"Returns number of nanoseconds it takes to execute body." [& body] `(let [t0# (System/nanoTime)] ~@body (- (System/nanoTime) t0#)))
[& body]
`(let [t0# (System/nanoTime)]
~@body
(- (System/nanoTime) t0#)))
(defmacro bench (defmacro bench
"Repeatedly executes form and returns time taken to complete execution." "Repeatedly executes form and returns time taken to complete execution."
[num-laps form & {:keys [warmup-laps num-threads as-ms?] [num-laps form & {:keys [warmup-laps num-threads as-ns?]}]
:or {as-ms? true}}]
`(try (when ~warmup-laps (dotimes [_# ~warmup-laps] ~form)) `(try (when ~warmup-laps (dotimes [_# ~warmup-laps] ~form))
(let [nanosecs# (let [nanosecs#
(if-not ~num-threads (if-not ~num-threads
@ -44,23 +43,17 @@
doall doall
(map deref) (map deref)
dorun))))] dorun))))]
(if ~as-ms? (Math/round (/ nanosecs# 1000000.0)) nanosecs#)) (if ~as-ns? nanosecs# (Math/round (/ nanosecs# 1000000.0))))
(catch Exception e# (str "DNF: " (.getMessage e#))))) (catch Exception e# (str "DNF: " (.getMessage e#)))))
(defn version-compare (defn version-compare "Comparator for version strings like x.y.z, etc."
"Comparator for version strings like x.y.z, etc." [x y] (let [vals (fn [s] (vec (map #(Integer/parseInt %) (str/split s #"\."))))]
[x y]
(let [vals (fn [s] (vec (map #(Integer/parseInt %) (str/split s #"\."))))]
(compare (vals x) (vals y)))) (compare (vals x) (vals y))))
(defn version-sufficient? (defn version-sufficient? [version-str min-version-str]
[version-str min-version-str]
(try (>= (version-compare version-str min-version-str) 0) (try (>= (version-compare version-str min-version-str) 0)
(catch Exception _ false))) (catch Exception _ false)))
(defn compress-bytes [^bytes ba] (Snappy/compress ba))
(defn uncompress-bytes [^bytes ba] (Snappy/uncompress ba 0 (alength ba)))
(defn memoized (defn memoized
"Like `memoize` but takes an explicit cache atom (possibly nil) and "Like `memoize` but takes an explicit cache atom (possibly nil) and
immediately applies memoized f to given arguments." immediately applies memoized f to given arguments."

51
test/taoensso/nippy/tests/main.clj Normal file
View file

@ -0,0 +1,51 @@
(ns taoensso.nippy.tests.main
(:require [expectations :as test :refer :all]
[taoensso.nippy :as nippy :refer (freeze thaw)]
[taoensso.nippy.benchmarks :as benchmarks]))
;; Remove stuff from stress-data that breaks roundtrip equality
(def test-data (dissoc nippy/stress-data :bytes))
(def roundtrip-defaults (comp thaw freeze))
(def roundtrip-encrypted (comp #(thaw % {:password [:salted "p"]})
#(freeze % {:password [:salted "p"]})))
(def roundtrip-defaults-legacy (comp #(thaw % {:legacy-mode? true})
#(freeze % {:legacy-mode? true})))
(def roundtrip-encrypted-legacy (comp #(thaw % {:password [:salted "p"]
:legacy-mode? true})
#(freeze % {:password [:salted "p"]
:legacy-mode? true})))
;;; Basic data integrity
(expect test-data (roundtrip-defaults test-data))
(expect test-data (roundtrip-encrypted test-data))
(expect test-data (roundtrip-defaults-legacy test-data))
(expect test-data (roundtrip-encrypted-legacy test-data))
(expect ; Snappy lib compatibility (for legacy versions of Nippy)
(let [^bytes raw-ba (freeze test-data {:compressor nil})
^bytes xerial-ba (org.xerial.snappy.Snappy/compress raw-ba)
^bytes iq80-ba (org.iq80.snappy.Snappy/compress raw-ba)]
(= (thaw raw-ba)
(thaw (org.xerial.snappy.Snappy/uncompress xerial-ba))
(thaw (org.xerial.snappy.Snappy/uncompress iq80-ba))
(thaw (org.iq80.snappy.Snappy/uncompress iq80-ba 0 (alength iq80-ba)))
(thaw (org.iq80.snappy.Snappy/uncompress xerial-ba 0 (alength xerial-ba))))))
;;; API stuff
;; Strict/auto mode - compression
(expect test-data (thaw (freeze test-data {:compressor nil})))
(expect Exception (thaw (freeze test-data {:compressor nil}) {:strict? true}))
;; Strict/auto mode - encryption
(expect test-data (thaw (freeze test-data) {:password [:salted "p"]}))
(expect Exception (thaw (freeze test-data) {:password [:salted "p"] :strict? true}))
;; Encryption - passwords
(expect Exception (thaw (freeze test-data {:password "malformed"})))
(expect Exception (thaw (freeze test-data {:password [:salted "p"]})))
(expect test-data (thaw (freeze test-data {:password [:salted "p"]})
{:password [:salted "p"]}))
(expect (benchmarks/autobench)) ; Also tests :cached passwords

30
test/test_nippy/main.clj
View file

@ -1,30 +0,0 @@
(ns test-nippy.main
(:use [clojure.test])
(:require [taoensso.nippy :as nippy]
[taoensso.nippy.benchmarks :as benchmarks]))
;; Remove stuff from stress-data that breaks roundtrip equality
(def test-data (dissoc nippy/stress-data :bytes))
(def roundtrip-defaults (comp nippy/thaw-from-bytes nippy/freeze-to-bytes))
(def roundtrip-encrypted (comp #(nippy/thaw-from-bytes % :password [:cached "secret"])
#(nippy/freeze-to-bytes % :password [:cached "secret"])))
(deftest test-roundtrip-defaults (is (= test-data (roundtrip-defaults test-data))))
(deftest test-roundtrip-encrypted (is (= test-data (roundtrip-encrypted test-data))))
(println "Benchmarking roundtrips (x3)")
(println "----------------------------")
(println (benchmarks/autobench))
(println (benchmarks/autobench))
(println (benchmarks/autobench))
(deftest test-snappy-library-compatibility
(let [thaw #(nippy/thaw-from-bytes % :compressed? false)
^bytes raw-ba (nippy/freeze-to-bytes test-data :compress? false)
^bytes xerial-ba (org.xerial.snappy.Snappy/compress raw-ba)
^bytes iq80-ba (org.iq80.snappy.Snappy/compress raw-ba)]
(is (= (thaw raw-ba)
(thaw (org.xerial.snappy.Snappy/uncompress xerial-ba))
(thaw (org.xerial.snappy.Snappy/uncompress iq80-ba))
(thaw (org.iq80.snappy.Snappy/uncompress iq80-ba 0 (alength iq80-ba)))
(thaw (org.iq80.snappy.Snappy/uncompress xerial-ba 0 (alength xerial-ba)))))))