taoensso.nippy
High-performance serialization library for Clojure
+
*auto-freeze-compressor*
dynamic
(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}),
+nil => default*custom-readers*
dynamic
{<hash-or-byte-id> (fn [data-input])}
+*final-freeze-fallback*
dynamic
*freeze-fallback*
dynamic
(fn [data-output x]), nil => default
+
->CacheWrapped
(->CacheWrapped value)
Positional factory function for class taoensso.nippy.CacheWrapped.
+
->StressRecord
(->StressRecord data)
Positional factory function for class taoensso.nippy.StressRecord.
+
aes128-encryptor
Default 128bit AES encryptor with many-round SHA-512 key-gen.
Password form [:salted "my-password"]
---------------------------------------
@@ -38,10 +40,11 @@ CONS: Using a small number of keys for many encrypted items means that if any
compromised.
Faster than `aes128-salted`, and harder to attack any particular key - but
-increased danger if a key is somehow compromised.byte-sized?
(byte-sized? n)
compress
(compress compressor ba)
decompress
(decompress compressor ba)
decrypt
(decrypt encryptor pwd ba)
default-freeze-compressor-selector
(default-freeze-compressor-selector ba)
Strategy:
-* Prioritize speed, but allow lz4.
-* Skip lz4 unless it's likely that lz4's space benefit will outweigh its
- space overhead.
encrypt
(encrypt encryptor pwd ba)
extend-freeze
macro
(extend-freeze type custom-type-id [x out] & body)
Extends Nippy to support freezing of a custom type (ideally concrete) with
+increased danger if a key is somehow compromised.
cache
(cache x)
Experimental! Wraps value so that future writes of the same wrapped
+value will be efficiently encoded as references to this one.
+
+(freeze [(cache "foo") (cache "foo") (cache "foo")])
+ will incl. a single "foo", plus 2x single-byte references to "foo".
compress
(compress compressor ba)
decompress
(decompress compressor ba)
decrypt
(decrypt encryptor pwd ba)
encrypt
(encrypt encryptor pwd ba)
extend-freeze
macro
(extend-freeze type custom-type-id [x out] & body)
Extends Nippy to support freezing of a custom type (ideally concrete) with
given id of form:
* Keyword - 2 byte overhead, resistent to id collisions
* Integer ∈[1, 128] - no overhead, subject to id collisions
@@ -51,23 +54,35 @@ given id of form:
(.writeUTF [data-output] (:data x)))
;; or
(extend-freeze MyType 1 [x data-output] ; Byte id
- (.writeUTF [data-output] (:data x)))
extend-thaw
macro
(extend-thaw custom-type-id [in] & body)
Extends Nippy to support thawing of a custom type with given id:
+ (.writeUTF [data-output] (:data x)))
extend-thaw
macro
(extend-thaw custom-type-id [in] & body)
Extends Nippy to support thawing of a custom type with given id:
(extend-thaw :foo/my-type [data-input] ; Keyword id
(->MyType (.readUTF data-input)))
;; or
(extend-thaw 1 [data-input] ; Byte id
- (->MyType (.readUTF data-input)))
Freezable
protocol
members
-freeze-to-out
(-freeze-to-out this out)
freezable?
(freezable? x & [{:keys [allow-clojure-reader? allow-java-serializable?]}])
Alpha - subject to change, MAY BE BUGGY!
-Returns truthy value iff Nippy supports de/serialization of given argument.
-Conservative with default options.
+ (->MyType (.readUTF data-input)))
fast-freeze
(fast-freeze x)
Like `freeze` but:
+ - Writes data without a Nippy header
+ - Drops all support for compression and encryption
+ - Must be thawed with `fast-thaw`
-`:allow-clojure-reader?` and `:allow-java-serializable?` options may be used
-to also enable the relevant roundtrip fallback test(s). These tests are only
-**moderately reliable** since they're cached by arg type and don't test for
-pre/post serialization equality (there's no good general way of doing so).
freeze
(freeze x)(freeze x {:keys [compressor encryptor password], :or {compressor :auto, encryptor aes128-encryptor}, :as opts})
Serializes arg (any Clojure data type) to a byte array. To freeze custom
-types, extend the Clojure reader or see `extend-freeze`.
freeze-fallback-as-str
(freeze-fallback-as-str out x)
freeze-to-out!
(freeze-to-out! data-output x)
Serializes arg (any Clojure data type) to a DataOutput
-
inspect-ba
(inspect-ba ba)(inspect-ba ba thaw-opts)
Alpha - subject to change
-
lz4-compressor
Default net.jpountz.lz4 compressor:
+Equivalent to (but a little faster than):
+ `(freeze x {:compressor nil :encryptor nil :no-header? true})fast-thaw
(fast-thaw ba)
Like `thaw` but:
+ - Drops all support for compression and encryption
+ - Supports only data frozen with `fast-freeze`
+
+Equivalent to (but a little faster than):
+ `(thaw x {:compressor nil :encryptor nil :no-header? true})Freezable
protocol
members
-freeze-to-out!
(-freeze-to-out! this out)
freezable?
(freezable? x)(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})
Alpha - subject to change.
+Returns truthy iff Nippy *appears* to support freezing the given argument.
+
+`:allow-clojure-reader?` and `:allow-java-serializable?` options may be
+used to enable the relevant roundtrip fallback test(s). These tests are
+only **moderately reliable** since they're cached by arg type and don't
+test for pre/post serialization value equality (there's no good general
+way of doing so).
freeze
(freeze x)(freeze x {:keys [compressor encryptor password], :or {compressor :auto, encryptor aes128-encryptor}, :as opts})
Serializes arg (any Clojure data type) to a byte array. To freeze custom
+types, extend the Clojure reader or see `extend-freeze`.
freeze-to-out!
(freeze-to-out! data-output x)
Serializes arg (any Clojure data type) to a DataOutput. Please note that
+this is a low-level util: in most cases you'll want `freeze` instead.
inspect-ba
(inspect-ba ba)(inspect-ba ba thaw-opts)
Alpha - subject to change
+
lz4-compressor
Default net.jpountz.lz4 compressor:
Ratio: low.
Write speed: very high.
Read speed: very high.
@@ -75,26 +90,25 @@ Write speed: very high.
A good general-purpose compressor, competitive with Snappy.
Thanks to Max Penet (@mpenet) for our first implementation,
-Ref. https://github.com/mpenet/nippy-lz4lz4hc-compressor
Like `lz4-compressor` but trades some write speed for ratio.
-
lz4hc-compressor
Like `lz4-compressor` but trades some write speed for ratio.
+
lzma2-compressor
Default org.tukaani.xz.LZMA2 compressor:
Ratio: high.
Write speed: _very_ slow (also currently single-threaded).
Read speed: slow.
A specialized compressor for large, low-write data in space-sensitive
-environments.map->Compressable-LZMA2
(map->Compressable-LZMA2 m__6522__auto__)
Factory function for class taoensso.nippy.Compressable-LZMA2, taking a map of keywords to field values.
-
map->StressRecord
(map->StressRecord m__6522__auto__)
Factory function for class taoensso.nippy.StressRecord, taking a map of keywords to field values.
-
read-biginteger
(read-biginteger in)
read-bytes
(read-bytes in)
read-coll
(read-coll in to-coll)
read-kvs
(read-kvs in to-coll)
read-kvs-depr1
(read-kvs-depr1 in to-coll)
read-sm-bytes
(read-sm-bytes in)
read-sm-coll
(read-sm-coll in to-coll)
read-sm-kvs
(read-sm-kvs in to-coll)
read-sm-utf8
(read-sm-utf8 in)
set-default-freeze-compressor-selector!
(set-default-freeze-compressor-selector! selector)
Sets root binding of `*default-freeze-compressor-selector*`
-
short-sized?
(short-sized? n)
snappy-compressor
Default org.iq80.snappy.Snappy compressor:
+environments.
map->CacheWrapped
(map->CacheWrapped m__8448__auto__)
Factory function for class taoensso.nippy.CacheWrapped, taking a map of keywords to field values.
+
map->StressRecord
(map->StressRecord m__8448__auto__)
Factory function for class taoensso.nippy.StressRecord, taking a map of keywords to field values.
+
set-auto-freeze-compressor!
(set-auto-freeze-compressor! x)
set-freeze-fallback!
(set-freeze-fallback! x)
snappy-compressor
Default org.iq80.snappy.Snappy compressor:
Ratio: low.
Write speed: very high.
Read speed: very high.
-A good general-purpose compressor.stress-data
Reference data used for tests & benchmarks
-
stress-data-benchable
Reference data with stuff removed that breaks reader or other utils we'll
-be benching against
stress-data-comparable
Reference data with stuff removed that breaks roundtrip equality
-
swap-custom-readers!
(swap-custom-readers! f)
thaw
(thaw ba)(thaw ba {:keys [v1-compatibility? compressor encryptor password], :or {compressor :auto, encryptor :auto}, :as opts})
Deserializes a frozen object from given byte array to its original Clojure
-data type. To thaw custom types, extend the Clojure reader or see `extend-thaw`.
+A good general-purpose compressor.
stress-data
Reference data used for tests & benchmarks
+
stress-data-benchable
Reference data with stuff removed that breaks reader or other utils we'll
+be benching against
stress-data-comparable
Reference data with stuff removed that breaks roundtrip equality
+
swap-custom-readers!
(swap-custom-readers! f)
thaw
(thaw ba)(thaw ba {:keys [v1-compatibility? compressor encryptor password], :or {compressor :auto, encryptor :auto}, :as opts})
Deserializes a frozen Nippy byte array to its original Clojure data type.
+To thaw custom types, extend the Clojure reader or see `extend-thaw`.
** By default, supports data frozen with Nippy v2+ ONLY **
Add `{:v1-compatibility? true}` option to support thawing of data frozen with
@@ -103,5 +117,6 @@ legacy versions of Nippy.
Options include:
:v1-compatibility? - support data frozen by legacy versions of Nippy?
:compressor - :auto (checks header, default) an ICompressor, or nil
- :encryptor - :auto (checks header, default), an IEncryptor, or nilthaw-from-in!
(thaw-from-in! data-input)
Deserializes a frozen object from given DataInput to its original Clojure
-data type
when-debug
macro
(when-debug & body)
write-biginteger
(write-biginteger out n)
write-bytes
(write-bytes out ba)
write-id
macro
(write-id out id)
write-ided-bytes
(write-ided-bytes out ba)(write-ided-bytes out id-sm id ba)
write-ided-coll
(write-ided-coll out ?id-sm id coll)
write-ided-keyword
(write-ided-keyword out kw)
write-ided-kvs
(write-ided-kvs out ?id-sm id coll)
write-ided-long
(write-ided-long out n)
write-ided-map
(write-ided-map out m)
write-ided-set
(write-ided-set out s)
write-ided-string
(write-ided-string out s)
write-ided-vec
(write-ided-vec out v)
write-sm-bytes
(write-sm-bytes out ba)
write-sm-utf8
(write-sm-utf8 out s)
write-utf8
(write-utf8 out s)