taoensso.nippy
High-performance serialization library for Clojure
*auto-freeze-compressor*
dynamic
(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}),
nil => default*custom-readers*
dynamic
{<hash-or-byte-id> (fn [data-input])->read}
*final-freeze-fallback*
dynamic
DEPRECATED: prefer `*freeze-fallback`.
*freeze-fallback*
dynamic
(fn [data-output x])->freeze, nil => default
*freeze-serializable-allowlist*
dynamic
Used when attempting to <freeze/thaw> an object that:
- Does not implement Nippy's Freezable protocol.
- Does implement Java's Serializable interface.
In this case, Java's Serializable interface will be permitted iff
(<allowlist> <class-name>) predicate call returns true.
This is a security measure to prevent possible Remote Code Execution
(RCE) when thawing malicious payloads. See [1] for details.
If `freeze` encounters a disallowed Serialized class, it will throw.
If `thaw` encounters a disallowed Serialized class, it will:
- Throw if it's not possible to safely quarantine the object
(object was frozen with Nippy < v2.15.0-final).
- Otherwise it will return a safely quarantined object of form
`{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}`.
- Quarantined objects may be manually unquarantined with
`read-quarantined-serializable-object-unsafe!`.
There are 2x allowlists: *<freeze/thaw>-serializable-allowlist*.
Example values:
- (fn allow-class? [class-name] true) ; Arbitrary fn
- #{"java.lang.Throwable", "clojure.lang.*"} ; Set of class-names
Note that class-names in sets may contain "*" wildcards.
Default allowlist values are:
- default-freeze-serializable-allowlist ; {"*"} => allow any class
- default-thaw-serializable-allowlist ; A set of common safe classes
Allowlist values may be overridden with `binding`, `alter-var-root`, or:
- `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property
- `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add` JVM property
- `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` env var
- `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD` env var
If present, these will be read as comma-separated lists of class names
and formed into sets. Each initial allowlist value will then be:
(into (or <?base> <default>) <?additions>).
I.e. you can use:
- The "base" property/var to replace Nippy's default allowlists.
- The "add" property/var to add to Nippy's default allowlists.
See also `taoensso.encore/compile-str-filter`, a util to help
easily build more advanced predicate functions.
Upgrading from an older version of Nippy and unsure whether you've been
using Nippy's Serializable support? Here's a snippet to ALLOW and RECORD
any class requesting Nippy's Serializable fallback:
;; Deref for set of all class names that made use of Nippy's Serializable support:
(defonce observed-serializables_ (atom #{}))
(let [f (fn allow-class? [class-name]
(swap! observed-serializables_ conj class-name) ; Record class name
true ; Allow any class
)]
(alter-var-root #'*freeze-serializable-allowlist* (fn [_] f))
(alter-var-root #'*thaw-serializable-allowlist* (fn [_] f)))
Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
identifying this vulnerability.
[1] https://github.com/ptaoussanis/nippy/issues/130*thaw-serializable-allowlist*
dynamic
Used when attempting to <freeze/thaw> an object that:
- Does not implement Nippy's Freezable protocol.
- Does implement Java's Serializable interface.
In this case, Java's Serializable interface will be permitted iff
(<allowlist> <class-name>) predicate call returns true.
This is a security measure to prevent possible Remote Code Execution
(RCE) when thawing malicious payloads. See [1] for details.
If `freeze` encounters a disallowed Serialized class, it will throw.
If `thaw` encounters a disallowed Serialized class, it will:
- Throw if it's not possible to safely quarantine the object
(object was frozen with Nippy < v2.15.0-final).
- Otherwise it will return a safely quarantined object of form
`{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}`.
- Quarantined objects may be manually unquarantined with
`read-quarantined-serializable-object-unsafe!`.
There are 2x allowlists: *<freeze/thaw>-serializable-allowlist*.
Example values:
- (fn allow-class? [class-name] true) ; Arbitrary fn
- #{"java.lang.Throwable", "clojure.lang.*"} ; Set of class-names
Note that class-names in sets may contain "*" wildcards.
Default allowlist values are:
- default-freeze-serializable-allowlist ; {"*"} => allow any class
- default-thaw-serializable-allowlist ; A set of common safe classes
Allowlist values may be overridden with `binding`, `alter-var-root`, or:
- `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property
- `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add` JVM property
- `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` env var
- `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD` env var
If present, these will be read as comma-separated lists of class names
and formed into sets. Each initial allowlist value will then be:
(into (or <?base> <default>) <?additions>).
I.e. you can use:
- The "base" property/var to replace Nippy's default allowlists.
- The "add" property/var to add to Nippy's default allowlists.
See also `taoensso.encore/compile-str-filter`, a util to help
easily build more advanced predicate functions.
Upgrading from an older version of Nippy and unsure whether you've been
using Nippy's Serializable support? Here's a snippet to ALLOW and RECORD
any class requesting Nippy's Serializable fallback:
;; Deref for set of all class names that made use of Nippy's Serializable support:
(defonce observed-serializables_ (atom #{}))
(let [f (fn allow-class? [class-name]
(swap! observed-serializables_ conj class-name) ; Record class name
true ; Allow any class
)]
(alter-var-root #'*freeze-serializable-allowlist* (fn [_] f))
(alter-var-root #'*thaw-serializable-allowlist* (fn [_] f)))
Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
identifying this vulnerability.
[1] https://github.com/ptaoussanis/nippy/issues/130-cache-proxy
{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing.
aes128-cbc-encryptor
Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen.
See also `aes-128-cbc-encryptor`.
aes128-encryptor
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
Password form [:salted "my-password"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
item encrypted will use a unique user-provided password).
IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
every key.
PROS: Each key is independent so would need to be attacked independently.
CONS: Key caching impossible, so there's an inherent trade-off between
encryption/decryption speed and the difficulty of attacking any
particular key.
Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.
Password form [:cached "my-password"]
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
number of staff/admins, or you'll be using a single password to
encrypt many items).
IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.
PROS: Great amortized encryption/decryption speed. Expensive key hash makes
attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
key _is_ somehow compromised, _all_ items encrypted with that key are
compromised.
Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised.aes128-gcm-encryptor
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
Password form [:salted "my-password"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
item encrypted will use a unique user-provided password).
IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
every key.
PROS: Each key is independent so would need to be attacked independently.
CONS: Key caching impossible, so there's an inherent trade-off between
encryption/decryption speed and the difficulty of attacking any
particular key.
Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.
Password form [:cached "my-password"]
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
number of staff/admins, or you'll be using a single password to
encrypt many items).
IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.
PROS: Great amortized encryption/decryption speed. Expensive key hash makes
attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
key _is_ somehow compromised, _all_ items encrypted with that key are
compromised.
Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised.cache
(cache x)
Experimental, subject to change.
Wraps value so that future writes of the same wrapped value with same
metadata will be efficiently encoded as references to this one.
(freeze [(cache "foo") (cache "foo") (cache "foo")])
will incl. a single "foo", plus 2x single-byte references to "foo".
compress
(compress compressor ba)
decompress
(decompress compressor ba)
decrypt
(decrypt encryptor pwd ba)
default-freeze-serializable-allowlist
Allows *any* class-name to be frozen using Java's Serializable interface.
This is generally safe since RCE risk is present only when thawing.
See also `*freeze-serializable-allowlist*`.
default-thaw-serializable-allowlist
A set of common safe class-names to allow to be frozen using Java's
Serializable interface. PRs welcome for additions.
See also `*thaw-serializable-allowlist*`.
encrypt
(encrypt encryptor pwd ba)
extend-freeze
macro
(extend-freeze type custom-type-id [x out] & body)
Extends Nippy to support freezing of a custom type (ideally concrete) with
given id of form:
* Keyword - 2 byte overhead, keywords hashed to 16 bit id
* ℕ∈[1, 128] - 0 byte overhead
NB: be careful about extending to interfaces, Ref. http://goo.gl/6gGRlU.
(defrecord MyRec [data])
(extend-freeze MyRec :foo/my-type [x data-output] ; Keyword id
(.writeUTF [data-output] (:data x)))
;; or
(extend-freeze MyRec 1 [x data-output] ; Byte id
(.writeUTF [data-output] (:data x)))
extend-thaw
macro
(extend-thaw custom-type-id [in] & body)
Extends Nippy to support thawing of a custom type with given id:
(extend-thaw :foo/my-type [data-input] ; Keyword id
(MyRec. (.readUTF data-input)))
;; or
(extend-thaw 1 [data-input] ; Byte id
(MyRec. (.readUTF data-input)))
fast-freeze
(fast-freeze x)
Like `freeze` but:
- Writes data without a Nippy header
- Drops all support for compression and encryption
- Must be thawed with `fast-thaw`
Equivalent to (but a little faster than) `freeze` with opts:
- :compressor nil
- :encryptor nil
- :no-header? true
fast-thaw
(fast-thaw ba)
Like `thaw` but:
- Drops all support for compression and encryption
- Supports only data frozen with `fast-freeze`
Equivalent to (but a little faster than) `thaw` with opts:
- :compressor nil
- :encryptor nil
- :no-header? true
freezable?
(freezable? x)(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})
Alpha - subject to change.
Returns truthy iff Nippy *appears* to support freezing the given argument.
`:allow-clojure-reader?` and `:allow-java-serializable?` options may be
used to enable the relevant roundtrip fallback test(s). These tests are
only **moderately reliable** since they're cached by arg type and don't
test for pre/post serialization value equality (there's no good general
way of doing so).
freeze
(freeze x)(freeze x {:as opts, :keys [compressor encryptor password serializable-allowlist incl-metadata?], :or {compressor :auto, encryptor aes128-gcm-encryptor}})
Serializes arg (any Clojure data type) to a byte array. To freeze custom
types, extend the Clojure reader or see `extend-freeze`.
freeze-to-file
(freeze-to-file file x)(freeze-to-file file x freeze-opts)
Convenience util: like `freeze`, but writes to `(clojure.java.io/file <file>)`
and returns the byte array written.
See also `thaw-from-file`.
freeze-to-out!
(freeze-to-out! data-output x)
Serializes arg (any Clojure data type) to a DataOutput.
This is a low-level util: in most cases you'll want `freeze` instead.
freeze-to-string
(freeze-to-string x)(freeze-to-string x freeze-opts)
Convenience util: like `freeze`, but returns a Base64-encoded string.
See also `thaw-from-string`.
inspect-ba
(inspect-ba ba)(inspect-ba ba thaw-opts)
Alpha - subject to change
lz4-compressor
Default net.jpountz.lz4 compressor:
Ratio: low.
Write speed: very high.
Read speed: very high.
A good general-purpose compressor, competitive with Snappy.
Thanks to Max Penet (@mpenet) for our first implementation,
Ref. https://github.com/mpenet/nippy-lz4lz4hc-compressor
Like `lz4-compressor` but trades some write speed for ratio.
lzma2-compressor
Default org.tukaani.xz.LZMA2 compressor:
Ratio: high.
Write speed: _very_ slow (also currently single-threaded).
Read speed: slow.
A specialized compressor for large, low-write data in space-sensitive
environments.read-quarantined-serializable-object-unsafe!
(read-quarantined-serializable-object-unsafe! m)
Given a quarantined Serializable object like
{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}, reads and
returns the object WITHOUT regard for `*thaw-serializable-allowlist*`.
**MAY BE UNSAFE!** Don't call this unless you absolutely trust the payload
to not contain any malicious code.
See `*thaw-serializable-allowlist*` for more info.snappy-compressor
Default org.iq80.snappy.Snappy compressor:
Ratio: low.
Write speed: very high.
Read speed: very high.
A good general-purpose compressor.stress-data
Reference data used for tests & benchmarks
stress-data-benchable
Reference data with stuff removed that breaks reader or other utils we'll
be benching against
stress-data-comparable
Reference data with stuff removed that breaks roundtrip equality
thaw
(thaw ba)(thaw ba {:as opts, :keys [v1-compatibility? compressor encryptor password serializable-allowlist incl-metadata?], :or {compressor :auto, encryptor :auto}})
Deserializes a frozen Nippy byte array to its original Clojure data type.
To thaw custom types, extend the Clojure reader or see `extend-thaw`.
** By default, supports data frozen with Nippy v2+ ONLY **
Add `{:v1-compatibility? true}` option to support thawing of data frozen with
legacy versions of Nippy.
Options include:
:v1-compatibility? - support data frozen by legacy versions of Nippy?
:compressor - :auto (checks header, default) an ICompressor, or nil
:encryptor - :auto (checks header, default), an IEncryptor, or nilthaw-from-file
(thaw-from-file file)(thaw-from-file file thaw-opts)
Convenience util: like `thaw`, but reads from `(clojure.java.io/file <file>)`.
To thaw from a resource on classpath (e.g in Leiningen `resources` dir):
(thaw-from-file (clojure.java.io/resource "my-resource-name.npy"))
See also `freeze-to-file`.
thaw-from-in!
(thaw-from-in! data-input)
Deserializes a frozen object from given DataInput to its original Clojure
data type.
This is a low-level util: in most cases you'll want `thaw` instead.
thaw-from-string
(thaw-from-string s)(thaw-from-string s thaw-opts)
Convenience util: like `thaw`, but takes a Base64-encoded string.
See also `freeze-to-string`.
throw-unfreezable
(throw-unfreezable x)
try-write-readable
(try-write-readable out x)
try-write-serializable
(try-write-serializable out x)
write-id
macro
(write-id out id)
write-unfreezable
(write-unfreezable out x)