Nippy 2.15.0
Released under the Eclipse Public License
High-performance serialization library for Clojure.
Installation
To install, add the following dependency to your project or build file:
[com.taoensso/nippy "2.15.0"]
Namespaces
taoensso.nippy
High-performance serialization library for Clojure
Public variables and functions:
- *auto-freeze-compressor*
- *custom-readers*
- *final-freeze-fallback*
- *freeze-fallback*
- *serializable-whitelist*
- -cache-proxy
- aes128-cbc-encryptor
- aes128-encryptor
- aes128-gcm-encryptor
- cache
- compress
- decompress
- decrypt
- encrypt
- extend-freeze
- extend-thaw
- fast-freeze
- fast-thaw
- freezable?
- freeze
- freeze-fallback-as-str
- freeze-to-file
- freeze-to-out!
- freeze-to-string
- IFreezable1
- IFreezable2
- inspect-ba
- lz4-compressor
- lz4hc-compressor
- lzma2-compressor
- set-auto-freeze-compressor!
- set-freeze-fallback!
- snappy-compressor
- stress-data
- stress-data-benchable
- stress-data-comparable
- swap-custom-readers!
- swap-serializable-whitelist!
- thaw
- thaw-from-file
- thaw-from-in!
- thaw-from-string
- throw-unfreezable
- try-write-readable
- try-write-serializable
- write-id
- write-unfreezable
taoensso.nippy.crypto
Low-level crypto utils. +Nippy 2.15.1 Generated by Codox
Nippy 2.15.1
\ No newline at end of file diff --git a/taoensso.nippy.benchmarks.html b/taoensso.nippy.benchmarks.html index 38b8d78..2d52689 100644 --- a/taoensso.nippy.benchmarks.html +++ b/taoensso.nippy.benchmarks.html @@ -1,3 +1,3 @@ -Nippy 2.15.1
Released under the Eclipse Public License
High-performance serialization library for Clojure.
Installation
To install, add the following dependency to your project or build file:
[com.taoensso/nippy "2.15.1"]Namespaces
taoensso.nippy
High-performance serialization library for ClojurePublic variables and functions:
- *auto-freeze-compressor*
- *custom-readers*
- *final-freeze-fallback*
- *freeze-fallback*
- *incl-metadata?*
- *serializable-whitelist*
- -cache-proxy
- aes128-cbc-encryptor
- aes128-encryptor
- aes128-gcm-encryptor
- cache
- compress
- decompress
- decrypt
- default-serializable-whitelist
- encrypt
- extend-freeze
- extend-thaw
- fast-freeze
- fast-thaw
- freezable?
- freeze
- freeze-fallback-as-str
- freeze-to-file
- freeze-to-out!
- freeze-to-string
- IFreezable1
- IFreezable2
- inspect-ba
- lz4-compressor
- lz4hc-compressor
- lzma2-compressor
- set-auto-freeze-compressor!
- set-freeze-fallback!
- snappy-compressor
- stress-data
- stress-data-benchable
- stress-data-comparable
- swap-custom-readers!
- swap-serializable-whitelist!
- thaw
- thaw-from-file
- thaw-from-in!
- thaw-from-string
- throw-unfreezable
- try-write-readable
- try-write-serializable
- write-id
- write-unfreezable
taoensso.nippy.crypto
Low-level crypto utils. Private & alpha, very likely to change!taoensso.nippy.encryption
Simple no-nonsense crypto with reasonable defaultsPublic variables and functions:
taoensso.nippy.tools
Utils for 3rd-party tools that want to add user-configurable Nippy support. Used by Carmine, Faraday, etc.Public variables and functions:
taoensso.nippy.benchmarks documentation Generated by Codox
Nippy 2.15.0
\ No newline at end of file +taoensso.nippy.benchmarks
taoensso.nippy.benchmarks documentation Generated by Codox
Nippy 2.15.1
\ No newline at end of file diff --git a/taoensso.nippy.compression.html b/taoensso.nippy.compression.html index 7d61352..37409b8 100644 --- a/taoensso.nippy.compression.html +++ b/taoensso.nippy.compression.html @@ -1,6 +1,6 @@ -taoensso.nippy.benchmarks
taoensso.nippy.compression documentation Generated by Codox
Nippy 2.15.0
taoensso.nippy.compression
ICompressor
protocol
members
compress
(compress compressor ba)decompress
(decompress compressor ba)header-id
(header-id compressor)lz4-compressor
Default net.jpountz.lz4 compressor: +taoensso.nippy.compression documentation Generated by Codox
Nippy 2.15.1
taoensso.nippy.compression
ICompressor
protocol
members
compress
(compress compressor ba)decompress
(decompress compressor ba)header-id
(header-id compressor)lz4-compressor
Default net.jpountz.lz4 compressor: Ratio: low. Write speed: very high. Read speed: very high. diff --git a/taoensso.nippy.crypto.html b/taoensso.nippy.crypto.html index beb9355..581d491 100644 --- a/taoensso.nippy.crypto.html +++ b/taoensso.nippy.crypto.html @@ -1,6 +1,6 @@ -taoensso.nippy.crypto documentation Generated by Codox
Nippy 2.15.0
taoensso.nippy.crypto
Low-level crypto utils. +taoensso.nippy.crypto documentation Generated by Codox
Nippy 2.15.1
taoensso.nippy.crypto
Low-level crypto utils. Private & alpha, very likely to change!decrypt
(decrypt {:keys [cipher-kit salt-size salt->key-fn enc-ba], :or {cipher-kit cipher-kit-aes-gcm}})encrypt
(encrypt {:keys [cipher-kit ?salt-ba key-ba plain-ba rand-bytes-fn], :or {cipher-kit cipher-kit-aes-gcm, rand-bytes-fn rand-bytes}})ICipherKit
protocol
members
get-cipher
(get-cipher _)Returns a thread-safe `javax.crypto.Cipher` instance. diff --git a/taoensso.nippy.encryption.html b/taoensso.nippy.encryption.html index 6fddd25..4a9d770 100644 --- a/taoensso.nippy.encryption.html +++ b/taoensso.nippy.encryption.html @@ -1,6 +1,6 @@ -taoensso.nippy.encryption documentation Generated by Codox
Nippy 2.15.0
taoensso.nippy.encryption
Simple no-nonsense crypto with reasonable defaults +taoensso.nippy.encryption documentation Generated by Codox
Nippy 2.15.1
taoensso.nippy.encryption
Simple no-nonsense crypto with reasonable defaultsaes128-cbc-encryptor
Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen. See also `aes-128-cbc-encryptor`.aes128-gcm-encryptor
\ No newline at end of file diff --git a/taoensso.nippy.tools.html b/taoensso.nippy.tools.html index 1a737eb..d252d7d 100644 --- a/taoensso.nippy.tools.html +++ b/taoensso.nippy.tools.html @@ -1,6 +1,6 @@ -Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen. diff --git a/taoensso.nippy.html b/taoensso.nippy.html index 2734130..f6cff12 100644 --- a/taoensso.nippy.html +++ b/taoensso.nippy.html @@ -1,29 +1,61 @@ -taoensso.nippy documentation Generated by Codox
Nippy 2.15.0
taoensso.nippy
High-performance serialization library for Clojure +taoensso.nippy documentation Generated by Codox
Nippy 2.15.1
\ No newline at end of file +This is a low-level util: in most cases you'll want `thaw` instead.taoensso.nippy
High-performance serialization library for Clojure*auto-freeze-compressor*
dynamic
(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}), -nil => default*serializable-whitelist*
dynamic
Used when attempting to freeze or thaw an object that: +nil => default*serializable-whitelist*
dynamic
Used when attempting to freeze or thaw an object that: - Does not implement Nippy's Freezable protocol. - Does implement Java's Serializable interface. In this case, Java's Serializable interface will be permitted iff the predicate (*serializable-whitelist* <class-name>) returns true. -I.e. this is a predicate (fn [class-name]) that specifies whether -Nippy may use a given class's Serializable implementation as +I.e. this is a predicate (fn allow-class? [class-name]) that specifies +whether Nippy may use a given class's Serializable implementation as fallback when its own protocol is unfamiliar with the type. +If `thaw` encounters an unwhitelisted Serialized class: + - `thaw` will throw if it's not possible to safely quarantine. + - Otherwise the object will be thawed as: + `{:nippy/unthawable {:class-name _ :content <quarantined-ba> ...}}`. + This is a security measure to prevent Remote Code Execution (RCE). -Context: +Default value is a set containing a number of known-safe classes, +see `default-serializable-whitelist` for details. PRs welcome to add +additional known-safe classes to default. + +Value may be overridden with `swap-serializable-whitelist!` or with: + + - `taoensso.nippy.serializable-whitelist-base` JVM property + - `taoensso.nippy.serializable-whitelist-add` JVM property + + - `TAOENSSO_NIPPY_SERIALIZABLE_WHITELIST_BASE` env var + - `TAOENSSO_NIPPY_SERIALIZABLE_WHITELIST_ADD` env var + + If present, these will be read as comma-separated lists of class + names and formed into sets. Initial whitelist value will then be: + (into (or <?base> <default>) <?additions>). + + I.e. you can use: + - The "base" property/var to override Nippy's default whitelist. + - The "add" property/var to add to Nippy's default whitelist. + +Strings in sets may contain "*" wildcards. + +See also `taoensso.encore/compile-str-filter`, a util to help +easily build more advanced predicate functions. + +================ +Further context: Reading arbitrary Serializable classes can be dangerous if they come from an untrusted source. - Specifically: if your classpath contains a vulnerable ("gadget") + Specifically: if your classpath contains a vulnerable ("gadget")[2] class - it is possible for an attacker to produce an object that can run arbitrary code when read via Serializable. @@ -45,25 +77,28 @@ What to use as a whitelist? source, you can use `(constantly true)` as predicate. This will whitelist everything, allowing Serializable for ANY class. -Default value as of v2.15.0 is: #{}. +Upgrading from an older version of Nippy and not sure whether you've +been using Nippy's Serializable support? Here's a code snippet that +will allow AND RECORD any class using Nippy's Serializable fallback: -PRs welcome for additional known-safe classes to be added to default -whitelist. + ;; Deref for set of all class names that made use of Nippy's Serializable support: + (defonce observed-serializables_ (atom #{})) -Note: if `thaw` encounters an unwhitelisted Serialized class: - - - `thaw` will throw if it's not possible to safely quarantine. - - Otherwise the object will be thawed as: - `{:nippy/unthawable {:class-name _ :content <quarantined-ba> ...}}`. + (swap-serializable-whitelist! + (fn [_] + (fn allow-class? [class-name] + (swap! observed-serializables_ conj class-name) ; Record class name + true ; Allow any class + ))) Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report identifying this vulnerability. -See also `swap-serializable-whitelist!`. - -[1] https://groups.google.com/forum/#!msg/clojure/WaL3hHzsevI/7zHU-L7LBQAJaes128-cbc-encryptor
Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen. -See also `aes-128-cbc-encryptor`.aes128-encryptor
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen. +[1] https://groups.google.com/forum/#!msg/clojure/WaL3hHzsevI/7zHU-L7LBQAJ +[2] Jackson maintains a list of common gadget classes at + https://github.com/FasterXML/jackson-databind/blob/master/src/main/java/com/fasterxml/jackson/databind/jsontype/impl/SubTypeValidator.javaaes128-cbc-encryptor
Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen. +See also `aes-128-cbc-encryptor`.aes128-encryptor
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen. Password form [:salted "my-password"] --------------------------------------- @@ -96,7 +131,7 @@ CONS: Using a small number of keys for many encrypted items means that if any compromised. Faster than `aes128-salted`, and harder to attack any particular key - but -increased danger if a key is somehow compromised.aes128-gcm-encryptor
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen. +increased danger if a key is somehow compromised.aes128-gcm-encryptor
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen. Password form [:salted "my-password"] --------------------------------------- @@ -129,16 +164,18 @@ CONS: Using a small number of keys for many encrypted items means that if any compromised. Faster than `aes128-salted`, and harder to attack any particular key - but -increased danger if a key is somehow compromised.cache
(cache x)Experimental, subject to change. +increased danger if a key is somehow compromised.cache
(cache x)Experimental, subject to change. Wraps value so that future writes of the same wrapped value with same metadata will be efficiently encoded as references to this one. (freeze [(cache "foo") (cache "foo") (cache "foo")]) - will incl. a single "foo", plus 2x single-byte references to "foo".extend-freeze
macro
(extend-freeze type custom-type-id [x out] & body)Extends Nippy to support freezing of a custom type (ideally concrete) with + will incl. a single "foo", plus 2x single-byte references to "foo".default-serializable-whitelist
PRs welcome to add additional known-safe classes to default. +extend-freeze
macro
(extend-freeze type custom-type-id [x out] & body)Extends Nippy to support freezing of a custom type (ideally concrete) with given id of form: - * Keyword - 2 byte overhead, resistent to id collisions - * Integer ∈[1, 128] - no overhead, subject to id collisions + + * Keyword - 2 byte overhead, keywords hashed to 16 bit id + * ℕ∈[1, 128] - 0 byte overhead NB: be careful about extending to interfaces, Ref. http://goo.gl/6gGRlU. @@ -147,37 +184,37 @@ NB: be careful about extending to interfaces, Ref. view sourceextend-thaw
macro
(extend-thaw custom-type-id [in] & body)Extends Nippy to support thawing of a custom type with given id: + (.writeUTF [data-output] (:data x)))extend-thaw
macro
(extend-thaw custom-type-id [in] & body)Extends Nippy to support thawing of a custom type with given id: (extend-thaw :foo/my-type [data-input] ; Keyword id (MyRec. (.readUTF data-input))) ;; or (extend-thaw 1 [data-input] ; Byte id - (MyRec. (.readUTF data-input)))fast-freeze
(fast-freeze x)(fast-freeze x opts)Like `freeze` but: - Writes data without a Nippy header - Drops all support for compression and encryption - Must be thawed with `fast-thaw` Equivalent to (but a little faster than): - `(freeze x {:compressor nil :encryptor nil :no-header? true})fast-thaw
(fast-thaw ba)Like `thaw` but: + `(freeze x {:compressor nil :encryptor nil :no-header? true})fast-thaw
(fast-thaw ba)(fast-thaw ba opts)Like `thaw` but: - Drops all support for compression and encryption - Supports only data frozen with `fast-freeze` Equivalent to (but a little faster than): - `(thaw x {:compressor nil :encryptor nil :no-header? true})freezable?
(freezable? x)(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})Alpha - subject to change. + `(thaw x {:compressor nil :encryptor nil :no-header? true})freezable?
(freezable? x)(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})Alpha - subject to change. Returns truthy iff Nippy *appears* to support freezing the given argument. `:allow-clojure-reader?` and `:allow-java-serializable?` options may be used to enable the relevant roundtrip fallback test(s). These tests are only **moderately reliable** since they're cached by arg type and don't test for pre/post serialization value equality (there's no good general -way of doing so).freeze
(freeze x)(freeze x {:as opts, :keys [compressor encryptor password], :or {compressor :auto, encryptor aes128-gcm-encryptor}})Serializes arg (any Clojure data type) to a byte array. To freeze custom -types, extend the Clojure reader or see `extend-freeze`.freeze-to-file
(freeze-to-file file x)(freeze-to-file file x freeze-opts)Convenience util: like `freeze`, but writes to `(clojure.java.io/file <file>)` +way of doing so).freeze
(freeze x)(freeze x {:as opts, :keys [compressor encryptor password serializable-whitelist incl-metadata?], :or {compressor :auto, encryptor aes128-gcm-encryptor}})Serializes arg (any Clojure data type) to a byte array. To freeze custom +types, extend the Clojure reader or see `extend-freeze`.freeze-to-file
(freeze-to-file file x)(freeze-to-file file x freeze-opts)Convenience util: like `freeze`, but writes to `(clojure.java.io/file <file>)` and returns the byte array written. -See also `thaw-from-file`.freeze-to-out!
(freeze-to-out! data-output x)Serializes arg (any Clojure data type) to a DataOutput. -This is a low-level util: in most cases you'll want `freeze` instead.freeze-to-string
(freeze-to-string x)(freeze-to-string x freeze-opts)Convenience util: like `freeze`, but returns a Base64-encoded string. -See also `thaw-from-string`.freeze-to-out!
(freeze-to-out! data-output x)Serializes arg (any Clojure data type) to a DataOutput. +This is a low-level util: in most cases you'll want `freeze` instead.freeze-to-string
(freeze-to-string x)(freeze-to-string x freeze-opts)Convenience util: like `freeze`, but returns a Base64-encoded string. +See also `thaw-from-string`.lz4-compressor
Default net.jpountz.lz4 compressor: Ratio: low. Write speed: very high. Read speed: very high. @@ -185,30 +222,33 @@ Write speed: very high. A good general-purpose compressor, competitive with Snappy. Thanks to Max Penet (@mpenet) for our first implementation, -Ref. https://github.com/mpenet/nippy-lz4lzma2-compressor
Default org.tukaani.xz.LZMA2 compressor: +Ref. https://github.com/mpenet/nippy-lz4lzma2-compressor
Default org.tukaani.xz.LZMA2 compressor: Ratio: high. Write speed: _very_ slow (also currently single-threaded). Read speed: slow. A specialized compressor for large, low-write data in space-sensitive -environments.snappy-compressor
Default org.iq80.snappy.Snappy compressor: Ratio: low. Write speed: very high. Read speed: very high. -A good general-purpose compressor.stress-data-benchable
Reference data with stuff removed that breaks reader or other utils we'll -be benching againstswap-serializable-whitelist!
(swap-serializable-whitelist! f)Changes root `*serializable-whitelist*` value to (f old-val). +A good general-purpose compressor.stress-data-benchable
Reference data with stuff removed that breaks reader or other utils we'll +be benching againstswap-serializable-whitelist!
(swap-serializable-whitelist! f)Changes root `*serializable-whitelist*` value to (f old-val). Example `f` arguments: - (fn [_old] true) ; Whitelist everything (allow all classes) - (fn [_old] #{}) ; Whitelist nothing (disallow all classes) - - (fn [_old] #{"java.lang.Throwable"}) ; Reset class whitelist - - (fn [ old] (conj old "java.lang.Throwable"))) ; Add class to whitelist + - (fn [_old] #{"java.lang.Throwable"}) ; Reset class whitelist set + - (fn [ old] (conj old "java.lang.Throwable"))) ; Add class to whitelist set + - (fn [ old] (conj old "java.lang.*")) ; Add classes to whitelist set (note wildcard) -See also `*serializable-whitelist*.thaw
(thaw ba)(thaw ba {:as opts, :keys [v1-compatibility? compressor encryptor password], :or {compressor :auto, encryptor :auto}})Deserializes a frozen Nippy byte array to its original Clojure data type. +Strings in sets may contain "*" wildcards. + +See also `*serializable-whitelist*.thaw
(thaw ba)(thaw ba {:as opts, :keys [v1-compatibility? compressor encryptor password serializable-whitelist incl-metadata?], :or {compressor :auto, encryptor :auto}})Deserializes a frozen Nippy byte array to its original Clojure data type. To thaw custom types, extend the Clojure reader or see `extend-thaw`. ** By default, supports data frozen with Nippy v2+ ONLY ** @@ -218,13 +258,13 @@ legacy versions of Nippy. Options include: :v1-compatibility? - support data frozen by legacy versions of Nippy? :compressor - :auto (checks header, default) an ICompressor, or nil - :encryptor - :auto (checks header, default), an IEncryptor, or nilthaw-from-file
(thaw-from-file file)(thaw-from-file file thaw-opts)Convenience util: like `thaw`, but reads from `(clojure.java.io/file <file>)`. + :encryptor - :auto (checks header, default), an IEncryptor, or nilthaw-from-file
(thaw-from-file file)(thaw-from-file file thaw-opts)Convenience util: like `thaw`, but reads from `(clojure.java.io/file <file>)`. To thaw from a resource on classpath (e.g in Leiningen `resources` dir): (thaw-from-file (clojure.java.io/resource "my-resource-name.npy")) -See also `freeze-to-file`.thaw-from-in!
(thaw-from-in! data-input)Deserializes a frozen object from given DataInput to its original Clojure +See also `freeze-to-file`.thaw-from-in!
(thaw-from-in! data-input)Deserializes a frozen object from given DataInput to its original Clojure data type. -This is a low-level util: in most cases you'll want `thaw` instead.thaw-from-string
(thaw-from-string s)(thaw-from-string s thaw-opts)Convenience util: like `thaw`, but takes a Base64-encoded string. -See also `freeze-to-string`.thaw-from-string
(thaw-from-string s)(thaw-from-string s thaw-opts)Convenience util: like `thaw`, but takes a Base64-encoded string. +See also `freeze-to-string`.taoensso.nippy.tools documentation Generated by Codox
Nippy 2.15.0
taoensso.nippy.tools
Utils for 3rd-party tools that want to add user-configurable Nippy support. +taoensso.nippy.tools documentation Generated by Codox
Nippy 2.15.1
taoensso.nippy.tools
Utils for 3rd-party tools that want to add user-configurable Nippy support. Used by Carmine, Faraday, etc.freeze
(freeze x)(freeze x default-opts)Like `nippy/freeze` but merges opts from *freeze-opts*, `wrap-for-freezing`.thaw
(thaw ba)(thaw ba default-opts)Like `nippy/thaw` but merges opts from `*thaw-opts*`.wrap-for-freezing
(wrap-for-freezing x)(wrap-for-freezing x opts)Ensures that given arg (any freezable data type) is wrapped so that diff --git a/taoensso.nippy.utils.html b/taoensso.nippy.utils.html index 6d39943..1793a5e 100644 --- a/taoensso.nippy.utils.html +++ b/taoensso.nippy.utils.html @@ -1,6 +1,6 @@ -taoensso.nippy.utils documentation Generated by Codox
Nippy 2.15.0
taoensso.nippy.utils
freezable?
(freezable? x)(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})Alpha - subject to change. +taoensso.nippy.utils documentation Generated by Codox
Nippy 2.15.1
taoensso.nippy.utils
freezable?
(freezable? x)(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})Alpha - subject to change. Returns truthy iff Nippy *appears* to support freezing the given argument. `:allow-clojure-reader?` and `:allow-java-serializable?` options may be