From 057dfc31c559b22d15c786375c27cb78117e3473 Mon Sep 17 00:00:00 2001 From: Peter Taoussanis Date: Fri, 11 Sep 2020 13:05:46 +0200 Subject: [PATCH] v3.0.0-RC1 2020-09-11 --- index.html | 2 +- taoensso.nippy.benchmarks.html | 2 +- taoensso.nippy.compression.html | 2 +- taoensso.nippy.crypto.html | 2 +- taoensso.nippy.encryption.html | 2 +- taoensso.nippy.html | 231 ++++++++++++++++++++------------ taoensso.nippy.tools.html | 2 +- taoensso.nippy.utils.html | 2 +- 8 files changed, 151 insertions(+), 94 deletions(-) diff --git a/index.html b/index.html index f0fc4d6..7a5e79b 100644 --- a/index.html +++ b/index.html @@ -1,5 +1,5 @@ -Nippy 2.15.3

Project

Namespaces

Nippy 2.15.3

Released under the Eclipse Public License

High-performance serialization library for Clojure.

Installation

To install, add the following dependency to your project or build file:

[com.taoensso/nippy "2.15.3"]

Namespaces

taoensso.nippy

High-performance serialization library for Clojure

Public variables and functions:

taoensso.nippy.benchmarks

Public variables and functions:
taoensso.nippy.compression
Public variables and functions:
taoensso.nippy.crypto
Low-level crypto utils.
+Nippy 3.0.0-RC1
Project
Namespaces
Nippy 3.0.0-RC1
Released under the Eclipse Public License
High-performance serialization library for Clojure.
Installation
To install, add the following dependency to your project or build file:
[com.taoensso/nippy "3.0.0-RC1"]
Namespaces
taoensso.nippy
High-performance serialization library for Clojure
Public variables and functions:
taoensso.nippy.benchmarks
Public variables and functions:
taoensso.nippy.compression
Public variables and functions:
taoensso.nippy.crypto
Low-level crypto utils.
 Private & alpha, very likely to change!
Public variables and functions:
taoensso.nippy.encryption
Simple no-nonsense crypto with reasonable defaults
Public variables and functions:
taoensso.nippy.tools
Utils for 3rd-party tools that want to add user-configurable Nippy support.
 Used by Carmine, Faraday, etc.
Public variables and functions:
taoensso.nippy.utils
Public variables and functions:

\ No newline at end of file
diff --git a/taoensso.nippy.benchmarks.html b/taoensso.nippy.benchmarks.html
index 6086a2f..176b7a7 100644
--- a/taoensso.nippy.benchmarks.html
+++ b/taoensso.nippy.benchmarks.html
@@ -1,3 +1,3 @@
 
-taoensso.nippy.benchmarks documentation
Project
Namespaces
Public Vars
taoensso.nippy.benchmarks
bench
(bench {:keys [reader? lzma2? fressian? laps], :or {laps 1}})
bench*
macro
(bench* & body)
bench1
(bench1 freezer thawer & [sizer])
data
fressian-freeze
(fressian-freeze value)
fressian-thaw
(fressian-thaw value)

\ No newline at end of file
+taoensso.nippy.benchmarks documentation
Project
Namespaces
Public Vars
taoensso.nippy.benchmarks
bench
(bench {:keys [reader? lzma2? fressian? laps], :or {laps 1}})
bench*
macro
(bench* & body)
bench1
(bench1 freezer thawer & [sizer])
data
fressian-freeze
(fressian-freeze value)
fressian-thaw
(fressian-thaw value)

\ No newline at end of file
diff --git a/taoensso.nippy.compression.html b/taoensso.nippy.compression.html
index 9dcf479..ab755a3 100644
--- a/taoensso.nippy.compression.html
+++ b/taoensso.nippy.compression.html
@@ -1,6 +1,6 @@
 
-taoensso.nippy.compression documentation
Project
Namespaces
Public Vars
taoensso.nippy.compression
ICompressor
protocol
members
compress
(compress compressor ba)
decompress
(decompress compressor ba)
header-id
(header-id compressor)
lz4-compressor
Default net.jpountz.lz4 compressor:
+taoensso.nippy.compression documentation
Project
Namespaces
Public Vars
taoensso.nippy.compression
ICompressor
protocol
members
compress
(compress compressor ba)
decompress
(decompress compressor ba)
header-id
(header-id compressor)
lz4-compressor
Default net.jpountz.lz4 compressor:
       Ratio: low.
 Write speed: very high.
  Read speed: very high.
diff --git a/taoensso.nippy.crypto.html b/taoensso.nippy.crypto.html
index 063d11e..9a6c995 100644
--- a/taoensso.nippy.crypto.html
+++ b/taoensso.nippy.crypto.html
@@ -1,6 +1,6 @@
 
-taoensso.nippy.crypto documentation
Project
Namespaces
Public Vars
taoensso.nippy.crypto
Low-level crypto utils.
+taoensso.nippy.crypto documentation
Project
Namespaces
Public Vars
taoensso.nippy.crypto
Low-level crypto utils.
 Private & alpha, very likely to change!
cbc-cipher
(cbc-cipher)
cipher-kit-aes-cbc
Default CipherKit for AES CBC
 
cipher-kit-aes-gcm
Default CipherKit for AES GCM
 
decrypt
(decrypt {:keys [cipher-kit salt-size salt->key-fn enc-ba], :or {cipher-kit cipher-kit-aes-gcm}})
encrypt
(encrypt {:keys [cipher-kit ?salt-ba key-ba plain-ba rand-bytes-fn], :or {cipher-kit cipher-kit-aes-gcm, rand-bytes-fn rand-bytes}})
gcm-cipher
(gcm-cipher)
ICipherKit
protocol
members
get-cipher
(get-cipher _)
Returns a thread-safe `javax.crypto.Cipher` instance.
diff --git a/taoensso.nippy.encryption.html b/taoensso.nippy.encryption.html
index 3363fd6..7dc67e6 100644
--- a/taoensso.nippy.encryption.html
+++ b/taoensso.nippy.encryption.html
@@ -1,6 +1,6 @@
 
-taoensso.nippy.encryption documentation
Project
Namespaces
Public Vars
taoensso.nippy.encryption
Simple no-nonsense crypto with reasonable defaults
+taoensso.nippy.encryption documentation
Project
Namespaces
Public Vars
taoensso.nippy.encryption
Simple no-nonsense crypto with reasonable defaults
 
aes128-cbc-encryptor
Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen.
 See also `aes-128-cbc-encryptor`.
aes128-gcm-encryptor
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
 
diff --git a/taoensso.nippy.html b/taoensso.nippy.html
index 0f67149..73a01fd 100644
--- a/taoensso.nippy.html
+++ b/taoensso.nippy.html
@@ -1,101 +1,156 @@
 
-taoensso.nippy documentation
Project
Namespaces
Public Vars
taoensso.nippy
High-performance serialization library for Clojure
+taoensso.nippy documentation
Project
Namespaces
Public Vars
taoensso.nippy
High-performance serialization library for Clojure
 
*auto-freeze-compressor*
dynamic
(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}),
 nil => default
*custom-readers*
dynamic
{<hash-or-byte-id> (fn [data-input])->read}
 
*final-freeze-fallback*
dynamic
DEPRECATED: prefer `*freeze-fallback`.
 
*freeze-fallback*
dynamic
(fn [data-output x])->freeze, nil => default
-
*serializable-whitelist*
dynamic
Used when attempting to freeze or thaw an object that:
+
*freeze-serializable-allowlist*
dynamic
Used when attempting to <freeze/thaw> an object that:
   - Does not implement Nippy's Freezable    protocol.
   - Does     implement Java's  Serializable interface.
 
 In this case, Java's Serializable interface will be permitted iff
-the predicate (*serializable-whitelist* <class-name>) returns true.
+(<allowlist> <class-name>) predicate call returns true.
 
-I.e. this is a predicate (fn allow-class? [class-name]) that specifies
-whether Nippy may use a given class's Serializable implementation as
-fallback when its own protocol is unfamiliar with the type.
+This is a security measure to prevent possible Remote Code Execution
+(RCE) when thawing malicious payloads. See [1] for details.
 
-If `thaw` encounters an unwhitelisted Serialized class:
-  - `thaw` will throw if it's not possible to safely quarantine.
-  - Otherwise the object will be thawed as:
-    `{:nippy/unthawable {:class-name _ :content <quarantined-ba> ...}}`.
+If `freeze` encounters a disallowed Serialized class, it will throw.
+If `thaw`   encounters a disallowed Serialized class, it will:
 
-This is a security measure to prevent Remote Code Execution (RCE).
+  - Throw if it's not possible to safely quarantine the object
+    (object was frozen with Nippy < v2.15.0-final).
 
-Default value is a set containing a number of known-safe classes,
-see `default-serializable-whitelist` for details. PRs welcome to add
-additional known-safe classes to default.
+  - Otherwise it will return a safely quarantined object of form
+    `{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}`.
+    - Quarantined objects may be manually unquarantined with
+      `read-quarantined-serializable-object-unsafe!`.
 
-Value may be overridden with `swap-serializable-whitelist!` or with:
+There are 2x allowlists: *<freeze/thaw>-serializable-allowlist*.
 
-  - `taoensso.nippy.serializable-whitelist-base` JVM property
-  - `taoensso.nippy.serializable-whitelist-add`  JVM property
+Example values:
+  - (fn allow-class? [class-name] true)            ; Arbitrary fn
+  - #{"java.lang.Throwable", "clojure.lang.*"} ; Set of class-names
 
-  - `TAOENSSO_NIPPY_SERIALIZABLE_WHITELIST_BASE` env var
-  - `TAOENSSO_NIPPY_SERIALIZABLE_WHITELIST_ADD`  env var
+  Note that class-names in sets may contain "*" wildcards.
 
-  If present, these will be read as comma-separated lists of class
-  names and formed into sets. Initial whitelist value will then be:
-  (into (or <?base> <default>) <?additions>).
+Default allowlist values are:
+  - default-freeze-serializable-allowlist ; {"*"} => allow any class
+  -   default-thaw-serializable-allowlist ; A set of common safe classes
+
+Allowlist values may be overridden with `binding`, `alter-var-root`, or:
+
+  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property
+  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add`  JVM property
+
+  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` env var
+  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD`  env var
+
+If present, these will be read as comma-separated lists of class names
+and formed into sets. Each initial allowlist value will then be:
+(into (or <?base> <default>) <?additions>).
 
   I.e. you can use:
-    - The "base" property/var to override Nippy's default whitelist.
-    - The "add"  property/var to add to   Nippy's default whitelist.
-
-Strings in sets may contain "*" wildcards.
+    - The "base" property/var to replace Nippy's default allowlists.
+    - The "add"  property/var to add to  Nippy's default allowlists.
 
 See also `taoensso.encore/compile-str-filter`, a util to help
 easily build more advanced predicate functions.
 
-================
-Further context:
 
-  Reading arbitrary Serializable classes can be dangerous if they
-  come from an untrusted source.
-
-  Specifically: if your classpath contains a vulnerable ("gadget")[2]
-  class - it is possible for an attacker to produce an object that
-  can run arbitrary code when read via Serializable.
-
-  Note that Clojure <= 1.8 itself contains such a class [1].
-
-What to use as a whitelist?
-
-  1. If you DO NOT wish to support Serializable: `#{}` is safest,
-     and just entirely disallows its use.
-
-  2. If you DO with to support Serializable:
-
-    2a. If you might serialize data from an untrusted source, or
-        if you'll only be serializing a limited number of known
-        classes: enumerate those class names, e.g.:
-          `#{"java.lang.Throwable", ...}`.
-
-    2b. If you're CERTAIN to NEVER serialize data from an untrusted
-        source, you can use `(constantly true)` as predicate. This
-        will whitelist everything, allowing Serializable for ANY class.
-
-Upgrading from an older version of Nippy and not sure whether you've
-been using Nippy's Serializable support? Here's a code snippet that
-will allow AND RECORD any class using Nippy's Serializable fallback:
+Upgrading from an older version of Nippy and unsure whether you've been
+using Nippy's Serializable support? Here's a snippet to ALLOW and RECORD
+any class requesting Nippy's Serializable fallback:
 
   ;; Deref for set of all class names that made use of Nippy's Serializable support:
   (defonce observed-serializables_ (atom #{}))
 
-  (swap-serializable-whitelist!
-    (fn [_]
-      (fn allow-class? [class-name]
-        (swap! observed-serializables_ conj class-name) ; Record class name
-        true ; Allow any class
-        )))
+  (let [f (fn allow-class? [class-name]
+            (swap! observed-serializables_ conj class-name) ; Record class name
+            true ; Allow any class
+            )]
+
+    (alter-var-root #'*freeze-serializable-allowlist* (fn [_] f))
+    (alter-var-root   #'*thaw-serializable-allowlist* (fn [_] f)))
+
 
 Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
 identifying this vulnerability.
 
-[1] https://groups.google.com/forum/#!msg/clojure/WaL3hHzsevI/7zHU-L7LBQAJ
-[2] Jackson maintains a list of common gadget classes at
-  https://github.com/FasterXML/jackson-databind/blob/master/src/main/java/com/fasterxml/jackson/databind/jsontype/impl/SubTypeValidator.java
-cache-proxy
{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing.
+[1] https://github.com/ptaoussanis/nippy/issues/130
*incl-metadata?*
dynamic
Include metadata when freezing/thawing?
+
*thaw-serializable-allowlist*
dynamic
Used when attempting to <freeze/thaw> an object that:
+  - Does not implement Nippy's Freezable    protocol.
+  - Does     implement Java's  Serializable interface.
+
+In this case, Java's Serializable interface will be permitted iff
+(<allowlist> <class-name>) predicate call returns true.
+
+This is a security measure to prevent possible Remote Code Execution
+(RCE) when thawing malicious payloads. See [1] for details.
+
+If `freeze` encounters a disallowed Serialized class, it will throw.
+If `thaw`   encounters a disallowed Serialized class, it will:
+
+  - Throw if it's not possible to safely quarantine the object
+    (object was frozen with Nippy < v2.15.0-final).
+
+  - Otherwise it will return a safely quarantined object of form
+    `{:nippy/unthawable {:class-name <> :content <quarantined-ba>}}`.
+    - Quarantined objects may be manually unquarantined with
+      `read-quarantined-serializable-object-unsafe!`.
+
+There are 2x allowlists: *<freeze/thaw>-serializable-allowlist*.
+
+Example values:
+  - (fn allow-class? [class-name] true)            ; Arbitrary fn
+  - #{"java.lang.Throwable", "clojure.lang.*"} ; Set of class-names
+
+  Note that class-names in sets may contain "*" wildcards.
+
+Default allowlist values are:
+  - default-freeze-serializable-allowlist ; {"*"} => allow any class
+  -   default-thaw-serializable-allowlist ; A set of common safe classes
+
+Allowlist values may be overridden with `binding`, `alter-var-root`, or:
+
+  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property
+  - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add`  JVM property
+
+  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` env var
+  - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD`  env var
+
+If present, these will be read as comma-separated lists of class names
+and formed into sets. Each initial allowlist value will then be:
+(into (or <?base> <default>) <?additions>).
+
+  I.e. you can use:
+    - The "base" property/var to replace Nippy's default allowlists.
+    - The "add"  property/var to add to  Nippy's default allowlists.
+
+See also `taoensso.encore/compile-str-filter`, a util to help
+easily build more advanced predicate functions.
+
+
+Upgrading from an older version of Nippy and unsure whether you've been
+using Nippy's Serializable support? Here's a snippet to ALLOW and RECORD
+any class requesting Nippy's Serializable fallback:
+
+  ;; Deref for set of all class names that made use of Nippy's Serializable support:
+  (defonce observed-serializables_ (atom #{}))
+
+  (let [f (fn allow-class? [class-name]
+            (swap! observed-serializables_ conj class-name) ; Record class name
+            true ; Allow any class
+            )]
+
+    (alter-var-root #'*freeze-serializable-allowlist* (fn [_] f))
+    (alter-var-root   #'*thaw-serializable-allowlist* (fn [_] f)))
+
+
+Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
+identifying this vulnerability.
+
+[1] https://github.com/ptaoussanis/nippy/issues/130
-cache-proxy
{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing.
 
aes128-cbc-encryptor
Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen.
 See also `aes-128-cbc-encryptor`.
aes128-encryptor
Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
 
@@ -169,8 +224,11 @@ Wraps value so that future writes of the same wrapped value with same
 metadata will be efficiently encoded as references to this one.
 
 (freeze [(cache "foo") (cache "foo") (cache "foo")])
-  will incl. a single "foo", plus 2x single-byte references to "foo".
compress
(compress compressor ba)
decompress
(decompress compressor ba)
decrypt
(decrypt encryptor pwd ba)
default-serializable-whitelist
PRs welcome to add additional known-safe classes to default.
-
encrypt
(encrypt encryptor pwd ba)
extend-freeze
macro
(extend-freeze type custom-type-id [x out] & body)
Extends Nippy to support freezing of a custom type (ideally concrete) with
+  will incl. a single "foo", plus 2x single-byte references to "foo".
compress
(compress compressor ba)
decompress
(decompress compressor ba)
decrypt
(decrypt encryptor pwd ba)
default-freeze-serializable-allowlist
Allows *any* class-name to be frozen using Java's Serializable interface.
+This is generally safe since RCE risk is present only when thawing.
+See also `*freeze-serializable-allowlist*`.
default-thaw-serializable-allowlist
A set of common safe class-names to allow to be frozen using Java's
+Serializable interface. PRs welcome for additions.
+See also `*thaw-serializable-allowlist*`.
encrypt
(encrypt encryptor pwd ba)
extend-freeze
macro
(extend-freeze type custom-type-id [x out] & body)
Extends Nippy to support freezing of a custom type (ideally concrete) with
 given id of form:
 
   * Keyword    - 2 byte overhead, keywords hashed to 16 bit id
@@ -193,22 +251,25 @@ NB: be careful about extending to interfaces, Ref. 
fast-thaw
(fast-thaw ba)
Like `thaw` but:
+Equivalent to (but a little faster than) `freeze` with opts:
+  - :compressor nil
+  - :encryptor  nil
+  - :no-header? true
thaw
(thaw ba)(thaw ba {:as opts, :keys [v1-compatibility? compressor encryptor password serializable-allowlist incl-metadata?], :or {compressor :auto, encryptor :auto}})
Deserializes a frozen Nippy byte array to its original Clojure data type.
 To thaw custom types, extend the Clojure reader or see `extend-thaw`.
 
 ** By default, supports data frozen with Nippy v2+ ONLY **
diff --git a/taoensso.nippy.tools.html b/taoensso.nippy.tools.html
index 02038f5..f11ce02 100644
--- a/taoensso.nippy.tools.html
+++ b/taoensso.nippy.tools.html
@@ -1,6 +1,6 @@
 
-taoensso.nippy.tools documentation
Project
Namespaces
Public Vars
taoensso.nippy.tools
Utils for 3rd-party tools that want to add user-configurable Nippy support.
+taoensso.nippy.tools documentation
Project
Namespaces
Public Vars
taoensso.nippy.tools
Utils for 3rd-party tools that want to add user-configurable Nippy support.
 Used by Carmine, Faraday, etc.
*freeze-opts*
dynamic
*thaw-opts*
dynamic
freeze
(freeze x)(freeze x default-opts)
Like `nippy/freeze` but merges opts from *freeze-opts*, `wrap-for-freezing`.
 
thaw
(thaw ba)(thaw ba default-opts)
Like `nippy/thaw` but merges opts  from `*thaw-opts*`.
 
with-freeze-opts
macro
(with-freeze-opts opts & body)
with-thaw-opts
macro
(with-thaw-opts opts & body)
wrap-for-freezing
(wrap-for-freezing x)(wrap-for-freezing x opts)
Ensures that given arg (any freezable data type) is wrapped so that
diff --git a/taoensso.nippy.utils.html b/taoensso.nippy.utils.html
index 9d2a2d0..bdcc55e 100644
--- a/taoensso.nippy.utils.html
+++ b/taoensso.nippy.utils.html
@@ -1,6 +1,6 @@
 
-taoensso.nippy.utils documentation
Project
Namespaces
Public Vars
taoensso.nippy.utils
freezable?
(freezable? x)(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})
Alpha - subject to change.
+taoensso.nippy.utils documentation
Project
Namespaces
Public Vars
taoensso.nippy.utils
freezable?
(freezable? x)(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})
Alpha - subject to change.
 Returns truthy iff Nippy *appears* to support freezing the given argument.
 
 `:allow-clojure-reader?` and `:allow-java-serializable?` options may be