taoensso.nippy

High-performance serialization library for Clojure

*auto-freeze-compressor*

dynamic

(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}),
nil => default

*custom-readers*

dynamic

{<hash-or-byte-id> (fn [data-input])->read}

*final-freeze-fallback*

dynamic

DEPRECATED: prefer `*freeze-fallback`.

*freeze-fallback*

dynamic

(fn [data-output x])->freeze, nil => default

*serializable-whitelist*

dynamic

Used when attempting to freeze or thaw an object that:
  - Does not implement Nippy's Freezable    protocol.
  - Does     implement Java's  Serializable interface.

In this case, Java's Serializable interface will be permitted iff
the predicate (*serializable-whitelist* <class-name>) returns true.

I.e. this is a predicate (fn [class-name]) that specifies whether
Nippy may use a given class's Serializable implementation as
fallback when its own protocol is unfamiliar with the type.

This is a security measure to prevent Remote Code Execution (RCE).

Context:

  Reading arbitrary Serializable classes can be dangerous if they
  come from an untrusted source.

  Specifically: if your classpath contains a vulnerable ("gadget")
  class - it is possible for an attacker to produce an object that
  can run arbitrary code when read via Serializable.

  Note that Clojure <= 1.8 itself contains such a class [1].

What to use as a whitelist?

  1. If you DO NOT wish to support Serializable: `#{}` is safest,
     and just entirely disallows its use.

  2. If you DO with to support Serializable:

    2a. If you might serialize data from an untrusted source, or
        if you'll only be serializing a limited number of known
        classes: enumerate those class names, e.g.:
          `#{"java.lang.Throwable", ...}`.

    2b. If you're CERTAIN to NEVER serialize data from an untrusted
        source, you can use `(constantly true)` as predicate. This
        will whitelist everything, allowing Serializable for ANY class.

Default value as of v2.15.0 is: #{}.

PRs welcome for additional known-safe classes to be added to default
whitelist.

Note: if `thaw` encounters an unwhitelisted Serialized class:

  - `thaw` will throw if it's not possible to safely quarantine.
  - Otherwise the object will be thawed as:
    `{:nippy/unthawable {:class-name _ :content <quarantined-ba> ...}}`.

Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
identifying this vulnerability.

See also `swap-serializable-whitelist!`.

[1] https://groups.google.com/forum/#!msg/clojure/WaL3hHzsevI/7zHU-L7LBQAJ

-cache-proxy

{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing.

aes128-cbc-encryptor

Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen.
See also `aes-128-cbc-encryptor`.

aes128-encryptor

Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.

Password form [:salted "my-password"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
          item encrypted will use a unique user-provided password).

IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
                every key.

PROS: Each key is independent so would need to be attacked independently.
CONS: Key caching impossible, so there's an inherent trade-off between
      encryption/decryption speed and the difficulty of attacking any
      particular key.

Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.

Password form [:cached "my-password"]
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
          number of staff/admins, or you'll be using a single password to
          encrypt many items).

IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.

PROS: Great amortized encryption/decryption speed. Expensive key hash makes
      attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
      key _is_ somehow compromised, _all_ items encrypted with that key are
      compromised.

Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised.

aes128-gcm-encryptor

Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.

Password form [:salted "my-password"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
          item encrypted will use a unique user-provided password).

IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
                every key.

PROS: Each key is independent so would need to be attacked independently.
CONS: Key caching impossible, so there's an inherent trade-off between
      encryption/decryption speed and the difficulty of attacking any
      particular key.

Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.

Password form [:cached "my-password"]
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
          number of staff/admins, or you'll be using a single password to
          encrypt many items).

IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.

PROS: Great amortized encryption/decryption speed. Expensive key hash makes
      attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
      key _is_ somehow compromised, _all_ items encrypted with that key are
      compromised.

Faster than `aes128-salted`, and harder to attack any particular key - but
increased danger if a key is somehow compromised.

cache

(cache x)
Experimental, subject to change.

Wraps value so that future writes of the same wrapped value with same
metadata will be efficiently encoded as references to this one.

(freeze [(cache "foo") (cache "foo") (cache "foo")])
  will incl. a single "foo", plus 2x single-byte references to "foo".

compress

(compress compressor ba)

decompress

(decompress compressor ba)

decrypt

(decrypt encryptor pwd ba)

encrypt

(encrypt encryptor pwd ba)

extend-freeze

macro

(extend-freeze type custom-type-id [x out] & body)
Extends Nippy to support freezing of a custom type (ideally concrete) with
given id of form:
  * Keyword           - 2 byte overhead, resistent to id collisions
  * Integer ∈[1, 128] - no overhead, subject to id collisions

NB: be careful about extending to interfaces, Ref. http://goo.gl/6gGRlU.

(defrecord MyRec [data])
(extend-freeze MyRec :foo/my-type [x data-output] ; Keyword id
  (.writeUTF [data-output] (:data x)))
;; or
(extend-freeze MyRec 1 [x data-output] ; Byte id
  (.writeUTF [data-output] (:data x)))

extend-thaw

macro

(extend-thaw custom-type-id [in] & body)
Extends Nippy to support thawing of a custom type with given id:
(extend-thaw :foo/my-type [data-input] ; Keyword id
  (MyRec. (.readUTF data-input)))
;; or
(extend-thaw 1 [data-input] ; Byte id
  (MyRec. (.readUTF data-input)))

fast-freeze

(fast-freeze x)
Like `freeze` but:
  - Writes data without a Nippy header
  - Drops all support for compression and encryption
  - Must be thawed with `fast-thaw`

Equivalent to (but a little faster than):
  `(freeze x {:compressor nil :encryptor nil :no-header? true})

fast-thaw

(fast-thaw ba)
Like `thaw` but:
  - Drops all support for compression and encryption
  - Supports only data frozen with `fast-freeze`

Equivalent to (but a little faster than):
  `(thaw x {:compressor nil :encryptor nil :no-header? true})

freezable?

(freezable? x)(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})
Alpha - subject to change.
Returns truthy iff Nippy *appears* to support freezing the given argument.

`:allow-clojure-reader?` and `:allow-java-serializable?` options may be
used to enable the relevant roundtrip fallback test(s). These tests are
only **moderately reliable** since they're cached by arg type and don't
test for pre/post serialization value equality (there's no good general
way of doing so).

freeze

(freeze x)(freeze x {:as opts, :keys [compressor encryptor password], :or {compressor :auto, encryptor aes128-gcm-encryptor}})
Serializes arg (any Clojure data type) to a byte array. To freeze custom
types, extend the Clojure reader or see `extend-freeze`.

freeze-fallback-as-str

DEPRECATED

freeze-to-file

(freeze-to-file file x)(freeze-to-file file x freeze-opts)
Convenience util: like `freeze`, but writes to `(clojure.java.io/file <file>)`
and returns the byte array written.
See also `thaw-from-file`.

freeze-to-out!

(freeze-to-out! data-output x)
Serializes arg (any Clojure data type) to a DataOutput.
This is a low-level util: in most cases you'll want `freeze` instead.

freeze-to-string

(freeze-to-string x)(freeze-to-string x freeze-opts)
Convenience util: like `freeze`, but returns a Base64-encoded string.
See also `thaw-from-string`.

IFreezable1

protocol

members

-freeze-without-meta!

(-freeze-without-meta! x data-output)

IFreezable2

protocol

members

-freeze-with-meta!

(-freeze-with-meta! x data-output)

inspect-ba

(inspect-ba ba)(inspect-ba ba thaw-opts)
Alpha - subject to change

lz4-compressor

Default net.jpountz.lz4 compressor:
      Ratio: low.
Write speed: very high.
 Read speed: very high.

A good general-purpose compressor, competitive with Snappy.

Thanks to Max Penet (@mpenet) for our first implementation,
Ref. https://github.com/mpenet/nippy-lz4

lz4hc-compressor

Like `lz4-compressor` but trades some write speed for ratio.

lzma2-compressor

Default org.tukaani.xz.LZMA2 compressor:
      Ratio: high.
Write speed: _very_ slow (also currently single-threaded).
 Read speed: slow.

A specialized compressor for large, low-write data in space-sensitive
environments.

set-auto-freeze-compressor!

(set-auto-freeze-compressor! x)

set-freeze-fallback!

(set-freeze-fallback! x)

snappy-compressor

Default org.iq80.snappy.Snappy compressor:
      Ratio: low.
Write speed: very high.
 Read speed: very high.

A good general-purpose compressor.

stress-data

Reference data used for tests & benchmarks

stress-data-benchable

Reference data with stuff removed that breaks reader or other utils we'll
be benching against

stress-data-comparable

Reference data with stuff removed that breaks roundtrip equality

swap-custom-readers!

(swap-custom-readers! f)

swap-serializable-whitelist!

(swap-serializable-whitelist! f)
Changes root `*serializable-whitelist*` value to (f old-val).
Example `f` arguments:

  - (fn [_old] true)                                ; Whitelist everything (allow    all classes)
  - (fn [_old] #{})                                 ; Whitelist nothing    (disallow all classes)
  - (fn [_old] #{"java.lang.Throwable"})          ; Reset class    whitelist
  - (fn [ old] (conj old "java.lang.Throwable"))) ; Add   class to whitelist

See also `*serializable-whitelist*.

thaw

(thaw ba)(thaw ba {:as opts, :keys [v1-compatibility? compressor encryptor password], :or {compressor :auto, encryptor :auto}})
Deserializes a frozen Nippy byte array to its original Clojure data type.
To thaw custom types, extend the Clojure reader or see `extend-thaw`.

** By default, supports data frozen with Nippy v2+ ONLY **
Add `{:v1-compatibility? true}` option to support thawing of data frozen with
legacy versions of Nippy.

Options include:
  :v1-compatibility? - support data frozen by legacy versions of Nippy?
  :compressor - :auto (checks header, default)  an ICompressor, or nil
  :encryptor  - :auto (checks header, default), an IEncryptor,  or nil

thaw-from-file

(thaw-from-file file)(thaw-from-file file thaw-opts)
Convenience util: like `thaw`, but reads from `(clojure.java.io/file <file>)`.

To thaw from a resource on classpath (e.g in Leiningen `resources` dir):
  (thaw-from-file (clojure.java.io/resource "my-resource-name.npy"))

See also `freeze-to-file`.

thaw-from-in!

(thaw-from-in! data-input)
Deserializes a frozen object from given DataInput to its original Clojure
data type.

This is a low-level util: in most cases you'll want `thaw` instead.

thaw-from-string

(thaw-from-string s)(thaw-from-string s thaw-opts)
Convenience util: like `thaw`, but takes a Base64-encoded string.
See also `freeze-to-string`.

throw-unfreezable

(throw-unfreezable x)

try-write-readable

(try-write-readable out x)

try-write-serializable

(try-write-serializable out x)

write-id

macro

(write-id out id)

write-unfreezable

(write-unfreezable out x)