nippy/taoensso.nippy.html

397 lines
39 KiB
HTML
Raw Normal View History

2016-02-25 11:50:00 +00:00
<!DOCTYPE html PUBLIC ""
"">
2023-08-02 12:39:18 +00:00
<html><head><meta charset="UTF-8" /><title>taoensso.nippy documentation</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">Nippy</span> <span class="project-version">3.3.0-RC1</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>taoensso</span></div></div></li><li class="depth-2 current"><a href="taoensso.nippy.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>nippy</span></div></a></li><li class="depth-3 branch"><a href="taoensso.nippy.benchmarks.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>benchmarks</span></div></a></li><li class="depth-3"><a href="taoensso.nippy.tools.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>tools</span></div></a></li></ul></div><div class="sidebar secondary"><h3><a href="#top"><span class="inner">Public Vars</span></a></h3><ul><li class="depth-1"><a href="taoensso.nippy.html#var-*auto-freeze-compressor*"><div class="inner"><span>*auto-freeze-compressor*</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-*custom-readers*"><div class="inner"><span>*custom-readers*</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-*final-freeze-fallback*"><div class="inner"><span>*final-freeze-fallback*</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-*freeze-fallback*"><div class="inner"><span>*freeze-fallback*</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-*freeze-serializable-allowlist*"><div class="inner"><span>*freeze-serializable-allowlist*</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-*incl-metadata.3F*"><div class="inner"><span>*incl-metadata?*</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-*serializable-whitelist*"><div class="inner"><span>*serializable-whitelist*</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-*thaw-serializable-allowlist*"><div class="inner"><span>*thaw-serializable-allowlist*</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-*thaw-xform*"><div class="inner"><span>*thaw-xform*</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var--cache-proxy"><div class="inner"><span>-cache-proxy</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-aes128-cbc-encryptor"><div class="inner"><span>aes128-cbc-encryptor</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-aes128-encryptor"><div class="inner"><span>aes128-encryptor</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-aes128-gcm-encryptor"><div class="inner"><span>aes128-gcm-encryptor</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-allow-and-record-any-serializable-class-unsafe"><div class="inner"><span>allow-and-record-any-serializable-class-unsafe</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-cache"><div class="inner"><span>cache</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.html#var-compress"><div class="inner"><span>co
2016-06-10 04:00:39 +00:00
</pre></div><div class="public anchor" id="var-*auto-freeze-compressor*"><h3>*auto-freeze-compressor*</h3><h4 class="dynamic">dynamic</h4><div class="usage"></div><div class="doc"><pre class="plaintext">(fn [byte-array])-&gt;compressor used by `(freeze &lt;x&gt; {:compressor :auto}),
2020-09-10 09:56:10 +00:00
nil =&gt; default</pre></div></div><div class="public anchor" id="var-*custom-readers*"><h3>*custom-readers*</h3><h4 class="dynamic">dynamic</h4><div class="usage"></div><div class="doc"><pre class="plaintext">{&lt;hash-or-byte-id&gt; (fn [data-input])-&gt;read}
2023-08-02 12:39:18 +00:00
</pre></div></div><div class="public anchor" id="var-*final-freeze-fallback*"><h3>*final-freeze-fallback*</h3><h4 class="dynamic">dynamic</h4><h4 class="deprecated">deprecated</h4><div class="usage"></div><div class="doc"><pre class="plaintext">DEPRECATED: prefer `*freeze-fallback`.
2020-09-10 09:56:10 +00:00
</pre></div></div><div class="public anchor" id="var-*freeze-fallback*"><h3>*freeze-fallback*</h3><h4 class="dynamic">dynamic</h4><div class="usage"></div><div class="doc"><pre class="plaintext">(fn [data-output x])-&gt;freeze, nil =&gt; default
2020-09-11 11:05:46 +00:00
</pre></div></div><div class="public anchor" id="var-*freeze-serializable-allowlist*"><h3>*freeze-serializable-allowlist*</h3><h4 class="dynamic">dynamic</h4><div class="usage"></div><div class="doc"><pre class="plaintext">Used when attempting to &lt;freeze/thaw&gt; an object that:
2020-09-12 10:18:18 +00:00
- Does NOT implement Nippy's Freezable protocol.
- DOES implement Java's Serializable interface.
2020-07-24 18:40:46 +00:00
2020-09-12 10:18:18 +00:00
In this case, the allowlist will be checked to see if Java's
Serializable interface may be used.
2020-07-24 18:40:46 +00:00
2020-09-11 11:05:46 +00:00
This is a security measure to prevent possible Remote Code Execution
(RCE) when thawing malicious payloads. See [1] for details.
2020-07-24 18:40:46 +00:00
2020-09-11 11:05:46 +00:00
If `freeze` encounters a disallowed Serialized class, it will throw.
If `thaw` encounters a disallowed Serialized class, it will:
2020-07-25 07:39:40 +00:00
2020-09-11 11:05:46 +00:00
- Throw if it's not possible to safely quarantine the object
(object was frozen with Nippy &lt; v2.15.0-final).
2020-07-24 18:40:46 +00:00
2020-09-11 11:05:46 +00:00
- Otherwise it will return a safely quarantined object of form
`{:nippy/unthawable {:class-name &lt;&gt; :content &lt;quarantined-ba&gt;}}`.
- Quarantined objects may be manually unquarantined with
`read-quarantined-serializable-object-unsafe!`.
2020-07-25 07:39:40 +00:00
2020-09-12 10:18:18 +00:00
There are 2x allowlists:
- `*freeze-serializable-allowlist*` ; Checked when freezing
- `*thaw-serializable-allowlist*` ; Checked when thawing
2020-07-25 07:39:40 +00:00
2020-09-12 10:18:18 +00:00
Example allowlist values:
- `(fn allow-class? [class-name] true)` ; Arbitrary predicate fn
- `#{"java.lang.Throwable", "clojure.lang.*"}` ; Set of class-names
- `"allow-and-record"` ; Special value, see [2]
2020-07-25 07:39:40 +00:00
2020-09-11 11:05:46 +00:00
Note that class-names in sets may contain "*" wildcards.
2020-07-25 07:39:40 +00:00
2020-09-11 11:05:46 +00:00
Default allowlist values are:
2020-09-12 10:18:18 +00:00
- default-freeze-serializable-allowlist ; `{"*"}` =&gt; allow any class
2020-09-11 11:05:46 +00:00
- default-thaw-serializable-allowlist ; A set of common safe classes
2020-07-25 07:39:40 +00:00
2020-09-11 11:05:46 +00:00
Allowlist values may be overridden with `binding`, `alter-var-root`, or:
- `taoensso.nippy.&lt;freeze/thaw&gt;-serializable-allowlist-base` JVM property
- `taoensso.nippy.&lt;freeze/thaw&gt;-serializable-allowlist-add` JVM property
- `TAOENSSO_NIPPY_&lt;FREEZE/THAW&gt;_SERIALIZABLE_ALLOWLIST_BASE` env var
- `TAOENSSO_NIPPY_&lt;FREEZE/THAW&gt;_SERIALIZABLE_ALLOWLIST_ADD` env var
2020-07-25 07:39:40 +00:00
2020-09-11 11:05:46 +00:00
If present, these will be read as comma-separated lists of class names
and formed into sets. Each initial allowlist value will then be:
(into (or &lt;?base&gt; &lt;default&gt;) &lt;?additions&gt;).
I.e. you can use:
- The "base" property/var to replace Nippy's default allowlists.
- The "add" property/var to add to Nippy's default allowlists.
2020-07-25 07:39:40 +00:00
2020-09-12 10:18:18 +00:00
The special `"allow-and-record"` value is also possible, see [2].
2020-07-24 18:40:46 +00:00
2020-09-11 11:05:46 +00:00
Upgrading from an older version of Nippy and unsure whether you've been
2020-09-12 10:18:18 +00:00
using Nippy's Serializable support, or which classes to allow? See [2].
2020-09-11 11:05:46 +00:00
2020-09-12 10:18:18 +00:00
See also `taoensso.encore/compile-str-filter` for a util to help easily
build more advanced predicate functions.
2020-09-11 11:05:46 +00:00
Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
identifying this vulnerability.
2020-09-12 10:18:18 +00:00
[1] <a href="https://github.com/ptaoussanis/nippy/issues/130">https://github.com/ptaoussanis/nippy/issues/130</a>
[2] See `allow-and-record-any-serializable-class-unsafe`.</pre></div></div><div class="public anchor" id="var-*incl-metadata.3F*"><h3>*incl-metadata?*</h3><h4 class="dynamic">dynamic</h4><div class="usage"></div><div class="doc"><pre class="plaintext">Include metadata when freezing/thawing?
2023-08-02 12:39:18 +00:00
</pre></div></div><div class="public anchor" id="var-*serializable-whitelist*"><h3>*serializable-whitelist*</h3><h4 class="dynamic">dynamic</h4><h4 class="deprecated">deprecated</h4><div class="usage"></div><div class="doc"><pre class="plaintext">DEPRECATED, now called `*thaw-serializable-allowlist*`
2020-09-11 11:05:46 +00:00
</pre></div></div><div class="public anchor" id="var-*thaw-serializable-allowlist*"><h3>*thaw-serializable-allowlist*</h3><h4 class="dynamic">dynamic</h4><div class="usage"></div><div class="doc"><pre class="plaintext">Used when attempting to &lt;freeze/thaw&gt; an object that:
2020-09-12 10:18:18 +00:00
- Does NOT implement Nippy's Freezable protocol.
- DOES implement Java's Serializable interface.
2020-09-11 11:05:46 +00:00
2020-09-12 10:18:18 +00:00
In this case, the allowlist will be checked to see if Java's
Serializable interface may be used.
2020-07-24 18:40:46 +00:00
2020-09-11 11:05:46 +00:00
This is a security measure to prevent possible Remote Code Execution
(RCE) when thawing malicious payloads. See [1] for details.
2020-07-24 18:40:46 +00:00
2020-09-11 11:05:46 +00:00
If `freeze` encounters a disallowed Serialized class, it will throw.
If `thaw` encounters a disallowed Serialized class, it will:
2020-07-24 18:40:46 +00:00
2020-09-11 11:05:46 +00:00
- Throw if it's not possible to safely quarantine the object
(object was frozen with Nippy &lt; v2.15.0-final).
2020-07-24 18:40:46 +00:00
2020-09-11 11:05:46 +00:00
- Otherwise it will return a safely quarantined object of form
`{:nippy/unthawable {:class-name &lt;&gt; :content &lt;quarantined-ba&gt;}}`.
- Quarantined objects may be manually unquarantined with
`read-quarantined-serializable-object-unsafe!`.
2020-07-24 18:40:46 +00:00
2020-09-12 10:18:18 +00:00
There are 2x allowlists:
- `*freeze-serializable-allowlist*` ; Checked when freezing
- `*thaw-serializable-allowlist*` ; Checked when thawing
2020-07-24 18:40:46 +00:00
2020-09-12 10:18:18 +00:00
Example allowlist values:
- `(fn allow-class? [class-name] true)` ; Arbitrary predicate fn
- `#{"java.lang.Throwable", "clojure.lang.*"}` ; Set of class-names
- `"allow-and-record"` ; Special value, see [2]
2020-07-24 18:40:46 +00:00
2020-09-11 11:05:46 +00:00
Note that class-names in sets may contain "*" wildcards.
2020-07-24 18:40:46 +00:00
2020-09-11 11:05:46 +00:00
Default allowlist values are:
2020-09-12 10:18:18 +00:00
- default-freeze-serializable-allowlist ; `{"*"}` =&gt; allow any class
2020-09-11 11:05:46 +00:00
- default-thaw-serializable-allowlist ; A set of common safe classes
Allowlist values may be overridden with `binding`, `alter-var-root`, or:
- `taoensso.nippy.&lt;freeze/thaw&gt;-serializable-allowlist-base` JVM property
- `taoensso.nippy.&lt;freeze/thaw&gt;-serializable-allowlist-add` JVM property
- `TAOENSSO_NIPPY_&lt;FREEZE/THAW&gt;_SERIALIZABLE_ALLOWLIST_BASE` env var
- `TAOENSSO_NIPPY_&lt;FREEZE/THAW&gt;_SERIALIZABLE_ALLOWLIST_ADD` env var
If present, these will be read as comma-separated lists of class names
and formed into sets. Each initial allowlist value will then be:
(into (or &lt;?base&gt; &lt;default&gt;) &lt;?additions&gt;).
I.e. you can use:
- The "base" property/var to replace Nippy's default allowlists.
- The "add" property/var to add to Nippy's default allowlists.
2020-09-12 10:18:18 +00:00
The special `"allow-and-record"` value is also possible, see [2].
2020-09-11 11:05:46 +00:00
Upgrading from an older version of Nippy and unsure whether you've been
2020-09-12 10:18:18 +00:00
using Nippy's Serializable support, or which classes to allow? See [2].
2020-09-11 11:05:46 +00:00
2020-09-12 10:18:18 +00:00
See also `taoensso.encore/compile-str-filter` for a util to help easily
build more advanced predicate functions.
2020-07-24 18:40:46 +00:00
Thanks to Timo Mihaljov (@solita-timo-mihaljov) for an excellent report
identifying this vulnerability.
2020-09-12 10:18:18 +00:00
[1] <a href="https://github.com/ptaoussanis/nippy/issues/130">https://github.com/ptaoussanis/nippy/issues/130</a>
2023-08-02 12:39:18 +00:00
[2] See `allow-and-record-any-serializable-class-unsafe`.</pre></div></div><div class="public anchor" id="var-*thaw-xform*"><h3>*thaw-xform*</h3><h4 class="dynamic">dynamic</h4><div class="usage"></div><div class="doc"><pre class="plaintext">Experimental, subject to change. Feedback welcome.
Transducer to use when thawing standard Clojure collection types
(vectors, maps, sets, lists, etc.).
Allows fast+flexible inspection and manipulation of data being thawed.
Key-val style data structures like maps will provide `MapEntry` args
to reducing function. Use `map-entry?`, `key`, `val` utils for these.
Example transducers:
(map (fn [x] (println x) x)) ; Print each coll item thawed
(comp
(map (fn [x] (if (= x :secret) :redacted x))) ; Replace secrets
(remove (fn [x] ; Remove maps with a truthy :remove?
(or
(and (map? x) (:remove? x))
(and (map-entry? x) (= (key x) :remove?) (val y)))))))
Note that while this is a very powerful feature, correctly writing
and debugging transducers and reducing fns can be tricky.
To help, if Nippy encounters an errors while applying your xform, it
will throw a detailed `ExceptionInfo` with message
"Error thrown via `*thaw-xform*`" to help you debug.</pre></div></div><div class="public anchor" id="var--cache-proxy"><h3>-cache-proxy</h3><div class="usage"></div><div class="doc"><pre class="plaintext">{[&lt;x&gt; &lt;meta&gt;] &lt;idx&gt;} for freezing, {&lt;idx&gt; &lt;x-with-meta&gt;} for thawing.
2020-09-10 09:56:10 +00:00
</pre></div></div><div class="public anchor" id="var-aes128-cbc-encryptor"><h3>aes128-cbc-encryptor</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen.
See also `aes-128-cbc-encryptor`.</pre></div></div><div class="public anchor" id="var-aes128-encryptor"><h3>aes128-encryptor</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
2014-07-06 06:37:03 +00:00
2016-02-25 11:50:00 +00:00
Password form [:salted "my-password"]
2014-07-06 06:37:03 +00:00
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
2015-09-17 04:19:19 +00:00
item encrypted will use a unique user-provided password).
2014-07-06 06:37:03 +00:00
IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
every key.
PROS: Each key is independent so would need to be attacked independently.
2016-02-25 11:50:00 +00:00
CONS: Key caching impossible, so there's an inherent trade-off between
2014-07-06 06:37:03 +00:00
encryption/decryption speed and the difficulty of attacking any
particular key.
Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.
2016-02-25 11:50:00 +00:00
Password form [:cached "my-password"]
2014-07-06 06:37:03 +00:00
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
2016-02-25 11:50:00 +00:00
number of staff/admins, or you'll be using a single password to
2014-07-06 06:37:03 +00:00
encrypt many items).
IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.
PROS: Great amortized encryption/decryption speed. Expensive key hash makes
attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
key _is_ somehow compromised, _all_ items encrypted with that key are
compromised.
Faster than `aes128-salted`, and harder to attack any particular key - but
2020-09-10 09:56:10 +00:00
increased danger if a key is somehow compromised.</pre></div></div><div class="public anchor" id="var-aes128-gcm-encryptor"><h3>aes128-gcm-encryptor</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
2020-07-24 18:40:46 +00:00
Password form [:salted "my-password"]
---------------------------------------
USE CASE: You want more than a small, finite number of passwords (e.g. each
item encrypted will use a unique user-provided password).
IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
every key.
PROS: Each key is independent so would need to be attacked independently.
CONS: Key caching impossible, so there's an inherent trade-off between
encryption/decryption speed and the difficulty of attacking any
particular key.
Slower than `aes128-cached`, and easier to attack any particular key - but
keys are independent.
Password form [:cached "my-password"]
---------------------------------------
USE CASE: You want only a small, finite number of passwords (e.g. a limited
number of staff/admins, or you'll be using a single password to
encrypt many items).
IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.
PROS: Great amortized encryption/decryption speed. Expensive key hash makes
attacking any particular key very difficult.
CONS: Using a small number of keys for many encrypted items means that if any
key _is_ somehow compromised, _all_ items encrypted with that key are
compromised.
Faster than `aes128-salted`, and harder to attack any particular key - but
2020-09-12 10:28:15 +00:00
increased danger if a key is somehow compromised.</pre></div></div><div class="public anchor" id="var-allow-and-record-any-serializable-class-unsafe"><h3>allow-and-record-any-serializable-class-unsafe</h3><div class="usage"><code>(allow-and-record-any-serializable-class-unsafe class-name)</code></div><div class="doc"><pre class="plaintext">A predicate (fn allow-class? [class-name]) fn that can be assigned
to `*freeze-serializable-allowlist*` and/or
`*thaw-serializable-allowlist*` that:
- Will allow ANY class to use Nippy's Serializable support (unsafe).
- And will record {&lt;class-name&gt; &lt;frequency-allowed&gt;} for the &lt;=1000
classes that ~most frequently made use of this support.
`get-recorded-serializable-classes` returns the recorded state.
This predicate is provided as a convenience for users upgrading from
previous versions of Nippy that allowed the use of Serializable for all
classes by default.
While transitioning from an unsafe-&gt;safe configuration, you can use
this predicate (unsafe) to record information about which classes have
been using Nippy's Serializable support in your environment.
Once some time has passed, you can check the recorded state. If you're
satisfied that all recorded classes are safely Serializable, you can
then merge the recorded classes into Nippy's default allowlist/s, e.g.:
(alter-var-root #'thaw-serializable-allowlist*
(fn [_] (into default-thaw-serializable-allowlist
2023-08-02 12:39:18 +00:00
(keys (get-recorded-serializable-classes)))))</pre></div></div><div class="public anchor" id="var-cache"><h3>cache</h3><div class="usage"><code>(cache x)</code></div><div class="doc"><pre class="plaintext">Experimental, subject to change. Feedback welcome.
2016-07-17 08:48:36 +00:00
Wraps value so that future writes of the same wrapped value with same
metadata will be efficiently encoded as references to this one.
2016-06-10 04:00:39 +00:00
(freeze [(cache "foo") (cache "foo") (cache "foo")])
2020-09-11 11:05:46 +00:00
will incl. a single "foo", plus 2x single-byte references to "foo".</pre></div></div><div class="public anchor" id="var-compress"><h3>compress</h3><div class="usage"><code>(compress compressor ba)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-decompress"><h3>decompress</h3><div class="usage"><code>(decompress compressor ba)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-decrypt"><h3>decrypt</h3><div class="usage"><code>(decrypt encryptor pwd ba)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-default-freeze-serializable-allowlist"><h3>default-freeze-serializable-allowlist</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Allows *any* class-name to be frozen using Java's Serializable interface.
This is generally safe since RCE risk is present only when thawing.
See also `*freeze-serializable-allowlist*`.</pre></div></div><div class="public anchor" id="var-default-thaw-serializable-allowlist"><h3>default-thaw-serializable-allowlist</h3><div class="usage"></div><div class="doc"><pre class="plaintext">A set of common safe class-names to allow to be frozen using Java's
Serializable interface. PRs welcome for additions.
See also `*thaw-serializable-allowlist*`.</pre></div></div><div class="public anchor" id="var-encrypt"><h3>encrypt</h3><div class="usage"><code>(encrypt encryptor pwd ba)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-extend-freeze"><h3>extend-freeze</h3><h4 class="type">macro</h4><div class="usage"><code>(extend-freeze type custom-type-id [x out] &amp; body)</code></div><div class="doc"><pre class="plaintext">Extends Nippy to support freezing of a custom type (ideally concrete) with
2014-07-06 06:37:03 +00:00
given id of form:
2020-07-25 07:39:40 +00:00
* Keyword - 2 byte overhead, keywords hashed to 16 bit id
* ℕ∈[1, 128] - 0 byte overhead
2014-07-06 06:37:03 +00:00
2016-07-17 08:48:36 +00:00
NB: be careful about extending to interfaces, Ref. <a href="http://goo.gl/6gGRlU">http://goo.gl/6gGRlU</a>.
(defrecord MyRec [data])
(extend-freeze MyRec :foo/my-type [x data-output] ; Keyword id
2014-07-06 06:37:03 +00:00
(.writeUTF [data-output] (:data x)))
;; or
2016-07-17 08:48:36 +00:00
(extend-freeze MyRec 1 [x data-output] ; Byte id
2020-09-10 09:56:10 +00:00
(.writeUTF [data-output] (:data x)))</pre></div></div><div class="public anchor" id="var-extend-thaw"><h3>extend-thaw</h3><h4 class="type">macro</h4><div class="usage"><code>(extend-thaw custom-type-id [in] &amp; body)</code></div><div class="doc"><pre class="plaintext">Extends Nippy to support thawing of a custom type with given id:
2014-07-06 06:37:03 +00:00
(extend-thaw :foo/my-type [data-input] ; Keyword id
2016-07-17 08:48:36 +00:00
(MyRec. (.readUTF data-input)))
2014-07-06 06:37:03 +00:00
;; or
(extend-thaw 1 [data-input] ; Byte id
2020-09-10 09:56:10 +00:00
(MyRec. (.readUTF data-input)))</pre></div></div><div class="public anchor" id="var-fast-freeze"><h3>fast-freeze</h3><div class="usage"><code>(fast-freeze x)</code></div><div class="doc"><pre class="plaintext">Like `freeze` but:
2016-06-10 04:00:39 +00:00
- Writes data without a Nippy header
- Drops all support for compression and encryption
- Must be thawed with `fast-thaw`
2020-09-11 11:05:46 +00:00
Equivalent to (but a little faster than) `freeze` with opts:
- :compressor nil
- :encryptor nil
- :no-header? true</pre></div></div><div class="public anchor" id="var-fast-thaw"><h3>fast-thaw</h3><div class="usage"><code>(fast-thaw ba)</code></div><div class="doc"><pre class="plaintext">Like `thaw` but:
2016-06-10 04:00:39 +00:00
- Drops all support for compression and encryption
- Supports only data frozen with `fast-freeze`
2020-09-11 11:05:46 +00:00
Equivalent to (but a little faster than) `thaw` with opts:
- :compressor nil
- :encryptor nil
- :no-header? true</pre></div></div><div class="public anchor" id="var-freezable.3F"><h3>freezable?</h3><div class="usage"><code>(freezable? x)</code><code>(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})</code></div><div class="doc"><pre class="plaintext">Alpha - subject to change.
2016-06-10 04:00:39 +00:00
Returns truthy iff Nippy *appears* to support freezing the given argument.
`:allow-clojure-reader?` and `:allow-java-serializable?` options may be
used to enable the relevant roundtrip fallback test(s). These tests are
only **moderately reliable** since they're cached by arg type and don't
test for pre/post serialization value equality (there's no good general
2023-08-02 12:39:18 +00:00
way of doing so).</pre></div></div><div class="public anchor" id="var-freeze"><h3>freeze</h3><div class="usage"><code>(freeze x)</code><code>(freeze x {:as opts, :keys [compressor encryptor password serializable-allowlist incl-metadata?], :or {compressor :auto, encryptor aes128-gcm-encryptor}})</code></div><div class="doc"><pre class="plaintext">Serializes arg (any Clojure data type) to a byte array.
To freeze custom types, extend the Clojure reader or see `extend-freeze`.</pre></div></div><div class="public anchor" id="var-freeze-to-file"><h3>freeze-to-file</h3><div class="usage"><code>(freeze-to-file file x)</code><code>(freeze-to-file file x freeze-opts)</code></div><div class="doc"><pre class="plaintext">Convenience util: like `freeze`, but writes to `(clojure.java.io/file &lt;file&gt;)`.
2020-11-06 14:46:26 +00:00
</pre></div></div><div class="public anchor" id="var-freeze-to-out.21"><h3>freeze-to-out!</h3><div class="usage"><code>(freeze-to-out! data-output x)</code></div><div class="doc"><pre class="plaintext">Serializes arg (any Clojure data type) to a DataOutput.
2020-09-10 09:56:10 +00:00
This is a low-level util: in most cases you'll want `freeze` instead.</pre></div></div><div class="public anchor" id="var-freeze-to-string"><h3>freeze-to-string</h3><div class="usage"><code>(freeze-to-string x)</code><code>(freeze-to-string x freeze-opts)</code></div><div class="doc"><pre class="plaintext">Convenience util: like `freeze`, but returns a Base64-encoded string.
2020-09-12 10:18:18 +00:00
See also `thaw-from-string`.</pre></div></div><div class="public anchor" id="var-get-recorded-serializable-classes"><h3>get-recorded-serializable-classes</h3><div class="usage"><code>(get-recorded-serializable-classes)</code></div><div class="doc"><pre class="plaintext">Returns {&lt;class-name&gt; &lt;frequency&gt;} of the &lt;=1000 classes that ~most
frequently made use of Nippy's Serializable support via
`allow-and-record-any-serializable-class-unsafe`.
2023-08-02 12:39:18 +00:00
See that function's docstring for more info.</pre></div></div><div class="public anchor" id="var-IFreezable1"><h3>IFreezable1</h3><h4 class="type">protocol</h4><div class="usage"></div><div class="doc"><pre class="plaintext"></pre></div><div class="members"><h4>members</h4><div class="inner"><div class="public anchor" id="var--freeze-without-meta.21"><h3>-freeze-without-meta!</h3><div class="usage"><code>(-freeze-without-meta! x data-output)</code></div><div class="doc"><pre class="plaintext"></pre></div></div></div></div></div><div class="public anchor" id="var-IFreezable2"><h3>IFreezable2</h3><h4 class="type">protocol</h4><div class="usage"></div><div class="doc"><pre class="plaintext"></pre></div><div class="members"><h4>members</h4><div class="inner"><div class="public anchor" id="var--freeze-with-meta.21"><h3>-freeze-with-meta!</h3><div class="usage"><code>(-freeze-with-meta! x data-output)</code></div><div class="doc"><pre class="plaintext"></pre></div></div></div></div></div><div class="public anchor" id="var-inspect-ba"><h3>inspect-ba</h3><div class="usage"><code>(inspect-ba ba)</code><code>(inspect-ba ba thaw-opts)</code></div><div class="doc"><pre class="plaintext">Experimental, subject to change. Feedback welcome.
2020-09-10 09:56:10 +00:00
</pre></div></div><div class="public anchor" id="var-lz4-compressor"><h3>lz4-compressor</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Default net.jpountz.lz4 compressor:
2014-07-06 06:37:03 +00:00
Ratio: low.
Write speed: very high.
Read speed: very high.
A good general-purpose compressor, competitive with Snappy.
Thanks to Max Penet (@mpenet) for our first implementation,
2020-09-10 09:56:10 +00:00
Ref. <a href="https://github.com/mpenet/nippy-lz4">https://github.com/mpenet/nippy-lz4</a></pre></div></div><div class="public anchor" id="var-lz4hc-compressor"><h3>lz4hc-compressor</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Like `lz4-compressor` but trades some write speed for ratio.
</pre></div></div><div class="public anchor" id="var-lzma2-compressor"><h3>lzma2-compressor</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Default org.tukaani.xz.LZMA2 compressor:
2014-07-06 06:37:03 +00:00
Ratio: high.
Write speed: _very_ slow (also currently single-threaded).
Read speed: slow.
A specialized compressor for large, low-write data in space-sensitive
2023-08-02 12:39:18 +00:00
environments.</pre></div></div><div class="public anchor" id="var-public-types-spec"><h3>public-types-spec</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Public representation of Nippy's internal type schema.
For use by tooling and advanced users.
**HIGHLY EXPERIMENTAL!**
Subject to breaking change without notice.
Currently completely untested, may contain bugs.
Intended for use only by early adopters to give design feedback.
Format:
{&lt;type-id&gt; {:keys [type-id type-kw payload-spec deprecated?]}},
- `type-id`: A +ive single-byte identifier like `110`.
-ive type ids are reserved for custom user-defined types.
- `type-kw`: A keyword like `:kw-sm`,
suffixes used to differentiate subtypes of different sizes:
-0 ; Empty =&gt; 0 byte payload / element-count
-sm ; Small =&gt; 1 byte (byte) payload / element-count
-md ; Medium =&gt; 2 byte (short) payload / element-count
-lg ; Large =&gt; 4 byte (int) payload / element-count
-xl ; Extra large =&gt; 8 byte (long) payload / element-count
- `payload-spec` examples:
- nil ; No spec available (e.g. unpredictable payload)
- [] ; Type has no payload
- [[:bytes 4]] ; Type has a payload of exactly 4 bytes
- [[:bytes 2] [:elements 2]] ; Type has a payload of exactly 2 bytes, then
; 2 elements
- [[:bytes {:read 2}]
[:elements {:read 4 :multiplier 2 :unsigned? true}]]
; Type has payload of &lt;short-count&gt; bytes, then
; &lt;unsigned-int-count&gt;*2 (multiplier) elements
Note that `payload-spec` can be handy for skipping over items in
data stream without fully reading every item.</pre></div></div><div class="public anchor" id="var-read-quarantined-serializable-object-unsafe.21"><h3>read-quarantined-serializable-object-unsafe!</h3><div class="usage"><code>(read-quarantined-serializable-object-unsafe! m)</code></div><div class="doc"><pre class="plaintext">Given a quarantined Serializable object like
2020-09-11 11:05:46 +00:00
{:nippy/unthawable {:class-name &lt;&gt; :content &lt;quarantined-ba&gt;}}, reads and
returns the object WITHOUT regard for `*thaw-serializable-allowlist*`.
**MAY BE UNSAFE!** Don't call this unless you absolutely trust the payload
to not contain any malicious code.
See `*thaw-serializable-allowlist*` for more info.</pre></div></div><div class="public anchor" id="var-snappy-compressor"><h3>snappy-compressor</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Default org.iq80.snappy.Snappy compressor:
2014-07-06 06:37:03 +00:00
Ratio: low.
Write speed: very high.
Read speed: very high.
2020-09-10 09:56:10 +00:00
A good general-purpose compressor.</pre></div></div><div class="public anchor" id="var-stress-data"><h3>stress-data</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Reference data used for tests &amp; benchmarks
</pre></div></div><div class="public anchor" id="var-stress-data-benchable"><h3>stress-data-benchable</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Reference data with stuff removed that breaks reader or other utils we'll
2023-08-02 12:39:18 +00:00
be benching with.</pre></div></div><div class="public anchor" id="var-stress-data-comparable"><h3>stress-data-comparable</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Reference data with stuff removed that breaks roundtrip equality.
</pre></div></div><div class="public anchor" id="var-thaw"><h3>thaw</h3><div class="usage"><code>(thaw ba)</code><code>(thaw ba {:as opts, :keys [v1-compatibility? compressor encryptor password serializable-allowlist incl-metadata? thaw-xform], :or {compressor :auto, encryptor :auto}})</code></div><div class="doc"><pre class="plaintext">Deserializes a frozen Nippy byte array to its original Clojure data type.
2016-06-10 04:00:39 +00:00
To thaw custom types, extend the Clojure reader or see `extend-thaw`.
2015-12-01 06:06:24 +00:00
** By default, supports data frozen with Nippy v2+ ONLY **
Add `{:v1-compatibility? true}` option to support thawing of data frozen with
legacy versions of Nippy.
2014-07-06 06:37:03 +00:00
Options include:
2015-12-01 06:06:24 +00:00
:v1-compatibility? - support data frozen by legacy versions of Nippy?
:compressor - :auto (checks header, default) an ICompressor, or nil
2020-09-10 09:56:10 +00:00
:encryptor - :auto (checks header, default), an IEncryptor, or nil</pre></div></div><div class="public anchor" id="var-thaw-from-file"><h3>thaw-from-file</h3><div class="usage"><code>(thaw-from-file file)</code><code>(thaw-from-file file thaw-opts)</code></div><div class="doc"><pre class="plaintext">Convenience util: like `thaw`, but reads from `(clojure.java.io/file &lt;file&gt;)`.
2020-11-06 14:46:26 +00:00
</pre></div></div><div class="public anchor" id="var-thaw-from-in.21"><h3>thaw-from-in!</h3><div class="usage"><code>(thaw-from-in! data-input)</code></div><div class="doc"><pre class="plaintext">Deserializes a frozen object from given DataInput to its original Clojure
2016-07-17 08:48:36 +00:00
data type.
2020-11-06 14:46:26 +00:00
This is a low-level util: in most cases you'll want `thaw` instead.</pre></div></div><div class="public anchor" id="var-thaw-from-resource"><h3>thaw-from-resource</h3><div class="usage"><code>(thaw-from-resource res)</code><code>(thaw-from-resource res thaw-opts)</code></div><div class="doc"><pre class="plaintext">Convenience util: like `thaw`, but reads from `(clojure.java.io/resource &lt;res&gt;)`.
</pre></div></div><div class="public anchor" id="var-thaw-from-string"><h3>thaw-from-string</h3><div class="usage"><code>(thaw-from-string s)</code><code>(thaw-from-string s thaw-opts)</code></div><div class="doc"><pre class="plaintext">Convenience util: like `thaw`, but takes a Base64-encoded string.
2020-09-10 09:56:10 +00:00
See also `freeze-to-string`.</pre></div></div><div class="public anchor" id="var-throw-unfreezable"><h3>throw-unfreezable</h3><div class="usage"><code>(throw-unfreezable x)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-try-write-readable"><h3>try-write-readable</h3><div class="usage"><code>(try-write-readable out x)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-try-write-serializable"><h3>try-write-serializable</h3><div class="usage"><code>(try-write-serializable out x)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-write-id"><h3>write-id</h3><h4 class="type">macro</h4><div class="usage"><code>(write-id out id)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-write-unfreezable"><h3>write-unfreezable</h3><div class="usage"><code>(write-unfreezable out x)</code></div><div class="doc"><pre class="plaintext"></pre></div></div></div></body></html>