Compare commits

...

18 commits

Author SHA1 Message Date
Peter Taoussanis
ae4463a85c [nop] Drop use of deprecated enc/binding 2025-05-04 15:17:13 +02:00
Peter Taoussanis
1b99516177 [doc] Misc improvements 2025-04-15 10:42:51 +02:00
Peter Taoussanis
e68c2d56fe v3.5.0 (2025-04-15) 2025-04-15 08:58:10 +02:00
Peter Taoussanis
bfba59483a [new] Add string array type to default thaw-serializable-allowlist 2025-04-14 23:03:17 +02:00
Peter Taoussanis
f7bb2824ac [new] Add value tests for non-comparable types 2025-04-14 23:03:17 +02:00
Peter Taoussanis
8d62dc2826 [new] Use Truss exceptions on errors
Ref. https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#*ctx*
2025-04-14 23:03:17 +02:00
Peter Taoussanis
da57206b0d [mod] Drop official Clojure v1.9 support
Clojure v1.9  was released December 8,  2017 (~7 years ago)
Clojure v1.10 was released December 17, 2018 (~6 years ago)

Official support continues for v1.10, v1.11, v1.12.
2025-04-14 23:03:17 +02:00
Peter Taoussanis
e0f49ced5a [nop] Bump deps 2025-04-14 23:03:17 +02:00
Peter Taoussanis
8d107650cd [new] [#184] Incl. cause on non-native freeze failures
Before this commit:

  - When freezing an item WITHOUT a native Nippy implementation,
    Nippy may try to use (1) Java Serializable or (2) Clojure's reader.
    If these also fail, an ex-info will be thrown.
    The ex-info does NOT include any info about possible exceptions
    from (1) or (2).

After this commit:

  - The thrown ex-info now includes info about possible exceptions
    from (1) and (2). These can be useful, e.g. when indicating
    an OOM error, etc.
2025-04-14 23:03:17 +02:00
Peter Taoussanis
1026ea0ae7 [doc] Clarify *freeze-fallback* docstring 2025-04-14 23:03:17 +02:00
Peter Taoussanis
c92457025f [nop] Housekeeping 2025-04-14 23:03:17 +02:00
Peter Taoussanis
3cb29f3c2e v3.5.0-RC1 (2024-10-28) 2024-10-28 11:42:59 +01:00
Peter Taoussanis
a9ea13618c [nop] Maintain new type id info 2024-10-28 11:42:59 +01:00
Peter Taoussanis
d415a2bf72 [new] [#178] Add support for native arrays: strings, longs, ints, doubles, floats 2024-10-28 11:42:59 +01:00
Peter Taoussanis
b217db5579 [new] [#175] Mark cache feature as stable 2024-10-28 10:23:31 +01:00
Peter Taoussanis
9022aad018 [doc] Misc doc improvements 2024-10-28 10:14:25 +01:00
Peter Taoussanis
c0d1da1bb4 [nop] Housekeeping 2024-10-28 10:14:25 +01:00
Peter Taoussanis
bb178f66fc [nop] Bump deps 2024-10-28 10:11:34 +01:00
13 changed files with 553 additions and 328 deletions

View file

@ -2,23 +2,121 @@ This project uses [**Break Versioning**](https://www.taoensso.com/break-versioni
---
# `v3.5.0` (2025-04-15)
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.5.0)
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
This is a **general maintenance release** focused on updating dependencies and laying groundwork (read support) for new array types coming in Nippy v3.6.
It **drops support for Clojure v1.9** but should otherwise be a safe update from (at least) all recent versions of Nippy.
## Since `v3.5.0-RC1` (2024-10-28)
- **\[mod]** Drop official Clojure v1.9 support \[da57206]
- \[new] Add string array type to default [thaw-serializable-allowlist](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*thaw-serializable-allowlist*) \[bfba594]
- \[new] Use [Truss exceptions](https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#ex-info) on errors \[8d62dc2]
- \[new] [#184] Incl. cause on non-native freeze failures \[8d10765]
- \[doc] Clarify `*freeze-fallback*` docstring \[1026ea0]
## Since `v3.4.2` (2024-05-26)
- **\[mod]** Drop official Clojure v1.9 support \[da57206]
- \[new] Add string array type to default [thaw-serializable-allowlist](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*thaw-serializable-allowlist*) \[bfba594]
- \[new] Use [Truss exceptions](https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#ex-info) on errors \[8d62dc2]
- \[new] [#184] Incl. cause on non-native freeze failures \[8d10765]
- \[new] [#175] Mark [cache](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#cache) feature as stable \[b217db5]
- \[doc] Clarify `*freeze-fallback*` docstring \[1026ea0]
## Migration info
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
| :------------------------ | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
| `v3.5.0-RC1` (2024-10-28) | - | - | - |
| `v3.4.2` (2024-05-26) | - | - | - |
| `v3.4.1` (2024-05-02) | - | - | - |
| `v3.4.0` (2024-04-30) | - | **Yes** | - |
| `v3.3.0` (2023-10-11) | - | - | - |
| `v3.2.0` (2022-07-18) | - | - | - |
| `v3.1.3` (2022-06-23) | - | - | - |
> [1] Relevant only when Nippy introduces support for new types **AND** you plan to update Nippy with a **rolling update** (coexisting new and old versions).
If updating from older versions of Nippy, please see the relevant release notes.
As always:
- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
- **Please report any unexpected problems** 🙏
\- [Peter Taoussanis](https://www.taoensso.com)
---
# `v3.5.0-RC1` (2024-10-28)
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.5.0-RC1)
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
This is a **non-breaking maintenance release** that updates dependencies and includes read support for more native array types to be introduced in a future v3.6 release.
It should be safe to update from (at least) all recent versions of Nippy.
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
| :-------------------- | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
| `v3.4.2` (2024-05-26) | - | - | - |
| `v3.4.1` (2024-05-02) | - | - | - |
| `v3.4.0` (2024-04-30) | - | Yes | - |
| `v3.3.0` (2023-10-11) | - | - | - |
| `v3.2.0` (2022-07-18) | - | - | - |
| `v3.1.3` (2022-06-23) | - | - | - |
> [1] Relevant only when introducing support for new types, and for users that do rolling updates
If updating from older versions of Nippy, please see the relevant release notes.
As always:
- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
- **Please report any unexpected problems** 🙏
\- [Peter Taoussanis](https://www.taoensso.com)
---
# `v3.4.2` (2024-05-26)
> **Dep**: Nippy is [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.2).
> **Versioning**: Nippy uses [Break Versioning](https://www.taoensso.com/break-versioning).
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.2)
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
This release updates some internal dependencies and is **recommended for all existing users**.
⚠️ This release addresses a [**security vulnerability**](https://github.com/taoensso/nippy/security/advisories/GHSA-vw78-267v-588h) in Nippy's upstream compression library and is **recommended for all existing users**.
It should be a **non-breaking update** for almost all users of Nippy `v3.4.x`, `v3.3.x`, and `v3.2.x`.
It should be a **straight-forward and non-breaking update** for almost everyone:
Notes:
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
| :-------------------- | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
| `v3.4.1` (2024-05-02) | - | - | - |
| `v3.4.0` (2024-04-30) | - | Yes | - |
| `v3.3.0` (2023-10-11) | - | - | - |
| `v3.2.0` (2022-07-18) | - | - | - |
| `v3.1.3` (2022-06-23) | - | - | - |
- May produce **different serialized output** to `v3.4.0` and `v3.3.0`. Most users won't care about this, but you could be affected if you depend on specific serialized byte values (for example by comparing serialized output between different versions of Nippy).
- When using Nippy version **X** to thaw data frozen by Nippy version **Y>X**, there is necessarily a risk of the thaw throwing when encountering unfamiliar types. This **can affect rolling updates** and/or **limit your ability to revert** a Nippy update - **so please ensure adequate testing** in your environment before updating against production data!
- As always, **please report any unexpected problems** 🙏
> [1] Relevant only when introducing support for new types, and for users that do rolling updates
If updating from older versions of Nippy, please see the relevant release notes.
As always:
- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
- **Please report any unexpected problems** 🙏
\- [Peter Taoussanis](https://www.taoensso.com)
---
# `v3.4.1` (2024-05-02)
> **Dep**: Nippy is [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.1).

View file

@ -1,20 +1,19 @@
<a href="https://www.taoensso.com/clojure" title="More stuff by @ptaoussanis at www.taoensso.com"><img src="https://www.taoensso.com/open-source.png" alt="Taoensso open source" width="340"/></a>
[**Documentation**](#documentation) | [Latest releases](#latest-releases) | [Get support][GitHub issues]
[**API**][cljdoc] | [**Wiki**][GitHub wiki] | [Latest releases](#latest-releases) | [Get support][GitHub issues]
# Nippy
### The fastest serialization library for Clojure
### Fast serialization library for Clojure
Clojure's rich data types are awesome. And its [reader](https://clojure.org/reference/reader) allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).
Nippy is an attempt to provide a reliable, high-performance **drop-in alternative to the reader**.
Nippy is a mature, high-performance **drop-in alternative to the reader**.
Used by [Carmine](https://www.taoensso.com/carmine), [Faraday](https://www.taoensso.com/faraday), [PigPen](https://github.com/Netflix/PigPen), [Onyx](https://github.com/onyx-platform/onyx),
[XTDB](https://github.com/xtdb/xtdb), [Datalevin](https://github.com/juji-io/datalevin), and others.
It is used at scale by [Carmine](https://www.taoensso.com/carmine), [Faraday](https://www.taoensso.com/faraday), [PigPen](https://github.com/Netflix/PigPen), [Onyx](https://github.com/onyx-platform/onyx), [XTDB](https://github.com/xtdb/xtdb), [Datalevin](https://github.com/juji-io/datalevin), and others.
## Latest release/s
- `2024-05-26` `v3.4.2`: [release info](../../releases/tag/v3.4.2)
- `2025-04-15` `v3.5.0`: [release info](../../releases/tag/v3.5.0)
[![Main tests][Main tests SVG]][Main tests URL]
[![Graal tests][Graal tests SVG]][Graal tests URL]
@ -23,28 +22,80 @@ See [here][GitHub releases] for earlier releases.
## Why Nippy?
- Small, simple **all-Clojure** library
- Small, simple **pure-Clojure** library
- **Terrific performance**: the [best](#performance) for Clojure that I'm aware of
- Comprehensive support for [all standard data types](../../wiki/1-Getting-started#deserializing)
- Easily extendable to [custom data types](../../wiki/1-Getting-started#custom-types)
- **Robust test suite**, incl. full coverage for every supported type
- Auto fallback to [Java Serializable](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*freeze-serializable-allowlist*) when available
- Auto fallback to Clojure Reader for all other types (including tagged literals)
- Pluggable **compression** with built-in [LZ4](https://code.google.com/p/lz4/), [Zstandard](https://facebook.github.io/zstd/), etc.
- Pluggable [encryption](../../wiki/1-Getting-started#encryption) with built-in AES128
- **Robust test suite** incl. coverage of every supported type
- **Mature** and widely used in production for 12+ years
- Optional auto fallback to [Java Serializable](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*freeze-serializable-allowlist*) for [safe](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*freeze-serializable-allowlist*) types
- Optional auto fallback to Clojure Reader (including tagged literals)
- Optional smart **compression** with [LZ4](https://code.google.com/p/lz4/) or [Zstandard](https://facebook.github.io/zstd/)
- Optional [encryption](../../wiki/1-Getting-started#encryption) with AES128
- [Tools](https://taoensso.github.io/nippy/taoensso.nippy.tools.html) for easy + robust **integration into 3rd-party libraries**, etc.
- Powerful [thaw transducer](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*thaw-xform*) for flexible data inspection and transformation
## Quick example
Nippy's super easy to use:
```clojure
(require '[taoensso.nippy :as nippy])
;; Freeze any Clojure value
(nippy/freeze <my-value>) ; => Serialized byte[]
;; Thaw the byte[] to get back the original value:
(nippy/thaw (nippy/freeze <my-value>)) ; => <my-value>
```
See the [wiki](https://github.com/taoensso/nippy/wiki/1-Getting-started#deserializing) for more.
## Operational considerations
### Data longevity
Nippy is widely used to store **long-lived** data and promises (as always) that **data serialized today should be readable by all future versions of Nippy**.
But please note that the **converse is not generally true**:
- Nippy `vX` **should** be able to read all data from Nippy `vY<=X` (backwards compatibility)
- Nippy `vX` **may/not** be able to read all data from Nippy `vY>X` (forwards compatibility)
### Rolling updates and rollback
From time to time, Nippy may introduce:
- Support for serializing **new types**
- Optimizations to the serialization of **pre-existing types**
To help ease **rolling updates** and to better support **rollback**, Nippy (since version v3.4) will always introduce such changes over **two version releases**:
- Release 1: to add **read support** for the new types
- Release 2: to add **write support** for the new types
Starting from v3.4, Nippy's release notes will **always clearly indicate** if a particular update sequence is recommended.
### Stability of byte output
It has **never been an objective** of Nippy to offer **predictable byte output**, and I'd generally **recommend against** depending on specific byte output.
However, I know that a small minority of users *do* have specialized needs in this area.
So starting with Nippy v3.4, Nippy's release notes will **always clearly indicate** if any changes to byte output are expected.
## Performance
Since its earliest versions, Nippy has consistently been the **fastest serialization library for Clojure** that I'm aware of. Latest [benchmark](../../blob/master/test/taoensso/nippy_benchmarks.clj) results:
Nippy is fast! Latest [benchmark](../../blob/master/test/taoensso/nippy_benchmarks.clj) results:
![benchmarks-png](../../raw/master/benchmarks.png)
PRs welcome to include other alternatives in the bench suite!
## Documentation
- [Wiki][GitHub wiki] (getting started, usage, etc.)
- API reference: [cljdoc][cljdoc docs], [Codox][Codox docs]
- API reference via [cljdoc][cljdoc]
## Funding
@ -52,7 +103,7 @@ You can [help support][sponsor] continued work on this project, thank you!! 🙏
## License
Copyright &copy; 2012-2024 [Peter Taoussanis][].
Copyright &copy; 2012-2025 [Peter Taoussanis][].
Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).
<!-- Common -->
@ -66,8 +117,7 @@ Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).
<!-- Project -->
[Codox docs]: https://taoensso.github.io/nippy/
[cljdoc docs]: https://cljdoc.org/d/com.taoensso/nippy/
[cljdoc]: https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy
[Clojars SVG]: https://img.shields.io/clojars/v/com.taoensso/nippy.svg
[Clojars URL]: https://clojars.org/com.taoensso/nippy

View file

@ -1,6 +1,6 @@
(defproject com.taoensso/nippy "3.4.2"
(defproject com.taoensso/nippy "3.5.0"
:author "Peter Taoussanis <https://www.taoensso.com>"
:description "The fastest serialization library for Clojure"
:description "Fast serialization library for Clojure"
:url "https://www.taoensso.com/nippy"
:license
@ -8,20 +8,19 @@
:url "https://www.eclipse.org/legal/epl-v10.html"}
:dependencies
[[org.clojure/tools.reader "1.4.2"]
[com.taoensso/encore "3.112.0"]
[org.tukaani/xz "1.9"]
[io.airlift/aircompressor "0.27"]]
[[org.clojure/tools.reader "1.5.2"]
[com.taoensso/encore "3.142.0"]
[org.tukaani/xz "1.10"]
[io.airlift/aircompressor "2.0.2"]]
:test-paths ["test" #_"src"]
:profiles
{;; :default [:base :system :user :provided :dev]
:provided {:dependencies [[org.clojure/clojure "1.11.3"]]}
:c1.12 {:dependencies [[org.clojure/clojure "1.12.0-alpha12"]]}
:c1.11 {:dependencies [[org.clojure/clojure "1.11.3"]]}
:c1.10 {:dependencies [[org.clojure/clojure "1.10.1"]]}
:c1.9 {:dependencies [[org.clojure/clojure "1.9.0"]]}
:provided {:dependencies [[org.clojure/clojure "1.11.4"]]}
:c1.12 {:dependencies [[org.clojure/clojure "1.12.0"]]}
:c1.11 {:dependencies [[org.clojure/clojure "1.11.4"]]}
:c1.10 {:dependencies [[org.clojure/clojure "1.10.3"]]}
:graal-tests
{:source-paths ["test"]
@ -38,7 +37,9 @@
"-Xms1024m" "-Xmx2048m"
"-Dtaoensso.elide-deprecated=true"
"-Dtaoensso.nippy.thaw-serializable-allowlist-base=base.1, base.2"
"-Dtaoensso.nippy.thaw-serializable-allowlist-add=add.1 , add.2"]
"-Dtaoensso.nippy.thaw-serializable-allowlist-add=add.1 , add.2"
#_"-Dtaoensso.nippy.target-release=320"
#_"-Dtaoensso.nippy.target-release=350"]
:global-vars
{*warn-on-reflection* true
@ -51,18 +52,13 @@
:plugins
[[lein-pprint "1.3.2"]
[lein-ancient "0.7.0"]
[com.taoensso.forks/lein-codox "0.10.11"]]
:codox
{:language #{:clojure #_:clojurescript}
:base-language :clojure}}}
[lein-ancient "0.7.0"]]}}
:aliases
{"start-dev" ["with-profile" "+dev" "repl" ":headless"]
;; "build-once" ["do" ["clean"] ["cljsbuild" "once"]]
"deploy-lib" ["do" #_["build-once"] ["deploy" "clojars"] ["install"]]
"test-clj" ["with-profile" "+c1.12:+c1.11:+c1.10:+c1.9" "test"]
"test-clj" ["with-profile" "+c1.12:+c1.11:+c1.10" "test"]
;; "test-cljs" ["with-profile" "+c1.12" "cljsbuild" "test"]
"test-all" ["do" ["clean"] ["test-clj"] #_["test-cljs"]]})

View file

@ -4,6 +4,7 @@
(:require
[clojure.string :as str]
[clojure.java.io :as jio]
[taoensso.truss :as truss]
[taoensso.encore :as enc]
[taoensso.nippy
[impl :as impl]
@ -25,7 +26,7 @@
PersistentQueue PersistentTreeMap PersistentTreeSet PersistentList
MapEntry LazySeq IRecord ISeq IType]))
(enc/assert-min-encore-version [3 112 0])
(enc/assert-min-encore-version [3 142 0])
(comment
(set! *unchecked-math* :warn-on-boxed)
@ -145,11 +146,6 @@
51 [:reader-md [[:bytes {:read 2}]]]
52 [:reader-lg [[:bytes {:read 4}]]]
53 [:bytes-0 []]
7 [:bytes-sm [[:bytes {:read 1}]]]
15 [:bytes-md [[:bytes {:read 2}]]]
2 [:bytes-lg [[:bytes {:read 4}]]]
17 [:vec-0 []]
113 [:vec-2 [[:elements 2]]]
114 [:vec-3 [[:elements 3]]]
@ -182,7 +178,6 @@
28 [:sorted-set-lg [[:elements {:read 4}]]]
31 [:sorted-map-lg [[:elements {:read 4 :multiplier 2}]]]
26 [:queue-lg [[:elements {:read 4}]]]
115 [:objects-lg [[:elements {:read 4}]]]
25 [:meta [[:elements 1]]]
58 [:regex [[:elements 1]]]
@ -196,6 +191,21 @@
70 [:ratio [[:bytes {:read 4}]
[:bytes {:read 4}]]]
;; Arrays
53 [:byte-array-0 []]
7 [:byte-array-sm [[:elements {:read 1}]]]
15 [:byte-array-md [[:elements {:read 2}]]]
2 [:byte-array-lg [[:elements {:read 4}]]]
109 [:int-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
108 [:long-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
117 [:float-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
116 [:double-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
107 [:string-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
115 [:object-array-lg [[:elements {:read 4}]]]
;; Serializable
75 [:sz-quarantined-sm [[:bytes {:read 1}] [:elements 1]]]
76 [:sz-quarantined-md [[:bytes {:read 2}] [:elements 1]]]
@ -300,14 +310,14 @@
- `payload-spec` examples:
- nil ; No spec available (e.g. unpredictable payload)
- [] ; Type has no payload
- [[:bytes 4]] ; Type has a payload of exactly 4 bytes
- [[:bytes 2] [:elements 2]] ; Type has a payload of exactly 2 bytes, then
; 2 elements
- [[:bytes 4]] ; Type has payload of exactly 4 bytes
- [[:bytes 2] [:elements 2]] ; Type has payload of exactly 2 bytes,
; followed by 2 elements
- [[:bytes {:read 2}]
[:elements {:read 4 :multiplier 2 :unsigned? true}]]
; Type has payload of <short-count> bytes, then
; Type has payload of <short-count> bytes, followed by
; <unsigned-int-count>*2 (multiplier) elements
Note that `payload-spec` can be handy for skipping over items in
@ -356,25 +366,26 @@
;; For back compatibility (incl. Timbre's Carmine appender)
(enc/defonce ^:dynamic ^:no-doc ^:deprecated *final-freeze-fallback* "Prefer `*freeze-fallback`." nil)
(enc/defonce ^:dynamic *freeze-fallback*
"Controls Nippy's behaviour when trying to freeze an item for which Nippy
doesn't currently have a native freeze/thaw implementation.
"Controls Nippy's behaviour when trying to freeze an object with a type for
which Nippy doesn't currently have a native (protocol) implementation.
Possible values:
1. `nil` (no freeze-fallback, default)
1. `nil` (no fallback, default)
Tries the following in order:
- Freeze with Java's `Serializable` interface if possible
- Freeze with Clojure's reader if possible
- Freeze with Java's `Serializable` interface if this seems possible
- Freeze with Clojure's reader if this seems possible
- Throw
2. `:write-unfreezable` keyword
Tries the following in order:
- Freeze with Java's `Serializable` interface if possible
- Freeze with Clojure's reader if possible
- Freeze with Java's `Serializable` interface if this seems possible
- Freeze with Clojure's reader if this seems possible
- Freeze a {:nippy/unfreezable {:type _}} placeholder value
3. [Advanced] Custom (fn [^java.io.DataOutput out item]) that must
write exactly one value to the given `DataOutput` stream"
3. [Advanced] Custom (fn [^java.io.DataOutput out obj]) that must
write an appropriate object type id and payload to the given
`DataOutput` stream."
nil)
@ -431,16 +442,16 @@
;; Unfortunately quite a bit of complexity to do this safely
(def default-freeze-serializable-allowlist
"Allows *any* class-name to be frozen using Java's `Serializable` interface.
"Allows *any* class name to be frozen using Java's `Serializable` interface.
This is generally safe since RCE risk is present only when thawing.
See also `*freeze-serializable-allowlist*`."
#{"*"})
(def default-thaw-serializable-allowlist
"A set of common safe class-names to allow to be frozen using Java's
"A set of common safe class names to allow to be frozen using Java's
`Serializable` interface. PRs welcome for additions.
See also `*thaw-serializable-allowlist*`."
#{"[I" "[F" "[Z" "[B" "[C" "[D" "[S" "[J"
#{"[Z" "[B" "[S" "[I" "[J" "[F" "[D" "[C" "[Ljava.lang.String;"
"java.lang.Throwable"
"java.lang.Exception"
@ -504,10 +515,10 @@
Example allowlist values:
- `(fn allow-class? [class-name] true)` ; Arbitrary predicate fn
- `#{\"java.lang.Throwable\", \"clojure.lang.*\"}` ; Set of class-names
- `#{\"java.lang.Throwable\", \"clojure.lang.*\"}` ; Set of class names
- `\"allow-and-record\"` ; Special value, see [2]
Note that class-names in sets may contain \"*\" wildcards.
Note that class names in sets may contain \"*\" wildcards.
Default allowlist values are:
- default-freeze-serializable-allowlist ; `{\"*\"}` => allow any class
@ -641,15 +652,23 @@
(defn- write-bytes [^DataOutput out ^bytes ba]
(let [len (alength ba)]
(if (zero? len)
(write-id out id-bytes-0)
(write-id out id-byte-array-0)
(do
(enc/cond
(sm-count? len) (do (write-id out id-bytes-sm) (write-sm-count out len))
(md-count? len) (do (write-id out id-bytes-md) (write-md-count out len))
:else (do (write-id out id-bytes-lg) (write-lg-count out len)))
(sm-count? len) (do (write-id out id-byte-array-sm) (write-sm-count out len))
(md-count? len) (do (write-id out id-byte-array-md) (write-md-count out len))
:else (do (write-id out id-byte-array-lg) (write-lg-count out len)))
(.write out ba 0 len)))))
(defmacro ^:private -run! [proc coll] `(do (reduce #(~proc %2) nil ~coll) nil))
(defmacro ^:private -run-kv! [proc m] `(do (reduce-kv #(~proc %2 %3) nil ~m) nil))
(defn- write-array-lg [^DataOutput out array array-len id]
(write-id out id)
(write-lg-count out array-len)
(-run! (fn [in] (-freeze-with-meta! in out)) array))
(defn- write-biginteger [out ^BigInteger n] (write-bytes-lg out (.toByteArray n)))
(defn- write-str-sm* [^DataOutput out ^String s] (write-bytes-sm* out (.getBytes s StandardCharsets/UTF_8)))
@ -677,7 +696,7 @@
(sm-count? len) (do (write-id out id-kw-sm) (write-sm-count out len))
(md-count? len) (do (write-id out id-kw-md) (write-md-count out len))
;; :else (do (write-id out id-kw-lg) (write-lg-count out len)) ; Unrealistic
:else (throw (ex-info "Keyword too long" {:name s})))
:else (truss/ex-info! "Keyword too long" {:name s}))
(.write out ba 0 len)))
@ -689,7 +708,7 @@
(sm-count? len) (do (write-id out id-sym-sm) (write-sm-count out len))
(md-count? len) (do (write-id out id-sym-md) (write-md-count out len))
;; :else (do (write-id out id-sym-lg) (write-lg-count out len)) ; Unrealistic
:else (throw (ex-info "Symbol too long" {:name s})))
:else (truss/ex-info! "Symbol too long" {:name s}))
(.write out ba 0 len)))
@ -729,9 +748,6 @@
(<= y range-uint) (do (write-id out id-long-neg-lg) (.writeInt out (+ y Integer/MIN_VALUE)))
:else (do (write-id out id-long-xl) (.writeLong out n))))))
(defmacro ^:private -run! [proc coll] `(do (reduce #(~proc %2) nil ~coll) nil))
(defmacro ^:private -run-kv! [proc m] `(do (reduce-kv #(~proc %2 %3) nil ~m) nil))
(defn- write-vec [^DataOutput out v]
(let [cnt (count v)]
(if (zero? cnt)
@ -879,14 +895,11 @@
(-run! (fn [in] (-freeze-with-meta! in out)) s)))))
(defn- write-objects [^DataOutput out ^objects ary]
(let [len (alength ary)]
(write-id out id-objects-lg)
(write-lg-count out len)
(-run! (fn [in] (-freeze-with-meta! in out)) ary)))
(defn- write-serializable [^DataOutput out x ^String class-name]
(defn- write-serializable [^DataOutput out x]
(when-debug (println (str "write-serializable: " (type x))))
(when (and (instance? Serializable x) (not (fn? x)))
(let [class-name (.getName (class x))] ; Reflect
(when (freeze-serializable-allowed? class-name)
(let [class-name-ba (.getBytes class-name StandardCharsets/UTF_8)
len (alength class-name-ba)]
@ -894,7 +907,7 @@
(sm-count? len) (do (write-id out id-sz-quarantined-sm) (write-bytes-sm out class-name-ba))
(md-count? len) (do (write-id out id-sz-quarantined-md) (write-bytes-md out class-name-ba))
;; :else (do (write-id out id-sz-quarantined-lg) (write-bytes-md out class-name-ba)) ; Unrealistic
:else (throw (ex-info "Serializable class name too long" {:name class-name})))
:else (truss/ex-info! "Serializable class name too long" {:name class-name}))
;; Legacy: write object directly to out.
;; (.writeObject (ObjectOutputStream. out) x)
@ -903,33 +916,24 @@
;; We'll have object length during thaw, allowing us to skip readObject.
(let [quarantined-ba (ByteArrayOutputStream.)]
(.writeObject (ObjectOutputStream. (DataOutputStream. quarantined-ba)) x)
(write-bytes out (.toByteArray quarantined-ba)))))
(write-bytes out (.toByteArray quarantined-ba)))
true)))))
(defn- write-readable [^DataOutput out x]
(when-debug (println (str "write-readable: " (type x))))
(when (impl/seems-readable? x)
(let [edn (enc/pr-edn x)
edn-ba (.getBytes ^String edn StandardCharsets/UTF_8)
len (alength edn-ba)]
(enc/cond
(sm-count? len) (do (write-id out id-reader-sm) (write-bytes-sm out edn-ba))
(md-count? len) (do (write-id out id-reader-md) (write-bytes-md out edn-ba))
:else (do (write-id out id-reader-lg) (write-bytes-lg out edn-ba)))))
:else (do (write-id out id-reader-lg) (write-bytes-lg out edn-ba)))
true)))
(defn try-write-serializable [out x]
(when (and (instance? Serializable x) (not (fn? x)))
(try
(let [class-name (.getName (class x))] ; Reflect
(when (freeze-serializable-allowed? class-name)
(write-serializable out x class-name)
true))
(catch Throwable _ nil))))
(defn try-write-readable [out x]
(when (impl/seems-readable? x)
(try
(write-readable out x)
true
(catch Throwable _ nil))))
(defn ^:deprecated try-write-serializable [out x] (truss/catching :all (write-serializable out x)))
(defn ^:deprecated try-write-readable [out x] (truss/catching :all (write-readable out x)))
(defn- try-pr-edn [x]
(try
@ -947,29 +951,22 @@
:content (try-pr-edn x)}}
out))
(defn throw-unfreezable [x]
(let [t (type x)]
(throw
(ex-info (str "Unfreezable type: " t)
{:type t
:as-str (try-pr-edn x)}))))
;; Public `-freeze-with-meta!` with different arg order
(defn freeze-to-out!
"Serializes arg (any Clojure data type) to a DataOutput.
This is a low-level util: in most cases you'll want `freeze` instead."
[^DataOutput data-output x] (-freeze-with-meta! x data-output))
;;;; Caching ; Experimental
;;;; Caching
;; Nb: don't use an auto initialValue; can cause thread-local state to
;; accidentally hang around with the use of `freeze-to-out!`, etc.
;; Safer to require explicit activation through `with-cache`.
(def ^ThreadLocal -cache-proxy
"{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing."
;; Nb: don't use an auto initialValue; can cause thread-local state to
;; accidentally hang around with the use of `freeze-to-out!`, etc.
;; Safer to require explicit activation through `with-cache`.
(proxy [ThreadLocal] []))
(defmacro ^:private with-cache
(defmacro with-cache
"Executes body with support for freezing/thawing cached values.
This is a low-level util: you won't need to use this yourself unless
@ -984,9 +981,7 @@
(deftype Cached [val])
(defn cache
"Experimental, subject to change. Feedback welcome!
Wraps value so that future writes of the same wrapped value with same
"Wraps value so that future writes of the same wrapped value with same
metadata will be efficiently encoded as references to this one.
(freeze [(cache \"foo\") (cache \"foo\") (cache \"foo\")])
@ -1034,7 +1029,7 @@
(when first-occurance? (-freeze-with-meta! x-val out)))
:else
;; (throw (ex-info "Max cache size exceeded" {:idx idx}))
;; (truss/ex-info! "Max cache size exceeded" {:idx idx})
(-freeze-with-meta! x-val out) ; Just freeze uncached
))
@ -1051,7 +1046,7 @@
(vswap! cache_ assoc idx x)
x)
v))
(throw (ex-info "No cache_ established, can't thaw. See `with-cache`." {}))))))
(truss/ex-info! "Can't thaw without cache available. See `with-cache`." {})))))
(comment
(thaw (freeze [(cache "foo") (cache "foo") (cache "foo")]))
@ -1096,8 +1091,6 @@
(.writeLong out (.getLeastSignificantBits x))))
(freezer Boolean nil true (if (boolean x) (write-id out id-true) (write-id out id-false)))
(freezer (Class/forName "[B") nil true (write-bytes out x))
(freezer (Class/forName "[Ljava.lang.Object;") nil true (write-objects out x))
(freezer String nil true (write-str out x))
(freezer Keyword nil true (write-kw out x))
(freezer Symbol nil true (write-sym out x))
@ -1107,6 +1100,17 @@
(do (write-id out id-double-0))
(do (write-id out id-double) (.writeDouble out x))))
;; Arrays
(freezer (Class/forName "[B") nil true (write-bytes out x))
(freezer (Class/forName "[Ljava.lang.Object;") nil true (write-array-lg out x (alength ^"[Ljava.lang.Object;" x) id-object-array-lg))
(when (impl/target-release>= 350)
(freezer (Class/forName "[I") nil true (write-array-lg out x (alength ^"[I" x) id-int-array-lg))
(freezer (Class/forName "[J") nil true (write-array-lg out x (alength ^"[J" x) id-long-array-lg))
(freezer (Class/forName "[F") nil true (write-array-lg out x (alength ^"[F" x) id-float-array-lg))
(freezer (Class/forName "[D") nil true (write-array-lg out x (alength ^"[D" x) id-double-array-lg))
(freezer (Class/forName "[Ljava.lang.String;") nil true (write-array-lg out x (alength ^"[Ljava.lang.String;" x) id-string-array-lg)))
(freezer PersistentQueue nil true (write-counted-coll out id-queue-lg x))
(freezer PersistentTreeSet nil true (write-counted-coll out id-sorted-set-lg x))
(freezer PersistentTreeMap nil true (write-kvs out id-sorted-map-lg x))
@ -1124,7 +1128,7 @@
(sm-count? len) (do (write-id out id-record-sm) (write-bytes-sm out class-name-ba))
(md-count? len) (do (write-id out id-record-md) (write-bytes-md out class-name-ba))
;; :else (do (write-id out id-record-lg) (write-bytes-md out class-name-ba)) ; Unrealistic
:else (throw (ex-info "Record class name too long" {:name class-name})))
:else (truss/ex-info! "Record class name too long" {:name class-name}))
(-freeze-without-meta! (into {} x) out)))
@ -1179,13 +1183,20 @@
(write-unfreezable out x)))
;; Without ff
(or
(try-write-serializable out x)
(try-write-readable out x)
(enc/cond
:let [[r1 e1] (try [(write-serializable out x)] (catch Throwable t [nil t]))], r1 r1
:let [[r2 e2] (try [(write-readable out x)] (catch Throwable t [nil t]))], r2 r2
(when-let [fff *final-freeze-fallback*] (fff out x) true) ; Deprecated
(throw-unfreezable x)))))
:if-let [fff *final-freeze-fallback*] (fff out x) ; Deprecated
:else
(let [t (type x)]
(truss/ex-info! (str "Failed to freeze type: " t)
(enc/assoc-some
{:type t
:as-str (try-pr-edn x)}
{:serializable-error e1
:readable-error e2})
(or e1 e2)))))))
;;;;
@ -1199,9 +1210,8 @@
(defn- wrap-header [data-ba head-meta]
(if-let [head-ba (get-head-ba head-meta)]
(enc/ba-concat head-ba data-ba)
(throw
(ex-info (str "Unrecognized header meta: " head-meta)
{:head-meta head-meta}))))
(truss/ex-info! (str "Unrecognized header meta: " head-meta)
{:head-meta head-meta})))
(comment (wrap-header (.getBytes "foo") {:compressor-id :lz4
:encryptor-id nil}))
@ -1250,11 +1260,16 @@
- Drops all support for compression and encryption
- Must be thawed with `fast-thaw`
Equivalent to (but a little faster than) `freeze` with opts:
- :compressor nil
- :encryptor nil
- :no-header? true"
[x]
Equivalent to (but a little faster than) `freeze` with opts
{:no-header? true, :compressor nil, :encryptor nil}.
Intended for use only by advanced users that clearly understand the tradeoffs.
I STRONGLY recommend that most users prefer the standard `freeze` since:
- The Nippy header is useful for data portability and preservation
- Compression is often benefitial at little/no cost
- The performance difference between `freeze` and `fast-freeze` is
often negligible in practice."
^bytes [x]
(let [baos (ByteArrayOutputStream. 64)
dos (DataOutputStream. baos)]
(with-cache (-freeze-with-meta! x dos))
@ -1263,8 +1278,10 @@
(defn freeze
"Serializes arg (any Clojure data type) to a byte array.
To freeze custom types, extend the Clojure reader or see `extend-freeze`."
([x] (freeze x nil))
([x {:as opts
(^bytes [x] (freeze x nil))
(^bytes
[x
{:as opts
:keys [compressor encryptor password serializable-allowlist incl-metadata?]
:or {compressor :auto
encryptor aes128-gcm-encryptor}}]
@ -1323,10 +1340,21 @@
([^DataInput in len] (let [ba (byte-array len)] (.readFully in ba 0 len) ba))
([^DataInput in ]
(enc/case-eval (.readByte in)
id-bytes-0 (byte-array 0)
id-bytes-sm (read-bytes in (read-sm-count in))
id-bytes-md (read-bytes in (read-md-count in))
id-bytes-lg (read-bytes in (read-lg-count in)))))
id-byte-array-0 (byte-array 0)
id-byte-array-sm (read-bytes in (read-sm-count in))
id-byte-array-md (read-bytes in (read-md-count in))
id-byte-array-lg (read-bytes in (read-lg-count in)))))
(defmacro ^:private read-array [in thaw-type array-type array]
(let [thawed-sym (with-meta 'thawed-sym {:tag thaw-type})
array-sym (with-meta 'array-sym {:tag array-type})]
`(let [~array-sym ~array]
(enc/reduce-n
(fn [_# idx#]
(let [~thawed-sym (thaw-from-in! ~in)]
(aset ~'array-sym idx# ~'thawed-sym)))
nil (alength ~'array-sym))
~'array-sym)))
(defn- read-str-sm* [^DataInput in] (String. ^bytes (read-bytes in (read-sm-count* in)) StandardCharsets/UTF_8))
(defn- read-str-sm [^DataInput in] (String. ^bytes (read-bytes in (read-sm-count in)) StandardCharsets/UTF_8))
@ -1346,7 +1374,7 @@
(defmacro ^:private editable? [coll] `(instance? clojure.lang.IEditableCollection ~coll))
(defn- xform* [xform] (enc/catching-xform {:error/msg "Error thrown via `*thaw-xform*`"} xform))
(defn- xform* [xform] (truss/catching-xform {:error/msg "Error thrown via `*thaw-xform*`"} xform))
(let [rf! (fn rf! ([x] (persistent! x)) ([acc x] (conj! acc x)))
rf* (fn rf* ([x] x) ([acc x] (conj acc x)))]
@ -1375,12 +1403,6 @@
(let [rf rf2 ] (rf (enc/reduce-n (fn [acc _] (rf acc (thaw-from-in! in) (thaw-from-in! in))) init n)))))))
(defn- read-kvs-depr [to ^DataInput in] (read-kvs-into to in (quot (.readInt in) 2)))
(defn- read-objects [^objects ary ^DataInput in]
(enc/reduce-n
(fn [^objects ary i]
(aset ary i (thaw-from-in! in))
ary)
ary (alength ary)))
(def ^:private class-method-sig (into-array Class [IPersistentMap]))
@ -1389,14 +1411,12 @@
(try
(custom-reader in)
(catch Exception e
(throw
(ex-info
(truss/ex-info!
(str "Reader exception for custom type id: " type-id)
{:type-id type-id, :prefixed? prefixed?} e))))
(throw
(ex-info
{:type-id type-id, :prefixed? prefixed?} e)))
(truss/ex-info!
(str "No reader provided for custom type id: " type-id)
{:type-id type-id, :prefixed? prefixed?}))))
{:type-id type-id, :prefixed? prefixed?})))
(defn- read-edn [edn]
(try
@ -1473,9 +1493,9 @@
[^DataInput in class-name]
(if (thaw-serializable-allowed? class-name)
(read-object in class-name)
(throw ; No way to skip bytes, so best we can do is throw
(ex-info "Cannot thaw object: `taoensso.nippy/*thaw-serializable-allowlist*` check failed. This is a security feature. See `*thaw-serializable-allowlist*` docstring or https://github.com/ptaoussanis/nippy/issues/130 for details!"
{:class-name class-name}))))
(truss/ex-info! ; No way to skip bytes, so best we can do is throw
"Cannot thaw object: `taoensso.nippy/*thaw-serializable-allowlist*` check failed. This is a security feature. See `*thaw-serializable-allowlist*` docstring or https://github.com/ptaoussanis/nippy/issues/130 for details!"
{:class-name class-name})))
(defn- read-record [in class-name]
(let [content (thaw-from-in! in)]
@ -1573,12 +1593,19 @@
id-cached-sm (thaw-cached (read-sm-count in) in)
id-cached-md (thaw-cached (read-md-count in) in)
id-bytes-0 (byte-array 0)
id-bytes-sm (read-bytes in (read-sm-count in))
id-bytes-md (read-bytes in (read-md-count in))
id-bytes-lg (read-bytes in (read-lg-count in))
id-byte-array-0 (byte-array 0)
id-byte-array-sm (read-bytes in (read-sm-count in))
id-byte-array-md (read-bytes in (read-md-count in))
id-byte-array-lg (read-bytes in (read-lg-count in))
id-objects-lg (read-objects (object-array (read-lg-count in)) in)
id-long-array-lg (read-array in long "[J" (long-array (read-lg-count in)))
id-int-array-lg (read-array in int "[I" (int-array (read-lg-count in)))
id-double-array-lg (read-array in double "[D" (double-array (read-lg-count in)))
id-float-array-lg (read-array in float "[F" (float-array (read-lg-count in)))
id-string-array-lg (read-array in String "[Ljava.lang.String;" (make-array String (read-lg-count in)))
id-object-array-lg (read-array in Object "[Ljava.lang.Object;" (object-array (read-lg-count in)))
id-str-0 ""
id-str-sm* (read-str in (read-sm-count* in))
@ -1721,15 +1748,13 @@
(if (neg? type-id)
(read-custom! in nil type-id) ; Unprefixed custom type
(throw
(ex-info
(truss/ex-info!
(str "Unrecognized type id (" type-id "). Data frozen with newer Nippy version?")
{:type-id type-id}))))
{:type-id type-id})))
(catch Throwable t
(throw
(ex-info (str "Thaw failed against type-id: " type-id)
{:type-id type-id} t))))))
(truss/ex-info! (str "Thaw failed against type-id: " type-id)
{:type-id type-id} t)))))
(let [head-sig head-sig] ; Not ^:const
(defn- try-parse-header [^bytes ba]
@ -1749,33 +1774,31 @@
:lzma2 lzma2-compressor
:lz4 lz4-compressor
:zstd zstd-compressor
:no-header (throw (ex-info ":auto not supported on headerless data." {}))
:else (throw (ex-info ":auto not supported for non-standard compressors." {}))
(do (throw (ex-info (str "Unrecognized :auto compressor id: " compressor-id)
{:compressor-id compressor-id})))))
:no-header (truss/ex-info! ":auto not supported on headerless data." {})
:else (truss/ex-info! ":auto not supported for non-standard compressors." {})
(do (truss/ex-info! (str "Unrecognized :auto compressor id: " compressor-id)
{:compressor-id compressor-id}))))
(defn- get-auto-encryptor [encryptor-id]
(case encryptor-id
nil nil
:aes128-gcm-sha512 aes128-gcm-encryptor
:aes128-cbc-sha512 aes128-cbc-encryptor
:no-header (throw (ex-info ":auto not supported on headerless data." {}))
:else (throw (ex-info ":auto not supported for non-standard encryptors." {}))
(do (throw (ex-info (str "Unrecognized :auto encryptor id: " encryptor-id)
{:encryptor-id encryptor-id})))))
:no-header (truss/ex-info! ":auto not supported on headerless data." {})
:else (truss/ex-info! ":auto not supported for non-standard encryptors." {})
(do (truss/ex-info! (str "Unrecognized :auto encryptor id: " encryptor-id)
{:encryptor-id encryptor-id}))))
(def ^:private err-msg-unknown-thaw-failure "Possible decryption/decompression error, unfrozen/damaged data, etc.")
(def ^:private err-msg-unrecognized-header "Unrecognized (but apparently well-formed) header. Data frozen with newer Nippy version?")
(defn fast-thaw
"Like `thaw` but:
- Drops all support for compression and encryption
- Supports only data frozen with `fast-freeze`
- Drops all support for compression and encryption
Equivalent to (but a little faster than) `thaw` with opts:
- :compressor nil
- :encryptor nil
- :no-header? true"
{:no-header? true, :compressor nil, :encryptor nil}."
[^bytes ba]
(let [dis (DataInputStream. (ByteArrayInputStream. ba))]
(with-cache (thaw-from-in! dis))))
@ -1811,8 +1834,7 @@
(fn ex
([ msg] (ex nil msg))
([e msg]
(throw
(ex-info (str "Thaw failed. " msg)
(truss/ex-info! (str "Thaw failed. " msg)
{:opts
(assoc opts
:compressor compressor
@ -1828,7 +1850,7 @@
'*thaw-serializable-allowlist* *thaw-serializable-allowlist*
'*thaw-xform* *thaw-xform*)}
e))))
e)))
thaw-data
(fn [data-ba compressor-id encryptor-id ex-fn]
@ -1985,7 +2007,7 @@
[{:keys [comparable?] :as opts}]
(let [rng (java.util.Random. 123456) ; Seeded for determinism
rand-nth (fn [coll] (nth coll (.nextInt rng (count coll))))
all
base
{:nil nil
:true true
:false false
@ -2022,7 +2044,6 @@
#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{[] ()}}] #{:a :b}}
[1 [1 2 [1 2 3 [1 2 3 4 [1 2 3 4 5 "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"] {} #{} [] ()]]]]]
:regex #"^(https?:)?//(www\?|\?)?"
:sorted-set (sorted-set 1 2 3 4 5)
:sorted-map (sorted-map :b 2 :a 1 :d 4 :c 3)
:lazy-seq-empty (map identity ())
@ -2034,8 +2055,6 @@
:uri (java.net.URI. "https://clojure.org")
:defrecord (StressRecord. "data")
:deftype (StressType. "data")
:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
:objects (object-array [1 "two" {:data "data"}])
:util-date (java.util.Date. 1577884455500)
:sql-date (java.sql.Date. 1577884455500)
@ -2043,10 +2062,6 @@
:duration (enc/compile-if java.time.Duration (java.time.Duration/ofSeconds 100 100) ::skip)
:period (enc/compile-if java.time.Period (java.time.Period/of 1 1 1) ::skip)
:throwable (Throwable. "Msg")
:exception (Exception. "Msg")
:ex-info (ex-info "Msg" {:data "data"})
:many-longs (vec (repeatedly 512 #(rand-nth (range 10))))
:many-doubles (vec (repeatedly 512 #(double (rand-nth (range 10)))))
:many-strings (vec (repeatedly 512 #(rand-nth ["foo" "bar" "baz" "qux"])))
@ -2056,8 +2071,24 @@
(rand-nth ["foo" "bar" "baz" "qux" ]))))}]
(if comparable?
(dissoc all :bytes :objects :throwable :exception :ex-info :regex)
(do all))))
base
(assoc base
:non-comparable
{:regex #"^(https?:)?//(www\?|\?)?"
:throwable (Throwable. "Msg")
:exception (Exception. "Msg")
:ex-info (ex-info "Msg" {:data "data"})
:arrays
{:boolean (boolean-array (mapv even? (range 32)))
:byte (byte-array (mapv byte (range 32)))
:short (short-array (mapv short (range 32)))
:int (int-array (mapv int (range 32)))
:long (long-array (mapv long (range 32)))
:float (float-array (mapv float (range 32)))
:double (double-array (mapv double (range 32)))
:char (char-array (mapv char (range 32)))
:str (into-array String (mapv str (range 32)))
:object (object-array (mapv vector (range 32)))}}))))
(comment
[(= (stress-data {:comparable? true}) (stress-data {:comparable? true}))
@ -2200,5 +2231,5 @@
(alter-var-root *thaw-serializable-allowlist* f) and/or
(alter-var-root *freeze-serializable-allow-list* f) instead."
[f]
(alter-var-root *freeze-serializable-allowlist* (fn [old] (f (enc/have set? old))))
(alter-var-root *thaw-serializable-allowlist* (fn [old] (f (enc/have set? old))))))
(alter-var-root *freeze-serializable-allowlist* (fn [old] (f (truss/have set? old))))
(alter-var-root *thaw-serializable-allowlist* (fn [old] (f (truss/have set? old))))))

View file

@ -1,6 +1,8 @@
(ns ^:no-doc taoensso.nippy.compression
"Private, implementation detail."
(:require [taoensso.encore :as enc])
(:require
[taoensso.truss :as truss]
[taoensso.encore :as enc])
(:import
[java.nio ByteBuffer]
[java.io
@ -156,7 +158,7 @@
(.read xzs ba 0 len-decomp)
(if (== -1 (.read xzs)) ; Good practice as extra safety measure
nil
(throw (ex-info "LZMA2 Decompress failed: corrupt data?" {:ba ba})))
(truss/ex-info! "LZMA2 Decompress failed: corrupt data?" {:ba ba}))
ba)))
;;;; Public API
@ -174,8 +176,8 @@
(def lz4-compressor
"Default `LZ4` compressor:
- Compression ratio: `C` (0.58 on reference benchmark).
- Compression speed: `A` (238 msecs on reference benchmark).
- Decompression speed: `A+` (31 msecs on reference benchmark).
- Compression speed: `A` (240 msecs on reference benchmark).
- Decompression speed: `A+` (30 msecs on reference benchmark).
Good general-purpose compressor, favours speed.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
@ -184,8 +186,8 @@
(def lzo-compressor
"Default `LZO` compressor:
- Compression ratio: `C` (0.58 on reference benchmark).
- Compression speed: `A` (216 msecs on reference benchmark).
- Decompression speed: `A` (43 msecs on reference benchmark).
- Compression speed: `A` (220 msecs on reference benchmark).
- Decompression speed: `A` (40 msecs on reference benchmark).
Good general-purpose compressor, favours speed.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
@ -195,7 +197,7 @@
"Default `LZMA2` compressor:
- Compression ratio: `A+` (0.4 on reference benchmark).
- Compression speed: `E` (18.5 secs on reference benchmark).
- Decompression speed: `D` (11.8 secs on reference benchmark).
- Decompression speed: `D` (12 secs on reference benchmark).
Specialized compressor, strongly favours ratio.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
@ -204,8 +206,8 @@
(enc/def* snappy-compressor
"Default `Snappy` compressor:
- Compression ratio: `C` (0.58 on reference benchmark).
- Compression speed: `A+` (206 msecs on reference benchmark).
- Decompression speed: `B` (134 msecs on reference benchmark).
- Compression speed: `A+` (210 msecs on reference benchmark).
- Decompression speed: `B` (130 msecs on reference benchmark).
Good general-purpose compressor, favours speed.
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
(SnappyCompressor. false))

View file

@ -2,7 +2,9 @@
"Low-level crypto utils.
Private & alpha, very likely to change!"
(:refer-clojure :exclude [rand-nth])
(:require [taoensso.encore :as enc]))
(:require
[taoensso.truss :as truss]
[taoensso.encore :as enc]))
;; Note that AES128 may be preferable to AES256 due to known attack
;; vectors specific to AES256, Ref. <https://goo.gl/qU4CCV>
@ -45,7 +47,7 @@
(defn take-ba ^bytes [n ^bytes ba] (java.util.Arrays/copyOf ba ^int n)) ; Pads if ba too small
(defn utf8->ba ^bytes [^String s] (.getBytes s "UTF-8"))
(defn- add-salt ^bytes [?salt-ba ba] (if ?salt-ba (enc/ba-concat ?salt-ba ba) ba))
(defn pwd-as-ba ^bytes [utf8-or-ba] (if (string? utf8-or-ba) (utf8->ba utf8-or-ba) (enc/have enc/bytes? utf8-or-ba)))
(defn pwd-as-ba ^bytes [utf8-or-ba] (if (string? utf8-or-ba) (utf8->ba utf8-or-ba) (truss/have enc/bytes? utf8-or-ba)))
(comment (seq (pwd-as-ba "foo")))

View file

@ -1,6 +1,7 @@
(ns ^:no-doc taoensso.nippy.encryption
"Private, implementation detail."
(:require
[taoensso.truss :as truss]
[taoensso.encore :as enc]
[taoensso.nippy.crypto :as crypto]))
@ -14,11 +15,11 @@
(decrypt ^bytes [encryptor pwd ba]))
(defn- throw-destructure-ex [typed-password]
(throw (ex-info
(truss/ex-info!
(str "Expected password form: "
"[<#{:salted :cached}> <password-string>].\n "
"See `aes128-encryptor` docstring for details!")
{:typed-password typed-password})))
{:typed-password typed-password}))
(defn- destructure-typed-pwd [typed-password]
(if (vector? typed-password)

View file

@ -2,6 +2,7 @@
"Private, implementation detail."
(:require
[clojure.string :as str]
[taoensso.truss :as truss]
[taoensso.encore :as enc]))
;;;; Fallback type tests
@ -63,7 +64,7 @@
(when x
(if (string? x)
(if (= x "") #{} (set (mapv str/trim (str/split x #"[,:]"))))
(enc/have set? x))))
(truss/have set? x))))
(comment
(mapv classname-set [nil #{"foo"} "" "foo, bar:baz"])
@ -151,25 +152,56 @@
See that function's docstring for more info."
[] (trim nmax (state_))))
(comment
(count (get-recorded-serializable-classes))
(enc/reduce-n
(fn [_ n] (allow-and-record-any-serializable-class-unsafe (str n)))
nil 0 1e5))
(let [compile
(enc/fmemoize
(fn [x]
(if (allow-and-record? x)
allow-and-record-any-serializable-class-unsafe
(enc/name-filter x))))
fn? fn?
conform?
(fn [x cn]
(if (fn? x)
(x cn) ; Intentionally uncached, can be handy
((compile x) cn)))]
(defn serializable-allowed? [allow-list class-name]
(conform? allow-list class-name)))
;;;; Release targeting
(comment
(set! *print-length* nil)
(vec (sort (keys taoensso.nippy/public-types-spec)))
;; To help support release targeting, we keep track of when new type ids are added
;; To help support release targeting, we track new type ids added over time
(let [id-history ; {<release> #{type-ids}}
{340 ; v3.4.0 (2024-04-30), added 2
;; New: map-entry meta-protocol-key
{350 ; v3.5.0 (2025-04-15), added 5x
;; #{int-array-lg long-array-lg float-array-lg double-array-lg string-array-lg}
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104
105 106 107 108 109 110 111 112 113 114 115 116 117}
340 ; v3.4.0 (2024-04-30), added 2x
;; #{map-entry meta-protocol-key}
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104
105 106 110 111 112 113 114 115}
330 ; v3.3.0 (2023-10-11), added 11
;; New: long-pos-sm long-pos-md long-pos-lg long-neg-sm long-neg-md long-neg-lg
;; str-sm* vec-sm* set-sm* map-sm* sql-date
330 ; v3.3.0 (2023-10-11), added 11x
;; #{long-pos-sm long-pos-md long-pos-lg long-neg-sm long-neg-md long-neg-lg
;; str-sm* vec-sm* set-sm* map-sm* sql-date}
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
@ -182,8 +214,8 @@
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
81 82 83 84 85 86 90 91 100 101 102 105 106 110 111 112 113 114 115}
313 ; v3.1.3 (2022-06-23), added 5
;; New: time-instant time-duration time-period kw-md sym-md
313 ; v3.1.3 (2022-06-23), added 5x
;; #{time-instant time-duration time-period kw-md sym-md}
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
@ -193,16 +225,15 @@
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 80
81 82 90 91 100 101 102 105 106 110 111 112 113 114 115}}]
81 82 90 91 100 101 102 105 106 110 111 112 113 114 115}}
(defn diff [new-release old-release]
(vec (sort (clojure.set/difference (id-history new-release) (id-history old-release))))))
diff
(fn [new-release old-release]
(vec (sort (clojure.set/difference (id-history new-release) (id-history old-release)))))]
(diff 340 330))
(diff 350 340)))
(let [;; Initially target compatibility with v3.2.0 (2020-07-18)
;; Next release will target v3.4.0 (2024-04-30), etc.
target-release
(let [target-release
(enc/get-env {:as :edn, :default 320}
:taoensso.nippy.target-release)
@ -240,28 +271,3 @@
[min-release] (target>= min-release)))
(comment (macroexpand '(target-release>= 340)))
;;;
(comment
(count (get-recorded-serializable-classes))
(enc/reduce-n
(fn [_ n] (allow-and-record-any-serializable-class-unsafe (str n)))
nil 0 1e5))
(let [compile
(enc/fmemoize
(fn [x]
(if (allow-and-record? x)
allow-and-record-any-serializable-class-unsafe
(enc/name-filter x))))
fn? fn?
conform?
(fn [x cn]
(if (fn? x)
(x cn) ; Intentionally uncached, can be handy
((compile x) cn)))]
(defn serializable-allowed? [allow-list class-name]
(conform? allow-list class-name)))

View file

@ -1,10 +1,8 @@
(ns taoensso.nippy.tools
"Utils for community tools that want to add user-configurable Nippy support.
Used by Carmine, Faraday, etc."
(:refer-clojure :exclude [binding])
(:require
[taoensso.encore :as enc :refer [binding]]
[taoensso.encore :as enc]
[taoensso.nippy :as nippy]))
(def ^:dynamic *freeze-opts* nil)
@ -12,9 +10,9 @@
(do
(defmacro with-freeze-opts [opts & body] `(binding [*freeze-opts* ~opts ] ~@body))
(defmacro with-freeze-opts+ [opts & body] `(binding [*freeze-opts* (enc/fast-merge *freeze-opts* ~opts)] ~@body))
(defmacro with-freeze-opts+ [opts & body] `(binding [*freeze-opts* (enc/merge *freeze-opts* ~opts)] ~@body))
(defmacro with-thaw-opts [opts & body] `(binding [*thaw-opts* ~opts ] ~@body))
(defmacro with-thaw-opts+ [opts & body] `(binding [*thaw-opts* (enc/fast-merge *thaw-opts* ~opts)] ~@body)))
(defmacro with-thaw-opts+ [opts & body] `(binding [*thaw-opts* (enc/merge *thaw-opts* ~opts)] ~@body)))
(deftype WrappedForFreezing [val opts])
(defn wrapped-for-freezing? [x] (instance? WrappedForFreezing x))
@ -27,7 +25,7 @@
See also `tools/freeze`."
([x ] (wrap-for-freezing x nil))
([x wrap-opts]
(let [captured-opts (enc/fast-merge *freeze-opts* wrap-opts)] ; wrap > dynamic
(let [captured-opts (enc/merge *freeze-opts* wrap-opts)] ; wrap > dynamic
(if (instance? WrappedForFreezing x)
(let [^WrappedForFreezing x x]
(if (= (.-opts x) captured-opts)
@ -47,11 +45,11 @@
([x ] (freeze x nil))
([x default-opts]
(let [default-opts (get default-opts :default-opts default-opts) ; Back compatibility
active-opts (enc/fast-merge default-opts *freeze-opts*)] ; dynamic > default
active-opts (enc/merge default-opts *freeze-opts*)] ; dynamic > default
(if (instance? WrappedForFreezing x)
(let [^WrappedForFreezing x x]
(nippy/freeze (.-val x) (enc/fast-merge active-opts (.-opts x)))) ; captured > active!
(nippy/freeze (.-val x) (enc/merge active-opts (.-opts x)))) ; captured > active!
(nippy/freeze x active-opts)))))
(defn thaw
@ -63,7 +61,7 @@
([ba ] (thaw ba nil))
([ba default-opts]
(let [default-opts (get default-opts :default-opts default-opts) ; Back compatibility
active-opts (enc/fast-merge default-opts *thaw-opts*)] ; dynamic > default
active-opts (enc/merge default-opts *thaw-opts*)] ; dynamic > default
(nippy/thaw ba active-opts))))

View file

@ -4,8 +4,11 @@
[clojure.test.check :as tc]
[clojure.test.check.generators :as tc-gens]
[clojure.test.check.properties :as tc-props]
[taoensso.truss :as truss :refer [throws?]]
[taoensso.encore :as enc :refer [ba=]]
[taoensso.nippy :as nippy :refer [freeze thaw]]
[taoensso.nippy.impl :as impl]
[taoensso.nippy.tools :as tools]
[taoensso.nippy.compression :as compr]
[taoensso.nippy.crypto :as crypto]
[taoensso.nippy-benchmarks :as benchmarks]))
@ -53,10 +56,6 @@
#(freeze % {:password [:salted "p"]}))
test-data)))
(let [d (nippy/stress-data {})]
[(is (= (vec (:bytes d)) ((comp vec thaw freeze) (:bytes d))))
(is (= (vec (:objects d)) ((comp vec thaw freeze) (:objects d))))])
(is (= test-data ((comp #(thaw % {:compressor nippy/lzma2-compressor})
#(freeze % {:compressor nippy/lzma2-compressor}))
test-data)))
@ -75,9 +74,9 @@
#(freeze % {:compressor nippy/zstd-compressor}))
test-data)))
(is (enc/throws? Exception (thaw (freeze test-data {:password "malformed"}))))
(is (enc/throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
(is (enc/throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
(is (throws? Exception (thaw (freeze test-data {:password "malformed"}))))
(is (throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
(is (throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
(is
(= "payload"
@ -94,14 +93,23 @@
(let [n range-uint+] (= (thaw (freeze n)) n))
(let [n (- range-uint+)] (= (thaw (freeze n)) n))]))
(is (enc/throws? :ex-info "Unfreezable type" (nippy/freeze (fn []))))
(is (throws? :ex-info "Failed to freeze type" (nippy/freeze (fn []))))
(testing "Clojure v1.10+ metadata protocol extensions"
[(is (enc/throws? :ex-info "Unfreezable type" (nippy/freeze (with-meta [] {:a :A, 'b (fn [])}))))
[(is (throws? :ex-info "Failed to freeze type" (nippy/freeze (with-meta [] {:a :A, 'b (fn [])}))))
(is (= {:a :A} (meta (nippy/thaw (nippy/freeze (with-meta [] {:a :A, 'b/c (fn [])}))))))
(is (= nil (meta (nippy/thaw (nippy/freeze (with-meta [] { 'b/c (fn [])})))))
"Don't attach empty metadata")])
(let [d (nippy/stress-data {})]
[(is (= (vec (:bytes d)) ((comp vec thaw freeze) (:bytes d))))
(is (= (vec (:objects d)) ((comp vec thaw freeze) (:objects d))))])
(testing "Arrays"
(binding [nippy/*thaw-serializable-allowlist* nippy/default-freeze-serializable-allowlist]
(mapv (fn [[k aval]] (is (= (vec aval) (-> aval nippy/freeze nippy/thaw vec)) (name k)))
(get-in (nippy/stress-data {}) [:non-comparable :arrays]))))
(is (gen-test 1600 [gen-data] (= gen-data (thaw (freeze gen-data)))) "Generative")])
;;;; Custom types & records
@ -112,7 +120,7 @@
(deftest _types
[(testing "Extend to custom type"
[(is
(enc/throws? Exception ; No thaw extension yet
(throws? Exception ; No thaw extension yet
(do
(alter-var-root #'nippy/*custom-readers* (constantly {}))
(nippy/extend-freeze MyType 1 [x s]
@ -323,7 +331,7 @@
[(is (= nippy/*thaw-serializable-allowlist* #{"base.1" "base.2" "add.1" "add.2"})
"JVM properties override initial allowlist values")
(is (enc/throws? Exception (nippy/freeze sem {:serializable-allowlist #{}}))
(is (throws? Exception (nippy/freeze sem {:serializable-allowlist #{}}))
"Can't freeze Serializable objects unless approved by allowlist")
(is (sem?
@ -435,8 +443,8 @@
(is (= (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze []))) []) "rf not run on empty colls")
(let [ex (enc/throws :default (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze [:a :b]))))]
(is (= (-> ex enc/ex-cause enc/ex-cause ex-data :call) '(rf acc in)) "Error thrown via `*thaw-xform*`"))])
(let [ex (truss/throws :default (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze [:a :b]))))]
(is (= (-> ex ex-cause ex-cause ex-data :call) '(rf acc in)) "Error thrown via `*thaw-xform*`"))])
;;;; Compressors
@ -453,7 +461,7 @@
(print ".") (flush)
(dotimes [_ 1000]
(is
(nil? (enc/catching (compr/decompress c (crypto/rand-bytes 1024))))
(nil? (truss/catching :all (compr/decompress c (crypto/rand-bytes 1024))))
"Decompression never crashes JVM, even against invalid data")))
(println)))

View file

@ -107,7 +107,10 @@ Deserialize it:
Couldn't be simpler!
See also the lower-level [`freeze-to-out!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-freeze-to-out.21) and [`thaw-from-in!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-thaw-from-in.21) fns for operating on `DataOutput` and `DataInput` types directly.
# Streaming
- To serialize directly to a `java.io.DataInput`, see [`freeze-to-out!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-freeze-to-out.21).
- To deserialize directly from a `java.io.DataOutput`, see [`thaw-from-in!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-thaw-from-in.21).
# Encryption

View file

@ -0,0 +1,30 @@
# Data longevity
Nippy is widely used to store **long-lived** data and promises (as always) that **data serialized today should be readable by all future versions of Nippy**.
But please note that the **converse is not generally true**:
- Nippy `vX` **should** be able to read all data from Nippy `vY<=X` (backwards compatibility)
- Nippy `vX` **may/not** be able to read all data from Nippy `vY>X` (forwards compatibility)
# Rolling updates and rollback
From time to time, Nippy may introduce:
- Support for serializing **new types**
- Optimizations to the serialization of **pre-existing types**
To help ease **rolling updates** and to better support **rollback**, Nippy (since version v3.4.1) will always introduce such changes over **two version releases**:
- Release 1: to add **read support** for the new types
- Release 2: to add **write support** for the new types
Starting from v3.4.1, Nippy's release notes will **always clearly indicate** if a particular update sequence is recommended.
# Stability of byte output
It has **never been an objective** of Nippy to offer **predictable byte output**, and I'd generally **recommend against** depending on specific byte output.
However, I know that a small minority of users *do* have specialized needs in this area.
So starting with Nippy v3.4, Nippy's release notes will **always clearly indicate** if any changes to byte output are expected.

View file

@ -1,4 +1,4 @@
> This article was kindly contributed by a Nippy user (@Outrovurt)
> This article is **community content** kindly contributed by a Nippy user (@Outrovurt)
This article describes a number of use cases where you need to make changes to your code which will have some impact on data you have already frozen using Nippy, and how best to manage each specific case. We will also discuss custom freezing and thawing.