[nop] Drop use of deprecated enc/binding

Ref. https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#*ctx*

.github/workflows/graal-tests.yml

@@ -10,7 +10,7 @@ jobs:
 
     runs-on: ${{ matrix.os }}
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: graalvm/setup-graalvm@v1
         with:
           version: 'latest'
@@ -18,12 +18,12 @@ jobs:
           components: 'native-image'
           github-token: ${{ secrets.GITHUB_TOKEN }}
 
-      - uses: DeLaGuardo/setup-clojure@10.0
+      - uses: DeLaGuardo/setup-clojure@12.5
         with:
           lein: latest
           bb: latest
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: ~/.m2/repository
           key: deps-${{ hashFiles('deps.edn') }}

.github/workflows/main-tests.yml

@@ -5,22 +5,22 @@ jobs:
   tests:
     strategy:
       matrix:
-        java: ['17', '18', '19']
+        java: ['17', '19', '21']
         os: [ubuntu-latest]
 
     runs-on: ${{ matrix.os }}
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
-      - uses: actions/setup-java@v3
+      - uses: actions/setup-java@v4
         with:
           distribution: 'corretto'
           java-version: ${{ matrix.java }}
 
-      - uses: DeLaGuardo/setup-clojure@10.0
+      - uses: DeLaGuardo/setup-clojure@12.5
         with:
           lein: latest
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         id: cache-deps
         with:
           path: ~/.m2/repository

CHANGELOG.md

@@ -2,26 +2,257 @@ This project uses [**Break Versioning**](https://www.taoensso.com/break-versioni
 
 ---
 
-# `v3.3.0-RC2` (2023-09-25)
+# `v3.5.0` (2025-04-15)
 
-> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.3.3-RC2)
+- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.5.0)
+- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
 
-Identical to `v3.3.0-RC1` except:
+This is a **general maintenance release** focused on updating dependencies and laying groundwork (read support) for new array types coming in Nippy v3.6.
 
-- Improves some docstrings
+It **drops support for Clojure v1.9** but should otherwise be a safe update from (at least) all recent versions of Nippy.
-- Improves generative unit tests
-- Updates internal dependencies
 
-If no unexpected problems come up, `v3.3.0` final is planned for release by the end of September.
+## Since `v3.5.0-RC1` (2024-10-28)
 
+- **\[mod]** Drop official Clojure v1.9 support \[da57206]
+- \[new] Add string array type to default [thaw-serializable-allowlist](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*thaw-serializable-allowlist*) \[bfba594]
+- \[new] Use [Truss exceptions](https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#ex-info) on errors \[8d62dc2]
+- \[new] [#184] Incl. cause on non-native freeze failures \[8d10765]
+- \[doc] Clarify `*freeze-fallback*` docstring \[1026ea0]
 
-# `v3.3.0-RC1` (2023-08-02)
+## Since `v3.4.2` (2024-05-26)
 
-> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.3.3-RC1)
+- **\[mod]** Drop official Clojure v1.9 support \[da57206]
+- \[new] Add string array type to default [thaw-serializable-allowlist](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*thaw-serializable-allowlist*) \[bfba594]
+- \[new] Use [Truss exceptions](https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#ex-info) on errors \[8d62dc2]
+- \[new] [#184] Incl. cause on non-native freeze failures \[8d10765]
+- \[new] [#175] Mark [cache](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#cache) feature as stable \[b217db5]
+- \[doc] Clarify `*freeze-fallback*` docstring \[1026ea0]
+
+## Migration info
+
+| Updating from Nippy       | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
+| :------------------------ | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
+| `v3.5.0-RC1` (2024-10-28) | -               | -                                                                                                                       | -                           |
+| `v3.4.2` (2024-05-26)     | -               | -                                                                                                                       | -                           |
+| `v3.4.1` (2024-05-02)     | -               | -                                                                                                                       | -                           |
+| `v3.4.0` (2024-04-30)     | -               | **Yes**                                                                                                                 | -                           |
+| `v3.3.0` (2023-10-11)     | -               | -                                                                                                                       | -                           |
+| `v3.2.0` (2022-07-18)     | -               | -                                                                                                                       | -                           |
+| `v3.1.3` (2022-06-23)     | -               | -                                                                                                                       | -                           |
+
+> [1] Relevant only when Nippy introduces support for new types **AND** you plan to update Nippy with a **rolling update** (coexisting new and old versions).
+
+If updating from older versions of Nippy, please see the relevant release notes.
+
+As always:
+
+- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
+- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
+- **Please report any unexpected problems** 🙏
+
+\- [Peter Taoussanis](https://www.taoensso.com)
+
+---
+
+# `v3.5.0-RC1` (2024-10-28)
+
+- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.5.0-RC1)
+- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
+
+This is a **non-breaking maintenance release** that updates dependencies and includes read support for more native array types to be introduced in a future v3.6 release.
+
+It should be safe to update from (at least) all recent versions of Nippy.
+
+| Updating from Nippy   | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
+| :-------------------- | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
+| `v3.4.2` (2024-05-26) | -               | -                                                                                                                       | -                           |
+| `v3.4.1` (2024-05-02) | -               | -                                                                                                                       | -                           |
+| `v3.4.0` (2024-04-30) | -               | Yes                                                                                                                     | -                           |
+| `v3.3.0` (2023-10-11) | -               | -                                                                                                                       | -                           |
+| `v3.2.0` (2022-07-18) | -               | -                                                                                                                       | -                           |
+| `v3.1.3` (2022-06-23) | -               | -                                                                                                                       | -                           |
+
+> [1] Relevant only when introducing support for new types, and for users that do rolling updates
+
+If updating from older versions of Nippy, please see the relevant release notes.
+
+As always:
+
+- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
+- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
+- **Please report any unexpected problems** 🙏
+
+\- [Peter Taoussanis](https://www.taoensso.com)
+
+---
+
+# `v3.4.2` (2024-05-26)
+
+- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.2)
+- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
+
+⚠️ This release addresses a [**security vulnerability**](https://github.com/taoensso/nippy/security/advisories/GHSA-vw78-267v-588h) in Nippy's upstream compression library and is **recommended for all existing users**.
+
+It should be a **straight-forward and non-breaking update** for almost everyone:
+
+| Updating from Nippy   | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
+| :-------------------- | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
+| `v3.4.1` (2024-05-02) | -               | -                                                                                                                       | -                           |
+| `v3.4.0` (2024-04-30) | -               | Yes                                                                                                                     | -                           |
+| `v3.3.0` (2023-10-11) | -               | -                                                                                                                       | -                           |
+| `v3.2.0` (2022-07-18) | -               | -                                                                                                                       | -                           |
+| `v3.1.3` (2022-06-23) | -               | -                                                                                                                       | -                           |
+
+> [1] Relevant only when introducing support for new types, and for users that do rolling updates
+
+If updating from older versions of Nippy, please see the relevant release notes.
+
+As always:
+
+- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
+- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
+- **Please report any unexpected problems** 🙏
+
+\- [Peter Taoussanis](https://www.taoensso.com)
+
+---
+
+# `v3.4.1` (2024-05-02)
+
+> **Dep**: Nippy is [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.1).  
+> **Versioning**: Nippy uses [Break Versioning](https://www.taoensso.com/break-versioning).
+
+Like [`v3.4.0`](https://github.com/taoensso/nippy/releases/tag/v3.4.0) but introduces an internal mechanism to help make it easier for some users that do **rolling updates** from earlier versions of Nippy.
+
+Still, the usual warning applies: data **frozen by Nippy version X** should ideally be **thawed by version >= X**, otherwise you run the risk of the thaw throwing when unfamiliar types are encountered. Please note that this can affect **rolling updates**, and can limit your ability to **revert a Nippy update**. Please ensure adequate testing in your environment before updating against production data!
+
+\- [Peter Taoussanis](https://www.taoensso.com)
+
+## Changes since `v3.4.0`
+
+* [mod]  Due to some internal format changes, Nippy `v3.4.1` may produce **different serialized output** to `v3.4.0` and earlier versions of Nippy. Most users won't care about this, but you could be affected if you depend on specific serialized byte values (for example by comparing serialized output between different versions of Nippy).
+
+---
+
+# `v3.4.0` (2024-04-30)
+
+> **Dep**: Nippy is [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0).  
+> **Versioning**: Nippy uses [Break Versioning](https://www.taoensso.com/break-versioning).
+
+This is a non-breaking **feature and maintenance** release and should be a safe update for existing users. But as always, please **test carefully and report any unexpected problems**, thank you! 🙏
+
+**IMPORTANT**: data **frozen by Nippy version X** should always be **thawed by version >= X**, otherwise you run the risk of the thaw throwing when encountering unfamiliar types. Please note that this can affect **rolling updates**, and can limit your ability to **revert a Nippy update**. Please ensure adequate testing in your environment before updating against production data.
+
+\- [Peter Taoussanis](https://www.taoensso.com)
+
+## Changes since `v3.3.0` (2023-10-11)
+
+* 82a050b [mod] Don't attach empty metadata (meta will now be `nil` rather than `{}`)
+
+## Fixes since `v3.3.0` (2023-10-11)
+
+* 92c4a83 [fix] Broken `*final-freeze-fallback*` default val
+
+## New since `v3.3.0` (2023-10-11)
+
+* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
+* 6ad5aeb [new] Add `:zstd` compressor, new compressor backend
+* 9db09e1 [new] [#163] Track serialized output in tests
+* dcc6b08 [new] [#164] Update benchmarks
+* f3ff7ae [new] Add native `MapEntry` freezer
+* 37cf415 [new] [#171] Auto strip metadata protocol extensions
+* Misc internal improvements
+
+## Everything since `v3.4.0-RC3` (2024-04-10)
+
+* Update dependencies
+
+---
+
+# `v3.4.0-RC3` (2024-04-10)
+
+> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0-RC3), this project uses [Break Versioning](https://www.taoensso.com/break-versioning).
 
 This is a non-breaking **feature and maintenance** pre-release.  
+Please **test carefully and report any unexpected problems**, thank you! 🙏
+
+## New since `v3.3.0`
+
+* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
+* 6ad5aeb [new] Add `:zstd` compressor, new compressor backend
+* 9db09e1 [new] [#163] Track serialized output in tests
+* dcc6b08 [new] [#164] Update benchmarks
+* f3ff7ae [new] Add native `MapEntry` freezer
+* 37cf415 [new] [#171] Auto strip metadata protocol extensions
+* Misc internal improvements
+
+## Everything since `v3.4.0-RC2`
+
+* 82a050b [mod] Don't attach empty metadata
+* 92c4a83 [fix] Broken `*final-freeze-fallback*` default val
+* 37cf415 [new] [#171] Auto strip metadata protocol extensions
+* Update dependencies
+
+
+# `v3.4.0-RC2` (2024-02-26)
+
+> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0-RC2), this project uses [Break Versioning](https://www.taoensso.com/break-versioning).
+
+This is a non-breaking **feature and maintenance** pre-release.  
+Please **test carefully and report any unexpected problems**, thank you! 🙏
+
+## New since `v3.3.0`
+
+* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
+* 6ad5aeb [new] Add `:zstd` compressor, new compressor backend
+* 9db09e1 [new] [#163] Track serialized output in tests
+* dcc6b08 [new] [#164] Update benchmarks
+* f3ff7ae [new] Add native `MapEntry` freezer
+* Misc internal improvements
+
+## Everything since `v3.4.0-RC1`
+
+* cb5b7cf [fix] [#169] Can't auto-identify `:zstd` compressor when decompressing
+* cb0b871 Revert [mod] 578c585 (upstream fix now available)
+* Update dependencies
+
+---
+
+# `v3.4.0-RC1` (2024-02-06)
+
+> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0-RC1), this project uses [Break Versioning](https://www.taoensso.com/break-versioning).
+
+This is a non-breaking **feature and maintenance** pre-release.  
+Please **test carefully and report any unexpected problems**, thank you! 🙏
+
+## Changes since `v3.3.0`
+
+* 578c585 [mod] Remove `nippy/snappy-compressor`
+
+## New since `v3.3.0`
+
+* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
+* 6ad5aeb [new] Add `:zstd` compressor, new compressor backend
+* 9db09e1 [new] [#163] Track serialized output in tests
+* dcc6b08 [new] [#164] Update benchmarks
+* f3ff7ae [new] Add native `MapEntry` freezer
+* Misc internal improvements
+
+---
+
+# `v3.3.0` (2023-10-11)
+
+> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.3.3), this project uses [Break Versioning](https://www.taoensso.com/break-versioning).
+
+Identical to `v3.3.0-RC2`.
+
+This is a non-breaking **feature and maintenance** release.  
 Please test carefully and report any unexpected problems, thank you! 🙏
 
+## Changes since `v3.2.0`
+
+* [mod] Due to micro-optimizations of some elementary types, Nippy v3.3 may produce **different serialized output** to earlier versions of Nippy. Most users won't care about this, but you could be affected if you depend on specific serialized byte values (for example by comparing serialized output between different versions of Nippy).
+
 ## Fixes since `v3.2.0`
 
 * fa1cc66 [fix] [#143] Don't freeze meta info for types that don't support `with-meta`
@@ -38,6 +269,75 @@ Please test carefully and report any unexpected problems, thank you! 🙏
 * 3ac06b6 [new] Refactor tools ns, embed dynamic `*freeze-opts*` in wrappers
 * 129ce95 [new] [#151] [#140] Add experimental `public-types-spec`
 
+## Other improvements since `v3.2.0`
+
+* Improved some docstrings
+* Improved generative unit tests
+* Updated internal dependencies
+
+---
+
+# `v3.4.0-beta1` (2023-09-26)
+
+> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0-beta1)
+
+This is a non-breaking **feature** pre-release.  
+Please **test carefully and report any unexpected problems**, thank you! 🙏
+
+## New since `v3.3.0-RC2`
+
+* 6ad5aeb [new] Add `:zstd` compressor, new (faster) compressor backend, better docstrings
+* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
+* f3ff7ae [new] Add native `MapEntry` freezer
+* fef079d [new] Add subvec to stress data
+* Misc internal improvements
+
+## Other improvements since `v3.3.0-RC2`
+
+* e0cd003 [nop] Update docs
+* 99970d5 [nop] Update benchmark results
+* bcf7673 [nop] Move benchmarks ns under tests dir
+
+---
+
+# `v3.3.0-RC2` (2023-09-25)
+
+> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.3.0-RC2)
+
+Identical to `v3.3.0-RC1` except:
+
+* Improves some docstrings
+* Improves generative unit tests
+* Updates internal dependencies
+
+If no unexpected problems come up, `v3.3.0` final is planned for release by the end of September.
+
+---
+
+# `v3.3.0-RC1` (2023-08-02)
+
+> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.3.0-RC1)
+
+This is a non-breaking **feature and maintenance** pre-release.  
+Please **test carefully and report any unexpected problems**, thank you! 🙏
+
+## Fixes since `v3.2.0`
+
+* fa1cc66 [fix] [#143] Don't freeze meta info for types that don't support `with-meta`
+
+## New since `v3.2.0`
+
+* 89f98b4 [new] [#153] PoC: transducer support on thaw
+* 60bc4e9 [new] [Storage efficiency] PoC: unsigned counts for small core colls
+* 0a9d670 [new] [Storage efficiency] PoC: separate signed long types
+* 8778fa4 [new] Include `:bindings` in ex-data of thaw failures
+* aba153e [new] [#159] Add native impln for `java.sql.Date` (@philomates)
+* d8b1825 [new] [#158] Add `java.lang.ClassCastException` to default thaw allow list (@carlosgeos)
+* 8b7186a [new] Update [benchmark results](https://github.com/taoensso/nippy#performance)
+* 3ac06b6 [new] Refactor tools ns, embed dynamic `*freeze-opts*` in wrappers
+* 129ce95 [new] [#151] [#140] Add experimental `public-types-spec`
+
+---
 
 # `v3.2.0` (2022-07-18)
 
@@ -68,6 +368,7 @@ Please test carefully and report any unexpected problems, thank you! 🙏
 The boxed Boolean bug has been around since the first version of Nippy and is mostly
 relevant to users doing Java interop. For more info see: https://github.com/taoensso/nippy/commit/8909a32bdd654a136da385e0e09c9cc44416f964
 
+---
 
 # `v3.2.0-RC3` (2022-06-27)
 
@@ -96,6 +397,7 @@ relevant to users doing Java interop. For more info see: https://github.com/taoe
 The boxed Boolean bug has been around since the first version of Nippy and is mostly
 relevant to users doing Java interop. For more info see: https://github.com/taoensso/nippy/commit/8909a32bdd654a136da385e0e09c9cc44416f964
 
+---
 
 # `v3.2.0-RC2` (2022-06-23)
 
@@ -123,6 +425,7 @@ relevant to users doing Java interop. For more info see: https://github.com/taoe
 The boxed Boolean bug has been around since the first version of Nippy and is mostly
 relevant to users doing Java interop. For more info see: https://github.com/taoensso/nippy/commit/8909a32bdd654a136da385e0e09c9cc44416f964
 
+---
 
 # `v3.1.3` (2022-06-23)
 
@@ -141,6 +444,7 @@ relevant to users doing Java interop. For more info see: https://github.com/taoe
 The boxed Boolean bug has been around since the first version of Nippy and is mostly
 relevant to users doing Java interop. For more info see: https://github.com/taoensso/nippy/commit/8909a32bdd654a136da385e0e09c9cc44416f964
 
+---
 
 # `v3.1.1` (2020-11-18)
 
@@ -157,7 +461,9 @@ relevant to users doing Java interop. For more info see: https://github.com/taoe
 
 [1] Keywords or symbols with >127 characters in their name
 
-# `v3.1.0` / (2020-11-06)
+---
+
+# `v3.1.0` (2020-11-06)
 
 ```clojure
 [com.taoensso/nippy "3.1.0"]
@@ -175,8 +481,9 @@ relevant to users doing Java interop. For more info see: https://github.com/taoe
 
 * Add several standard `java.time` classes to default `*thaw-serializable-whitelist*`.
 
+---
 
-## `v3.1.0-RC1` (2020-10-24)
+# `v3.1.0-RC1` (2020-10-24)
 
 ```clojure
 [com.taoensso/nippy "3.1.0-RC1"]
@@ -188,6 +495,7 @@ relevant to users doing Java interop. For more info see: https://github.com/taoe
 
 * [#135 #128] Added native `freeze/thaw` support for `java.time.Instant` on JVM 8+ (@cnuernber).
 
+---
 
 # `v3.0.0` (2020-09-20)
 
@@ -247,4 +555,4 @@ Likely breaking. Please see [#130] for **detailed upgrade instructions**.
 
 # Earlier releases
 
-See [here](https://github.com/taoensso/nippy/releases) for earlier releases.
+See [here](https://github.com/taoensso/nippy/releases) for earlier releases.

README.md

@@ -1,21 +1,19 @@
 <a href="https://www.taoensso.com/clojure" title="More stuff by @ptaoussanis at www.taoensso.com"><img src="https://www.taoensso.com/open-source.png" alt="Taoensso open source" width="340"/></a>  
-[**Documentation**](#documentation) | [Latest releases](#latest-releases) | [Get support][GitHub issues]
+[**API**][cljdoc] | [**Wiki**][GitHub wiki] | [Latest releases](#latest-releases) | [Get support][GitHub issues]
 
 # Nippy
 
-### The fastest serialization library for Clojure
+### Fast serialization library for Clojure
 
 Clojure's rich data types are awesome. And its [reader](https://clojure.org/reference/reader) allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).
 
-Nippy is an attempt to provide a reliable, high-performance **drop-in alternative to the reader**.
+Nippy is a mature, high-performance **drop-in alternative to the reader**.
 
-Used by [Carmine](https://www.taoensso.com/carmine), [Faraday](https://www.taoensso.com/faraday), [PigPen](https://github.com/Netflix/PigPen), [Onyx](https://github.com/onyx-platform/onyx), 
+It is used at scale by [Carmine](https://www.taoensso.com/carmine), [Faraday](https://www.taoensso.com/faraday), [PigPen](https://github.com/Netflix/PigPen), [Onyx](https://github.com/onyx-platform/onyx), [XTDB](https://github.com/xtdb/xtdb), [Datalevin](https://github.com/juji-io/datalevin), and others.
-[XTDB](https://github.com/xtdb/xtdb), [Datalevin](https://github.com/juji-io/datalevin), and others.
 
 ## Latest release/s
 
-- `2023-09-25` `3.3.0-RC2` (dev): [changes](../../releases/tag/v3.3.0-RC2)
+- `2025-04-15` `v3.5.0`: [release info](../../releases/tag/v3.5.0)
-- `2022-07-18` `3.2.0` (stable):  [changes](../../releases/tag/v3.2.0)
 
 [![Main tests][Main tests SVG]][Main tests URL]
 [![Graal tests][Graal tests SVG]][Graal tests URL]
@@ -24,33 +22,80 @@ See [here][GitHub releases] for earlier releases.
 
 ## Why Nippy?
 
-- Small, simple **all-Clojure** library
+- Small, simple **pure-Clojure** library
 - **Terrific performance**: the [best](#performance) for Clojure that I'm aware of
 - Comprehensive support for [all standard data types](../../wiki/1-Getting-started#deserializing)
 - Easily extendable to [custom data types](../../wiki/1-Getting-started#custom-types)
-- **Robust test suite**, incl. full coverage for every supported type
+- **Robust test suite** incl. coverage of every supported type
-- Auto fallback to Java Serializable when available
+- **Mature** and widely used in production for 12+ years
-- Auto fallback to Clojure Reader for all other types (including tagged literals
+- Optional auto fallback to [Java Serializable](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*freeze-serializable-allowlist*) for [safe](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*freeze-serializable-allowlist*) types
-- Pluggable **compression** with built-in [LZ4](https://code.google.com/p/lz4/)
+- Optional auto fallback to Clojure Reader (including tagged literals)
-- Pluggable [encryption](../../wiki/1-Getting-started#encryption) with built-in AES128
+- Optional smart **compression** with [LZ4](https://code.google.com/p/lz4/) or [Zstandard](https://facebook.github.io/zstd/)
+- Optional [encryption](../../wiki/1-Getting-started#encryption) with AES128
 - [Tools](https://taoensso.github.io/nippy/taoensso.nippy.tools.html) for easy + robust **integration into 3rd-party libraries**, etc.
 - Powerful [thaw transducer](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*thaw-xform*) for flexible data inspection and transformation
 
+## Quick example
+
+Nippy's super easy to use:
+
+```clojure
+(require '[taoensso.nippy :as nippy])
+
+;; Freeze any Clojure value
+(nippy/freeze <my-value>) ; => Serialized byte[]
+
+;; Thaw the byte[] to get back the original value:
+(nippy/thaw (nippy/freeze <my-value>)) ; => <my-value>
+```
+
+See the [wiki](https://github.com/taoensso/nippy/wiki/1-Getting-started#deserializing) for more.
+
+## Operational considerations
+
+### Data longevity
+
+Nippy is widely used to store **long-lived** data and promises (as always) that **data serialized today should be readable by all future versions of Nippy**.
+
+But please note that the **converse is not generally true**:
+
+- Nippy `vX` **should** be able to read all data from Nippy `vY<=X` (backwards compatibility)
+- Nippy `vX` **may/not** be able to read all data from Nippy `vY>X` (forwards compatibility)
+
+### Rolling updates and rollback
+
+From time to time, Nippy may introduce:
+
+- Support for serializing **new types**
+- Optimizations to the serialization of **pre-existing types**
+
+To help ease **rolling updates** and to better support **rollback**, Nippy (since version v3.4) will always introduce such changes over **two version releases**:
+
+- Release 1: to add **read support** for the new types
+- Release 2: to add **write support** for the new types
+
+Starting from v3.4, Nippy's release notes will **always clearly indicate** if a particular update sequence is recommended.
+
+### Stability of byte output
+
+It has **never been an objective** of Nippy to offer **predictable byte output**, and I'd generally **recommend against** depending on specific byte output.
+
+However, I know that a small minority of users *do* have specialized needs in this area.
+
+So starting with Nippy v3.4, Nippy's release notes will **always clearly indicate** if any changes to byte output are expected.
+
 ## Performance
 
-Since its earliest versions, Nippy has consistently been the **fastest serialization library for Clojure** that I'm aware of. It offers:
+Nippy is fast! Latest [benchmark](../../blob/master/test/taoensso/nippy_benchmarks.clj) results:
-
-- Roundtrip times **>12x faster** than `tools.reader` with **60% smaller** data size.
-- Roundtrip times **>2x faster** than `data.fressian` with **30% smaller** data size.
 
 ![benchmarks-png](../../raw/master/benchmarks.png)
 
-The [benchmark code](../../blob/master/src/taoensso/nippy/benchmarks.clj) can be easily run in your own environment.
+PRs welcome to include other alternatives in the bench suite!
 
 ## Documentation
 
-- [Full documentation][GitHub wiki] (**getting started** and more)
+- [Wiki][GitHub wiki] (getting started, usage, etc.)
-- Auto-generated API reference: [Codox][Codox docs], [clj-doc][clj-doc docs]
+- API reference via [cljdoc][cljdoc]
 
 ## Funding
 
@@ -58,7 +103,7 @@ You can [help support][sponsor] continued work on this project, thank you!! 🙏
 
 ## License
 
-Copyright &copy; 2012-2023 [Peter Taoussanis][].  
+Copyright &copy; 2012-2025 [Peter Taoussanis][].  
 Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).
 
 <!-- Common -->
@@ -72,8 +117,7 @@ Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).
 
 <!-- Project -->
 
-[Codox docs]:   https://taoensso.github.io/nippy/
+[cljdoc]: https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy
-[clj-doc docs]: https://cljdoc.org/d/com.taoensso/nippy/
 
 [Clojars SVG]: https://img.shields.io/clojars/v/com.taoensso/nippy.svg
 [Clojars URL]: https://clojars.org/com.taoensso/nippy

SECURITY.md

@@ -0,0 +1,13 @@
+# Security policy
+
+## Advisories
+
+All security advisories will be posted [on GitHub](https://github.com/taoensso/nippy/security/advisories).
+
+## Reporting a vulnerability
+
+Please report possible security vulnerabilities [via GitHub](https://github.com/taoensso/nippy/security/advisories), or by emailing me at `my first name at taoensso.com`. You may encrypt emails with [my public PGP/GPG key](https://www.taoensso.com/pgp).
+
+Thank you!
+
+\- [Peter Taoussanis](https://www.taoensso.com)

project.clj

@@ -1,33 +1,45 @@
-(defproject com.taoensso/nippy "3.3.0-RC2"
+(defproject com.taoensso/nippy "3.5.0"
   :author "Peter Taoussanis <https://www.taoensso.com>"
-  :description "The fastest serialization library for Clojure"
+  :description "Fast serialization library for Clojure"
-  :url "https://github.com/taoensso/nippy"
+  :url "https://www.taoensso.com/nippy"
 
   :license
   {:name "Eclipse Public License - v 1.0"
    :url  "https://www.eclipse.org/legal/epl-v10.html"}
 
   :dependencies
-  [[org.clojure/tools.reader "1.3.6"]
+  [[org.clojure/tools.reader "1.5.2"]
-   [com.taoensso/encore      "3.68.0"]
+   [com.taoensso/encore      "3.142.0"]
-   [org.iq80.snappy/snappy   "0.4"]
+   [org.tukaani/xz           "1.10"]
-   [org.tukaani/xz           "1.9"]
+   [io.airlift/aircompressor "2.0.2"]]
-   [org.lz4/lz4-java         "1.8.0"]]
+
+  :test-paths ["test" #_"src"]
 
   :profiles
   {;; :default [:base :system :user :provided :dev]
-   :provided {:dependencies [[org.clojure/clojure "1.11.1"]]}
+   :provided {:dependencies [[org.clojure/clojure "1.11.4"]]}
-   :c1.11    {:dependencies [[org.clojure/clojure "1.11.1"]]}
+   :c1.12    {:dependencies [[org.clojure/clojure "1.12.0"]]}
-   :c1.10    {:dependencies [[org.clojure/clojure "1.10.1"]]}
+   :c1.11    {:dependencies [[org.clojure/clojure "1.11.4"]]}
-   :c1.9     {:dependencies [[org.clojure/clojure "1.9.0"]]}
+   :c1.10    {:dependencies [[org.clojure/clojure "1.10.3"]]}
 
-   :test
+   :graal-tests
+   {:source-paths ["test"]
+    :main taoensso.graal-tests
+    :aot [taoensso.graal-tests]
+    :uberjar-name "graal-tests.jar"
+    :dependencies
+    [[org.clojure/clojure                  "1.11.3"]
+     [com.github.clj-easy/graal-build-time "1.0.5"]]}
+
+   :dev
    {:jvm-opts
     ["-server"
      "-Xms1024m" "-Xmx2048m"
      "-Dtaoensso.elide-deprecated=true"
      "-Dtaoensso.nippy.thaw-serializable-allowlist-base=base.1, base.2"
-     "-Dtaoensso.nippy.thaw-serializable-allowlist-add=add.1 , add.2"]
+     "-Dtaoensso.nippy.thaw-serializable-allowlist-add=add.1 , add.2"
+     #_"-Dtaoensso.nippy.target-release=320"
+     #_"-Dtaoensso.nippy.target-release=350"]
 
     :global-vars
     {*warn-on-reflection* true
@@ -35,35 +47,18 @@
      *unchecked-math*     false #_:warn-on-boxed}
 
     :dependencies
-    [[org.clojure/test.check        "1.1.1"]
+    [[org.clojure/test.check    "1.1.1"]
-     [org.clojure/data.fressian     "1.0.0"]
+     [org.clojure/data.fressian "1.1.0"]]
-     [org.xerial.snappy/snappy-java "1.1.10.4"]]}
 
-   :graal-tests
+    :plugins
-   {:dependencies [[org.clojure/clojure "1.11.1"]
+    [[lein-pprint  "1.3.2"]
-                   [com.github.clj-easy/graal-build-time "1.0.5"]]
+     [lein-ancient "0.7.0"]]}}
-    :main taoensso.graal-tests
-    :aot [taoensso.graal-tests]
-    :uberjar-name "graal-tests.jar"}
-
-   :dev
-   [:c1.11 :test
-    {:plugins
-     [[lein-pprint  "1.3.2"]
-      [lein-ancient "0.7.0"]
-      [com.taoensso.forks/lein-codox "0.10.10"]]
-
-     :codox
-     {:language #{:clojure #_:clojurescript}
-      :base-language :clojure}}]}
-
-  :test-paths ["test" #_"src"]
 
   :aliases
   {"start-dev"     ["with-profile" "+dev" "repl" ":headless"]
    ;; "build-once" ["do" ["clean"] ["cljsbuild" "once"]]
    "deploy-lib"    ["do" #_["build-once"] ["deploy" "clojars"] ["install"]]
 
-   "test-clj"     ["with-profile" "+c1.11:+c1.10:+c1.9" "test"]
+   "test-clj"     ["with-profile" "+c1.12:+c1.11:+c1.10" "test"]
-   ;; "test-cljs" ["with-profile" "+test" "cljsbuild"   "test"]
+   ;; "test-cljs" ["with-profile" "+c1.12" "cljsbuild"   "test"]
    "test-all"     ["do" ["clean"] ["test-clj"] #_["test-cljs"]]})

src/taoensso/nippy/benchmarks.clj

@@ -1,126 +0,0 @@
-(ns taoensso.nippy.benchmarks
-  "Nippy benchmarks."
-  (:require
-   [clojure.data.fressian :as fress]
-   [taoensso.encore       :as enc]
-   [taoensso.nippy        :as nippy]))
-
-;;;; Reader
-
-(defn- freeze-reader [x]   (enc/pr-edn   x))
-(defn- thaw-reader   [edn] (enc/read-edn edn))
-
-;;;; Fressian
-
-(defn- freeze-fress [x]
-  (let [^java.nio.ByteBuffer bb (fress/write x)
-        len (.remaining bb)
-        ba  (byte-array len)]
-    (.get bb ba 0 len)
-    (do      ba)))
-
-(defn- thaw-fress [^bytes ba]
-  (let [bb (java.nio.ByteBuffer/wrap ba)]
-    (fress/read bb)))
-
-(comment (-> data freeze-fress thaw-fress))
-
-;;;; Benchable data
-
-(def data
-  "Map of data suitable for benching, a subset of
-  `nippy/stress-data-comparable`."
-  (reduce-kv
-    (fn [m k v]
-      (try
-        (-> v freeze-reader thaw-reader)
-        (-> v freeze-fress  thaw-fress)
-        m
-        (catch Throwable _ (dissoc m k))))
-    nippy/stress-data-comparable
-    nippy/stress-data-comparable))
-
-(comment
-  (clojure.set/difference
-    (set (keys nippy/stress-data-comparable))
-    (set (keys data))))
-
-;;;;
-
-(defn- bench1
-  [{:keys [laps warmup] :or {laps 1e4, warmup 25e3}} freezer thawer sizer]
-  (let [data-frozen (freezer data)
-        time-freeze (enc/bench laps {:warmup-laps warmup} (freezer data))
-        time-thaw   (enc/bench laps {:warmup-laps warmup} (thawer  data-frozen))
-        data-size   (sizer data-frozen)]
-
-    {:round (+ time-freeze time-thaw)
-     :freeze   time-freeze
-     :thaw                 time-thaw
-     :size  data-size}))
-
-(comment (bench1 {} nippy/freeze nippy/thaw count))
-
-(defn bench
-  [{:keys [all? reader? fressian? fressian? lzma2? laps warmup]
-    :or   {laps      1e4
-           warmup    25e3}}]
-
-  (println "\nStarting benchmarks")
-
-  (let [results_ (atom {})]
-    (when (or all? reader?)
-      (println "Benching Reader...")
-      (swap! results_ assoc :reader
-        (bench1 {:laps laps, :warmup warmup}
-          freeze-reader thaw-reader
-          (fn [^String s] (count (.getBytes s "UTF-8"))))))
-
-    (when (or all? fressian?)
-      (println "- Benching Fressian...")
-      (swap! results_ assoc :fressian (bench1 {:laps laps, :warmup warmup} freeze-fress thaw-fress count)))
-
-    (when (or all? lzma2?)
-      (println "- Benching Nippy/LZMA2...")
-      (swap! results_ assoc :nippy/lzma2
-        (bench1 {:laps laps, :warmup warmup}
-          #(nippy/freeze % {:compressor nippy/lzma2-compressor})
-          #(nippy/thaw   % {:compressor nippy/lzma2-compressor})
-          count)))
-
-    (println "- Benching Nippy/encrypted...")
-    (swap! results_ assoc :nippy/encrypted
-      (bench1 {:laps laps, :warmup warmup}
-        #(nippy/freeze % {:password [:cached "p"]})
-        #(nippy/thaw   % {:password [:cached "p"]})
-        count))
-
-    (println "- Benching Nippy/default...")
-    (swap! results_ assoc :nippy/default (bench1 {:laps laps, :warmup warmup} nippy/freeze      nippy/thaw      count))
-
-    (println "- Benching Nippy/fast...")
-    (swap! results_ assoc :nippy/fast (bench1 {:laps laps, :warmup warmup} nippy/fast-freeze nippy/fast-thaw count))
-
-    (println "- Benchmarks complete! (Time for cake?)")
-
-    (let [results @results_]
-      (println "\nBenchmark results:")
-      (doseq [[k v] results] (println k " " v))
-      (do           results))))
-
-(comment
-  (do
-    (set! *unchecked-math* false)
-    (bench {:all? true}))
-
-  ;; 2023 Aug 1, 2020 Apple MBP M1
-  ;; [com.taoensso/nippy        "3.2.0"]
-  ;; [org.clojure/tools.reader  "1.3.6"]
-  ;; [org.clojure/data.fressian "1.0.0"]
-
-  {:reader          {:round 35041, :freeze 5942,  :thaw 29099, :size 39389}
-   :fressian        {:round 6241,  :freeze 3429,  :thaw 2812,  :size 22850}
-   :nippy/lzma2     {:round 33561, :freeze 20530, :thaw 13031, :size 11444}
-   :nippy/encrypted {:round 3390,  :freeze 1807,  :thaw 1583,  :size 16468}
-   :nippy/default   {:round 2845,  :freeze 1513,  :thaw 1332,  :size 16440}
-   :nippy/fast      {:round 2634,  :freeze 1338,  :thaw 1296,  :size 28454}})

src/taoensso/nippy/compression.clj

@@ -1,8 +1,13 @@
 (ns ^:no-doc taoensso.nippy.compression
   "Private, implementation detail."
-  (:require [taoensso.encore :as enc])
+  (:require
-  (:import  [java.io ByteArrayInputStream ByteArrayOutputStream DataInputStream
+   [taoensso.truss  :as truss]
-             DataOutputStream]))
+   [taoensso.encore :as enc])
+  (:import
+   [java.nio ByteBuffer]
+   [java.io
+    ByteArrayInputStream ByteArrayOutputStream
+    DataInputStream DataOutputStream]))
 
 ;;;; Interface
 
@@ -11,24 +16,120 @@
   (compress   ^bytes [compressor ba])
   (decompress ^bytes [compressor ba]))
 
-;;;; Default implementations
+(def ^:const standard-header-ids
+  "These support `:auto` thaw."
+  #{:zstd :lz4 #_:lzo :lzma2 :snappy})
 
-(def standard-header-ids "These'll support :auto thaw" #{:snappy :lzma2 :lz4})
+;;;; Misc utils
 
-(deftype SnappyCompressor []
+(defn- int-size->ba ^bytes [size]
-  ICompressor
+  (let [ba   (byte-array 4)
-  (header-id  [_] :snappy)
+        baos (ByteArrayOutputStream. 4)
-  (compress   [_ ba] (org.iq80.snappy.Snappy/compress   ba))
+        dos  (DataOutputStream. baos)]
-  (decompress [_ ba] (org.iq80.snappy.Snappy/uncompress ba 0 (alength ^bytes ba))))
+    (.writeInt dos (int size))
+    (.toByteArray baos)))
 
-(def snappy-compressor
+(defn- ba->int-size [ba]
-  "Default org.iq80.snappy.Snappy compressor:
+  (let [bais (ByteArrayInputStream. ba)
-        Ratio: low.
+        dis  (DataInputStream. bais)]
-  Write speed: very high.
+    (.readInt dis)))
-   Read speed: very high.
 
-  A good general-purpose compressor."
+(comment (ba->int-size (int-size->ba 3737)))
-  (SnappyCompressor.))
+
+;;;; Airlift
+
+(defn- airlift-compress
+  ^bytes [^io.airlift.compress.Compressor c ^bytes ba prepend-size?]
+  (let [in-len      (alength ba)
+        max-out-len (.maxCompressedLength c in-len)]
+
+    (if prepend-size?
+      (let [ba-max-out  (byte-array (int (+ 4 max-out-len)))
+            int-size-ba (int-size->ba in-len)
+            _           (System/arraycopy int-size-ba 0 ba-max-out 0 4)
+            out-len
+            (.compress c
+              ba         0 in-len
+              ba-max-out 4 max-out-len)]
+
+        (if (== out-len max-out-len)
+          (do                           ba-max-out)
+          (java.util.Arrays/copyOfRange ba-max-out 0 (+ 4 out-len))))
+
+      (let [ba-max-out (byte-array max-out-len)
+            out-len
+            (.compress c
+              ba         0 in-len
+              ba-max-out 0 max-out-len)]
+
+        (if (== out-len max-out-len)
+          (do                           ba-max-out)
+          (java.util.Arrays/copyOfRange ba-max-out 0 out-len))))))
+
+(defn- airlift-decompress
+  ^bytes [^io.airlift.compress.Decompressor d ^bytes ba max-out-len]
+  (if max-out-len
+    (let [max-out-len (int        max-out-len)
+          ba-max-out  (byte-array max-out-len)
+          out-len
+          (.decompress d
+            ba         0 (alength ba)
+            ba-max-out 0 max-out-len)]
+
+      (if (== out-len max-out-len)
+        (do                           ba-max-out)
+        (java.util.Arrays/copyOfRange ba-max-out 0 out-len)))
+
+    ;; Prepended size
+    (let [out-len (ba->int-size ba)
+          ba-out  (byte-array (int out-len))]
+      (.decompress d
+        ba     4 (- (alength ba) 4)
+        ba-out 0 out-len)
+      ba-out)))
+
+(do
+  (enc/def* ^:private airlift-zstd-compressor_   (enc/thread-local (io.airlift.compress.zstd.ZstdCompressor.)))
+  (enc/def* ^:private airlift-zstd-decompressor_ (enc/thread-local (io.airlift.compress.zstd.ZstdDecompressor.)))
+  (deftype ZstdCompressor [prepend-size?]
+    ICompressor
+    (header-id  [_] :zstd)
+    (compress   [_ ba] (airlift-compress   @airlift-zstd-compressor_   ba prepend-size?))
+    (decompress [_ ba] (airlift-decompress @airlift-zstd-decompressor_ ba
+                         (when-not prepend-size?
+                           (io.airlift.compress.zstd.ZstdDecompressor/getDecompressedSize ba
+                             0 (alength ^bytes ba)))))))
+
+(do
+  (enc/def* ^:private airlift-lz4-compressor_   (enc/thread-local (io.airlift.compress.lz4.Lz4Compressor.)))
+  (enc/def* ^:private airlift-lz4-decompressor_ (enc/thread-local (io.airlift.compress.lz4.Lz4Decompressor.)))
+  (deftype LZ4Compressor []
+    ICompressor
+    (header-id  [_] :lz4)
+    (compress   [_ ba] (airlift-compress   @airlift-lz4-compressor_   ba true))
+    (decompress [_ ba] (airlift-decompress @airlift-lz4-decompressor_ ba nil))))
+
+(do
+  (enc/def* ^:private airlift-lzo-compressor_   (enc/thread-local (io.airlift.compress.lzo.LzoCompressor.)))
+  (enc/def* ^:private airlift-lzo-decompressor_ (enc/thread-local (io.airlift.compress.lzo.LzoDecompressor.)))
+  (deftype LZOCompressor []
+    ICompressor
+    (header-id  [_] :lzo)
+    (compress   [_ ba] (airlift-compress   @airlift-lzo-compressor_   ba true))
+    (decompress [_ ba] (airlift-decompress @airlift-lzo-decompressor_ ba nil))))
+
+(do
+  (enc/def* ^:private airlift-snappy-compressor_   (enc/thread-local (io.airlift.compress.snappy.SnappyCompressor.)))
+  (enc/def* ^:private airlift-snappy-decompressor_ (enc/thread-local (io.airlift.compress.snappy.SnappyDecompressor.)))
+  (deftype SnappyCompressor [prepend-size?]
+    ICompressor
+    (header-id  [_] :snappy)
+    (compress   [_ ba] (airlift-compress   @airlift-snappy-compressor_   ba prepend-size?))
+    (decompress [_ ba] (airlift-decompress @airlift-snappy-decompressor_ ba
+                         (when-not prepend-size?
+                           (io.airlift.compress.snappy.SnappyDecompressor/getUncompressedLength ba 0))))))
+
+;;;; LZMA2
 
 (deftype LZMA2Compressor [compression-level]
   ;; Compression level ∈ℕ[0,9] (low->high) with 6 LZMA2 default (we use 0)
@@ -57,86 +158,61 @@
       (.read xzs ba 0 len-decomp)
       (if (== -1 (.read xzs)) ; Good practice as extra safety measure
         nil
-        (throw (ex-info "LZMA2 Decompress failed: corrupt data?" {:ba ba})))
+        (truss/ex-info! "LZMA2 Decompress failed: corrupt data?" {:ba ba}))
       ba)))
 
-(def lzma2-compressor
+;;;; Public API
-  "Default org.tukaani.xz.LZMA2 compressor:
-        Ratio: high.
-  Write speed: _very_ slow (also currently single-threaded).
-   Read speed: slow.
 
-  A specialized compressor for large, low-write data in space-sensitive
+(def zstd-compressor
-  environments."
+  "Default `Zstd` (`Zstandard`) compressor:
+    -   Compression ratio: `B` (0.53       on reference benchmark).
+    -   Compression speed: `C` (1300 msecs on reference benchmark).
+    - Decompression speed: `B` (400  msecs on reference benchmark).
+
+  Good general-purpose compressor, balances ratio & speed.
+  See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
+  (ZstdCompressor. false))
+
+(def lz4-compressor
+  "Default `LZ4` compressor:
+    -   Compression ratio: `C`  (0.58      on reference benchmark).
+    -   Compression speed: `A`  (240 msecs on reference benchmark).
+    - Decompression speed: `A+` (30  msecs on reference benchmark).
+
+  Good general-purpose compressor, favours speed.
+  See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
+  (LZ4Compressor.))
+
+(def lzo-compressor
+  "Default `LZO` compressor:
+    -   Compression ratio: `C` (0.58      on reference benchmark).
+    -   Compression speed: `A` (220 msecs on reference benchmark).
+    - Decompression speed: `A` (40  msecs on reference benchmark).
+
+  Good general-purpose compressor, favours speed.
+  See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
+  (LZOCompressor.))
+
+(def lzma2-compressor
+  "Default `LZMA2` compressor:
+    -   Compression ratio: `A+` (0.4       on reference benchmark).
+    -   Compression speed: `E`  (18.5 secs on reference benchmark).
+    - Decompression speed: `D`  (12   secs on reference benchmark).
+
+  Specialized compressor, strongly favours ratio.
+  See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
   (LZMA2Compressor. 0))
 
-(deftype LZ4Compressor [compressor_ decompressor_]
+(enc/def* snappy-compressor
-  ICompressor
+  "Default `Snappy` compressor:
-  (header-id [_] :lz4)
+    -   Compression ratio: `C`  (0.58      on reference benchmark).
-  (compress  [_ ba]
+    -   Compression speed: `A+` (210 msecs on reference benchmark).
-    (let [^net.jpountz.lz4.LZ4Compressor compressor @compressor_
+    - Decompression speed: `B`  (130 msecs on reference benchmark).
-          len-decomp   (alength ^bytes ba)
+  Good general-purpose compressor, favours speed.
-          max-len-comp (.maxCompressedLength compressor len-decomp)
+  See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
-          ba-comp*     (byte-array max-len-comp) ; Over-sized
+  (SnappyCompressor. false))
-          len-comp     (.compress compressor ^bytes ba 0 len-decomp
-                         ba-comp* 0 max-len-comp)
-          ;;
-          baos (ByteArrayOutputStream. (+ len-comp 4))
-          dos  (DataOutputStream. baos)]
-      (.writeInt dos len-decomp) ; Prefix with uncompressed length
-      (.write    dos ba-comp* 0 len-comp)
-      (.toByteArray baos)))
 
-  (decompress [_ ba]
+(enc/def* ^:no-doc lz4hc-compressor
-    (let [^net.jpountz.lz4.LZ4Decompressor decompressor @decompressor_
+  "Different LZ4 modes no longer supported, prefer `lz4-compressor`."
-          bais (ByteArrayInputStream. ba)
+  {:deprecated "v3.4.0-RC1 (2024-02-06)"}
-          dis  (DataInputStream. bais)
+  (LZ4Compressor.))
-          ;;
-          len-decomp (.readInt dis)
-          len-comp   (- (alength ^bytes ba) 4)
-          ;; ba-comp (byte-array len-comp)
-          ;; _       (.readFully dis ba-comp 0 len-comp)
-          ba-decomp  (byte-array len-decomp)
-          _          (.decompress decompressor ba 4 ba-decomp 0 len-decomp)]
-      ba-decomp)))
-
-(def ^:private lz4-factory_ (delay (net.jpountz.lz4.LZ4Factory/fastestInstance)))
-(def lz4-compressor
-  "Default net.jpountz.lz4 compressor:
-        Ratio: low.
-  Write speed: very high.
-   Read speed: very high.
-
-  A good general-purpose compressor, competitive with Snappy.
-
-  Thanks to Max Penet (@mpenet) for our first implementation,
-  Ref. https://github.com/mpenet/nippy-lz4"
-  (LZ4Compressor.
-    (delay (.fastCompressor   ^net.jpountz.lz4.LZ4Factory @lz4-factory_))
-    (delay (.fastDecompressor ^net.jpountz.lz4.LZ4Factory @lz4-factory_))))
-
-(def lz4hc-compressor
-  "Like `lz4-compressor` but trades some write speed for ratio."
-  (LZ4Compressor.
-    (delay (.highCompressor   ^net.jpountz.lz4.LZ4Factory @lz4-factory_))
-    (delay (.fastDecompressor ^net.jpountz.lz4.LZ4Factory @lz4-factory_))))
-
-(comment
-  (def ba-bench (.getBytes (apply str (repeatedly 1000 rand)) "UTF-8"))
-  (defn bench1 [compressor]
-    {:time  (enc/bench 10000 {:nlaps-warmup 10000}
-              (->> ba-bench (compress compressor) (decompress compressor)))
-     :ratio (enc/round2 (/ (count (compress compressor ba-bench))
-                           (count ba-bench)))})
-
-  (println
-   {:snappy (bench1 snappy-compressor)
-    ;:lzma2  (bench1 lzma2-compressor) ; Slow!
-    :lz4    (bench1 lz4-compressor)
-    :lz4hc  (bench1 lz4hc-compressor)})
-
-  ;;; 2014 April 7
-  {:snappy {:time 2251, :ratio 0.852},
-   :lzma2  {:time 46684 :ratio 0.494}
-   :lz4    {:time 1184, :ratio 0.819},
-   :lz4hc  {:time 5422, :ratio 0.761}})

src/taoensso/nippy/crypto.clj

@@ -2,11 +2,13 @@
   "Low-level crypto utils.
   Private & alpha, very likely to change!"
   (:refer-clojure :exclude [rand-nth])
-  (:require [taoensso.encore :as enc]))
+  (:require
+   [taoensso.truss  :as truss]
+   [taoensso.encore :as enc]))
 
 ;; Note that AES128 may be preferable to AES256 due to known attack
-;; vectors specific to AES256, Ref. https://goo.gl/qU4CCV
+;; vectors specific to AES256, Ref. <https://goo.gl/qU4CCV>
-;;  or for a counter argument, Ref. https://goo.gl/9LA9Yb
+;;  or for a counter argument, Ref. <https://goo.gl/9LA9Yb>
 
 ;;;; Randomness
 
@@ -45,7 +47,7 @@
 (defn   take-ba ^bytes [n ^bytes ba] (java.util.Arrays/copyOf ba ^int n)) ; Pads if ba too small
 (defn  utf8->ba ^bytes [^String s] (.getBytes s "UTF-8"))
 (defn- add-salt ^bytes [?salt-ba ba] (if ?salt-ba (enc/ba-concat ?salt-ba ba) ba))
-(defn pwd-as-ba ^bytes [utf8-or-ba] (if (string? utf8-or-ba) (utf8->ba utf8-or-ba) (enc/have enc/bytes? utf8-or-ba)))
+(defn pwd-as-ba ^bytes [utf8-or-ba] (if (string? utf8-or-ba) (utf8->ba utf8-or-ba) (truss/have enc/bytes? utf8-or-ba)))
 
 (comment (seq (pwd-as-ba "foo")))
 
@@ -70,7 +72,7 @@
   (get-key-spec   ^javax.crypto.spec.SecretKeySpec           [_    ba] "Returns a `javax.crypto.spec.SecretKeySpec`.")
   (get-param-spec ^java.security.spec.AlgorithmParameterSpec [_ iv-ba] "Returns a `java.security.spec.AlgorithmParameters`."))
 
-;; Prefer GCM > CBC, Ref. https://goo.gl/jpZoj8
+;; Prefer GCM > CBC, Ref. <https://goo.gl/jpZoj8>
 (def ^:private gcm-cipher* (enc/thread-local-proxy (javax.crypto.Cipher/getInstance "AES/GCM/NoPadding")))
 (def ^:private cbc-cipher* (enc/thread-local-proxy (javax.crypto.Cipher/getInstance "AES/CBC/PKCS5Padding")))
 

src/taoensso/nippy/encryption.clj

@@ -1,13 +1,13 @@
 (ns ^:no-doc taoensso.nippy.encryption
   "Private, implementation detail."
   (:require
+   [taoensso.truss        :as truss]
    [taoensso.encore       :as enc]
    [taoensso.nippy.crypto :as crypto]))
 
-(def standard-header-ids
+(def ^:const standard-header-ids
-  "These'll support `:auto` thaw."
+  "These support `:auto` thaw."
-  #{:aes128-cbc-sha512
+  #{:aes128-cbc-sha512 :aes128-gcm-sha512})
-    :aes128-gcm-sha512})
 
 (defprotocol IEncryptor
   (header-id      [encryptor])
@@ -15,11 +15,11 @@
   (decrypt ^bytes [encryptor pwd ba]))
 
 (defn- throw-destructure-ex [typed-password]
-  (throw (ex-info
+  (truss/ex-info!
-           (str "Expected password form: "
+    (str "Expected password form: "
-             "[<#{:salted :cached}> <password-string>].\n "
+      "[<#{:salted :cached}> <password-string>].\n "
-             "See `aes128-encryptor` docstring for details!")
+      "See `aes128-encryptor` docstring for details!")
-           {:typed-password typed-password})))
+    {:typed-password typed-password}))
 
 (defn- destructure-typed-pwd [typed-password]
   (if (vector? typed-password)

src/taoensso/nippy/impl.clj

@@ -0,0 +1,273 @@
+(ns ^:no-doc taoensso.nippy.impl
+  "Private, implementation detail."
+  (:require
+   [clojure.string  :as str]
+   [taoensso.truss  :as truss]
+   [taoensso.encore :as enc]))
+
+;;;; Fallback type tests
+
+(defn cache-by-type [f]
+  (let [cache_ (enc/latom {})] ; {<type> <result_>}
+    (fn [x]
+      (let [t (if (fn? x) ::fn (type x))]
+        (if-let [result_ (get (cache_) t)]
+          @result_
+          (if-let [uncacheable-type? (re-find #"\d" (str t))]
+            (do                      (f x))
+            @(cache_ t #(or % (delay (f x))))))))))
+
+(def seems-readable?
+  (cache-by-type
+    (fn [x]
+      (try
+        (enc/read-edn (enc/pr-edn x))
+        true
+        (catch Throwable _ false)))))
+
+(def seems-serializable?
+  (cache-by-type
+    (fn [x]
+      (enc/cond
+        (fn? x) false ; Falsely reports as Serializable
+
+        (instance? java.io.Serializable x)
+        (try
+          (let [c   (Class/forName (.getName (class x))) ; Try 1st (fail fast)
+                bas (java.io.ByteArrayOutputStream.)
+                _   (.writeObject (java.io.ObjectOutputStream. bas) x)
+                ba  (.toByteArray bas)]
+            #_
+            (cast c
+              (.readObject ; Unsafe + usu. unnecessary to check
+                (ObjectInputStream. (ByteArrayInputStream. ba))))
+            true)
+          (catch Throwable _ false))
+
+        :else false))))
+
+(comment
+  (enc/qb 1e6 ; [60.83 61.16 59.86 57.37]
+    (seems-readable?     "Hello world")
+    (seems-serializable? "Hello world")
+    (seems-readable?     (fn []))
+    (seems-serializable? (fn []))))
+
+;;;; Java Serializable
+
+(def ^:const ^:private allow-and-record          "allow-and-record")
+(defn-                 allow-and-record? [x] (= x allow-and-record))
+
+(defn- classname-set
+  "Returns ?#{<classname>}."
+  [x]
+  (when x
+    (if (string? x)
+      (if (= x "") #{} (set (mapv str/trim (str/split x #"[,:]"))))
+      (truss/have set? x))))
+
+(comment
+  (mapv classname-set [nil #{"foo"} "" "foo, bar:baz"])
+  (.getName (.getSuperclass (.getClass (java.util.concurrent.TimeoutException.)))))
+
+(defn parse-allowlist
+  "Returns #{<classname>}, or `allow-and-record`."
+  [default base add]
+  (if (or
+        (allow-and-record? base)
+        (allow-and-record? add))
+    allow-and-record
+    (into
+      (or (classname-set base) default)
+      (do (classname-set add)))))
+
+(comment (parse-allowlist #{"default"} "base1,base2" "add1"))
+
+(let [nmax    1000
+      ngc     16000
+      state_  (enc/latom {})  ; {<class-name> <frequency>}
+      lock_   (enc/latom nil) ; ?promise
+      trim
+      (fn [nmax state]
+        (persistent!
+          (enc/reduce-top nmax val enc/rcompare conj!
+            (transient {}) state)))]
+
+  ;; Note: trim strategy isn't perfect: it can be tough for new
+  ;; classes to break into the top set since frequencies are being
+  ;; reset only for classes outside the top set.
+  ;;
+  ;; In practice this is probably good enough since the main objective
+  ;; is to discard one-off anonymous classes to protect state from
+  ;; endlessly growing. Also `gc-rate` allows state to temporarily grow
+  ;; significantly beyond `nmax` size, which helps to give new classes
+  ;; some chance to accumulate a competitive frequency before next GC.
+
+  (defn ^{:-state_ state_} ; Undocumented
+    allow-and-record-any-serializable-class-unsafe
+    "A predicate (fn allow-class? [class-name]) fn that can be assigned
+    to `*freeze-serializable-allowlist*` and/or
+         `*thaw-serializable-allowlist*` that:
+
+      - Will allow ANY class to use Nippy's `Serializable` support (unsafe).
+      - And will record {<class-name> <frequency-allowed>} for the <=1000
+        classes that ~most frequently made use of this support.
+
+    `get-recorded-serializable-classes` returns the recorded state.
+
+    This predicate is provided as a convenience for users upgrading from
+    previous versions of Nippy that allowed the use of `Serializable` for all
+    classes by default.
+
+    While transitioning from an unsafe->safe configuration, you can use
+    this predicate (unsafe) to record information about which classes have
+    been using Nippy's `Serializable` support in your environment.
+
+    Once some time has passed, you can check the recorded state. If you're
+    satisfied that all recorded classes are safely `Serializable`, you can
+    then merge the recorded classes into Nippy's default allowlist/s, e.g.:
+
+    (alter-var-root #'thaw-serializable-allowlist*
+      (fn [_] (into default-thaw-serializable-allowlist
+                (keys (get-recorded-serializable-classes)))))"
+
+    [class-name]
+    (when-let [p (lock_)] @p)
+    (let [n (count (state_ #(assoc % class-name (inc (long (or (get % class-name) 0))))))]
+      ;; Garbage collection (GC): may be serializing anonymous classes, etc.
+      ;; so input domain could be infinite
+      (when (> n ngc) ; Too many classes recorded, uncommon
+        (let [p (promise)]
+          (when (compare-and-set! lock_ nil p) ; Acquired GC lock
+            (try
+              (do      (reset! state_ (trim nmax (state_)))) ; GC state
+              (finally (reset! lock_  nil) (deliver p nil))))))
+      n))
+
+  (defn get-recorded-serializable-classes
+    "Returns {<class-name> <frequency>} of the <=1000 classes that ~most
+    frequently made use of Nippy's `Serializable` support via
+    `allow-and-record-any-serializable-class-unsafe`.
+
+    See that function's docstring for more info."
+    [] (trim nmax (state_))))
+
+(comment
+  (count (get-recorded-serializable-classes))
+  (enc/reduce-n
+    (fn [_ n] (allow-and-record-any-serializable-class-unsafe (str n)))
+    nil 0 1e5))
+
+(let [compile
+      (enc/fmemoize
+        (fn [x]
+          (if (allow-and-record? x)
+            allow-and-record-any-serializable-class-unsafe
+            (enc/name-filter x))))
+
+      fn? fn?
+      conform?
+      (fn [x cn]
+        (if (fn? x)
+          (x cn) ; Intentionally uncached, can be handy
+          ((compile x) cn)))]
+
+  (defn serializable-allowed? [allow-list class-name]
+    (conform? allow-list class-name)))
+
+;;;; Release targeting
+
+(comment
+  (set! *print-length* nil)
+  (vec (sort (keys taoensso.nippy/public-types-spec)))
+
+  ;; To help support release targeting, we track new type ids added over time
+  (let [id-history ; {<release> #{type-ids}}
+        {350 ; v3.5.0 (2025-04-15), added 5x
+         ;; #{int-array-lg long-array-lg float-array-lg double-array-lg string-array-lg}
+         #{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
+           29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
+           55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
+           81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104
+           105 106 107 108 109 110 111 112 113 114 115 116 117}
+
+         340 ; v3.4.0 (2024-04-30), added 2x
+         ;; #{map-entry meta-protocol-key}
+         #{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
+           29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
+           55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
+           81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104
+           105 106 110 111 112 113 114 115}
+
+         330 ; v3.3.0 (2023-10-11), added 11x
+         ;; #{long-pos-sm long-pos-md long-pos-lg long-neg-sm long-neg-md long-neg-lg
+         ;;   str-sm* vec-sm* set-sm* map-sm* sql-date}
+         #{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
+           29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
+           55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
+           81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 105 106
+           110 111 112 113 114 115}
+
+         320 ; v3.2.0 (2022-07-18), added none
+         #{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
+           29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
+           55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
+           81 82 83 84 85 86 90 91 100 101 102 105 106 110 111 112 113 114 115}
+
+         313 ; v3.1.3 (2022-06-23), added 5x
+         ;; #{time-instant time-duration time-period kw-md sym-md}
+         #{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
+           29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
+           55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
+           81 82 83 84 85 86 90 91 100 101 102 105 106 110 111 112 113 114 115}
+
+         300 ; v3.0.0 (2020-09-20), baseline
+         #{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
+           29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
+           55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 80
+           81 82 90 91 100 101 102 105 106 110 111 112 113 114 115}}
+
+        diff
+        (fn [new-release old-release]
+          (vec (sort (clojure.set/difference (id-history new-release) (id-history old-release)))))]
+
+    (diff 350 340)))
+
+(let [target-release
+      (enc/get-env {:as :edn, :default 320}
+        :taoensso.nippy.target-release)
+
+      target>=
+      (fn [min-release]
+        (if target-release
+          (>= (long target-release) (long min-release))
+          true))]
+
+  (defmacro target-release< [min-release] (not (target>= min-release)))
+  (defmacro target-release>=
+    "Returns true iff `target-release` is nil or >= given `min-release`.
+    Used to help ease data migration for changes to core data types.
+
+    When support is added for a new type in Nippy version X, it necessarily means
+    that data containing that new type and frozen with Nippy version X is unthawable
+    with Nippy versions < X.
+
+    Earlier versions of Nippy will throw an exception on thawing affected data:
+      \"Unrecognized type id (<n>). Data frozen with newer Nippy version?\"
+
+    This can present a challenge when updating to new versions of Nippy, e.g.:
+
+      - Rolling updates could lead to old and new versions of Nippy temporarily co-existing.
+      - Data written with new types could limit your ability to revert a Nippy update.
+
+    There's no easy solution to this in GENERAL, but we CAN at least help reduce the
+    burden related to CHANGES in core data types by introducing changes over 2 phases:
+
+      1. Nippy vX   reads  new (changed) type, writes old type
+      2. Nippy vX+1 writes new (changed) type
+
+    When relevant, we can then warn users in the CHANGELOG to not leapfrog
+    (e.g. Nippy vX -> Nippy vX+2) when doing rolling updates."
+    [min-release] (target>= min-release)))
+
+(comment (macroexpand '(target-release>= 340)))

src/taoensso/nippy/tools.clj

@@ -1,45 +1,40 @@
 (ns taoensso.nippy.tools
   "Utils for community tools that want to add user-configurable Nippy support.
   Used by Carmine, Faraday, etc."
-  (:require [taoensso.nippy :as nippy]))
+  (:require
+   [taoensso.encore :as enc]
+   [taoensso.nippy  :as nippy]))
 
 (def ^:dynamic *freeze-opts* nil)
 (def ^:dynamic *thaw-opts*   nil)
 
-(defn ^:no-doc -merge-opts
-  "Private, implementation detail."
-  ([x y  ] (if x (conj x y) y))
-  ([x y z] (-merge-opts (-merge-opts x y) z)))
-
 (do
-  (defmacro with-freeze-opts  [opts & body] `(binding [*freeze-opts*                            ~opts ] ~@body))
+  (defmacro with-freeze-opts  [opts & body] `(binding [*freeze-opts*                          ~opts ] ~@body))
-  (defmacro with-freeze-opts+ [opts & body] `(binding [*freeze-opts* (-merge-opts *freeze-opts* ~opts)] ~@body))
+  (defmacro with-freeze-opts+ [opts & body] `(binding [*freeze-opts* (enc/merge *freeze-opts* ~opts)] ~@body))
-  (defmacro with-thaw-opts    [opts & body] `(binding [*thaw-opts*                              ~opts ] ~@body))
+  (defmacro with-thaw-opts    [opts & body] `(binding [*thaw-opts*                            ~opts ] ~@body))
-  (defmacro with-thaw-opts+   [opts & body] `(binding [*thaw-opts*   (-merge-opts *thaw-opts*   ~opts)] ~@body)))
+  (defmacro with-thaw-opts+   [opts & body] `(binding [*thaw-opts*   (enc/merge *thaw-opts*   ~opts)] ~@body)))
 
 (deftype WrappedForFreezing [val opts])
 (defn wrapped-for-freezing? [x] (instance? WrappedForFreezing x))
 
-(let [-merge-opts -merge-opts]
+(defn wrap-for-freezing
-  (defn wrap-for-freezing
+  "Captures (merge `tools/*thaw-opts*` `wrap-opts`), and returns
-    "Captures (merge `tools/*thaw-opts*` `wrap-opts`), and returns
     the given argument in a wrapped form so that `tools/freeze` will
     use the captured options when freezing the wrapper argument.
 
     See also `tools/freeze`."
-    ([x          ] (wrap-for-freezing x nil))
+  ([x          ] (wrap-for-freezing x nil))
-    ([x wrap-opts]
+  ([x wrap-opts]
-     (let [captured-opts (-merge-opts *freeze-opts* wrap-opts)] ; wrap > dynamic
+   (let [captured-opts (enc/merge *freeze-opts* wrap-opts)] ; wrap > dynamic
-       (if (instance? WrappedForFreezing x)
+     (if (instance? WrappedForFreezing x)
-         (let [^WrappedForFreezing x x]
+       (let [^WrappedForFreezing x x]
-           (if (= (.-opts x) captured-opts)
+         (if (= (.-opts x) captured-opts)
-             x
+           x
-             (WrappedForFreezing. (.-val x) captured-opts)))
+           (WrappedForFreezing. (.-val x) captured-opts)))
-         (WrappedForFreezing.            x  captured-opts))))))
+       (WrappedForFreezing.            x  captured-opts)))))
 
-(let [-merge-opts -merge-opts]
+(defn freeze
-  (defn freeze
+  "Like `nippy/freeze` but uses as options the following, merged in
-    "Like `nippy/freeze` but uses as options the following, merged in
     order of ascending preference:
 
       1. `default-opts` given to this fn            (default nil).
@@ -47,28 +42,27 @@
       3. Opts captured by `tools/wrap-for-freezing` (default nil).
 
     See also `tools/wrap-for-freezing`."
-    ([x             ] (freeze x nil))
+  ([x             ] (freeze x nil))
-    ([x default-opts]
+  ([x default-opts]
-     (let [default-opts (get default-opts :default-opts default-opts) ; Back compatibility
+   (let [default-opts (get       default-opts :default-opts default-opts) ; Back compatibility
-           active-opts  (-merge-opts default-opts *freeze-opts*)] ; dynamic > default
+         active-opts  (enc/merge default-opts *freeze-opts*)] ; dynamic > default
 
-       (if (instance? WrappedForFreezing x)
+     (if (instance? WrappedForFreezing x)
-         (let [^WrappedForFreezing x x]
+       (let [^WrappedForFreezing x x]
-           (nippy/freeze (.-val x) (-merge-opts active-opts (.-opts x)))) ; captured > active!
+         (nippy/freeze (.-val x) (enc/merge active-opts (.-opts x)))) ; captured > active!
-         (nippy/freeze          x               active-opts))))))
+       (nippy/freeze          x             active-opts)))))
 
-(let [-merge-opts -merge-opts]
+(defn thaw
-  (defn thaw
+  "Like `nippy/thaw` but uses as options the following, merged in
-    "Like `nippy/thaw` but uses as options the following, merged in
     order of ascending preference:
 
       1. `default-opts` given to this fn   (default nil).
       2. `tools/*thaw-opts*` dynamic value (default nil)."
-    ([ba             ] (thaw ba nil))
+  ([ba             ] (thaw ba nil))
-    ([ba default-opts]
+  ([ba default-opts]
-     (let [default-opts (get default-opts :default-opts default-opts) ; Back compatibility
+   (let [default-opts (get       default-opts :default-opts default-opts) ; Back compatibility
-           active-opts  (-merge-opts default-opts *thaw-opts*)] ; dynamic > default
+         active-opts  (enc/merge default-opts *thaw-opts*)] ; dynamic > default
 
-       (nippy/thaw ba active-opts)))))
+     (nippy/thaw ba active-opts))))
 
 (comment (thaw (freeze (wrap-for-freezing "wrapped"))))

src/taoensso/nippy/utils.clj

@@ -1,144 +0,0 @@
-(ns ^:no-doc taoensso.nippy.utils
-  "Private, implementation detail."
-  (:require [clojure.string  :as str]
-            [taoensso.encore :as enc])
-  (:import  [java.io ByteArrayInputStream ByteArrayOutputStream Serializable
-             ObjectOutputStream ObjectInputStream]))
-
-;;;; Fallback type tests
-;; Unfortunately the only ~reliable way we can tell if something's
-;; really serializable/readable is to actually try a full roundtrip.
-
-(defn- memoize-type-test [test-fn]
-  (let [cache_ (enc/latom {})] ; {<type> <type-okay?>}
-    (fn [x]
-      (let [t          (type x)
-            gensym?    (re-find #"__\d+" (str t))
-            cacheable? (not gensym?) ; Hack, but no obviously better solutions
-            test       (fn [] (try (test-fn x) (catch Exception _ false)))]
-
-        (if cacheable?
-          @(cache_ t #(if % % (delay (test))))
-          (do                        (test)))))))
-
-(def     readable? (memoize-type-test (fn [x] (-> x enc/pr-edn enc/read-edn) true)))
-(def serializable?
-  (let [mtt
-        (memoize-type-test
-          (fn [x]
-            (let [class-name (.getName (class x))
-                  c   (Class/forName class-name) ; Try 1st (fail fast)
-                  bas (ByteArrayOutputStream.)
-                  _   (.writeObject (ObjectOutputStream. bas) x)
-                  ba  (.toByteArray bas)]
-
-              #_
-              (cast c
-                (.readObject ; Unsafe + usu. unnecessary to check
-                  (ObjectInputStream.
-                    (ByteArrayInputStream. ba))))
-
-              true)))]
-
-    (fn [x]
-      (if (instance? Serializable x)
-        (if (fn? x)
-          false ; Reports as true but is unreliable
-          (mtt x))
-       false))))
-
-(comment
-  (enc/qb 1e4
-    (readable?     "Hello world") ; Cacheable
-    (serializable? "Hello world") ; Cacheable
-    (readable?     (fn []))       ; Uncacheable
-    (serializable? (fn []))       ; Uncacheable
-    )) ; [2.52 2.53 521.34 0.63]
-
-;;;;
-
-(defn- is-coll?
-  "Checks for _explicit_ IPersistentCollection types with Nippy support.
-  Checking for explicit concrete types is tedious but preferable since a
-  `freezable?` false positive would be much worse than a false negative."
-  [x]
-  (let [is? #(when (instance? % x) %)]
-    (or
-     (is? clojure.lang.APersistentVector)
-     (is? clojure.lang.APersistentMap)
-     (is? clojure.lang.APersistentSet)
-     (is? clojure.lang.PersistentList)
-     (is? clojure.lang.PersistentList$EmptyList) ; (type '())
-     (is? clojure.lang.PersistentQueue)
-     (is? clojure.lang.PersistentTreeSet)
-     (is? clojure.lang.PersistentTreeMap)
-     (is? clojure.lang.PersistentVector$ChunkedSeq)
-
-     (is? clojure.lang.IRecord) ; TODO Possible to avoid the interface check?
-     (is? clojure.lang.LazySeq)
-
-     ;; Too non-specific: could result in false positives (which would be a
-     ;; serious problem here):
-     ;; (is? clojure.lang.ISeq)
-
-     )))
-
-(comment (is-coll? (clojure.lang.PersistentVector$ChunkedSeq. [1 2 3] 0 0)))
-
-(defmacro ^:private is? [x c] `(when (instance? ~c ~x) ~c))
-
-(defn freezable?
-  "Alpha - subject to change.
-  Returns truthy iff Nippy *appears* to support freezing the given argument.
-
-  `:allow-clojure-reader?` and `:allow-java-serializable?` options may be
-  used to enable the relevant roundtrip fallback test(s). These tests are
-  only **moderately reliable** since they're cached by arg type and don't
-  test for pre/post serialization value equality (there's no good general
-  way of doing so)."
-
-  ;; TODO Not happy with this approach in general, could do with a refactor.
-  ;; Maybe return true/false/nil (nil => maybe)?
-
-  ([x] (freezable? x nil))
-  ([x {:keys [allow-clojure-reader? allow-java-serializable?]}]
-   (if (is-coll? x)
-     (when (enc/revery? freezable? x) (type x))
-     (or
-       (is? x clojure.lang.Keyword)
-       (is? x java.lang.String)
-       (is? x java.lang.Long)
-       (is? x java.lang.Double)
-       (nil? x)
-
-       (is? x clojure.lang.BigInt)
-       (is? x clojure.lang.Ratio)
-
-       (is? x java.lang.Boolean)
-       (is? x java.lang.Integer)
-       (is? x java.lang.Short)
-       (is? x java.lang.Byte)
-       (is? x java.lang.Character)
-       (is? x java.math.BigInteger)
-       (is? x java.math.BigDecimal)
-       (is? x #=(java.lang.Class/forName "[B"))
-
-       (is? x clojure.lang.Symbol)
-
-       (is? x java.util.Date)
-       (is? x java.util.UUID)
-       (is? x java.util.regex.Pattern)
-
-       (when (and allow-clojure-reader?    (readable?     x)) :clojure-reader)
-       (when (and allow-java-serializable? (serializable? x)) :java-serializable)))))
-
-(comment
-  (enc/qb 10000 (freezable? "hello")) ; 0.79
-  (freezable? [:a :b])
-  (freezable? [:a (fn [x] (* x x))])
-  (freezable? (.getBytes "foo"))
-  (freezable? (java.util.Date.) {:allow-clojure-reader?    true})
-  (freezable? (Exception. "_")  {:allow-clojure-reader?    true})
-  (freezable? (Exception. "_")  {:allow-java-serializable? true})
-  (freezable? (atom {}) {:allow-clojure-reader?    true
-                         :allow-java-serializable? true}))

test/taoensso/nippy/tests/main.clj

@@ -1,363 +0,0 @@
-(ns taoensso.nippy.tests.main
-  (:require
-   [clojure.test :as test :refer [deftest testing is]]
-   [clojure.test.check            :as tc]
-   [clojure.test.check.generators :as tc-gens]
-   [clojure.test.check.properties :as tc-props]
-   [taoensso.encore :as enc   :refer []]
-   [taoensso.nippy  :as nippy :refer [freeze thaw]]
-   [taoensso.nippy.benchmarks :as benchmarks]))
-
-(comment (test/run-tests))
-
-(def test-data nippy/stress-data-comparable)
-(def tc-num-tests 120)
-(def tc-gens
-  "Like `tc-gens/any` but removes NaN (which breaks equality tests)"
-  (tc-gens/recursive-gen tc-gens/container-type #_simple-type
-    (tc-gens/one-of
-      [tc-gens/int tc-gens/large-integer #_tc-gens/double
-       (tc-gens/double* {:NaN? false})
-       tc-gens/char tc-gens/string tc-gens/ratio tc-gens/boolean tc-gens/keyword
-       tc-gens/keyword-ns tc-gens/symbol tc-gens/symbol-ns tc-gens/uuid])))
-
-(comment (tc-gens/sample tc-gens 10))
-
-;;;; Core
-
-(deftest _core
-  (is (do (println (str "Clojure version: " *clojure-version*)) true))
-  (is (= test-data ((comp thaw freeze) test-data)))
-  (is (= test-data ((comp #(thaw   % {:no-header? true
-                                      :compressor nippy/lz4-compressor
-                                      :encryptor  nil})
-                          #(freeze % {:no-header? true}))
-                    test-data)))
-
-  (is (= test-data ((comp #(thaw   % {:password [:salted "p"]})
-                          #(freeze % {:password [:salted "p"]}))
-                    test-data)))
-
-  (is (= (vec (:objects nippy/stress-data))
-         ((comp vec thaw freeze) (:objects nippy/stress-data))))
-
-  (is (= test-data ((comp #(thaw   % {:compressor nippy/lzma2-compressor})
-                          #(freeze % {:compressor nippy/lzma2-compressor}))
-                    test-data)))
-
-  (is (= test-data ((comp #(thaw   % {:compressor nippy/lzma2-compressor
-                                      :password [:salted "p"]})
-                          #(freeze % {:compressor nippy/lzma2-compressor
-                                      :password [:salted "p"]}))
-                    test-data)))
-
-  (is (= test-data ((comp #(thaw   % {:compressor nippy/lz4-compressor})
-                          #(freeze % {:compressor nippy/lz4hc-compressor}))
-                    test-data)))
-
-  (is ; Try roundtrip anything that simple-check can dream up
-    (:result (tc/quick-check tc-num-tests
-               (tc-props/for-all [val tc-gens]
-                 (= val (thaw (freeze val)))))))
-
-  (is (thrown? Exception (thaw (freeze test-data {:password "malformed"}))))
-  (is (thrown? Exception (thaw (freeze test-data {:password [:salted "p"]})
-                           {;; Necessary to prevent against JVM segfault due to
-                            ;; https://goo.gl/t0OUIo:
-                            :v1-compatibility? false})))
-  (is (thrown? Exception (thaw (freeze test-data {:password [:salted "p"]})
-                           {:v1-compatibility? false ; Ref. https://goo.gl/t0OUIo
-                            :compressor nil})))
-
-  (is ; Snappy lib compatibility (for legacy versions of Nippy)
-    (let [^bytes raw-ba    (freeze test-data {:compressor nil})
-          ^bytes xerial-ba (org.xerial.snappy.Snappy/compress raw-ba)
-          ^bytes iq80-ba   (org.iq80.snappy.Snappy/compress   raw-ba)]
-      (= (thaw raw-ba)
-        (thaw (org.xerial.snappy.Snappy/uncompress xerial-ba))
-        (thaw (org.xerial.snappy.Snappy/uncompress iq80-ba))
-        (thaw (org.iq80.snappy.Snappy/uncompress   iq80-ba    0 (alength iq80-ba)))
-        (thaw (org.iq80.snappy.Snappy/uncompress   xerial-ba  0 (alength xerial-ba))))))
-
-  (is ; CBC auto-encryptor compatibility
-    (= "payload"
-      (thaw (freeze "payload" {:password [:salted "pwd"] :encryptor nippy/aes128-cbc-encryptor})
-        (do                   {:password [:salted "pwd"]})))))
-
-;;;; Custom types & records
-
-(deftype   MyType [basic_field fancy-field!]) ; Note `fancy-field!` field name will be munged
-(defrecord MyRec  [basic_field fancy-field!])
-
-(deftest _types
-  (testing "Extend to custom type"
-    (is
-      (thrown? Exception ; No thaw extension yet
-        (do
-          (alter-var-root #'nippy/*custom-readers* (constantly {}))
-          (nippy/extend-freeze MyType 1 [x s]
-            (.writeUTF s (.basic_field  x))
-            (.writeUTF s (.fancy-field! x)))
-
-          (thaw (freeze (MyType. "basic" "fancy"))))))
-
-    (is
-      (do
-        (nippy/extend-thaw 1 [s] (MyType. (.readUTF s) (.readUTF s)))
-        (let [mt1 (MyType. "basic" "fancy")
-              ^MyType mt2 (thaw (freeze mt1))]
-          (=
-            [(.basic_field mt1) (.fancy-field! mt1)]
-            [(.basic_field mt2) (.fancy-field! mt2)])))))
-
-  (testing "Extend to custom Record"
-    (is
-      (do
-        (nippy/extend-freeze MyRec 2 [x s]
-          (.writeUTF s (str "foo-" (:basic_field  x)))
-          (.writeUTF s (str "foo-" (:fancy-field! x))))
-
-        (nippy/extend-thaw 2 [s] (MyRec. (.readUTF s) (.readUTF s)))
-        (=
-          (do           (MyRec. "foo-basic" "foo-fancy"))
-          (thaw (freeze (MyRec.     "basic"     "fancy")))))))
-
-  (testing "Keyword (prefixed) extensions"
-    (is
-      (do
-        (nippy/extend-freeze MyRec :nippy-tests/MyRec [x s]
-          (.writeUTF s (:basic_field  x))
-          (.writeUTF s (:fancy-field! x)))
-
-        (nippy/extend-thaw :nippy-tests/MyRec [s] (MyRec. (.readUTF s) (.readUTF s)))
-        (let [mr (MyRec. "basic" "fancy")]
-          (=  mr (thaw (freeze mr))))))))
-
-;;;; Caching
-
-(deftest _caching
-  (let [stress [nippy/stress-data-comparable
-                nippy/stress-data-comparable
-                nippy/stress-data-comparable
-                nippy/stress-data-comparable]
-        cached (mapv nippy/cache stress)
-        cached (mapv nippy/cache stress) ; <=1 wrap auto-enforced
-        ]
-
-    (is (= stress (thaw (freeze stress {:compressor nil}))))
-    (is (= stress (thaw (freeze cached {:compressor nil}))))
-    (let [size-stress (count (freeze stress {:compressor nil}))
-          size-cached (count (freeze cached {:compressor nil}))]
-      (is (>= size-stress (* 3 size-cached)))
-      (is (<  size-stress (* 4 size-cached))))))
-
-(deftest _caching-metadata
-  (let [v1 (with-meta [] {:id :v1})
-        v2 (with-meta [] {:id :v2})
-
-        frozen-without-caching (freeze [v1 v2 v1 v2])
-        frozen-with-caching
-        (freeze [(nippy/cache v1)
-                 (nippy/cache v2)
-                 (nippy/cache v1)
-                 (nippy/cache v2)])]
-
-    (is (> (count frozen-without-caching)
-           (count frozen-with-caching)))
-
-    (is (= (thaw frozen-without-caching)
-           (thaw frozen-with-caching)))
-
-    (is (= (mapv meta (thaw frozen-with-caching))
-           [{:id :v1} {:id :v2} {:id :v1} {:id :v2}]))))
-
-;;;; Stable binary representation of vals
-
-(deftest _stable-bin
-
-  (is (= (seq (freeze test-data))
-         (seq (freeze test-data)))) ; f(x)=f(y) | x=y
-
-  ;; As above, but try multiple times to catch possible protocol interface races:
-  (is (every? true?
-        (repeatedly 1000 (fn [] (= (seq (freeze test-data))
-                                  (seq (freeze test-data)))))))
-
-  ;; NB abandoning - no way to do this reliably w/o appropriate contracts from
-  ;; (seq <unordered-coll>):
-  ;;
-  ;; (is (=  (seq (-> test-data freeze))
-  ;;         (seq (-> test-data freeze thaw freeze)))) ; f(x)=f(f-1(f(x)))
-  ;;
-  ;; As above, but with repeated refreeze to catch possible protocol interface races:
-  ;; (is (= (seq (freeze test-data))
-  ;;        (seq (reduce (fn [frozen _] (freeze (thaw frozen)))
-  ;;               (freeze test-data) (range 1000)))))
-  )
-
-(defn qc-prop-bijection [& [n]]
-  (let [bin->val (atom {})
-        val->bin (atom {})]
-    (merge
-      (tc/quick-check (or n 1)
-        (tc-props/for-all [val tc-gens]
-          (let [;; Nb need `seq` for Clojure hash equality:
-                bin (hash (seq (freeze val)))]
-            (and
-              (if (contains? val->bin val)
-                (= (get val->bin val) bin) ; x=y => f(x)=f(y) by clj=
-                (do (swap! val->bin assoc val bin)
-                    true))
-
-              (if (contains? bin->val bin)
-                (= (get bin->val bin) val) ; f(x)=f(y) => x=y by clj=
-                (do (swap! bin->val assoc bin val)
-                    true))))))
-      #_{:bin->val @bin->val
-         :val->bin @val->bin}
-      nil)))
-
-(comment
-  (tc-gens/sample tc-gens 10)
-  (:result (qc-prop-bijection 80))
-  (let [{:keys [result bin->val val->bin]} (qc-prop-bijection 10)]
-    [result (vals bin->val)]))
-
-(deftest _gc-prop-bijection
-  (is (:result (qc-prop-bijection tc-num-tests))))
-
-;;;; Thread safety
-
-(deftest _thread-safe
-  (is
-    (let [futures
-          (mapv
-            (fn [_]
-              (future
-                (= (thaw (freeze test-data)) test-data)))
-            (range 50))]
-      (every? deref futures)))
-
-  (is
-    (let [futures
-          (mapv
-            (fn [_]
-              (future
-                (= (thaw (freeze test-data {:password [:salted "password"]})
-                                           {:password [:salted "password"]})
-                  test-data)))
-            (range 50))]
-      (every? deref futures)))
-
-  (is
-    (let [futures
-          (mapv
-            (fn [_]
-              (future
-                (= (thaw (freeze test-data {:password [:cached "password"]})
-                                           {:password [:cached "password"]})
-                  test-data)))
-            (range 50))]
-      (every? deref futures))))
-
-;;;; Redefs
-
-(defrecord MyFoo [] Object (toString [_] "v1"))
-(str (thaw (freeze (MyFoo.))))
-(defrecord MyFoo [] Object (toString [_] "v2"))
-
-(deftest _redefs
-  (is (= (str (thaw (freeze (MyFoo.)))) "v2")))
-
-;;;; Serializable
-
-(do
-  (def ^:private semcn              "java.util.concurrent.Semaphore")
-  (def ^:private sem                (java.util.concurrent.Semaphore. 1))
-  (defn-         sem? [x] (instance? java.util.concurrent.Semaphore x)))
-
-(deftest _serializable
-  (is (= nippy/*thaw-serializable-allowlist* #{"base.1" "base.2" "add.1" "add.2"})
-    "JVM properties override initial allowlist values")
-
-  (is (thrown? Exception (nippy/freeze sem {:serializable-allowlist #{}}))
-    "Can't freeze Serializable objects unless approved by allowlist")
-
-  (is (sem?
-        (nippy/thaw
-          (nippy/freeze sem {:serializable-allowlist #{semcn}})
-          {:serializable-allowlist #{semcn}}))
-
-    "Can freeze and thaw Serializable objects if approved by allowlist")
-
-  (is (sem?
-        (nippy/thaw
-          (nippy/freeze sem {:serializable-allowlist #{"java.util.concurrent.*"}})
-          {:serializable-allowlist #{"java.util.concurrent.*"}}))
-
-    "Strings in allowlist sets may contain \"*\" wildcards")
-
-  (let [ba     (nippy/freeze sem #_{:serializable-allowlist "*"})
-        thawed (nippy/thaw   ba    {:serializable-allowlist #{}})]
-
-    (is (= :quarantined (get-in thawed [:nippy/unthawable :cause]))
-      "Serializable objects will be quarantined when approved for freezing but not thawing.")
-
-    (is (sem? (nippy/read-quarantined-serializable-object-unsafe! thawed))
-      "Quarantined Serializable objects can still be manually force-read.")
-
-    (is (sem? (nippy/read-quarantined-serializable-object-unsafe!
-                (nippy/thaw (nippy/freeze thawed))))
-      "Quarantined Serializable objects are themselves safely transportable."))
-
-  (let [obj
-        (nippy/thaw
-          (nippy/freeze sem)
-          {:serializable-allowlist "allow-and-record"})]
-
-    (is (sem? obj)
-      "Special \"allow-and-record\" allowlist permits any class")
-
-    (is
-      (contains? (nippy/get-recorded-serializable-classes) semcn)
-      "Special \"allow-and-record\" allowlist records classes")))
-
-;;;; Metadata
-
-(deftest _metadata
-
-  (is
-    (:has-meta?
-     (meta
-       (nippy/thaw
-         (nippy/freeze (with-meta [] {:has-meta? true}) {:incl-metadata? true})
-         {:incl-metadata? true}
-         )))
-
-    "Metadata successfully included")
-
-  (is
-    (nil?
-      (meta
-        (nippy/thaw
-          (nippy/freeze (with-meta [] {:has-meta? true}) {:incl-metadata? true})
-          {:incl-metadata? false}
-          )))
-
-    "Metadata successfully excluded by thaw")
-
-  (is
-    (nil?
-      (meta
-        (nippy/thaw
-          (nippy/freeze (with-meta [] {:has-meta? true}) {:incl-metadata? false})
-          {:incl-metadata? true}
-          )))
-
-    "Metadata successfully excluded by freeze"))
-
-;;;; Benchmarks
-
-(deftest _benchmarks
-  (is (benchmarks/bench {})) ; Also tests :cached passwords
-  )

test/taoensso/nippy_benchmarks.clj

@@ -0,0 +1,207 @@
+(ns taoensso.nippy-benchmarks
+  (:require
+   [clojure.data.fressian      :as fress]
+   [taoensso.encore            :as enc]
+   [taoensso.nippy             :as nippy]
+   [taoensso.nippy.compression :as compr]))
+
+;;;; Reader
+
+(defn- freeze-reader [x]   (enc/pr-edn   x))
+(defn- thaw-reader   [edn] (enc/read-edn edn))
+
+;;;; Fressian
+
+(defn- freeze-fress [x]
+  (let [^java.nio.ByteBuffer bb (fress/write x)
+        len (.remaining bb)
+        ba  (byte-array len)]
+    (.get bb ba 0 len)
+    (do      ba)))
+
+(defn- thaw-fress [^bytes ba]
+  (let [bb (java.nio.ByteBuffer/wrap ba)]
+    (fress/read bb)))
+
+(comment (-> data freeze-fress thaw-fress))
+
+;;;; Bench data
+
+(def default-bench-data
+  "Subset of stress data suitable for benching."
+  (let [sd (nippy/stress-data {:comparable? true})]
+    (reduce-kv
+      (fn [m k v]
+        (try
+          (-> v freeze-reader thaw-reader)
+          (-> v freeze-fress  thaw-fress)
+          m
+          (catch Throwable _ (dissoc m k))))
+      sd sd)))
+
+(comment
+  (clojure.set/difference
+    (set (keys (nippy/stress-data {:comparable? true})))
+    (set (keys default-bench-data))))
+
+;;;;  Serialization
+
+(defn- bench1-serialization
+  [freezer thawer sizer
+   {:keys [laps warmup bench-data]
+    :or
+    {laps       1e4
+     warmup     25e3
+     bench-data default-bench-data}}]
+
+  (let [data-frozen                                       (freezer bench-data)
+        time-freeze (enc/bench laps {:warmup-laps warmup} (freezer bench-data))
+        time-thaw   (enc/bench laps {:warmup-laps warmup} (thawer  data-frozen))
+        data-size   (sizer data-frozen)]
+
+    {:round (+ time-freeze time-thaw)
+     :freeze   time-freeze
+     :thaw                 time-thaw
+     :size  data-size}))
+
+(comment (bench1-serialization nippy/freeze nippy/thaw count {}))
+
+(defn- printed-results [results]
+  (println "\nBenchmark results:")
+  (doseq [[k v] results] (println k " " v))
+  (do           results))
+
+(defn bench-serialization
+  [{:keys [all? reader? fressian? fressian? lzma2? laps warmup bench-data]
+    :as   opts
+    :or
+    {laps   1e4
+     warmup 25e3}}]
+
+  (println "\nRunning benchmarks...")
+
+  (let [results_ (atom {})]
+    (when (or all? reader?)
+      (println "  With Reader...")
+      (swap! results_ assoc :reader
+        (bench1-serialization freeze-reader thaw-reader
+          (fn [^String s] (count (.getBytes s "UTF-8")))
+          (assoc opts :laps laps, :warmup warmup))))
+
+    (when (or all? fressian?)
+      (println "  With Fressian...")
+      (swap! results_ assoc :fressian
+        (bench1-serialization freeze-fress thaw-fress count
+          (assoc opts :laps laps, :warmup warmup))))
+
+    (when (or all? lzma2?)
+      (println "  With Nippy/LZMA2...")
+      (swap! results_ assoc :nippy/lzma2
+        (bench1-serialization
+          #(nippy/freeze % {:compressor nippy/lzma2-compressor})
+          #(nippy/thaw   % {:compressor nippy/lzma2-compressor})
+          count
+          (assoc opts :laps laps, :warmup warmup))))
+
+    (println "  With Nippy/encrypted...")
+    (swap! results_ assoc :nippy/encrypted
+      (bench1-serialization
+        #(nippy/freeze % {:password [:cached "p"]})
+        #(nippy/thaw   % {:password [:cached "p"]})
+        count
+        (assoc opts :laps laps, :warmup warmup)))
+
+    (println "  With Nippy/default...")
+    (swap! results_ assoc :nippy/default
+      (bench1-serialization nippy/freeze nippy/thaw count
+        (assoc opts :laps laps, :warmup warmup)))
+
+    (println "  With Nippy/fast...")
+    (swap! results_ assoc :nippy/fast
+      (bench1-serialization nippy/fast-freeze nippy/fast-thaw count
+        (assoc opts :laps laps, :warmup warmup)))
+
+    (println "\nBenchmarks done:")
+    (printed-results @results_)))
+
+;;;; Compression
+
+(defn- bench1-compressor
+  [compressor
+   {:keys [laps warmup bench-data]
+    :or
+    {laps       1e4
+     warmup     2e4
+     bench-data default-bench-data}}]
+
+  (let [data-frozen     (nippy/freeze bench-data {:compressor nil})
+        data-compressed                                       (compr/compress   compressor data-frozen)
+        time-compress   (enc/bench laps {:warmup-laps warmup} (compr/compress   compressor data-frozen))
+        time-decompress (enc/bench laps {:warmup-laps warmup} (compr/decompress compressor data-compressed))]
+
+    {:round   (+ time-compress time-decompress)
+     :compress   time-compress
+     :decompress time-decompress
+     :ratio      (enc/round2 (/ (count data-compressed) (count data-frozen)))}))
+
+(defn bench-compressors [opts lzma-opts]
+  (printed-results
+    (merge
+      (let [bench1 #(bench1-compressor % opts)]
+        {:zstd/prepended     (bench1 (compr/->ZstdCompressor true))
+         :zstd/unprepended   (bench1 (compr/->ZstdCompressor false))
+         :lz4                (bench1 (compr/->LZ4Compressor))
+         :lzo                (bench1 (compr/->LZOCompressor))
+         :snappy/prepended   (bench1 (compr/->SnappyCompressor true))
+         :snappy/unprepended (bench1 (compr/->SnappyCompressor false))})
+
+      (let [bench1 #(bench1-compressor % (merge opts lzma-opts))]
+        {:lzma2/level0 (bench1 (compr/->LZMA2Compressor 0))
+         :lzma2/level3 (bench1 (compr/->LZMA2Compressor 3))
+         :lzma2/level6 (bench1 (compr/->LZMA2Compressor 6))
+         :lzma2/level9 (bench1 (compr/->LZMA2Compressor 9))}))))
+
+;;;; Results
+
+(comment
+  {:last-updated    "2024-01-16"
+   :system          "2020 Macbook Pro M1, 16 GB memory"
+   :clojure-version "1.11.1"
+   :java-version    "OpenJDK 21"
+   :deps
+   '[[com.taoensso/nippy        "3.4.0-beta1"]
+     [org.clojure/tools.reader  "1.3.7"]
+     [org.clojure/data.fressian "1.0.0"]
+     [org.tukaani/xz            "1.9"]
+     [io.airlift/aircompressor  "0.25"]]}
+
+  (bench-serialization {:all? true})
+
+  {:reader          {:round 13496, :freeze 5088, :thaw 8408, :size 15880}
+   :fressian        {:round 3898,  :freeze 2350, :thaw 1548, :size 12222}
+   :nippy/lzma2     {:round 12341, :freeze 7809, :thaw 4532, :size 3916}
+   :nippy/encrypted {:round 2939,  :freeze 1505, :thaw 1434, :size 8547}
+   :nippy/default   {:round 2704,  :freeze 1330, :thaw 1374, :size 8519}
+   :nippy/fast      {:round 2425,  :freeze 1117, :thaw 1308, :size 17088}}
+
+  (enc/round2 (/ 2704 13496)) ; 0.20 of reader   roundtrip time
+  (enc/round2 (/ 2704  3898)) ; 0.69 of fressian roundtrip time
+
+  (enc/round2 (/ 8519 15880)) ; 0.54 of reader   output size
+  (enc/round2 (/ 8519 12222)) ; 0.70 of fressian output size
+
+  (bench-compressors
+    {:laps 1e4 :warmup 2e4}
+    {:laps 1e2 :warmup 2e2})
+
+  ;; Note that ratio depends on compressibility of stress data
+  {:lz4                {:round 293,  :compress 234,  :decompress 59,  :ratio 0.5}
+   :lzo                {:round 483,  :compress 349,  :decompress 134, :ratio 0.46}
+   :snappy/prepended   {:round 472,  :compress 296,  :decompress 176, :ratio 0.43}
+   :snappy/unprepended {:round 420,  :compress 260,  :decompress 160, :ratio 0.43}
+   :zstd/prepended     {:round 2105, :compress 1419, :decompress 686, :ratio 0.3}
+   :zstd/unprepended   {:round 1261, :compress 921,  :decompress 340, :ratio 0.3}
+   :lzma2/level0       {:round 158,  :compress 121,  :decompress 37,  :ratio 0.24}
+   :lzma2/level3       {:round 536,  :compress 436,  :decompress 100, :ratio 0.22}
+   :lzma2/level6       {:round 1136, :compress 1075, :decompress 61,  :ratio 0.21}
+   :lzma2/level9       {:round 2391, :compress 2096, :decompress 295, :ratio 0.21}})

test/taoensso/nippy_tests.clj

@@ -4,9 +4,14 @@
    [clojure.test.check            :as tc]
    [clojure.test.check.generators :as tc-gens]
    [clojure.test.check.properties :as tc-props]
-   [taoensso.encore :as enc   :refer []]
+   [taoensso.truss                :as truss :refer [throws?]]
-   [taoensso.nippy  :as nippy :refer [freeze thaw]]
+   [taoensso.encore               :as enc   :refer [ba=]]
-   [taoensso.nippy.benchmarks :as benchmarks]))
+   [taoensso.nippy                :as nippy :refer [freeze thaw]]
+   [taoensso.nippy.impl           :as impl]
+   [taoensso.nippy.tools          :as tools]
+   [taoensso.nippy.compression    :as compr]
+   [taoensso.nippy.crypto         :as crypto]
+   [taoensso.nippy-benchmarks     :as benchmarks]))
 
 (comment
   (remove-ns      'taoensso.nippy-tests)
@@ -14,7 +19,7 @@
 
 ;;;; Config, etc.
 
-(def test-data nippy/stress-data-comparable)
+(def test-data (nippy/stress-data {:comparable? true}))
 (def tc-gen-recursive-any-equatable
   (tc-gens/recursive-gen tc-gens/container-type
     tc-gens/any-equatable))
@@ -33,7 +38,13 @@
 ;;;; Core
 
 (deftest _core
-  [(println (str "Clojure version: " *clojure-version*))
+  (println (str "Clojure version: " *clojure-version*))
+  [(is (= test-data test-data) "Test data is comparable")
+   (is (=
+         (nippy/stress-data {:comparable? true})
+         (nippy/stress-data {:comparable? true}))
+     "Stress data is deterministic")
+
    (is (= test-data ((comp thaw freeze) test-data)))
    (is (= test-data ((comp #(thaw   % {:no-header? true
                                        :compressor nippy/lz4-compressor
@@ -45,9 +56,6 @@
                            #(freeze % {:password [:salted "p"]}))
                      test-data)))
 
-   (is (= (vec (:objects nippy/stress-data))
-          ((comp vec thaw freeze) (:objects nippy/stress-data))))
-
    (is (= test-data ((comp #(thaw   % {:compressor nippy/lzma2-compressor})
                            #(freeze % {:compressor nippy/lzma2-compressor}))
                      test-data)))
@@ -59,29 +67,16 @@
                      test-data)))
 
    (is (= test-data ((comp #(thaw   % {:compressor nippy/lz4-compressor})
-                           #(freeze % {:compressor nippy/lz4hc-compressor}))
+                           #(freeze % {:compressor nippy/lz4-compressor}))
                      test-data)))
 
-   (is (enc/throws? Exception (thaw (freeze test-data {:password "malformed"}))))
+   (is (= test-data ((comp #(thaw   % {:compressor nippy/zstd-compressor})
-   (is (enc/throws? Exception (thaw (freeze test-data {:password [:salted "p"]})
+                           #(freeze % {:compressor nippy/zstd-compressor}))
-                                {;; Necessary to prevent against JVM segfault due to
+                     test-data)))
-                                 ;; https://goo.gl/t0OUIo:
-                                 :v1-compatibility? false})))
-   (is (enc/throws? Exception (thaw (freeze test-data {:password [:salted "p"]})
-                                {:v1-compatibility? false ; Ref. https://goo.gl/t0OUIo
-                                 :compressor nil})))
 
-   (is
+   (is (throws? Exception (thaw (freeze test-data {:password "malformed"}))))
-     (let [^bytes raw-ba    (freeze test-data {:compressor nil})
+   (is (throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
-           ^bytes xerial-ba (org.xerial.snappy.Snappy/compress raw-ba)
+   (is (throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
-           ^bytes iq80-ba   (org.iq80.snappy.Snappy/compress   raw-ba)]
-
-       (= (thaw raw-ba)
-          (thaw (org.xerial.snappy.Snappy/uncompress xerial-ba))
-          (thaw (org.xerial.snappy.Snappy/uncompress iq80-ba))
-          (thaw (org.iq80.snappy.Snappy/uncompress   iq80-ba    0 (alength iq80-ba)))
-          (thaw (org.iq80.snappy.Snappy/uncompress   xerial-ba  0 (alength xerial-ba)))))
-     "Snappy lib compatibility (for legacy versions of Nippy)")
 
    (is
      (= "payload"
@@ -89,7 +84,7 @@
          (do                   {:password [:salted "pwd"]})))
      "CBC auto-encryptor compatibility")
 
-   (testing "Signed long types"
+   (testing "Unsigned long types"
      (let [range-ushort+ (+ (long @#'nippy/range-ushort) 128)
            range-uint+   (+ (long @#'nippy/range-uint)   128)]
 
@@ -98,6 +93,23 @@
         (let [n    range-uint+]                          (= (thaw (freeze n)) n))
         (let [n (- range-uint+)]                         (= (thaw (freeze n)) n))]))
 
+   (is (throws? :ex-info "Failed to freeze type" (nippy/freeze (fn []))))
+
+   (testing "Clojure v1.10+ metadata protocol extensions"
+     [(is (throws? :ex-info "Failed to freeze type" (nippy/freeze (with-meta [] {:a :A, 'b   (fn [])}))))
+      (is (= {:a :A}              (meta (nippy/thaw (nippy/freeze (with-meta [] {:a :A, 'b/c (fn [])}))))))
+      (is (= nil                  (meta (nippy/thaw (nippy/freeze (with-meta [] {       'b/c (fn [])})))))
+        "Don't attach empty metadata")])
+
+   (let [d (nippy/stress-data {})]
+     [(is (= (vec (:bytes   d)) ((comp vec thaw freeze) (:bytes   d))))
+      (is (= (vec (:objects d)) ((comp vec thaw freeze) (:objects d))))])
+
+   (testing "Arrays"
+     (binding [nippy/*thaw-serializable-allowlist* nippy/default-freeze-serializable-allowlist]
+       (mapv (fn [[k aval]] (is (= (vec aval) (-> aval nippy/freeze nippy/thaw vec)) (name k)))
+         (get-in (nippy/stress-data {}) [:non-comparable :arrays]))))
+
    (is (gen-test 1600 [gen-data] (= gen-data (thaw (freeze gen-data)))) "Generative")])
 
 ;;;; Custom types & records
@@ -108,7 +120,7 @@
 (deftest _types
   [(testing "Extend to custom type"
      [(is
-        (enc/throws? Exception ; No thaw extension yet
+        (throws? Exception ; No thaw extension yet
           (do
             (alter-var-root #'nippy/*custom-readers* (constantly {}))
             (nippy/extend-freeze MyType 1 [x s]
@@ -152,18 +164,15 @@
 ;;;; Caching
 
 (deftest _caching
-  (let [stress [nippy/stress-data-comparable
+  (let [test-data* [test-data test-data test-data test-data] ; Data with duplicates
-                nippy/stress-data-comparable
+        cached (mapv nippy/cache test-data*)
-                nippy/stress-data-comparable
+        cached (mapv nippy/cache test-data*) ; <=1 wrap auto-enforced
-                nippy/stress-data-comparable]
-        cached (mapv nippy/cache stress)
-        cached (mapv nippy/cache stress) ; <=1 wrap auto-enforced
         ]
 
-    [(is (= stress (thaw (freeze stress {:compressor nil}))))
+    [(is (= test-data*  (thaw (freeze test-data* {:compressor nil}))))
-     (is (= stress (thaw (freeze cached {:compressor nil}))))
+     (is (= test-data*  (thaw (freeze cached     {:compressor nil}))))
-     (let [size-stress (count (freeze stress {:compressor nil}))
+     (let [size-stress (count (freeze test-data* {:compressor nil}))
-           size-cached (count (freeze cached {:compressor nil}))]
+           size-cached (count (freeze cached     {:compressor nil}))]
        (is (>= size-stress (* 3 size-cached)))
        (is (<  size-stress (* 4 size-cached))))]))
 
@@ -187,52 +196,91 @@
      (is (= (mapv meta (thaw frozen-with-caching))
             [{:id :v1} {:id :v2} {:id :v1} {:id :v2}]))]))
 
-;;;; Stable binary representation of vals
+;;;; Serialized output
 
-(deftest _stable-bin
+(defn ba-hash [^bytes ba] (hash (seq ba)))
-  (testing "Stable binary representation of vals"
 
-    (testing "x=y !=> f(x)=f(y)"
+(defn gen-hashes [] (enc/map-vals (fn [v] (ba-hash (freeze v))) test-data))
-      ;; `x=y => f(x)=f(y)` is unfortunately NOT true in general
+(defn cmp-hashes [new old] (vec (sort (reduce-kv (fn [s k v] (if (= (get old k) v) s (conj s k))) #{} new))))
-      ;; Be careful to never assume the above relationship!
+
-      [(is (not= (vec (freeze {:a 1 :b 1})) (vec (freeze {:b 1 :a 1}))) "Small (array) map (not= (seq {:a 1 :b 1}) (seq {:b 1 :a 1}))")
+(def ref-hashes-v341
-       (is (not= (vec (freeze [[]]))        (vec (freeze ['()])))       "(= [] '()) is true")
+  {:deftype -148586793, :lazy-seq-empty 1277437598, :true -1809580601, :long 598276629, :double -454270428, :lazy-seq -1039619789, :short 1152993378, :meta -858252893, :str-long -1970041891, :instant -1401948864, :many-keywords 665654816, :bigint 2033662230, :sym-ns 769802402, :queue 447747779, :float 603100813, :sorted-set 2005004017, :many-strings 1738215727, :nested -1350538572, :queue-empty 1760934486, :duration -775528642, :false 1506926383, :vector 813550992, :util-date 1326218051, :kw 389651898, :sym -1742024487, :str-short -921330463, :subvec 709331681, :kw-long 852232872, :integer 624865727, :sym-long -1535730190, :list -1207486853, :ratio 1186850097, :byte -1041979678, :bigdec -1846988137, :nil 2005042235, :defrecord -553848560, :sorted-map -1160380145, :sql-date 80018667, :map-entry 1219306839, :false-boxed 1506926383, :uri 870148616, :period -2043530540, :many-longs -1109794519, :uuid -338331115, :set 1649942133, :kw-ns 1050084331, :map 1989337680, :many-doubles -827569787, :char 858269588})
-       (is (= (vec (freeze (sorted-map :a 1 :b 1)))
+
-              (vec (freeze (sorted-map :b 1 :a 1)))) "Sorted structures are immune")])
+(def ref-hashes-v340
+  {:deftype 1529147805, :lazy-seq-empty 1277437598, :true -1809580601, :long 219451189, :double -454270428, :lazy-seq -1039619789, :short 1152993378, :meta 352218350, :str-long -1970041891, :instant -1401948864, :many-keywords 665654816, :bigint 2033662230, :sym-ns 769802402, :queue 447747779, :float 603100813, :sorted-set 1443292905, :many-strings 1777678883, :nested -1590473924, :queue-empty 1760934486, :duration -775528642, :false 1506926383, :vector 89425525, :util-date 1326218051, :kw 389651898, :sym -1742024487, :str-short -1097575232, :subvec -2047667173, :kw-long 852232872, :integer 624865727, :sym-long -1535730190, :list -1113199651, :ratio 1186850097, :byte -1041979678, :bigdec -1846988137, :nil 2005042235, :defrecord 287634761, :sorted-map 1464032648, :sql-date 80018667, :map-entry -1353323498, :false-boxed 1506926383, :uri -1374752165, :period -2043530540, :many-longs 759118414, :uuid -338331115, :set -1515144175, :kw-ns 1050084331, :map 358912619, :many-doubles -827569787, :char 858269588})
+
+(comment
+  (cmp-hashes ref-hashes-v341 ref-hashes-v340)
+  [:defrecord :deftype :list :long :many-longs :many-strings :map :map-entry :meta :nested :set :sorted-map :sorted-set :str-short :subvec :uri :vector])
+
+(deftest _stable-serialized-output
+  (testing "Stable serialized output"
+
+    (testing "x=y => f(x)=f(y) for SOME inputs, SOMETIMES"
+      ;; `x=y => f(x)=f(y)` is unfortunately NOT true in general, and NOT something we
+      ;; promise. Still, we do unofficially try our best to maintain this property when
+      ;; possible - and to warn when it'll be violated for common/elementary types.
+      [(is (not (ba= (freeze {:a 1 :b 1}) (freeze {:b 1 :a 1}))) "Small (array) map (not= (seq {:a 1 :b 1}) (seq {:b 1 :a 1}))")
+       (is (not (ba= (freeze [[]])        (freeze ['()])))       "(= [] '()) is true")
+       (is      (ba= (freeze (sorted-map :a 1 :b 1))
+                     (freeze (sorted-map :b 1 :a 1))) "Sorted structures are generally safe")
+
+       ;; Track serialized output of stress data so that we can detect unintentional changes,
+       ;; and warn about intended ones. Hashes will need to be recalculated on changes to stress data.
+       (let [reference-hashes ref-hashes-v341
+             failures ; #{{:keys [k v]}}
+             (reduce-kv
+               (fn [failures k v]
+                 (or
+                   (when (not= v :taoensso.nippy/skip)
+                     (let [frozen (freeze v)
+                           actual (ba-hash frozen)
+                           ref    (get reference-hashes k)]
+                       (when (not= actual ref)
+                         (conj failures
+                           {:k k,
+                            :v {:type (type v), :value v}
+                            :actual actual
+                            :ref    ref
+                            :frozen (vec frozen)}))))
+                   failures))
+               #{}
+               test-data)]
+
+         (is (empty? failures)))])
 
     (testing "x==y => f(x)=f(y)"
       ;; This weaker version of `x=y => f(x)=f(y)` does hold
-      [(is (= (vec (freeze test-data))
+      [(is (ba= (freeze test-data)
-              (vec (freeze test-data))))
+                (freeze test-data)))
 
-       (is (every? true? (repeatedly 1000 (fn [] (= (vec (freeze test-data))
+       (is (every? true? (repeatedly 1000 (fn [] (ba= (freeze test-data)
-                                                    (vec (freeze test-data))))))
+                                                      (freeze test-data)))))
          "Try repeatedly to catch possible protocol interface races")
 
        (is (gen-test 400 [gen-data]
-              (= (vec (freeze gen-data))
+              (ba= (freeze gen-data)
-                 (vec (freeze gen-data)))) "Generative")])
+                   (freeze gen-data))) "Generative")])
 
     (testing "f(x)=f(f-1(f(x)))"
-      [(is (= (vec (-> test-data freeze))
+      [(is (ba= (-> test-data freeze)
-              (vec (-> test-data freeze thaw freeze))))
+                (-> test-data freeze thaw freeze)))
 
-       (is (= (seq (freeze test-data))
+       (is (ba= (freeze test-data)
-              (seq (reduce (fn [frozen _] (freeze (thaw frozen)))
+                (reduce (fn [frozen _] (freeze (thaw frozen))) (freeze test-data) (range 1000)))
-                     (freeze test-data) (range 1000))))
          "Try repeatedly to catch possible protocol interface races")
 
        (is (gen-test 400 [gen-data]
-             (= (vec (-> gen-data freeze))
+             (ba= (-> gen-data freeze)
-                (vec (-> gen-data freeze thaw freeze)))) "Generative")])
+                  (-> gen-data freeze thaw freeze))) "Generative")])
 
     (testing "f(x)=f(y) => x=y"
       (let [vals_ (atom {})]
         (gen-test 400 [gen-data]
-          (let [bin (freeze     gen-data)
+          (let [out (freeze     gen-data)
                 ref (get @vals_ gen-data ::nx)]
-            (swap! vals_ assoc bin gen-data)
+            (swap! vals_ assoc out gen-data)
-            (or (= ref ::nx) (= ref bin))))))))
+            (or (= ref ::nx) (= ref out))))))))
 
 ;;;; Thread safety
 
@@ -283,7 +331,7 @@
   [(is (= nippy/*thaw-serializable-allowlist* #{"base.1" "base.2" "add.1" "add.2"})
      "JVM properties override initial allowlist values")
 
-   (is (enc/throws? Exception (nippy/freeze sem {:serializable-allowlist #{}}))
+   (is (throws? Exception (nippy/freeze sem {:serializable-allowlist #{}}))
      "Can't freeze Serializable objects unless approved by allowlist")
 
    (is (sem?
@@ -365,6 +413,20 @@
 
      "Don't try to preserve metadata on vars")])
 
+;;;; Freezable?
+
+(deftest _freezable?
+  [(is (= (nippy/freezable? :foo)                      :native))
+   (is (= (nippy/freezable? [:a :b])                   :native))
+   (is (= (nippy/freezable? [:a (fn [])])                  nil))
+   (is (= (nippy/freezable? [:a (byte-array [1 2 3])]) :native))
+   (is (= (nippy/freezable? [:a (java.util.Date.)])    :native))
+   (is (= (nippy/freezable? (Exception.))                  nil))
+   (is (= (nippy/freezable? (MyType. "a" "b"))         :native))
+   (is (= (nippy/freezable? (MyRec.  "a" "b"))         :native))
+   (is (= (nippy/freezable? (Exception.) {:allow-java-serializable? true})
+         :maybe-java-serializable))])
+
 ;;;; thaw-xform
 
 (deftest _thaw-xform
@@ -381,9 +443,29 @@
 
    (is (= (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze []))) []) "rf not run on empty colls")
 
-   (let [ex (enc/throws :default (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze [:a :b]))))]
+   (let [ex (truss/throws :default (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze [:a :b]))))]
-     (is (= (-> ex enc/ex-cause enc/ex-cause ex-data :call) '(rf acc in)) "Error thrown via `*thaw-xform*`"))])
+     (is (= (-> ex ex-cause ex-cause ex-data :call) '(rf acc in)) "Error thrown via `*thaw-xform*`"))])
+
+;;;; Compressors
+
+(deftest _compressors
+  (println "\nTesting decompression of random data...")
+  (doseq [c [compr/zstd-compressor
+             compr/lz4-compressor
+             compr/lzo-compressor
+             compr/snappy-compressor
+             compr/lzma2-compressor]]
+
+    (print (str "  With " (name (compr/header-id c)))) (flush)
+    (dotimes [_ 5] ; Slow, a few k laps should be sufficient for CI
+      (print ".") (flush)
+      (dotimes [_ 1000]
+        (is
+          (nil? (truss/catching :all (compr/decompress c (crypto/rand-bytes 1024))))
+          "Decompression never crashes JVM, even against invalid data")))
+    (println)))
 
 ;;;; Benchmarks
 
-(deftest _benchmarks (is (benchmarks/bench {})))
+(deftest _benchmarks
+  (is (benchmarks/bench-serialization {:all? true})))

wiki/.gitignore

@@ -0,0 +1 @@
+README.md

wiki/1 Getting-started.md

@@ -15,94 +15,83 @@ And setup your namespace imports:
 
 # De/serializing
 
-As an example of what it can do, let's take a look at Nippy's own reference stress data:
+As an example of what it can do, let's take a look at Nippy's own reference [stress data](https://taoensso.github.io/nippy/taoensso.nippy.html#var-stress-data):
 
 ```clojure
-nippy/stress-data
-=>
 {:nil                   nil
  :true                  true
  :false                 false
- :boxed-false (Boolean. false)
+ :false-boxed (Boolean. false)
 
  :char      \ಬ
  :str-short "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"
- :str-long  (apply str (range 1000))
+ :str-long  (reduce str (range 1024))
  :kw        :keyword
  :kw-ns     ::keyword
- :kw-long   (keyword
-              (apply str "kw" (range 1000))
-              (apply str "kw" (range 1000)))
-
  :sym       'foo
  :sym-ns    'foo/bar
- :sym-long   (symbol
+ :kw-long   (keyword (reduce str "_" (range 128)) (reduce str "_" (range 128)))
-               (apply str "sym" (range 1000))
+ :sym-long  (symbol  (reduce str "_" (range 128)) (reduce str "_" (range 128)))
-               (apply str "sym" (range 1000)))
 
- :regex     #"^(https?:)?//(www\?|\?)?"
+ :byte      (byte   16)
+ :short     (short  42)
+ :integer   (int    3)
+ :long      (long   3)
+ :float     (float  3.1415926535897932384626433832795)
+ :double    (double 3.1415926535897932384626433832795)
+ :bigdec    (bigdec 3.1415926535897932384626433832795)
+ :bigint    (bigint  31415926535897932384626433832795)
+ :ratio     22/7
 
- ;;; Try reflect real-world data:
+ :list      (list 1 2 3 4 5 (list 6 7 8 (list 9 10 (list) ())))
- :many-small-numbers  (vec (range 200))
+ :vector    [1 2 3 4 5 [6 7 8 [9 10 [[]]]]]
- :many-small-keywords (->> (java.util.Locale/getISOLanguages)
+ :subvec    (subvec [1 2 3 4 5 6 7 8] 2 8)
-                           (mapv keyword))
+ :map       {:a 1 :b 2 :c 3 :d {:e 4 :f {:g 5 :h 6 :i 7 :j {{} {}}}}}
- :many-small-strings  (->> (java.util.Locale/getISOCountries)
+ :map-entry (clojure.lang.MapEntry/create "key" "val")
-                           (mapv #(.getDisplayCountry (java.util.Locale. "en" %))))
+ :set       #{1 2 3 4 5 #{6 7 8 #{9 10 #{#{}}}}}
+ :meta      (with-meta {:a :A} {:metakey :metaval})
+ :nested    [#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{[] ()}}] #{:a :b}}
+             #{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{[] ()}}] #{:a :b}}
+             [1 [1 2 [1 2 3 [1 2 3 4 [1 2 3 4 5 "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"] {} #{} [] ()]]]]]
 
- :queue        (enc/queue [:a :b :c :d :e :f :g])
+ :regex          #"^(https?:)?//(www\?|\?)?"
- :queue-empty  (enc/queue)
+ :sorted-set     (sorted-set 1 2 3 4 5)
- :sorted-set   (sorted-set 1 2 3 4 5)
+ :sorted-map     (sorted-map :b 2 :a 1 :d 4 :c 3)
- :sorted-map   (sorted-map :b 2 :a 1 :d 4 :c 3)
+ :lazy-seq-empty (map identity ())
+ :lazy-seq       (repeatedly 64 #(do nil))
+ :queue-empty    (into clojure.lang.PersistentQueue/EMPTY [:a :b :c :d :e :f :g])
+ :queue                clojure.lang.PersistentQueue/EMPTY
 
- :list         (list 1 2 3 4 5 (list 6 7 8 (list 9 10 '(()))))
+ :uuid       (java.util.UUID. 7232453380187312026 -7067939076204274491)
- :vector       [1 2 3 4 5 [6 7 8 [9 10 [[]]]]]
+ :uri        (java.net.URI. "https://clojure.org")
- :map          {:a 1 :b 2 :c 3 :d {:e 4 :f {:g 5 :h 6 :i 7 :j {{} {}}}}}
+ :defrecord  (nippy/StressRecord. "data")
- :set          #{1 2 3 4 5 #{6 7 8 #{9 10 #{#{}}}}}
+ :deftype    (nippy/StressType.   "data")
- :meta         (with-meta {:a :A} {:metakey :metaval})
+ :bytes      (byte-array   [(byte 1) (byte 2) (byte 3)])
- :nested       [#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{}}] #{:a :b}}
+ :objects    (object-array [1 "two" {:data "data"}])
-                #{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{}}] #{:a :b}}
-                  [1 [1 2 [1 2 3 [1 2 3 4 [1 2 3 4 5]]]]]]
 
- :lazy-seq       (repeatedly 1000 rand)
+ :util-date (java.util.Date. 1577884455500)
- :lazy-seq-empty (map identity '())
+ :sql-date  (java.sql.Date.  1577884455500)
+ :instant   (java.time.Instant/parse "2020-01-01T13:14:15.50Z")
+ :duration  (java.time.Duration/ofSeconds 100 100)
+ :period    (java.time.Period/of 1 1 1)
 
- :byte         (byte   16)
+ :throwable (Throwable. "Msg")
- :short        (short  42)
+ :exception (Exception. "Msg")
- :integer      (int    3)
+ :ex-info   (ex-info    "Msg" {:data "data"})
- :long         (long   3)
- :bigint       (bigint 31415926535897932384626433832795)
 
- :float        (float  3.14)
+ :many-longs    (vec (repeatedly 512         #(rand-nth (range 10))))
- :double       (double 3.14)
+ :many-doubles  (vec (repeatedly 512 #(double (rand-nth (range 10)))))
- :bigdec       (bigdec 3.1415926535897932384626433832795)
+ :many-strings  (vec (repeatedly 512         #(rand-nth ["foo" "bar" "baz" "qux"])))
-
+ :many-keywords (vec (repeatedly 512
- :ratio        22/7
+                       #(keyword
- :uri          (java.net.URI. "https://clojure.org/reference/data_structures")
+                          (rand-nth ["foo" "bar" "baz" "qux" nil])
- :uuid         (java.util.UUID/randomUUID)
+                          (rand-nth ["foo" "bar" "baz" "qux"    ]))))}
- :util-date    (java.util.Date.)
- :sql-date     (java.sql.Date/valueOf "2023-06-21")
-
- ;;; JVM 8+
- :time-instant  (enc/compile-if java.time.Instant  (java.time.Instant/now)                nil)
- :time-duration (enc/compile-if java.time.Duration (java.time.Duration/ofSeconds 100 100) nil)
- :time-period   (enc/compile-if java.time.Period   (java.time.Period/of 1 1 1)            nil)
-
- :bytes         (byte-array [(byte 1) (byte 2) (byte 3)])
- :objects       (object-array [1 "two" {:data "data"}])
-
- :stress-record (StressRecord. "data")
- :stress-type   (StressType.   "data")
-
- ;; Serializable
- :throwable    (Throwable. "Yolo")
- :exception    (try (/ 1 0) (catch Exception e e))
- :ex-info      (ex-info "ExInfo" {:data "data"})}
 ```
 
 Serialize it:
 
 ```clojure
-(def frozen-stress-data (nippy/freeze nippy/stress-data))
+(def frozen-stress-data (nippy/freeze (nippy/stress-data {})))
 => #<byte[] [B@3253bcf3>
 ```
 
@@ -118,7 +107,10 @@ Deserialize it:
 
 Couldn't be simpler!
 
-See also the lower-level [`freeze-to-out!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-freeze-to-out.21) and [`thaw-from-in!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-thaw-from-in.21) fns for operating on `DataOutput` and `DataInput` types directly.
+# Streaming
+
+- To serialize directly to a `java.io.DataInput`, see [`freeze-to-out!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-freeze-to-out.21).
+- To deserialize directly from a `java.io.DataOutput`, see [`thaw-from-in!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-thaw-from-in.21).
 
 # Encryption
 
@@ -128,8 +120,8 @@ Nippy also gives you **dead simple data encryption**.
 Add a single option to your usual freeze/thaw calls like so:
 
 ```clojure
-(nippy/freeze nippy/stress-data {:password [:salted "my-password"]}) ; Encrypt
+(nippy/freeze (nippy/stress-data {}) {:password [:salted "my-password"]}) ; Encrypt
-(nippy/thaw   <encrypted-data>  {:password [:salted "my-password"]}) ; Decrypt
+(nippy/thaw   <encrypted-data>       {:password [:salted "my-password"]}) ; Decrypt
 ```
 
 There's two default forms of encryption on offer: `:salted` and `:cached`. Each of these makes carefully-chosen trade-offs and is suited to one of two common use cases. See [`aes128-encryptor`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-aes128-encryptor) for a detailed explanation of why/when you'd want one or the other.

wiki/2 Operational-considerations.md

@@ -0,0 +1,30 @@
+# Data longevity
+
+Nippy is widely used to store **long-lived** data and promises (as always) that **data serialized today should be readable by all future versions of Nippy**.
+
+But please note that the **converse is not generally true**:
+
+- Nippy `vX` **should** be able to read all data from Nippy `vY<=X` (backwards compatibility)
+- Nippy `vX` **may/not** be able to read all data from Nippy `vY>X` (forwards compatibility)
+
+# Rolling updates and rollback
+
+From time to time, Nippy may introduce:
+
+- Support for serializing **new types**
+- Optimizations to the serialization of **pre-existing types**
+
+To help ease **rolling updates** and to better support **rollback**, Nippy (since version v3.4.1) will always introduce such changes over **two version releases**:
+
+- Release 1: to add **read support** for the new types
+- Release 2: to add **write support** for the new types
+
+Starting from v3.4.1, Nippy's release notes will **always clearly indicate** if a particular update sequence is recommended.
+
+# Stability of byte output
+
+It has **never been an objective** of Nippy to offer **predictable byte output**, and I'd generally **recommend against** depending on specific byte output.
+
+However, I know that a small minority of users *do* have specialized needs in this area.
+
+So starting with Nippy v3.4, Nippy's release notes will **always clearly indicate** if any changes to byte output are expected.

wiki/3 Evolving-your-Nippy-data.md

@@ -1,4 +1,4 @@
-> This article was kindly contributed by a Nippy user (@Outrovurt)
+> This article is **community content** kindly contributed by a Nippy user (@Outrovurt)
 
 This article describes a number of use cases where you need to make changes to your code which will have some impact on data you have already frozen using Nippy, and how best to manage each specific case. We will also discuss custom freezing and thawing.
 

wiki/Home.md

@@ -1,4 +1,4 @@
-See the menu to the right for content 👉
+See the **menu to the right** for content 👉
 
 # Contributions welcome
 

wiki/README.md

@@ -0,0 +1,5 @@
+# Attention!
+
+This wiki is designed for viewing from [here](../../../wiki)!
+
+Viewing from GitHub's file browser will result in **broken links**.