Compare commits
42 commits
v3.4.0-RC1
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
ae4463a85c | ||
|
|
1b99516177 | ||
|
|
e68c2d56fe | ||
|
|
bfba59483a | ||
|
|
f7bb2824ac | ||
|
|
8d62dc2826 | ||
|
|
da57206b0d | ||
|
|
e0f49ced5a | ||
|
|
8d107650cd | ||
|
|
1026ea0ae7 | ||
|
|
c92457025f | ||
|
|
3cb29f3c2e | ||
|
|
a9ea13618c | ||
|
|
d415a2bf72 | ||
|
|
b217db5579 | ||
|
|
9022aad018 | ||
|
|
c0d1da1bb4 | ||
|
|
bb178f66fc | ||
|
|
9b380821cd | ||
|
|
c5209e32ce | ||
|
|
f6240582e1 | ||
|
|
4cb2a14adf | ||
|
|
dc52356106 | ||
|
|
bd4d5205d5 | ||
|
|
229ab94c14 | ||
|
|
535d4e5ab0 | ||
|
|
51298e9252 | ||
|
|
738023764c | ||
|
|
1b05c9b8f9 | ||
|
|
82a050b925 | ||
|
|
37cf415c02 | ||
|
|
92c4a83d61 | ||
|
|
af928ed6a4 | ||
|
|
f749e07eed | ||
|
|
4d96757447 | ||
|
|
03c4cf1784 | ||
|
|
ea7d9ae9de | ||
|
|
cb0b871fe8 | ||
|
|
7be9b4f789 | ||
|
|
cb5b7cf063 | ||
|
|
40143e71ee | ||
|
|
7e84f58ee4 |
18 changed files with 883 additions and 367 deletions
6
.github/workflows/graal-tests.yml
vendored
6
.github/workflows/graal-tests.yml
vendored
|
|
@ -10,7 +10,7 @@ jobs:
|
|||
|
||||
runs-on: ${{ matrix.os }}
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/checkout@v4
|
||||
- uses: graalvm/setup-graalvm@v1
|
||||
with:
|
||||
version: 'latest'
|
||||
|
|
@ -18,12 +18,12 @@ jobs:
|
|||
components: 'native-image'
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- uses: DeLaGuardo/setup-clojure@10.0
|
||||
- uses: DeLaGuardo/setup-clojure@12.5
|
||||
with:
|
||||
lein: latest
|
||||
bb: latest
|
||||
|
||||
- uses: actions/cache@v3
|
||||
- uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.m2/repository
|
||||
key: deps-${{ hashFiles('deps.edn') }}
|
||||
|
|
|
|||
10
.github/workflows/main-tests.yml
vendored
10
.github/workflows/main-tests.yml
vendored
|
|
@ -5,22 +5,22 @@ jobs:
|
|||
tests:
|
||||
strategy:
|
||||
matrix:
|
||||
java: ['17', '18', '19']
|
||||
java: ['17', '19', '21']
|
||||
os: [ubuntu-latest]
|
||||
|
||||
runs-on: ${{ matrix.os }}
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-java@v3
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-java@v4
|
||||
with:
|
||||
distribution: 'corretto'
|
||||
java-version: ${{ matrix.java }}
|
||||
|
||||
- uses: DeLaGuardo/setup-clojure@10.0
|
||||
- uses: DeLaGuardo/setup-clojure@12.5
|
||||
with:
|
||||
lein: latest
|
||||
|
||||
- uses: actions/cache@v3
|
||||
- uses: actions/cache@v4
|
||||
id: cache-deps
|
||||
with:
|
||||
path: ~/.m2/repository
|
||||
|
|
|
|||
216
CHANGELOG.md
216
CHANGELOG.md
|
|
@ -2,6 +2,222 @@ This project uses [**Break Versioning**](https://www.taoensso.com/break-versioni
|
|||
|
||||
---
|
||||
|
||||
# `v3.5.0` (2025-04-15)
|
||||
|
||||
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.5.0)
|
||||
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
|
||||
|
||||
This is a **general maintenance release** focused on updating dependencies and laying groundwork (read support) for new array types coming in Nippy v3.6.
|
||||
|
||||
It **drops support for Clojure v1.9** but should otherwise be a safe update from (at least) all recent versions of Nippy.
|
||||
|
||||
## Since `v3.5.0-RC1` (2024-10-28)
|
||||
|
||||
- **\[mod]** Drop official Clojure v1.9 support \[da57206]
|
||||
- \[new] Add string array type to default [thaw-serializable-allowlist](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*thaw-serializable-allowlist*) \[bfba594]
|
||||
- \[new] Use [Truss exceptions](https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#ex-info) on errors \[8d62dc2]
|
||||
- \[new] [#184] Incl. cause on non-native freeze failures \[8d10765]
|
||||
- \[doc] Clarify `*freeze-fallback*` docstring \[1026ea0]
|
||||
|
||||
## Since `v3.4.2` (2024-05-26)
|
||||
|
||||
- **\[mod]** Drop official Clojure v1.9 support \[da57206]
|
||||
- \[new] Add string array type to default [thaw-serializable-allowlist](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*thaw-serializable-allowlist*) \[bfba594]
|
||||
- \[new] Use [Truss exceptions](https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#ex-info) on errors \[8d62dc2]
|
||||
- \[new] [#184] Incl. cause on non-native freeze failures \[8d10765]
|
||||
- \[new] [#175] Mark [cache](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#cache) feature as stable \[b217db5]
|
||||
- \[doc] Clarify `*freeze-fallback*` docstring \[1026ea0]
|
||||
|
||||
## Migration info
|
||||
|
||||
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
|
||||
| :------------------------ | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
|
||||
| `v3.5.0-RC1` (2024-10-28) | - | - | - |
|
||||
| `v3.4.2` (2024-05-26) | - | - | - |
|
||||
| `v3.4.1` (2024-05-02) | - | - | - |
|
||||
| `v3.4.0` (2024-04-30) | - | **Yes** | - |
|
||||
| `v3.3.0` (2023-10-11) | - | - | - |
|
||||
| `v3.2.0` (2022-07-18) | - | - | - |
|
||||
| `v3.1.3` (2022-06-23) | - | - | - |
|
||||
|
||||
> [1] Relevant only when Nippy introduces support for new types **AND** you plan to update Nippy with a **rolling update** (coexisting new and old versions).
|
||||
|
||||
If updating from older versions of Nippy, please see the relevant release notes.
|
||||
|
||||
As always:
|
||||
|
||||
- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
|
||||
- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
|
||||
- **Please report any unexpected problems** 🙏
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
||||
---
|
||||
|
||||
# `v3.5.0-RC1` (2024-10-28)
|
||||
|
||||
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.5.0-RC1)
|
||||
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
|
||||
|
||||
This is a **non-breaking maintenance release** that updates dependencies and includes read support for more native array types to be introduced in a future v3.6 release.
|
||||
|
||||
It should be safe to update from (at least) all recent versions of Nippy.
|
||||
|
||||
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
|
||||
| :-------------------- | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
|
||||
| `v3.4.2` (2024-05-26) | - | - | - |
|
||||
| `v3.4.1` (2024-05-02) | - | - | - |
|
||||
| `v3.4.0` (2024-04-30) | - | Yes | - |
|
||||
| `v3.3.0` (2023-10-11) | - | - | - |
|
||||
| `v3.2.0` (2022-07-18) | - | - | - |
|
||||
| `v3.1.3` (2022-06-23) | - | - | - |
|
||||
|
||||
> [1] Relevant only when introducing support for new types, and for users that do rolling updates
|
||||
|
||||
If updating from older versions of Nippy, please see the relevant release notes.
|
||||
|
||||
As always:
|
||||
|
||||
- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
|
||||
- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
|
||||
- **Please report any unexpected problems** 🙏
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
||||
---
|
||||
|
||||
# `v3.4.2` (2024-05-26)
|
||||
|
||||
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.2)
|
||||
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
|
||||
|
||||
⚠️ This release addresses a [**security vulnerability**](https://github.com/taoensso/nippy/security/advisories/GHSA-vw78-267v-588h) in Nippy's upstream compression library and is **recommended for all existing users**.
|
||||
|
||||
It should be a **straight-forward and non-breaking update** for almost everyone:
|
||||
|
||||
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
|
||||
| :-------------------- | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
|
||||
| `v3.4.1` (2024-05-02) | - | - | - |
|
||||
| `v3.4.0` (2024-04-30) | - | Yes | - |
|
||||
| `v3.3.0` (2023-10-11) | - | - | - |
|
||||
| `v3.2.0` (2022-07-18) | - | - | - |
|
||||
| `v3.1.3` (2022-06-23) | - | - | - |
|
||||
|
||||
> [1] Relevant only when introducing support for new types, and for users that do rolling updates
|
||||
|
||||
If updating from older versions of Nippy, please see the relevant release notes.
|
||||
|
||||
As always:
|
||||
|
||||
- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
|
||||
- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
|
||||
- **Please report any unexpected problems** 🙏
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
||||
---
|
||||
|
||||
# `v3.4.1` (2024-05-02)
|
||||
|
||||
> **Dep**: Nippy is [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.1).
|
||||
> **Versioning**: Nippy uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
|
||||
Like [`v3.4.0`](https://github.com/taoensso/nippy/releases/tag/v3.4.0) but introduces an internal mechanism to help make it easier for some users that do **rolling updates** from earlier versions of Nippy.
|
||||
|
||||
Still, the usual warning applies: data **frozen by Nippy version X** should ideally be **thawed by version >= X**, otherwise you run the risk of the thaw throwing when unfamiliar types are encountered. Please note that this can affect **rolling updates**, and can limit your ability to **revert a Nippy update**. Please ensure adequate testing in your environment before updating against production data!
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
||||
## Changes since `v3.4.0`
|
||||
|
||||
* [mod] Due to some internal format changes, Nippy `v3.4.1` may produce **different serialized output** to `v3.4.0` and earlier versions of Nippy. Most users won't care about this, but you could be affected if you depend on specific serialized byte values (for example by comparing serialized output between different versions of Nippy).
|
||||
|
||||
---
|
||||
|
||||
# `v3.4.0` (2024-04-30)
|
||||
|
||||
> **Dep**: Nippy is [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0).
|
||||
> **Versioning**: Nippy uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
|
||||
This is a non-breaking **feature and maintenance** release and should be a safe update for existing users. But as always, please **test carefully and report any unexpected problems**, thank you! 🙏
|
||||
|
||||
**IMPORTANT**: data **frozen by Nippy version X** should always be **thawed by version >= X**, otherwise you run the risk of the thaw throwing when encountering unfamiliar types. Please note that this can affect **rolling updates**, and can limit your ability to **revert a Nippy update**. Please ensure adequate testing in your environment before updating against production data.
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
||||
## Changes since `v3.3.0` (2023-10-11)
|
||||
|
||||
* 82a050b [mod] Don't attach empty metadata (meta will now be `nil` rather than `{}`)
|
||||
|
||||
## Fixes since `v3.3.0` (2023-10-11)
|
||||
|
||||
* 92c4a83 [fix] Broken `*final-freeze-fallback*` default val
|
||||
|
||||
## New since `v3.3.0` (2023-10-11)
|
||||
|
||||
* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
|
||||
* 6ad5aeb [new] Add `:zstd` compressor, new compressor backend
|
||||
* 9db09e1 [new] [#163] Track serialized output in tests
|
||||
* dcc6b08 [new] [#164] Update benchmarks
|
||||
* f3ff7ae [new] Add native `MapEntry` freezer
|
||||
* 37cf415 [new] [#171] Auto strip metadata protocol extensions
|
||||
* Misc internal improvements
|
||||
|
||||
## Everything since `v3.4.0-RC3` (2024-04-10)
|
||||
|
||||
* Update dependencies
|
||||
|
||||
---
|
||||
|
||||
# `v3.4.0-RC3` (2024-04-10)
|
||||
|
||||
> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0-RC3), this project uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
|
||||
This is a non-breaking **feature and maintenance** pre-release.
|
||||
Please **test carefully and report any unexpected problems**, thank you! 🙏
|
||||
|
||||
## New since `v3.3.0`
|
||||
|
||||
* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
|
||||
* 6ad5aeb [new] Add `:zstd` compressor, new compressor backend
|
||||
* 9db09e1 [new] [#163] Track serialized output in tests
|
||||
* dcc6b08 [new] [#164] Update benchmarks
|
||||
* f3ff7ae [new] Add native `MapEntry` freezer
|
||||
* 37cf415 [new] [#171] Auto strip metadata protocol extensions
|
||||
* Misc internal improvements
|
||||
|
||||
## Everything since `v3.4.0-RC2`
|
||||
|
||||
* 82a050b [mod] Don't attach empty metadata
|
||||
* 92c4a83 [fix] Broken `*final-freeze-fallback*` default val
|
||||
* 37cf415 [new] [#171] Auto strip metadata protocol extensions
|
||||
* Update dependencies
|
||||
|
||||
|
||||
# `v3.4.0-RC2` (2024-02-26)
|
||||
|
||||
> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0-RC2), this project uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
|
||||
This is a non-breaking **feature and maintenance** pre-release.
|
||||
Please **test carefully and report any unexpected problems**, thank you! 🙏
|
||||
|
||||
## New since `v3.3.0`
|
||||
|
||||
* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
|
||||
* 6ad5aeb [new] Add `:zstd` compressor, new compressor backend
|
||||
* 9db09e1 [new] [#163] Track serialized output in tests
|
||||
* dcc6b08 [new] [#164] Update benchmarks
|
||||
* f3ff7ae [new] Add native `MapEntry` freezer
|
||||
* Misc internal improvements
|
||||
|
||||
## Everything since `v3.4.0-RC1`
|
||||
|
||||
* cb5b7cf [fix] [#169] Can't auto-identify `:zstd` compressor when decompressing
|
||||
* cb0b871 Revert [mod] 578c585 (upstream fix now available)
|
||||
* Update dependencies
|
||||
|
||||
---
|
||||
|
||||
# `v3.4.0-RC1` (2024-02-06)
|
||||
|
||||
> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0-RC1), this project uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
|
|
|
|||
85
README.md
85
README.md
|
|
@ -1,21 +1,19 @@
|
|||
<a href="https://www.taoensso.com/clojure" title="More stuff by @ptaoussanis at www.taoensso.com"><img src="https://www.taoensso.com/open-source.png" alt="Taoensso open source" width="340"/></a>
|
||||
[**Documentation**](#documentation) | [Latest releases](#latest-releases) | [Get support][GitHub issues]
|
||||
[**API**][cljdoc] | [**Wiki**][GitHub wiki] | [Latest releases](#latest-releases) | [Get support][GitHub issues]
|
||||
|
||||
# Nippy
|
||||
|
||||
### The fastest serialization library for Clojure
|
||||
### Fast serialization library for Clojure
|
||||
|
||||
Clojure's rich data types are awesome. And its [reader](https://clojure.org/reference/reader) allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).
|
||||
|
||||
Nippy is an attempt to provide a reliable, high-performance **drop-in alternative to the reader**.
|
||||
Nippy is a mature, high-performance **drop-in alternative to the reader**.
|
||||
|
||||
Used by [Carmine](https://www.taoensso.com/carmine), [Faraday](https://www.taoensso.com/faraday), [PigPen](https://github.com/Netflix/PigPen), [Onyx](https://github.com/onyx-platform/onyx),
|
||||
[XTDB](https://github.com/xtdb/xtdb), [Datalevin](https://github.com/juji-io/datalevin), and others.
|
||||
It is used at scale by [Carmine](https://www.taoensso.com/carmine), [Faraday](https://www.taoensso.com/faraday), [PigPen](https://github.com/Netflix/PigPen), [Onyx](https://github.com/onyx-platform/onyx), [XTDB](https://github.com/xtdb/xtdb), [Datalevin](https://github.com/juji-io/datalevin), and others.
|
||||
|
||||
## Latest release/s
|
||||
|
||||
- `2023-10-11` `3.3.0` (stable): [changes](../../releases/tag/v3.3.0)
|
||||
- `2024-02-06` `3.4.0-RC1` (dev): [changes](../../releases/tag/v3.4.0-RC1)
|
||||
- `2025-04-15` `v3.5.0`: [release info](../../releases/tag/v3.5.0)
|
||||
|
||||
[![Main tests][Main tests SVG]][Main tests URL]
|
||||
[![Graal tests][Graal tests SVG]][Graal tests URL]
|
||||
|
|
@ -24,28 +22,80 @@ See [here][GitHub releases] for earlier releases.
|
|||
|
||||
## Why Nippy?
|
||||
|
||||
- Small, simple **all-Clojure** library
|
||||
- Small, simple **pure-Clojure** library
|
||||
- **Terrific performance**: the [best](#performance) for Clojure that I'm aware of
|
||||
- Comprehensive support for [all standard data types](../../wiki/1-Getting-started#deserializing)
|
||||
- Easily extendable to [custom data types](../../wiki/1-Getting-started#custom-types)
|
||||
- **Robust test suite**, incl. full coverage for every supported type
|
||||
- Auto fallback to [Java Serializable](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*freeze-serializable-allowlist*) when available
|
||||
- Auto fallback to Clojure Reader for all other types (including tagged literals)
|
||||
- Pluggable **compression** with built-in [LZ4](https://code.google.com/p/lz4/), [Zstandard](https://facebook.github.io/zstd/), etc.
|
||||
- Pluggable [encryption](../../wiki/1-Getting-started#encryption) with built-in AES128
|
||||
- **Robust test suite** incl. coverage of every supported type
|
||||
- **Mature** and widely used in production for 12+ years
|
||||
- Optional auto fallback to [Java Serializable](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*freeze-serializable-allowlist*) for [safe](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*freeze-serializable-allowlist*) types
|
||||
- Optional auto fallback to Clojure Reader (including tagged literals)
|
||||
- Optional smart **compression** with [LZ4](https://code.google.com/p/lz4/) or [Zstandard](https://facebook.github.io/zstd/)
|
||||
- Optional [encryption](../../wiki/1-Getting-started#encryption) with AES128
|
||||
- [Tools](https://taoensso.github.io/nippy/taoensso.nippy.tools.html) for easy + robust **integration into 3rd-party libraries**, etc.
|
||||
- Powerful [thaw transducer](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*thaw-xform*) for flexible data inspection and transformation
|
||||
|
||||
## Quick example
|
||||
|
||||
Nippy's super easy to use:
|
||||
|
||||
```clojure
|
||||
(require '[taoensso.nippy :as nippy])
|
||||
|
||||
;; Freeze any Clojure value
|
||||
(nippy/freeze <my-value>) ; => Serialized byte[]
|
||||
|
||||
;; Thaw the byte[] to get back the original value:
|
||||
(nippy/thaw (nippy/freeze <my-value>)) ; => <my-value>
|
||||
```
|
||||
|
||||
See the [wiki](https://github.com/taoensso/nippy/wiki/1-Getting-started#deserializing) for more.
|
||||
|
||||
## Operational considerations
|
||||
|
||||
### Data longevity
|
||||
|
||||
Nippy is widely used to store **long-lived** data and promises (as always) that **data serialized today should be readable by all future versions of Nippy**.
|
||||
|
||||
But please note that the **converse is not generally true**:
|
||||
|
||||
- Nippy `vX` **should** be able to read all data from Nippy `vY<=X` (backwards compatibility)
|
||||
- Nippy `vX` **may/not** be able to read all data from Nippy `vY>X` (forwards compatibility)
|
||||
|
||||
### Rolling updates and rollback
|
||||
|
||||
From time to time, Nippy may introduce:
|
||||
|
||||
- Support for serializing **new types**
|
||||
- Optimizations to the serialization of **pre-existing types**
|
||||
|
||||
To help ease **rolling updates** and to better support **rollback**, Nippy (since version v3.4) will always introduce such changes over **two version releases**:
|
||||
|
||||
- Release 1: to add **read support** for the new types
|
||||
- Release 2: to add **write support** for the new types
|
||||
|
||||
Starting from v3.4, Nippy's release notes will **always clearly indicate** if a particular update sequence is recommended.
|
||||
|
||||
### Stability of byte output
|
||||
|
||||
It has **never been an objective** of Nippy to offer **predictable byte output**, and I'd generally **recommend against** depending on specific byte output.
|
||||
|
||||
However, I know that a small minority of users *do* have specialized needs in this area.
|
||||
|
||||
So starting with Nippy v3.4, Nippy's release notes will **always clearly indicate** if any changes to byte output are expected.
|
||||
|
||||
## Performance
|
||||
|
||||
Since its earliest versions, Nippy has consistently been the **fastest serialization library for Clojure** that I'm aware of. Latest [benchmark](../../blob/master/test/taoensso/nippy_benchmarks.clj) results:
|
||||
Nippy is fast! Latest [benchmark](../../blob/master/test/taoensso/nippy_benchmarks.clj) results:
|
||||
|
||||
|
||||
|
||||
PRs welcome to include other alternatives in the bench suite!
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Wiki][GitHub wiki] (getting started, usage, etc.)
|
||||
- API reference: [Codox][Codox docs], [clj-doc][clj-doc docs]
|
||||
- API reference via [cljdoc][cljdoc]
|
||||
|
||||
## Funding
|
||||
|
||||
|
|
@ -53,7 +103,7 @@ You can [help support][sponsor] continued work on this project, thank you!! 🙏
|
|||
|
||||
## License
|
||||
|
||||
Copyright © 2012-2024 [Peter Taoussanis][].
|
||||
Copyright © 2012-2025 [Peter Taoussanis][].
|
||||
Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).
|
||||
|
||||
<!-- Common -->
|
||||
|
|
@ -67,8 +117,7 @@ Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).
|
|||
|
||||
<!-- Project -->
|
||||
|
||||
[Codox docs]: https://taoensso.github.io/nippy/
|
||||
[clj-doc docs]: https://cljdoc.org/d/com.taoensso/nippy/
|
||||
[cljdoc]: https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy
|
||||
|
||||
[Clojars SVG]: https://img.shields.io/clojars/v/com.taoensso/nippy.svg
|
||||
[Clojars URL]: https://clojars.org/com.taoensso/nippy
|
||||
|
|
|
|||
13
SECURITY.md
Normal file
13
SECURITY.md
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
# Security policy
|
||||
|
||||
## Advisories
|
||||
|
||||
All security advisories will be posted [on GitHub](https://github.com/taoensso/nippy/security/advisories).
|
||||
|
||||
## Reporting a vulnerability
|
||||
|
||||
Please report possible security vulnerabilities [via GitHub](https://github.com/taoensso/nippy/security/advisories), or by emailing me at `my first name at taoensso.com`. You may encrypt emails with [my public PGP/GPG key](https://www.taoensso.com/pgp).
|
||||
|
||||
Thank you!
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
67
project.clj
67
project.clj
|
|
@ -1,32 +1,45 @@
|
|||
(defproject com.taoensso/nippy "3.4.0-RC1"
|
||||
(defproject com.taoensso/nippy "3.5.0"
|
||||
:author "Peter Taoussanis <https://www.taoensso.com>"
|
||||
:description "The fastest serialization library for Clojure"
|
||||
:url "https://github.com/taoensso/nippy"
|
||||
:description "Fast serialization library for Clojure"
|
||||
:url "https://www.taoensso.com/nippy"
|
||||
|
||||
:license
|
||||
{:name "Eclipse Public License - v 1.0"
|
||||
:url "https://www.eclipse.org/legal/epl-v10.html"}
|
||||
|
||||
:dependencies
|
||||
[[org.clojure/tools.reader "1.3.7"]
|
||||
[com.taoensso/encore "3.77.0"]
|
||||
[org.tukaani/xz "1.9"]
|
||||
[io.airlift/aircompressor "0.25"]]
|
||||
[[org.clojure/tools.reader "1.5.2"]
|
||||
[com.taoensso/encore "3.142.0"]
|
||||
[org.tukaani/xz "1.10"]
|
||||
[io.airlift/aircompressor "2.0.2"]]
|
||||
|
||||
:test-paths ["test" #_"src"]
|
||||
|
||||
:profiles
|
||||
{;; :default [:base :system :user :provided :dev]
|
||||
:provided {:dependencies [[org.clojure/clojure "1.11.1"]]}
|
||||
:c1.11 {:dependencies [[org.clojure/clojure "1.11.1"]]}
|
||||
:c1.10 {:dependencies [[org.clojure/clojure "1.10.1"]]}
|
||||
:c1.9 {:dependencies [[org.clojure/clojure "1.9.0"]]}
|
||||
:provided {:dependencies [[org.clojure/clojure "1.11.4"]]}
|
||||
:c1.12 {:dependencies [[org.clojure/clojure "1.12.0"]]}
|
||||
:c1.11 {:dependencies [[org.clojure/clojure "1.11.4"]]}
|
||||
:c1.10 {:dependencies [[org.clojure/clojure "1.10.3"]]}
|
||||
|
||||
:test
|
||||
:graal-tests
|
||||
{:source-paths ["test"]
|
||||
:main taoensso.graal-tests
|
||||
:aot [taoensso.graal-tests]
|
||||
:uberjar-name "graal-tests.jar"
|
||||
:dependencies
|
||||
[[org.clojure/clojure "1.11.3"]
|
||||
[com.github.clj-easy/graal-build-time "1.0.5"]]}
|
||||
|
||||
:dev
|
||||
{:jvm-opts
|
||||
["-server"
|
||||
"-Xms1024m" "-Xmx2048m"
|
||||
"-Dtaoensso.elide-deprecated=true"
|
||||
"-Dtaoensso.nippy.thaw-serializable-allowlist-base=base.1, base.2"
|
||||
"-Dtaoensso.nippy.thaw-serializable-allowlist-add=add.1 , add.2"]
|
||||
"-Dtaoensso.nippy.thaw-serializable-allowlist-add=add.1 , add.2"
|
||||
#_"-Dtaoensso.nippy.target-release=320"
|
||||
#_"-Dtaoensso.nippy.target-release=350"]
|
||||
|
||||
:global-vars
|
||||
{*warn-on-reflection* true
|
||||
|
|
@ -35,35 +48,17 @@
|
|||
|
||||
:dependencies
|
||||
[[org.clojure/test.check "1.1.1"]
|
||||
[org.clojure/data.fressian "1.0.0"]]}
|
||||
[org.clojure/data.fressian "1.1.0"]]
|
||||
|
||||
:graal-tests
|
||||
{:source-paths ["test"]
|
||||
:main taoensso.graal-tests
|
||||
:aot [taoensso.graal-tests]
|
||||
:uberjar-name "graal-tests.jar"
|
||||
:dependencies
|
||||
[[org.clojure/clojure "1.11.1"]
|
||||
[com.github.clj-easy/graal-build-time "1.0.5"]]}
|
||||
|
||||
:dev [:c1.11 :test :dev+]
|
||||
:dev+
|
||||
{:plugins
|
||||
:plugins
|
||||
[[lein-pprint "1.3.2"]
|
||||
[lein-ancient "0.7.0"]
|
||||
[com.taoensso.forks/lein-codox "0.10.10"]]
|
||||
|
||||
:codox
|
||||
{:language #{:clojure #_:clojurescript}
|
||||
:base-language :clojure}}}
|
||||
|
||||
:test-paths ["test" #_"src"]
|
||||
[lein-ancient "0.7.0"]]}}
|
||||
|
||||
:aliases
|
||||
{"start-dev" ["with-profile" "+dev" "repl" ":headless"]
|
||||
;; "build-once" ["do" ["clean"] ["cljsbuild" "once"]]
|
||||
"deploy-lib" ["do" #_["build-once"] ["deploy" "clojars"] ["install"]]
|
||||
|
||||
"test-clj" ["with-profile" "+c1.11:+c1.10:+c1.9" "test"]
|
||||
;; "test-cljs" ["with-profile" "+test" "cljsbuild" "test"]
|
||||
"test-clj" ["with-profile" "+c1.12:+c1.11:+c1.10" "test"]
|
||||
;; "test-cljs" ["with-profile" "+c1.12" "cljsbuild" "test"]
|
||||
"test-all" ["do" ["clean"] ["test-clj"] #_["test-cljs"]]})
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@
|
|||
(:require
|
||||
[clojure.string :as str]
|
||||
[clojure.java.io :as jio]
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc]
|
||||
[taoensso.nippy
|
||||
[impl :as impl]
|
||||
|
|
@ -25,7 +26,7 @@
|
|||
PersistentQueue PersistentTreeMap PersistentTreeSet PersistentList
|
||||
MapEntry LazySeq IRecord ISeq IType]))
|
||||
|
||||
(enc/assert-min-encore-version [3 77 0])
|
||||
(enc/assert-min-encore-version [3 142 0])
|
||||
|
||||
(comment
|
||||
(set! *unchecked-math* :warn-on-boxed)
|
||||
|
|
@ -145,11 +146,6 @@
|
|||
51 [:reader-md [[:bytes {:read 2}]]]
|
||||
52 [:reader-lg [[:bytes {:read 4}]]]
|
||||
|
||||
53 [:bytes-0 []]
|
||||
7 [:bytes-sm [[:bytes {:read 1}]]]
|
||||
15 [:bytes-md [[:bytes {:read 2}]]]
|
||||
2 [:bytes-lg [[:bytes {:read 4}]]]
|
||||
|
||||
17 [:vec-0 []]
|
||||
113 [:vec-2 [[:elements 2]]]
|
||||
114 [:vec-3 [[:elements 3]]]
|
||||
|
|
@ -182,7 +178,6 @@
|
|||
28 [:sorted-set-lg [[:elements {:read 4}]]]
|
||||
31 [:sorted-map-lg [[:elements {:read 4 :multiplier 2}]]]
|
||||
26 [:queue-lg [[:elements {:read 4}]]]
|
||||
115 [:objects-lg [[:elements {:read 4}]]]
|
||||
|
||||
25 [:meta [[:elements 1]]]
|
||||
58 [:regex [[:elements 1]]]
|
||||
|
|
@ -196,6 +191,21 @@
|
|||
70 [:ratio [[:bytes {:read 4}]
|
||||
[:bytes {:read 4}]]]
|
||||
|
||||
;; Arrays
|
||||
53 [:byte-array-0 []]
|
||||
7 [:byte-array-sm [[:elements {:read 1}]]]
|
||||
15 [:byte-array-md [[:elements {:read 2}]]]
|
||||
2 [:byte-array-lg [[:elements {:read 4}]]]
|
||||
|
||||
109 [:int-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
|
||||
108 [:long-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
|
||||
|
||||
117 [:float-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
|
||||
116 [:double-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
|
||||
|
||||
107 [:string-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
|
||||
115 [:object-array-lg [[:elements {:read 4}]]]
|
||||
|
||||
;; Serializable
|
||||
75 [:sz-quarantined-sm [[:bytes {:read 1}] [:elements 1]]]
|
||||
76 [:sz-quarantined-md [[:bytes {:read 2}] [:elements 1]]]
|
||||
|
|
@ -203,6 +213,8 @@
|
|||
48 [:record-sm [[:bytes {:read 1}] [:elements 1]]]
|
||||
49 [:record-md [[:bytes {:read 2}] [:elements 1]]]
|
||||
|
||||
104 [:meta-protocol-key []]
|
||||
|
||||
;; Necessarily without size information
|
||||
81 [:type nil]
|
||||
82 [:prefixed-custom-md nil]
|
||||
|
|
@ -298,14 +310,14 @@
|
|||
- `payload-spec` examples:
|
||||
- nil ; No spec available (e.g. unpredictable payload)
|
||||
- [] ; Type has no payload
|
||||
- [[:bytes 4]] ; Type has a payload of exactly 4 bytes
|
||||
- [[:bytes 2] [:elements 2]] ; Type has a payload of exactly 2 bytes, then
|
||||
; 2 elements
|
||||
- [[:bytes 4]] ; Type has payload of exactly 4 bytes
|
||||
- [[:bytes 2] [:elements 2]] ; Type has payload of exactly 2 bytes,
|
||||
; followed by 2 elements
|
||||
|
||||
- [[:bytes {:read 2}]
|
||||
[:elements {:read 4 :multiplier 2 :unsigned? true}]]
|
||||
|
||||
; Type has payload of <short-count> bytes, then
|
||||
; Type has payload of <short-count> bytes, followed by
|
||||
; <unsigned-int-count>*2 (multiplier) elements
|
||||
|
||||
Note that `payload-spec` can be handy for skipping over items in
|
||||
|
|
@ -337,7 +349,7 @@
|
|||
compression/lz4-compressor
|
||||
compression/lz4hc-compressor
|
||||
#_compression/lzo-compressor
|
||||
#_compression/snappy-compressor ; Can be unsafe
|
||||
compression/snappy-compressor
|
||||
compression/lzma2-compressor
|
||||
|
||||
encryption/encrypt
|
||||
|
|
@ -352,27 +364,28 @@
|
|||
;; See also `nippy.tools` ns for further dynamic config support
|
||||
|
||||
;; For back compatibility (incl. Timbre's Carmine appender)
|
||||
(enc/defonce ^:dynamic ^:no-doc ^:deprecated *final-freeze-fallback* "Prefer `*freeze-fallback`.")
|
||||
(enc/defonce ^:dynamic ^:no-doc ^:deprecated *final-freeze-fallback* "Prefer `*freeze-fallback`." nil)
|
||||
(enc/defonce ^:dynamic *freeze-fallback*
|
||||
"Controls Nippy's behaviour when trying to freeze an item for which Nippy
|
||||
doesn't currently have a native freeze/thaw implementation.
|
||||
"Controls Nippy's behaviour when trying to freeze an object with a type for
|
||||
which Nippy doesn't currently have a native (protocol) implementation.
|
||||
|
||||
Possible values:
|
||||
|
||||
1. `nil` (no freeze-fallback, default)
|
||||
1. `nil` (no fallback, default)
|
||||
Tries the following in order:
|
||||
- Freeze with Java's `Serializable` interface if possible
|
||||
- Freeze with Clojure's reader if possible
|
||||
- Freeze with Java's `Serializable` interface if this seems possible
|
||||
- Freeze with Clojure's reader if this seems possible
|
||||
- Throw
|
||||
|
||||
2. `:write-unfreezable` keyword
|
||||
Tries the following in order:
|
||||
- Freeze with Java's `Serializable` interface if possible
|
||||
- Freeze with Clojure's reader if possible
|
||||
- Freeze with Java's `Serializable` interface if this seems possible
|
||||
- Freeze with Clojure's reader if this seems possible
|
||||
- Freeze a {:nippy/unfreezable {:type _}} placeholder value
|
||||
|
||||
3. [Advanced] Custom (fn [^java.io.DataOutput out item]) that must
|
||||
write exactly one value to the given `DataOutput` stream"
|
||||
3. [Advanced] Custom (fn [^java.io.DataOutput out obj]) that must
|
||||
write an appropriate object type id and payload to the given
|
||||
`DataOutput` stream."
|
||||
|
||||
nil)
|
||||
|
||||
|
|
@ -429,16 +442,16 @@
|
|||
;; Unfortunately quite a bit of complexity to do this safely
|
||||
|
||||
(def default-freeze-serializable-allowlist
|
||||
"Allows *any* class-name to be frozen using Java's `Serializable` interface.
|
||||
"Allows *any* class name to be frozen using Java's `Serializable` interface.
|
||||
This is generally safe since RCE risk is present only when thawing.
|
||||
See also `*freeze-serializable-allowlist*`."
|
||||
#{"*"})
|
||||
|
||||
(def default-thaw-serializable-allowlist
|
||||
"A set of common safe class-names to allow to be frozen using Java's
|
||||
"A set of common safe class names to allow to be frozen using Java's
|
||||
`Serializable` interface. PRs welcome for additions.
|
||||
See also `*thaw-serializable-allowlist*`."
|
||||
#{"[I" "[F" "[Z" "[B" "[C" "[D" "[S" "[J"
|
||||
#{"[Z" "[B" "[S" "[I" "[J" "[F" "[D" "[C" "[Ljava.lang.String;"
|
||||
|
||||
"java.lang.Throwable"
|
||||
"java.lang.Exception"
|
||||
|
|
@ -502,10 +515,10 @@
|
|||
|
||||
Example allowlist values:
|
||||
- `(fn allow-class? [class-name] true)` ; Arbitrary predicate fn
|
||||
- `#{\"java.lang.Throwable\", \"clojure.lang.*\"}` ; Set of class-names
|
||||
- `#{\"java.lang.Throwable\", \"clojure.lang.*\"}` ; Set of class names
|
||||
- `\"allow-and-record\"` ; Special value, see [2]
|
||||
|
||||
Note that class-names in sets may contain \"*\" wildcards.
|
||||
Note that class names in sets may contain \"*\" wildcards.
|
||||
|
||||
Default allowlist values are:
|
||||
- default-freeze-serializable-allowlist ; `{\"*\"}` => allow any class
|
||||
|
|
@ -589,13 +602,15 @@
|
|||
|
||||
(defmacro write-id [out id] `(.writeByte ~out ~id))
|
||||
|
||||
(declare write-map)
|
||||
|
||||
(extend-protocol IFreezableWithMeta
|
||||
clojure.lang.IObj ; IMeta => `meta` will work, IObj => `with-meta` will work
|
||||
(-freeze-with-meta! [x ^DataOutput data-output]
|
||||
(when-let [m (when *incl-metadata?* (meta x))]
|
||||
(write-id data-output id-meta)
|
||||
(-freeze-without-meta! m data-output))
|
||||
(-freeze-without-meta! x data-output))
|
||||
(when-let [m (when *incl-metadata?* (not-empty (meta x)))]
|
||||
(write-id data-output id-meta)
|
||||
(write-map data-output m :is-metadata))
|
||||
(-freeze-without-meta! x data-output))
|
||||
|
||||
nil (-freeze-with-meta! [x data-output] (-freeze-without-meta! x data-output))
|
||||
Object (-freeze-with-meta! [x data-output] (-freeze-without-meta! x data-output)))
|
||||
|
|
@ -637,15 +652,23 @@
|
|||
(defn- write-bytes [^DataOutput out ^bytes ba]
|
||||
(let [len (alength ba)]
|
||||
(if (zero? len)
|
||||
(write-id out id-bytes-0)
|
||||
(write-id out id-byte-array-0)
|
||||
(do
|
||||
(enc/cond
|
||||
(sm-count? len) (do (write-id out id-bytes-sm) (write-sm-count out len))
|
||||
(md-count? len) (do (write-id out id-bytes-md) (write-md-count out len))
|
||||
:else (do (write-id out id-bytes-lg) (write-lg-count out len)))
|
||||
(sm-count? len) (do (write-id out id-byte-array-sm) (write-sm-count out len))
|
||||
(md-count? len) (do (write-id out id-byte-array-md) (write-md-count out len))
|
||||
:else (do (write-id out id-byte-array-lg) (write-lg-count out len)))
|
||||
|
||||
(.write out ba 0 len)))))
|
||||
|
||||
(defmacro ^:private -run! [proc coll] `(do (reduce #(~proc %2) nil ~coll) nil))
|
||||
(defmacro ^:private -run-kv! [proc m] `(do (reduce-kv #(~proc %2 %3) nil ~m) nil))
|
||||
|
||||
(defn- write-array-lg [^DataOutput out array array-len id]
|
||||
(write-id out id)
|
||||
(write-lg-count out array-len)
|
||||
(-run! (fn [in] (-freeze-with-meta! in out)) array))
|
||||
|
||||
(defn- write-biginteger [out ^BigInteger n] (write-bytes-lg out (.toByteArray n)))
|
||||
|
||||
(defn- write-str-sm* [^DataOutput out ^String s] (write-bytes-sm* out (.getBytes s StandardCharsets/UTF_8)))
|
||||
|
|
@ -658,9 +681,10 @@
|
|||
(let [ba (.getBytes s StandardCharsets/UTF_8)
|
||||
len (alength ba)]
|
||||
(enc/cond
|
||||
(sm-count?* len) (do (write-id out id-str-sm*) (write-sm-count* out len))
|
||||
(md-count? len) (do (write-id out id-str-md) (write-md-count out len))
|
||||
:else (do (write-id out id-str-lg) (write-lg-count out len)))
|
||||
(and (impl/target-release>= 330) (sm-count?* len)) (do (write-id out id-str-sm*) (write-sm-count* out len))
|
||||
(and (impl/target-release< 330) (sm-count? len)) (do (write-id out id-str-sm_) (write-sm-count out len))
|
||||
(md-count? len) (do (write-id out id-str-md) (write-md-count out len))
|
||||
:else (do (write-id out id-str-lg) (write-lg-count out len)))
|
||||
|
||||
(.write out ba 0 len))))
|
||||
|
||||
|
|
@ -672,7 +696,7 @@
|
|||
(sm-count? len) (do (write-id out id-kw-sm) (write-sm-count out len))
|
||||
(md-count? len) (do (write-id out id-kw-md) (write-md-count out len))
|
||||
;; :else (do (write-id out id-kw-lg) (write-lg-count out len)) ; Unrealistic
|
||||
:else (throw (ex-info "Keyword too long" {:name s})))
|
||||
:else (truss/ex-info! "Keyword too long" {:name s}))
|
||||
|
||||
(.write out ba 0 len)))
|
||||
|
||||
|
|
@ -684,12 +708,30 @@
|
|||
(sm-count? len) (do (write-id out id-sym-sm) (write-sm-count out len))
|
||||
(md-count? len) (do (write-id out id-sym-md) (write-md-count out len))
|
||||
;; :else (do (write-id out id-sym-lg) (write-lg-count out len)) ; Unrealistic
|
||||
:else (throw (ex-info "Symbol too long" {:name s})))
|
||||
:else (truss/ex-info! "Symbol too long" {:name s}))
|
||||
|
||||
(.write out ba 0 len)))
|
||||
|
||||
(defn- write-long-legacy [^DataOutput out ^long n]
|
||||
(enc/cond
|
||||
(zero? n) (write-id out id-long-0)
|
||||
(pos? n)
|
||||
(enc/cond
|
||||
(<= n Byte/MAX_VALUE) (do (write-id out id-long-sm_) (.writeByte out n))
|
||||
(<= n Short/MAX_VALUE) (do (write-id out id-long-md_) (.writeShort out n))
|
||||
(<= n Integer/MAX_VALUE) (do (write-id out id-long-lg_) (.writeInt out n))
|
||||
:else (do (write-id out id-long-xl) (.writeLong out n)))
|
||||
|
||||
:else
|
||||
(enc/cond
|
||||
(>= n Byte/MIN_VALUE) (do (write-id out id-long-sm_) (.writeByte out n))
|
||||
(>= n Short/MIN_VALUE) (do (write-id out id-long-md_) (.writeShort out n))
|
||||
(>= n Integer/MIN_VALUE) (do (write-id out id-long-lg_) (.writeInt out n))
|
||||
:else (do (write-id out id-long-xl) (.writeLong out n)))))
|
||||
|
||||
(defn- write-long [^DataOutput out ^long n]
|
||||
(enc/cond
|
||||
(impl/target-release< 330) (write-long-legacy out n)
|
||||
(zero? n) (write-id out id-long-0)
|
||||
(pos? n)
|
||||
(enc/cond
|
||||
|
|
@ -706,23 +748,16 @@
|
|||
(<= y range-uint) (do (write-id out id-long-neg-lg) (.writeInt out (+ y Integer/MIN_VALUE)))
|
||||
:else (do (write-id out id-long-xl) (.writeLong out n))))))
|
||||
|
||||
(defmacro ^:private -run! [proc coll] `(do (reduce #(~proc %2) nil ~coll) nil))
|
||||
(defmacro ^:private -run-kv! [proc m] `(do (reduce-kv #(~proc %2 %3) nil ~m) nil))
|
||||
|
||||
(defn- write-vec [^DataOutput out v]
|
||||
(let [cnt (count v)]
|
||||
(if (zero? cnt)
|
||||
(write-id out id-vec-0)
|
||||
(do
|
||||
(enc/cond
|
||||
(sm-count?* cnt)
|
||||
(enc/cond
|
||||
(== cnt 2) (write-id out id-vec-2)
|
||||
(== cnt 3) (write-id out id-vec-3)
|
||||
:else (do (write-id out id-vec-sm*) (write-sm-count* out cnt)))
|
||||
|
||||
(md-count? cnt) (do (write-id out id-vec-md) (write-md-count out cnt))
|
||||
:else (do (write-id out id-vec-lg) (write-lg-count out cnt)))
|
||||
(and (impl/target-release>= 330) (sm-count?* cnt)) (do (write-id out id-vec-sm*) (write-sm-count* out cnt))
|
||||
(and (impl/target-release< 330) (sm-count? cnt)) (do (write-id out id-vec-sm_) (write-sm-count out cnt))
|
||||
(md-count? cnt) (do (write-id out id-vec-md) (write-md-count out cnt))
|
||||
:else (do (write-id out id-vec-lg) (write-lg-count out cnt)))
|
||||
|
||||
(-run! (fn [in] (-freeze-with-meta! in out)) v)))))
|
||||
|
||||
|
|
@ -813,24 +848,38 @@
|
|||
(write-counted-coll out id-empty id-sm id-md id-lg coll)
|
||||
(write-uncounted-coll out id-empty id-sm id-md id-lg coll))))
|
||||
|
||||
(def ^:private ^:const meta-protocol-key ::meta-protocol-key)
|
||||
|
||||
;; Micro-optimization:
|
||||
;; As (write-kvs out id-map-0 id-map-sm id-map-md id-map-lg x)
|
||||
(defn- write-map [^DataOutput out m]
|
||||
(defn- write-map [^DataOutput out m is-metadata?]
|
||||
(let [cnt (count m)]
|
||||
(if (zero? cnt)
|
||||
(write-id out id-map-0)
|
||||
(do
|
||||
(enc/cond
|
||||
(sm-count?* cnt) (do (write-id out id-map-sm*) (write-sm-count* out cnt))
|
||||
(md-count? cnt) (do (write-id out id-map-md) (write-md-count out cnt))
|
||||
:else (do (write-id out id-map-lg) (write-lg-count out cnt)))
|
||||
(and (impl/target-release>= 330) (sm-count?* cnt)) (do (write-id out id-map-sm*) (write-sm-count* out cnt))
|
||||
(and (impl/target-release< 330) (sm-count? cnt)) (do (write-id out id-map-sm_) (write-sm-count out cnt))
|
||||
(md-count? cnt) (do (write-id out id-map-md) (write-md-count out cnt))
|
||||
:else (do (write-id out id-map-lg) (write-lg-count out cnt)))
|
||||
|
||||
(-run-kv!
|
||||
(fn [k v]
|
||||
(-freeze-with-meta! k out)
|
||||
(-freeze-with-meta! v out))
|
||||
(if (and is-metadata? (fn? v) (qualified-symbol? k))
|
||||
(do
|
||||
;; Strip Clojure v1.10+ metadata protocol extensions
|
||||
;; (used by defprotocol `:extend-via-metadata`)
|
||||
(if (impl/target-release>= 340)
|
||||
(write-id out id-meta-protocol-key)
|
||||
(-freeze-without-meta! meta-protocol-key out))
|
||||
(write-id out id-nil))
|
||||
(do
|
||||
(-freeze-with-meta! k out)
|
||||
(-freeze-with-meta! v out))))
|
||||
m)))))
|
||||
|
||||
(comment (meta (thaw (freeze (with-meta [] {:a :A, 'b/c (fn [])})))))
|
||||
|
||||
;; Micro-optimization:
|
||||
;; As (write-counted-coll out id-set-0 id-set-sm id-set-md id-set-lg x)
|
||||
(defn- write-set [^DataOutput out s]
|
||||
|
|
@ -839,63 +888,52 @@
|
|||
(write-id out id-set-0)
|
||||
(do
|
||||
(enc/cond
|
||||
(sm-count?* cnt) (do (write-id out id-set-sm*) (write-sm-count* out cnt))
|
||||
(md-count? cnt) (do (write-id out id-set-md) (write-md-count out cnt))
|
||||
:else (do (write-id out id-set-lg) (write-lg-count out cnt)))
|
||||
(and (impl/target-release>= 330) (sm-count?* cnt)) (do (write-id out id-set-sm*) (write-sm-count* out cnt))
|
||||
(and (impl/target-release< 330) (sm-count? cnt)) (do (write-id out id-set-sm_) (write-sm-count out cnt))
|
||||
(md-count? cnt) (do (write-id out id-set-md) (write-md-count out cnt))
|
||||
:else (do (write-id out id-set-lg) (write-lg-count out cnt)))
|
||||
|
||||
(-run! (fn [in] (-freeze-with-meta! in out)) s)))))
|
||||
|
||||
(defn- write-objects [^DataOutput out ^objects ary]
|
||||
(let [len (alength ary)]
|
||||
(write-id out id-objects-lg)
|
||||
(write-lg-count out len)
|
||||
(-run! (fn [in] (-freeze-with-meta! in out)) ary)))
|
||||
|
||||
(defn- write-serializable [^DataOutput out x ^String class-name]
|
||||
(defn- write-serializable [^DataOutput out x]
|
||||
(when-debug (println (str "write-serializable: " (type x))))
|
||||
(let [class-name-ba (.getBytes class-name StandardCharsets/UTF_8)
|
||||
len (alength class-name-ba)]
|
||||
(when (and (instance? Serializable x) (not (fn? x)))
|
||||
(let [class-name (.getName (class x))] ; Reflect
|
||||
(when (freeze-serializable-allowed? class-name)
|
||||
(let [class-name-ba (.getBytes class-name StandardCharsets/UTF_8)
|
||||
len (alength class-name-ba)]
|
||||
|
||||
(enc/cond
|
||||
(sm-count? len) (do (write-id out id-sz-quarantined-sm) (write-bytes-sm out class-name-ba))
|
||||
(md-count? len) (do (write-id out id-sz-quarantined-md) (write-bytes-md out class-name-ba))
|
||||
;; :else (do (write-id out id-sz-quarantined-lg) (write-bytes-md out class-name-ba)) ; Unrealistic
|
||||
:else (throw (ex-info "Serializable class name too long" {:name class-name})))
|
||||
(enc/cond
|
||||
(sm-count? len) (do (write-id out id-sz-quarantined-sm) (write-bytes-sm out class-name-ba))
|
||||
(md-count? len) (do (write-id out id-sz-quarantined-md) (write-bytes-md out class-name-ba))
|
||||
;; :else (do (write-id out id-sz-quarantined-lg) (write-bytes-md out class-name-ba)) ; Unrealistic
|
||||
:else (truss/ex-info! "Serializable class name too long" {:name class-name}))
|
||||
|
||||
;; Legacy: write object directly to out.
|
||||
;; (.writeObject (ObjectOutputStream. out) x)
|
||||
;; Legacy: write object directly to out.
|
||||
;; (.writeObject (ObjectOutputStream. out) x)
|
||||
|
||||
;; Quarantined: write object to ba, then ba to out.
|
||||
;; We'll have object length during thaw, allowing us to skip readObject.
|
||||
(let [quarantined-ba (ByteArrayOutputStream.)]
|
||||
(.writeObject (ObjectOutputStream. (DataOutputStream. quarantined-ba)) x)
|
||||
(write-bytes out (.toByteArray quarantined-ba)))))
|
||||
;; Quarantined: write object to ba, then ba to out.
|
||||
;; We'll have object length during thaw, allowing us to skip readObject.
|
||||
(let [quarantined-ba (ByteArrayOutputStream.)]
|
||||
(.writeObject (ObjectOutputStream. (DataOutputStream. quarantined-ba)) x)
|
||||
(write-bytes out (.toByteArray quarantined-ba)))
|
||||
|
||||
true)))))
|
||||
|
||||
(defn- write-readable [^DataOutput out x]
|
||||
(when-debug (println (str "write-readable: " (type x))))
|
||||
(let [edn (enc/pr-edn x)
|
||||
edn-ba (.getBytes ^String edn StandardCharsets/UTF_8)
|
||||
len (alength edn-ba)]
|
||||
(enc/cond
|
||||
(sm-count? len) (do (write-id out id-reader-sm) (write-bytes-sm out edn-ba))
|
||||
(md-count? len) (do (write-id out id-reader-md) (write-bytes-md out edn-ba))
|
||||
:else (do (write-id out id-reader-lg) (write-bytes-lg out edn-ba)))))
|
||||
|
||||
(defn try-write-serializable [out x]
|
||||
(when (and (instance? Serializable x) (not (fn? x)))
|
||||
(try
|
||||
(let [class-name (.getName (class x))] ; Reflect
|
||||
(when (freeze-serializable-allowed? class-name)
|
||||
(write-serializable out x class-name)
|
||||
true))
|
||||
(catch Throwable _ nil))))
|
||||
|
||||
(defn try-write-readable [out x]
|
||||
(when (impl/seems-readable? x)
|
||||
(try
|
||||
(write-readable out x)
|
||||
true
|
||||
(catch Throwable _ nil))))
|
||||
(let [edn (enc/pr-edn x)
|
||||
edn-ba (.getBytes ^String edn StandardCharsets/UTF_8)
|
||||
len (alength edn-ba)]
|
||||
(enc/cond
|
||||
(sm-count? len) (do (write-id out id-reader-sm) (write-bytes-sm out edn-ba))
|
||||
(md-count? len) (do (write-id out id-reader-md) (write-bytes-md out edn-ba))
|
||||
:else (do (write-id out id-reader-lg) (write-bytes-lg out edn-ba)))
|
||||
true)))
|
||||
|
||||
(defn ^:deprecated try-write-serializable [out x] (truss/catching :all (write-serializable out x)))
|
||||
(defn ^:deprecated try-write-readable [out x] (truss/catching :all (write-readable out x)))
|
||||
|
||||
(defn- try-pr-edn [x]
|
||||
(try
|
||||
|
|
@ -913,29 +951,22 @@
|
|||
:content (try-pr-edn x)}}
|
||||
out))
|
||||
|
||||
(defn throw-unfreezable [x]
|
||||
(let [t (type x)]
|
||||
(throw
|
||||
(ex-info (str "Unfreezable type: " t)
|
||||
{:type t
|
||||
:as-str (try-pr-edn x)}))))
|
||||
|
||||
;; Public `-freeze-with-meta!` with different arg order
|
||||
(defn freeze-to-out!
|
||||
"Serializes arg (any Clojure data type) to a DataOutput.
|
||||
This is a low-level util: in most cases you'll want `freeze` instead."
|
||||
[^DataOutput data-output x] (-freeze-with-meta! x data-output))
|
||||
|
||||
;;;; Caching ; Experimental
|
||||
;;;; Caching
|
||||
|
||||
;; Nb: don't use an auto initialValue; can cause thread-local state to
|
||||
;; accidentally hang around with the use of `freeze-to-out!`, etc.
|
||||
;; Safer to require explicit activation through `with-cache`.
|
||||
(def ^ThreadLocal -cache-proxy
|
||||
"{[<x> <meta>] <idx>} for freezing, {<idx> <x-with-meta>} for thawing."
|
||||
;; Nb: don't use an auto initialValue; can cause thread-local state to
|
||||
;; accidentally hang around with the use of `freeze-to-out!`, etc.
|
||||
;; Safer to require explicit activation through `with-cache`.
|
||||
(proxy [ThreadLocal] []))
|
||||
|
||||
(defmacro ^:private with-cache
|
||||
(defmacro with-cache
|
||||
"Executes body with support for freezing/thawing cached values.
|
||||
|
||||
This is a low-level util: you won't need to use this yourself unless
|
||||
|
|
@ -950,9 +981,7 @@
|
|||
|
||||
(deftype Cached [val])
|
||||
(defn cache
|
||||
"Experimental, subject to change. Feedback welcome!
|
||||
|
||||
Wraps value so that future writes of the same wrapped value with same
|
||||
"Wraps value so that future writes of the same wrapped value with same
|
||||
metadata will be efficiently encoded as references to this one.
|
||||
|
||||
(freeze [(cache \"foo\") (cache \"foo\") (cache \"foo\")])
|
||||
|
|
@ -1000,7 +1029,7 @@
|
|||
(when first-occurance? (-freeze-with-meta! x-val out)))
|
||||
|
||||
:else
|
||||
;; (throw (ex-info "Max cache size exceeded" {:idx idx}))
|
||||
;; (truss/ex-info! "Max cache size exceeded" {:idx idx})
|
||||
(-freeze-with-meta! x-val out) ; Just freeze uncached
|
||||
))
|
||||
|
||||
|
|
@ -1017,7 +1046,7 @@
|
|||
(vswap! cache_ assoc idx x)
|
||||
x)
|
||||
v))
|
||||
(throw (ex-info "No cache_ established, can't thaw. See `with-cache`." {}))))))
|
||||
(truss/ex-info! "Can't thaw without cache available. See `with-cache`." {})))))
|
||||
|
||||
(comment
|
||||
(thaw (freeze [(cache "foo") (cache "foo") (cache "foo")]))
|
||||
|
|
@ -1062,8 +1091,6 @@
|
|||
(.writeLong out (.getLeastSignificantBits x))))
|
||||
|
||||
(freezer Boolean nil true (if (boolean x) (write-id out id-true) (write-id out id-false)))
|
||||
(freezer (Class/forName "[B") nil true (write-bytes out x))
|
||||
(freezer (Class/forName "[Ljava.lang.Object;") nil true (write-objects out x))
|
||||
(freezer String nil true (write-str out x))
|
||||
(freezer Keyword nil true (write-kw out x))
|
||||
(freezer Symbol nil true (write-sym out x))
|
||||
|
|
@ -1073,12 +1100,23 @@
|
|||
(do (write-id out id-double-0))
|
||||
(do (write-id out id-double) (.writeDouble out x))))
|
||||
|
||||
;; Arrays
|
||||
(freezer (Class/forName "[B") nil true (write-bytes out x))
|
||||
(freezer (Class/forName "[Ljava.lang.Object;") nil true (write-array-lg out x (alength ^"[Ljava.lang.Object;" x) id-object-array-lg))
|
||||
|
||||
(when (impl/target-release>= 350)
|
||||
(freezer (Class/forName "[I") nil true (write-array-lg out x (alength ^"[I" x) id-int-array-lg))
|
||||
(freezer (Class/forName "[J") nil true (write-array-lg out x (alength ^"[J" x) id-long-array-lg))
|
||||
(freezer (Class/forName "[F") nil true (write-array-lg out x (alength ^"[F" x) id-float-array-lg))
|
||||
(freezer (Class/forName "[D") nil true (write-array-lg out x (alength ^"[D" x) id-double-array-lg))
|
||||
(freezer (Class/forName "[Ljava.lang.String;") nil true (write-array-lg out x (alength ^"[Ljava.lang.String;" x) id-string-array-lg)))
|
||||
|
||||
(freezer PersistentQueue nil true (write-counted-coll out id-queue-lg x))
|
||||
(freezer PersistentTreeSet nil true (write-counted-coll out id-sorted-set-lg x))
|
||||
(freezer PersistentTreeMap nil true (write-kvs out id-sorted-map-lg x))
|
||||
(freezer APersistentVector nil true (write-vec out x))
|
||||
(freezer APersistentSet nil true (write-set out x))
|
||||
(freezer APersistentMap nil true (write-map out x))
|
||||
(freezer APersistentMap nil true (write-map out x false))
|
||||
(freezer PersistentList nil true (write-counted-coll out id-list-0 id-list-sm id-list-md id-list-lg x))
|
||||
(freezer LazySeq nil true (write-uncounted-coll out id-seq-0 id-seq-sm id-seq-md id-seq-lg x))
|
||||
(freezer ISeq nil true (write-coll out id-seq-0 id-seq-sm id-seq-md id-seq-lg x))
|
||||
|
|
@ -1090,26 +1128,29 @@
|
|||
(sm-count? len) (do (write-id out id-record-sm) (write-bytes-sm out class-name-ba))
|
||||
(md-count? len) (do (write-id out id-record-md) (write-bytes-md out class-name-ba))
|
||||
;; :else (do (write-id out id-record-lg) (write-bytes-md out class-name-ba)) ; Unrealistic
|
||||
:else (throw (ex-info "Record class name too long" {:name class-name})))
|
||||
:else (truss/ex-info! "Record class name too long" {:name class-name}))
|
||||
|
||||
(-freeze-without-meta! (into {} x) out)))
|
||||
|
||||
(let [munge-cached (enc/fmemoize munge)]
|
||||
(let [munged-name (enc/fmemoize #(munge (name %)))
|
||||
get-basis
|
||||
(do #_enc/fmemoize ; Small perf benefit not worth the loss of dynamism
|
||||
(fn [^java.lang.Class aclass]
|
||||
(let [basis-method (.getMethod aclass "getBasis" nil)]
|
||||
(.invoke basis-method nil nil))))]
|
||||
|
||||
(freezer IType nil true
|
||||
(let [aclass (class x)
|
||||
class-name (.getName aclass)]
|
||||
(write-id out id-type)
|
||||
(write-str out class-name)
|
||||
;; Could cache basis generation for given class-name with generalized
|
||||
;; `-cache-proxy` or something like it, but probably not worth the extra complexity.
|
||||
(let [basis-method (.getMethod aclass "getBasis" nil)
|
||||
basis (.invoke basis-method nil nil)]
|
||||
(-run!
|
||||
(fn [b]
|
||||
(let [^Field cfield (.getField aclass (munge-cached (name b)))]
|
||||
(let [fvalue (.get cfield x)]
|
||||
(-freeze-without-meta! fvalue out))))
|
||||
basis)))))
|
||||
(-run!
|
||||
(fn [b]
|
||||
(let [^Field cfield (.getField aclass (munged-name b))]
|
||||
(-freeze-without-meta! (.get cfield x) out)))
|
||||
(get-basis aclass)))))
|
||||
|
||||
(comment (do (deftype T1 [x]) (.invoke (.getMethod (class (T1. :x)) "getBasis" nil) nil nil)))
|
||||
|
||||
(enc/compile-if java.time.Instant
|
||||
(freezer java.time.Instant id-time-instant true
|
||||
|
|
@ -1142,13 +1183,20 @@
|
|||
(write-unfreezable out x)))
|
||||
|
||||
;; Without ff
|
||||
(or
|
||||
(try-write-serializable out x)
|
||||
(try-write-readable out x)
|
||||
(enc/cond
|
||||
:let [[r1 e1] (try [(write-serializable out x)] (catch Throwable t [nil t]))], r1 r1
|
||||
:let [[r2 e2] (try [(write-readable out x)] (catch Throwable t [nil t]))], r2 r2
|
||||
|
||||
(when-let [fff *final-freeze-fallback*] (fff out x) true) ; Deprecated
|
||||
|
||||
(throw-unfreezable x)))))
|
||||
:if-let [fff *final-freeze-fallback*] (fff out x) ; Deprecated
|
||||
:else
|
||||
(let [t (type x)]
|
||||
(truss/ex-info! (str "Failed to freeze type: " t)
|
||||
(enc/assoc-some
|
||||
{:type t
|
||||
:as-str (try-pr-edn x)}
|
||||
{:serializable-error e1
|
||||
:readable-error e2})
|
||||
(or e1 e2)))))))
|
||||
|
||||
;;;;
|
||||
|
||||
|
|
@ -1162,9 +1210,8 @@
|
|||
(defn- wrap-header [data-ba head-meta]
|
||||
(if-let [head-ba (get-head-ba head-meta)]
|
||||
(enc/ba-concat head-ba data-ba)
|
||||
(throw
|
||||
(ex-info (str "Unrecognized header meta: " head-meta)
|
||||
{:head-meta head-meta}))))
|
||||
(truss/ex-info! (str "Unrecognized header meta: " head-meta)
|
||||
{:head-meta head-meta})))
|
||||
|
||||
(comment (wrap-header (.getBytes "foo") {:compressor-id :lz4
|
||||
:encryptor-id nil}))
|
||||
|
|
@ -1213,11 +1260,16 @@
|
|||
- Drops all support for compression and encryption
|
||||
- Must be thawed with `fast-thaw`
|
||||
|
||||
Equivalent to (but a little faster than) `freeze` with opts:
|
||||
- :compressor nil
|
||||
- :encryptor nil
|
||||
- :no-header? true"
|
||||
[x]
|
||||
Equivalent to (but a little faster than) `freeze` with opts
|
||||
{:no-header? true, :compressor nil, :encryptor nil}.
|
||||
|
||||
Intended for use only by advanced users that clearly understand the tradeoffs.
|
||||
I STRONGLY recommend that most users prefer the standard `freeze` since:
|
||||
- The Nippy header is useful for data portability and preservation
|
||||
- Compression is often benefitial at little/no cost
|
||||
- The performance difference between `freeze` and `fast-freeze` is
|
||||
often negligible in practice."
|
||||
^bytes [x]
|
||||
(let [baos (ByteArrayOutputStream. 64)
|
||||
dos (DataOutputStream. baos)]
|
||||
(with-cache (-freeze-with-meta! x dos))
|
||||
|
|
@ -1226,11 +1278,13 @@
|
|||
(defn freeze
|
||||
"Serializes arg (any Clojure data type) to a byte array.
|
||||
To freeze custom types, extend the Clojure reader or see `extend-freeze`."
|
||||
([x] (freeze x nil))
|
||||
([x {:as opts
|
||||
:keys [compressor encryptor password serializable-allowlist incl-metadata?]
|
||||
:or {compressor :auto
|
||||
encryptor aes128-gcm-encryptor}}]
|
||||
(^bytes [x] (freeze x nil))
|
||||
(^bytes
|
||||
[x
|
||||
{:as opts
|
||||
:keys [compressor encryptor password serializable-allowlist incl-metadata?]
|
||||
:or {compressor :auto
|
||||
encryptor aes128-gcm-encryptor}}]
|
||||
|
||||
(call-with-bindings :freeze opts
|
||||
(fn []
|
||||
|
|
@ -1286,10 +1340,21 @@
|
|||
([^DataInput in len] (let [ba (byte-array len)] (.readFully in ba 0 len) ba))
|
||||
([^DataInput in ]
|
||||
(enc/case-eval (.readByte in)
|
||||
id-bytes-0 (byte-array 0)
|
||||
id-bytes-sm (read-bytes in (read-sm-count in))
|
||||
id-bytes-md (read-bytes in (read-md-count in))
|
||||
id-bytes-lg (read-bytes in (read-lg-count in)))))
|
||||
id-byte-array-0 (byte-array 0)
|
||||
id-byte-array-sm (read-bytes in (read-sm-count in))
|
||||
id-byte-array-md (read-bytes in (read-md-count in))
|
||||
id-byte-array-lg (read-bytes in (read-lg-count in)))))
|
||||
|
||||
(defmacro ^:private read-array [in thaw-type array-type array]
|
||||
(let [thawed-sym (with-meta 'thawed-sym {:tag thaw-type})
|
||||
array-sym (with-meta 'array-sym {:tag array-type})]
|
||||
`(let [~array-sym ~array]
|
||||
(enc/reduce-n
|
||||
(fn [_# idx#]
|
||||
(let [~thawed-sym (thaw-from-in! ~in)]
|
||||
(aset ~'array-sym idx# ~'thawed-sym)))
|
||||
nil (alength ~'array-sym))
|
||||
~'array-sym)))
|
||||
|
||||
(defn- read-str-sm* [^DataInput in] (String. ^bytes (read-bytes in (read-sm-count* in)) StandardCharsets/UTF_8))
|
||||
(defn- read-str-sm [^DataInput in] (String. ^bytes (read-bytes in (read-sm-count in)) StandardCharsets/UTF_8))
|
||||
|
|
@ -1309,7 +1374,7 @@
|
|||
|
||||
(defmacro ^:private editable? [coll] `(instance? clojure.lang.IEditableCollection ~coll))
|
||||
|
||||
(defn- xform* [xform] (enc/catching-xform {:error/msg "Error thrown via `*thaw-xform*`"} xform))
|
||||
(defn- xform* [xform] (truss/catching-xform {:error/msg "Error thrown via `*thaw-xform*`"} xform))
|
||||
|
||||
(let [rf! (fn rf! ([x] (persistent! x)) ([acc x] (conj! acc x)))
|
||||
rf* (fn rf* ([x] x) ([acc x] (conj acc x)))]
|
||||
|
|
@ -1338,12 +1403,6 @@
|
|||
(let [rf rf2 ] (rf (enc/reduce-n (fn [acc _] (rf acc (thaw-from-in! in) (thaw-from-in! in))) init n)))))))
|
||||
|
||||
(defn- read-kvs-depr [to ^DataInput in] (read-kvs-into to in (quot (.readInt in) 2)))
|
||||
(defn- read-objects [^objects ary ^DataInput in]
|
||||
(enc/reduce-n
|
||||
(fn [^objects ary i]
|
||||
(aset ary i (thaw-from-in! in))
|
||||
ary)
|
||||
ary (alength ary)))
|
||||
|
||||
(def ^:private class-method-sig (into-array Class [IPersistentMap]))
|
||||
|
||||
|
|
@ -1352,14 +1411,12 @@
|
|||
(try
|
||||
(custom-reader in)
|
||||
(catch Exception e
|
||||
(throw
|
||||
(ex-info
|
||||
(str "Reader exception for custom type id: " type-id)
|
||||
{:type-id type-id, :prefixed? prefixed?} e))))
|
||||
(throw
|
||||
(ex-info
|
||||
(str "No reader provided for custom type id: " type-id)
|
||||
{:type-id type-id, :prefixed? prefixed?}))))
|
||||
(truss/ex-info!
|
||||
(str "Reader exception for custom type id: " type-id)
|
||||
{:type-id type-id, :prefixed? prefixed?} e)))
|
||||
(truss/ex-info!
|
||||
(str "No reader provided for custom type id: " type-id)
|
||||
{:type-id type-id, :prefixed? prefixed?})))
|
||||
|
||||
(defn- read-edn [edn]
|
||||
(try
|
||||
|
|
@ -1436,9 +1493,9 @@
|
|||
[^DataInput in class-name]
|
||||
(if (thaw-serializable-allowed? class-name)
|
||||
(read-object in class-name)
|
||||
(throw ; No way to skip bytes, so best we can do is throw
|
||||
(ex-info "Cannot thaw object: `taoensso.nippy/*thaw-serializable-allowlist*` check failed. This is a security feature. See `*thaw-serializable-allowlist*` docstring or https://github.com/ptaoussanis/nippy/issues/130 for details!"
|
||||
{:class-name class-name}))))
|
||||
(truss/ex-info! ; No way to skip bytes, so best we can do is throw
|
||||
"Cannot thaw object: `taoensso.nippy/*thaw-serializable-allowlist*` check failed. This is a security feature. See `*thaw-serializable-allowlist*` docstring or https://github.com/ptaoussanis/nippy/issues/130 for details!"
|
||||
{:class-name class-name})))
|
||||
|
||||
(defn- read-record [in class-name]
|
||||
(let [content (thaw-from-in! in)]
|
||||
|
|
@ -1516,10 +1573,14 @@
|
|||
id-true true
|
||||
id-false false
|
||||
id-char (.readChar in)
|
||||
id-meta (let [m (thaw-from-in! in)]
|
||||
(if *incl-metadata?*
|
||||
(with-meta (thaw-from-in! in) m)
|
||||
(do (thaw-from-in! in))))
|
||||
|
||||
id-meta-protocol-key meta-protocol-key
|
||||
id-meta
|
||||
(let [m (thaw-from-in! in) ; Always consume from stream
|
||||
x (thaw-from-in! in)]
|
||||
(if-let [m (when *incl-metadata?* (not-empty (dissoc m meta-protocol-key)))]
|
||||
(with-meta x m)
|
||||
(do x)))
|
||||
|
||||
id-cached-0 (thaw-cached 0 in)
|
||||
id-cached-1 (thaw-cached 1 in)
|
||||
|
|
@ -1532,12 +1593,19 @@
|
|||
id-cached-sm (thaw-cached (read-sm-count in) in)
|
||||
id-cached-md (thaw-cached (read-md-count in) in)
|
||||
|
||||
id-bytes-0 (byte-array 0)
|
||||
id-bytes-sm (read-bytes in (read-sm-count in))
|
||||
id-bytes-md (read-bytes in (read-md-count in))
|
||||
id-bytes-lg (read-bytes in (read-lg-count in))
|
||||
id-byte-array-0 (byte-array 0)
|
||||
id-byte-array-sm (read-bytes in (read-sm-count in))
|
||||
id-byte-array-md (read-bytes in (read-md-count in))
|
||||
id-byte-array-lg (read-bytes in (read-lg-count in))
|
||||
|
||||
id-objects-lg (read-objects (object-array (read-lg-count in)) in)
|
||||
id-long-array-lg (read-array in long "[J" (long-array (read-lg-count in)))
|
||||
id-int-array-lg (read-array in int "[I" (int-array (read-lg-count in)))
|
||||
|
||||
id-double-array-lg (read-array in double "[D" (double-array (read-lg-count in)))
|
||||
id-float-array-lg (read-array in float "[F" (float-array (read-lg-count in)))
|
||||
|
||||
id-string-array-lg (read-array in String "[Ljava.lang.String;" (make-array String (read-lg-count in)))
|
||||
id-object-array-lg (read-array in Object "[Ljava.lang.Object;" (object-array (read-lg-count in)))
|
||||
|
||||
id-str-0 ""
|
||||
id-str-sm* (read-str in (read-sm-count* in))
|
||||
|
|
@ -1680,15 +1748,13 @@
|
|||
|
||||
(if (neg? type-id)
|
||||
(read-custom! in nil type-id) ; Unprefixed custom type
|
||||
(throw
|
||||
(ex-info
|
||||
(str "Unrecognized type id (" type-id "). Data frozen with newer Nippy version?")
|
||||
{:type-id type-id}))))
|
||||
(truss/ex-info!
|
||||
(str "Unrecognized type id (" type-id "). Data frozen with newer Nippy version?")
|
||||
{:type-id type-id})))
|
||||
|
||||
(catch Throwable t
|
||||
(throw
|
||||
(ex-info (str "Thaw failed against type-id: " type-id)
|
||||
{:type-id type-id} t))))))
|
||||
(truss/ex-info! (str "Thaw failed against type-id: " type-id)
|
||||
{:type-id type-id} t)))))
|
||||
|
||||
(let [head-sig head-sig] ; Not ^:const
|
||||
(defn- try-parse-header [^bytes ba]
|
||||
|
|
@ -1707,33 +1773,32 @@
|
|||
:snappy compression/snappy-compressor
|
||||
:lzma2 lzma2-compressor
|
||||
:lz4 lz4-compressor
|
||||
:no-header (throw (ex-info ":auto not supported on headerless data." {}))
|
||||
:else (throw (ex-info ":auto not supported for non-standard compressors." {}))
|
||||
(do (throw (ex-info (str "Unrecognized :auto compressor id: " compressor-id)
|
||||
{:compressor-id compressor-id})))))
|
||||
:zstd zstd-compressor
|
||||
:no-header (truss/ex-info! ":auto not supported on headerless data." {})
|
||||
:else (truss/ex-info! ":auto not supported for non-standard compressors." {})
|
||||
(do (truss/ex-info! (str "Unrecognized :auto compressor id: " compressor-id)
|
||||
{:compressor-id compressor-id}))))
|
||||
|
||||
(defn- get-auto-encryptor [encryptor-id]
|
||||
(case encryptor-id
|
||||
nil nil
|
||||
:aes128-gcm-sha512 aes128-gcm-encryptor
|
||||
:aes128-cbc-sha512 aes128-cbc-encryptor
|
||||
:no-header (throw (ex-info ":auto not supported on headerless data." {}))
|
||||
:else (throw (ex-info ":auto not supported for non-standard encryptors." {}))
|
||||
(do (throw (ex-info (str "Unrecognized :auto encryptor id: " encryptor-id)
|
||||
{:encryptor-id encryptor-id})))))
|
||||
:no-header (truss/ex-info! ":auto not supported on headerless data." {})
|
||||
:else (truss/ex-info! ":auto not supported for non-standard encryptors." {})
|
||||
(do (truss/ex-info! (str "Unrecognized :auto encryptor id: " encryptor-id)
|
||||
{:encryptor-id encryptor-id}))))
|
||||
|
||||
(def ^:private err-msg-unknown-thaw-failure "Possible decryption/decompression error, unfrozen/damaged data, etc.")
|
||||
(def ^:private err-msg-unrecognized-header "Unrecognized (but apparently well-formed) header. Data frozen with newer Nippy version?")
|
||||
|
||||
(defn fast-thaw
|
||||
"Like `thaw` but:
|
||||
- Drops all support for compression and encryption
|
||||
- Supports only data frozen with `fast-freeze`
|
||||
- Drops all support for compression and encryption
|
||||
|
||||
Equivalent to (but a little faster than) `thaw` with opts:
|
||||
- :compressor nil
|
||||
- :encryptor nil
|
||||
- :no-header? true"
|
||||
{:no-header? true, :compressor nil, :encryptor nil}."
|
||||
[^bytes ba]
|
||||
(let [dis (DataInputStream. (ByteArrayInputStream. ba))]
|
||||
(with-cache (thaw-from-in! dis))))
|
||||
|
|
@ -1769,24 +1834,23 @@
|
|||
(fn ex
|
||||
([ msg] (ex nil msg))
|
||||
([e msg]
|
||||
(throw
|
||||
(ex-info (str "Thaw failed. " msg)
|
||||
{:opts
|
||||
(assoc opts
|
||||
:compressor compressor
|
||||
:encryptor encryptor)
|
||||
(truss/ex-info! (str "Thaw failed. " msg)
|
||||
{:opts
|
||||
(assoc opts
|
||||
:compressor compressor
|
||||
:encryptor encryptor)
|
||||
|
||||
:bindings
|
||||
(enc/assoc-some {}
|
||||
'*freeze-fallback* *freeze-fallback*
|
||||
'*final-freeze-fallback* *final-freeze-fallback*
|
||||
'*auto-freeze-compressor* *auto-freeze-compressor*
|
||||
'*custom-readers* *custom-readers*
|
||||
'*incl-metadata?* *incl-metadata?*
|
||||
'*thaw-serializable-allowlist* *thaw-serializable-allowlist*
|
||||
'*thaw-xform* *thaw-xform*)}
|
||||
:bindings
|
||||
(enc/assoc-some {}
|
||||
'*freeze-fallback* *freeze-fallback*
|
||||
'*final-freeze-fallback* *final-freeze-fallback*
|
||||
'*auto-freeze-compressor* *auto-freeze-compressor*
|
||||
'*custom-readers* *custom-readers*
|
||||
'*incl-metadata?* *incl-metadata?*
|
||||
'*thaw-serializable-allowlist* *thaw-serializable-allowlist*
|
||||
'*thaw-xform* *thaw-xform*)}
|
||||
|
||||
e))))
|
||||
e)))
|
||||
|
||||
thaw-data
|
||||
(fn [data-ba compressor-id encryptor-id ex-fn]
|
||||
|
|
@ -1876,8 +1940,9 @@
|
|||
"Extends Nippy to support freezing of a custom type (ideally concrete) with
|
||||
given id of form:
|
||||
|
||||
* Keyword - 2 byte overhead, keywords hashed to 16 bit id
|
||||
* ℕ∈[1, 128] - 0 byte overhead
|
||||
* ℕ∈[1, 128] - 0 byte overhead. You are responsible for managing ids.
|
||||
* (Namespaced) keyword - 2 byte overhead. Keyword will be hashed to 16 bit int,
|
||||
collisions will throw at compile-time.
|
||||
|
||||
NB: be careful about extending to interfaces, Ref. <http://goo.gl/6gGRlU>.
|
||||
|
||||
|
|
@ -1942,7 +2007,7 @@
|
|||
[{:keys [comparable?] :as opts}]
|
||||
(let [rng (java.util.Random. 123456) ; Seeded for determinism
|
||||
rand-nth (fn [coll] (nth coll (.nextInt rng (count coll))))
|
||||
all
|
||||
base
|
||||
{:nil nil
|
||||
:true true
|
||||
:false false
|
||||
|
|
@ -1979,7 +2044,6 @@
|
|||
#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{[] ()}}] #{:a :b}}
|
||||
[1 [1 2 [1 2 3 [1 2 3 4 [1 2 3 4 5 "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"] {} #{} [] ()]]]]]
|
||||
|
||||
:regex #"^(https?:)?//(www\?|\?)?"
|
||||
:sorted-set (sorted-set 1 2 3 4 5)
|
||||
:sorted-map (sorted-map :b 2 :a 1 :d 4 :c 3)
|
||||
:lazy-seq-empty (map identity ())
|
||||
|
|
@ -1987,12 +2051,10 @@
|
|||
:queue (into clojure.lang.PersistentQueue/EMPTY [:a :b :c :d :e :f :g])
|
||||
:queue-empty clojure.lang.PersistentQueue/EMPTY
|
||||
|
||||
:uuid (java.util.UUID. 7232453380187312026 -7067939076204274491)
|
||||
:uri (java.net.URI. "https://clojure.org")
|
||||
:defrecord (StressRecord. "data")
|
||||
:deftype (StressType. "data")
|
||||
:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
|
||||
:objects (object-array [1 "two" {:data "data"}])
|
||||
:uuid (java.util.UUID. 7232453380187312026 -7067939076204274491)
|
||||
:uri (java.net.URI. "https://clojure.org")
|
||||
:defrecord (StressRecord. "data")
|
||||
:deftype (StressType. "data")
|
||||
|
||||
:util-date (java.util.Date. 1577884455500)
|
||||
:sql-date (java.sql.Date. 1577884455500)
|
||||
|
|
@ -2000,10 +2062,6 @@
|
|||
:duration (enc/compile-if java.time.Duration (java.time.Duration/ofSeconds 100 100) ::skip)
|
||||
:period (enc/compile-if java.time.Period (java.time.Period/of 1 1 1) ::skip)
|
||||
|
||||
:throwable (Throwable. "Msg")
|
||||
:exception (Exception. "Msg")
|
||||
:ex-info (ex-info "Msg" {:data "data"})
|
||||
|
||||
:many-longs (vec (repeatedly 512 #(rand-nth (range 10))))
|
||||
:many-doubles (vec (repeatedly 512 #(double (rand-nth (range 10)))))
|
||||
:many-strings (vec (repeatedly 512 #(rand-nth ["foo" "bar" "baz" "qux"])))
|
||||
|
|
@ -2013,8 +2071,24 @@
|
|||
(rand-nth ["foo" "bar" "baz" "qux" ]))))}]
|
||||
|
||||
(if comparable?
|
||||
(dissoc all :bytes :objects :throwable :exception :ex-info :regex)
|
||||
(do all))))
|
||||
base
|
||||
(assoc base
|
||||
:non-comparable
|
||||
{:regex #"^(https?:)?//(www\?|\?)?"
|
||||
:throwable (Throwable. "Msg")
|
||||
:exception (Exception. "Msg")
|
||||
:ex-info (ex-info "Msg" {:data "data"})
|
||||
:arrays
|
||||
{:boolean (boolean-array (mapv even? (range 32)))
|
||||
:byte (byte-array (mapv byte (range 32)))
|
||||
:short (short-array (mapv short (range 32)))
|
||||
:int (int-array (mapv int (range 32)))
|
||||
:long (long-array (mapv long (range 32)))
|
||||
:float (float-array (mapv float (range 32)))
|
||||
:double (double-array (mapv double (range 32)))
|
||||
:char (char-array (mapv char (range 32)))
|
||||
:str (into-array String (mapv str (range 32)))
|
||||
:object (object-array (mapv vector (range 32)))}}))))
|
||||
|
||||
(comment
|
||||
[(= (stress-data {:comparable? true}) (stress-data {:comparable? true}))
|
||||
|
|
@ -2157,5 +2231,5 @@
|
|||
(alter-var-root *thaw-serializable-allowlist* f) and/or
|
||||
(alter-var-root *freeze-serializable-allow-list* f) instead."
|
||||
[f]
|
||||
(alter-var-root *freeze-serializable-allowlist* (fn [old] (f (enc/have set? old))))
|
||||
(alter-var-root *thaw-serializable-allowlist* (fn [old] (f (enc/have set? old))))))
|
||||
(alter-var-root *freeze-serializable-allowlist* (fn [old] (f (truss/have set? old))))
|
||||
(alter-var-root *thaw-serializable-allowlist* (fn [old] (f (truss/have set? old))))))
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
(ns ^:no-doc taoensso.nippy.compression
|
||||
"Private, implementation detail."
|
||||
(:require [taoensso.encore :as enc])
|
||||
(:require
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc])
|
||||
(:import
|
||||
[java.nio ByteBuffer]
|
||||
[java.io
|
||||
|
|
@ -156,7 +158,7 @@
|
|||
(.read xzs ba 0 len-decomp)
|
||||
(if (== -1 (.read xzs)) ; Good practice as extra safety measure
|
||||
nil
|
||||
(throw (ex-info "LZMA2 Decompress failed: corrupt data?" {:ba ba})))
|
||||
(truss/ex-info! "LZMA2 Decompress failed: corrupt data?" {:ba ba}))
|
||||
ba)))
|
||||
|
||||
;;;; Public API
|
||||
|
|
@ -174,8 +176,8 @@
|
|||
(def lz4-compressor
|
||||
"Default `LZ4` compressor:
|
||||
- Compression ratio: `C` (0.58 on reference benchmark).
|
||||
- Compression speed: `A` (238 msecs on reference benchmark).
|
||||
- Decompression speed: `A+` (31 msecs on reference benchmark).
|
||||
- Compression speed: `A` (240 msecs on reference benchmark).
|
||||
- Decompression speed: `A+` (30 msecs on reference benchmark).
|
||||
|
||||
Good general-purpose compressor, favours speed.
|
||||
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
|
||||
|
|
@ -184,8 +186,8 @@
|
|||
(def lzo-compressor
|
||||
"Default `LZO` compressor:
|
||||
- Compression ratio: `C` (0.58 on reference benchmark).
|
||||
- Compression speed: `A` (216 msecs on reference benchmark).
|
||||
- Decompression speed: `A` (43 msecs on reference benchmark).
|
||||
- Compression speed: `A` (220 msecs on reference benchmark).
|
||||
- Decompression speed: `A` (40 msecs on reference benchmark).
|
||||
|
||||
Good general-purpose compressor, favours speed.
|
||||
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
|
||||
|
|
@ -195,19 +197,22 @@
|
|||
"Default `LZMA2` compressor:
|
||||
- Compression ratio: `A+` (0.4 on reference benchmark).
|
||||
- Compression speed: `E` (18.5 secs on reference benchmark).
|
||||
- Decompression speed: `D` (11.8 secs on reference benchmark).
|
||||
- Decompression speed: `D` (12 secs on reference benchmark).
|
||||
|
||||
Specialized compressor, strongly favours ratio.
|
||||
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
|
||||
(LZMA2Compressor. 0))
|
||||
|
||||
(enc/def* snappy-compressor
|
||||
"Default `Snappy` compressor:
|
||||
- Compression ratio: `C` (0.58 on reference benchmark).
|
||||
- Compression speed: `A+` (210 msecs on reference benchmark).
|
||||
- Decompression speed: `B` (130 msecs on reference benchmark).
|
||||
Good general-purpose compressor, favours speed.
|
||||
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
|
||||
(SnappyCompressor. false))
|
||||
|
||||
(enc/def* ^:no-doc lz4hc-compressor
|
||||
"Different LZ4 modes no longer supported, prefer `lz4-compressor`."
|
||||
{:deprecated "v3.4.0-RC1 (2024-02-06)"}
|
||||
(LZ4Compressor.))
|
||||
|
||||
(enc/def* ^:no-doc snappy-compressor
|
||||
"Snappy compressor no longer recommended, prefer `lz4-compressor`.
|
||||
Decompression can be unsafe against untrusted data!"
|
||||
{:deprecated "v3.4.0-RC1 (2024-02-06)"}
|
||||
(SnappyCompressor. false))
|
||||
|
|
|
|||
|
|
@ -2,7 +2,9 @@
|
|||
"Low-level crypto utils.
|
||||
Private & alpha, very likely to change!"
|
||||
(:refer-clojure :exclude [rand-nth])
|
||||
(:require [taoensso.encore :as enc]))
|
||||
(:require
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc]))
|
||||
|
||||
;; Note that AES128 may be preferable to AES256 due to known attack
|
||||
;; vectors specific to AES256, Ref. <https://goo.gl/qU4CCV>
|
||||
|
|
@ -45,7 +47,7 @@
|
|||
(defn take-ba ^bytes [n ^bytes ba] (java.util.Arrays/copyOf ba ^int n)) ; Pads if ba too small
|
||||
(defn utf8->ba ^bytes [^String s] (.getBytes s "UTF-8"))
|
||||
(defn- add-salt ^bytes [?salt-ba ba] (if ?salt-ba (enc/ba-concat ?salt-ba ba) ba))
|
||||
(defn pwd-as-ba ^bytes [utf8-or-ba] (if (string? utf8-or-ba) (utf8->ba utf8-or-ba) (enc/have enc/bytes? utf8-or-ba)))
|
||||
(defn pwd-as-ba ^bytes [utf8-or-ba] (if (string? utf8-or-ba) (utf8->ba utf8-or-ba) (truss/have enc/bytes? utf8-or-ba)))
|
||||
|
||||
(comment (seq (pwd-as-ba "foo")))
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
(ns ^:no-doc taoensso.nippy.encryption
|
||||
"Private, implementation detail."
|
||||
(:require
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc]
|
||||
[taoensso.nippy.crypto :as crypto]))
|
||||
|
||||
|
|
@ -14,11 +15,11 @@
|
|||
(decrypt ^bytes [encryptor pwd ba]))
|
||||
|
||||
(defn- throw-destructure-ex [typed-password]
|
||||
(throw (ex-info
|
||||
(str "Expected password form: "
|
||||
"[<#{:salted :cached}> <password-string>].\n "
|
||||
"See `aes128-encryptor` docstring for details!")
|
||||
{:typed-password typed-password})))
|
||||
(truss/ex-info!
|
||||
(str "Expected password form: "
|
||||
"[<#{:salted :cached}> <password-string>].\n "
|
||||
"See `aes128-encryptor` docstring for details!")
|
||||
{:typed-password typed-password}))
|
||||
|
||||
(defn- destructure-typed-pwd [typed-password]
|
||||
(if (vector? typed-password)
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@
|
|||
"Private, implementation detail."
|
||||
(:require
|
||||
[clojure.string :as str]
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc]))
|
||||
|
||||
;;;; Fallback type tests
|
||||
|
|
@ -63,7 +64,7 @@
|
|||
(when x
|
||||
(if (string? x)
|
||||
(if (= x "") #{} (set (mapv str/trim (str/split x #"[,:]"))))
|
||||
(enc/have set? x))))
|
||||
(truss/have set? x))))
|
||||
|
||||
(comment
|
||||
(mapv classname-set [nil #{"foo"} "" "foo, bar:baz"])
|
||||
|
|
@ -151,8 +152,6 @@
|
|||
See that function's docstring for more info."
|
||||
[] (trim nmax (state_))))
|
||||
|
||||
;;;
|
||||
|
||||
(comment
|
||||
(count (get-recorded-serializable-classes))
|
||||
(enc/reduce-n
|
||||
|
|
@ -175,3 +174,100 @@
|
|||
|
||||
(defn serializable-allowed? [allow-list class-name]
|
||||
(conform? allow-list class-name)))
|
||||
|
||||
;;;; Release targeting
|
||||
|
||||
(comment
|
||||
(set! *print-length* nil)
|
||||
(vec (sort (keys taoensso.nippy/public-types-spec)))
|
||||
|
||||
;; To help support release targeting, we track new type ids added over time
|
||||
(let [id-history ; {<release> #{type-ids}}
|
||||
{350 ; v3.5.0 (2025-04-15), added 5x
|
||||
;; #{int-array-lg long-array-lg float-array-lg double-array-lg string-array-lg}
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
|
||||
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104
|
||||
105 106 107 108 109 110 111 112 113 114 115 116 117}
|
||||
|
||||
340 ; v3.4.0 (2024-04-30), added 2x
|
||||
;; #{map-entry meta-protocol-key}
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
|
||||
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104
|
||||
105 106 110 111 112 113 114 115}
|
||||
|
||||
330 ; v3.3.0 (2023-10-11), added 11x
|
||||
;; #{long-pos-sm long-pos-md long-pos-lg long-neg-sm long-neg-md long-neg-lg
|
||||
;; str-sm* vec-sm* set-sm* map-sm* sql-date}
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
|
||||
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 105 106
|
||||
110 111 112 113 114 115}
|
||||
|
||||
320 ; v3.2.0 (2022-07-18), added none
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
|
||||
81 82 83 84 85 86 90 91 100 101 102 105 106 110 111 112 113 114 115}
|
||||
|
||||
313 ; v3.1.3 (2022-06-23), added 5x
|
||||
;; #{time-instant time-duration time-period kw-md sym-md}
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
|
||||
81 82 83 84 85 86 90 91 100 101 102 105 106 110 111 112 113 114 115}
|
||||
|
||||
300 ; v3.0.0 (2020-09-20), baseline
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 80
|
||||
81 82 90 91 100 101 102 105 106 110 111 112 113 114 115}}
|
||||
|
||||
diff
|
||||
(fn [new-release old-release]
|
||||
(vec (sort (clojure.set/difference (id-history new-release) (id-history old-release)))))]
|
||||
|
||||
(diff 350 340)))
|
||||
|
||||
(let [target-release
|
||||
(enc/get-env {:as :edn, :default 320}
|
||||
:taoensso.nippy.target-release)
|
||||
|
||||
target>=
|
||||
(fn [min-release]
|
||||
(if target-release
|
||||
(>= (long target-release) (long min-release))
|
||||
true))]
|
||||
|
||||
(defmacro target-release< [min-release] (not (target>= min-release)))
|
||||
(defmacro target-release>=
|
||||
"Returns true iff `target-release` is nil or >= given `min-release`.
|
||||
Used to help ease data migration for changes to core data types.
|
||||
|
||||
When support is added for a new type in Nippy version X, it necessarily means
|
||||
that data containing that new type and frozen with Nippy version X is unthawable
|
||||
with Nippy versions < X.
|
||||
|
||||
Earlier versions of Nippy will throw an exception on thawing affected data:
|
||||
\"Unrecognized type id (<n>). Data frozen with newer Nippy version?\"
|
||||
|
||||
This can present a challenge when updating to new versions of Nippy, e.g.:
|
||||
|
||||
- Rolling updates could lead to old and new versions of Nippy temporarily co-existing.
|
||||
- Data written with new types could limit your ability to revert a Nippy update.
|
||||
|
||||
There's no easy solution to this in GENERAL, but we CAN at least help reduce the
|
||||
burden related to CHANGES in core data types by introducing changes over 2 phases:
|
||||
|
||||
1. Nippy vX reads new (changed) type, writes old type
|
||||
2. Nippy vX+1 writes new (changed) type
|
||||
|
||||
When relevant, we can then warn users in the CHANGELOG to not leapfrog
|
||||
(e.g. Nippy vX -> Nippy vX+2) when doing rolling updates."
|
||||
[min-release] (target>= min-release)))
|
||||
|
||||
(comment (macroexpand '(target-release>= 340)))
|
||||
|
|
|
|||
|
|
@ -9,10 +9,10 @@
|
|||
(def ^:dynamic *thaw-opts* nil)
|
||||
|
||||
(do
|
||||
(defmacro with-freeze-opts [opts & body] `(binding [*freeze-opts* ~opts ] ~@body))
|
||||
(defmacro with-freeze-opts+ [opts & body] `(binding [*freeze-opts* (enc/fast-merge *freeze-opts* ~opts)] ~@body))
|
||||
(defmacro with-thaw-opts [opts & body] `(binding [*thaw-opts* ~opts ] ~@body))
|
||||
(defmacro with-thaw-opts+ [opts & body] `(binding [*thaw-opts* (enc/fast-merge *thaw-opts* ~opts)] ~@body)))
|
||||
(defmacro with-freeze-opts [opts & body] `(binding [*freeze-opts* ~opts ] ~@body))
|
||||
(defmacro with-freeze-opts+ [opts & body] `(binding [*freeze-opts* (enc/merge *freeze-opts* ~opts)] ~@body))
|
||||
(defmacro with-thaw-opts [opts & body] `(binding [*thaw-opts* ~opts ] ~@body))
|
||||
(defmacro with-thaw-opts+ [opts & body] `(binding [*thaw-opts* (enc/merge *thaw-opts* ~opts)] ~@body)))
|
||||
|
||||
(deftype WrappedForFreezing [val opts])
|
||||
(defn wrapped-for-freezing? [x] (instance? WrappedForFreezing x))
|
||||
|
|
@ -25,7 +25,7 @@
|
|||
See also `tools/freeze`."
|
||||
([x ] (wrap-for-freezing x nil))
|
||||
([x wrap-opts]
|
||||
(let [captured-opts (enc/fast-merge *freeze-opts* wrap-opts)] ; wrap > dynamic
|
||||
(let [captured-opts (enc/merge *freeze-opts* wrap-opts)] ; wrap > dynamic
|
||||
(if (instance? WrappedForFreezing x)
|
||||
(let [^WrappedForFreezing x x]
|
||||
(if (= (.-opts x) captured-opts)
|
||||
|
|
@ -44,13 +44,13 @@
|
|||
See also `tools/wrap-for-freezing`."
|
||||
([x ] (freeze x nil))
|
||||
([x default-opts]
|
||||
(let [default-opts (get default-opts :default-opts default-opts) ; Back compatibility
|
||||
active-opts (enc/fast-merge default-opts *freeze-opts*)] ; dynamic > default
|
||||
(let [default-opts (get default-opts :default-opts default-opts) ; Back compatibility
|
||||
active-opts (enc/merge default-opts *freeze-opts*)] ; dynamic > default
|
||||
|
||||
(if (instance? WrappedForFreezing x)
|
||||
(let [^WrappedForFreezing x x]
|
||||
(nippy/freeze (.-val x) (enc/fast-merge active-opts (.-opts x)))) ; captured > active!
|
||||
(nippy/freeze x active-opts)))))
|
||||
(nippy/freeze (.-val x) (enc/merge active-opts (.-opts x)))) ; captured > active!
|
||||
(nippy/freeze x active-opts)))))
|
||||
|
||||
(defn thaw
|
||||
"Like `nippy/thaw` but uses as options the following, merged in
|
||||
|
|
@ -60,8 +60,8 @@
|
|||
2. `tools/*thaw-opts*` dynamic value (default nil)."
|
||||
([ba ] (thaw ba nil))
|
||||
([ba default-opts]
|
||||
(let [default-opts (get default-opts :default-opts default-opts) ; Back compatibility
|
||||
active-opts (enc/fast-merge default-opts *thaw-opts*)] ; dynamic > default
|
||||
(let [default-opts (get default-opts :default-opts default-opts) ; Back compatibility
|
||||
active-opts (enc/merge default-opts *thaw-opts*)] ; dynamic > default
|
||||
|
||||
(nippy/thaw ba active-opts))))
|
||||
|
||||
|
|
|
|||
|
|
@ -78,24 +78,24 @@
|
|||
{laps 1e4
|
||||
warmup 25e3}}]
|
||||
|
||||
(println "\nStarting benchmarks")
|
||||
(println "\nRunning benchmarks...")
|
||||
|
||||
(let [results_ (atom {})]
|
||||
(when (or all? reader?)
|
||||
(println "- Benching Reader...")
|
||||
(println " With Reader...")
|
||||
(swap! results_ assoc :reader
|
||||
(bench1-serialization freeze-reader thaw-reader
|
||||
(fn [^String s] (count (.getBytes s "UTF-8")))
|
||||
(assoc opts :laps laps, :warmup warmup))))
|
||||
|
||||
(when (or all? fressian?)
|
||||
(println "- Benching Fressian...")
|
||||
(println " With Fressian...")
|
||||
(swap! results_ assoc :fressian
|
||||
(bench1-serialization freeze-fress thaw-fress count
|
||||
(assoc opts :laps laps, :warmup warmup))))
|
||||
|
||||
(when (or all? lzma2?)
|
||||
(println "- Benching Nippy/LZMA2...")
|
||||
(println " With Nippy/LZMA2...")
|
||||
(swap! results_ assoc :nippy/lzma2
|
||||
(bench1-serialization
|
||||
#(nippy/freeze % {:compressor nippy/lzma2-compressor})
|
||||
|
|
@ -103,7 +103,7 @@
|
|||
count
|
||||
(assoc opts :laps laps, :warmup warmup))))
|
||||
|
||||
(println "- Benching Nippy/encrypted...")
|
||||
(println " With Nippy/encrypted...")
|
||||
(swap! results_ assoc :nippy/encrypted
|
||||
(bench1-serialization
|
||||
#(nippy/freeze % {:password [:cached "p"]})
|
||||
|
|
@ -111,17 +111,17 @@
|
|||
count
|
||||
(assoc opts :laps laps, :warmup warmup)))
|
||||
|
||||
(println "- Benching Nippy/default...")
|
||||
(println " With Nippy/default...")
|
||||
(swap! results_ assoc :nippy/default
|
||||
(bench1-serialization nippy/freeze nippy/thaw count
|
||||
(assoc opts :laps laps, :warmup warmup)))
|
||||
|
||||
(println "- Benching Nippy/fast...")
|
||||
(println " With Nippy/fast...")
|
||||
(swap! results_ assoc :nippy/fast
|
||||
(bench1-serialization nippy/fast-freeze nippy/fast-thaw count
|
||||
(assoc opts :laps laps, :warmup warmup)))
|
||||
|
||||
(println "- Benchmarks complete! (Time for cake?)")
|
||||
(println "\nBenchmarks done:")
|
||||
(printed-results @results_)))
|
||||
|
||||
;;;; Compression
|
||||
|
|
@ -165,6 +165,7 @@
|
|||
|
||||
(comment
|
||||
{:last-updated "2024-01-16"
|
||||
:system "2020 Macbook Pro M1, 16 GB memory"
|
||||
:clojure-version "1.11.1"
|
||||
:java-version "OpenJDK 21"
|
||||
:deps
|
||||
|
|
|
|||
|
|
@ -4,8 +4,11 @@
|
|||
[clojure.test.check :as tc]
|
||||
[clojure.test.check.generators :as tc-gens]
|
||||
[clojure.test.check.properties :as tc-props]
|
||||
[taoensso.truss :as truss :refer [throws?]]
|
||||
[taoensso.encore :as enc :refer [ba=]]
|
||||
[taoensso.nippy :as nippy :refer [freeze thaw]]
|
||||
[taoensso.nippy.impl :as impl]
|
||||
[taoensso.nippy.tools :as tools]
|
||||
[taoensso.nippy.compression :as compr]
|
||||
[taoensso.nippy.crypto :as crypto]
|
||||
[taoensso.nippy-benchmarks :as benchmarks]))
|
||||
|
|
@ -53,10 +56,6 @@
|
|||
#(freeze % {:password [:salted "p"]}))
|
||||
test-data)))
|
||||
|
||||
(let [d (nippy/stress-data {})]
|
||||
[(is (= (vec (:bytes d)) ((comp vec thaw freeze) (:bytes d))))
|
||||
(is (= (vec (:objects d)) ((comp vec thaw freeze) (:objects d))))])
|
||||
|
||||
(is (= test-data ((comp #(thaw % {:compressor nippy/lzma2-compressor})
|
||||
#(freeze % {:compressor nippy/lzma2-compressor}))
|
||||
test-data)))
|
||||
|
|
@ -75,9 +74,9 @@
|
|||
#(freeze % {:compressor nippy/zstd-compressor}))
|
||||
test-data)))
|
||||
|
||||
(is (enc/throws? Exception (thaw (freeze test-data {:password "malformed"}))))
|
||||
(is (enc/throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
|
||||
(is (enc/throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
|
||||
(is (throws? Exception (thaw (freeze test-data {:password "malformed"}))))
|
||||
(is (throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
|
||||
(is (throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
|
||||
|
||||
(is
|
||||
(= "payload"
|
||||
|
|
@ -94,6 +93,23 @@
|
|||
(let [n range-uint+] (= (thaw (freeze n)) n))
|
||||
(let [n (- range-uint+)] (= (thaw (freeze n)) n))]))
|
||||
|
||||
(is (throws? :ex-info "Failed to freeze type" (nippy/freeze (fn []))))
|
||||
|
||||
(testing "Clojure v1.10+ metadata protocol extensions"
|
||||
[(is (throws? :ex-info "Failed to freeze type" (nippy/freeze (with-meta [] {:a :A, 'b (fn [])}))))
|
||||
(is (= {:a :A} (meta (nippy/thaw (nippy/freeze (with-meta [] {:a :A, 'b/c (fn [])}))))))
|
||||
(is (= nil (meta (nippy/thaw (nippy/freeze (with-meta [] { 'b/c (fn [])})))))
|
||||
"Don't attach empty metadata")])
|
||||
|
||||
(let [d (nippy/stress-data {})]
|
||||
[(is (= (vec (:bytes d)) ((comp vec thaw freeze) (:bytes d))))
|
||||
(is (= (vec (:objects d)) ((comp vec thaw freeze) (:objects d))))])
|
||||
|
||||
(testing "Arrays"
|
||||
(binding [nippy/*thaw-serializable-allowlist* nippy/default-freeze-serializable-allowlist]
|
||||
(mapv (fn [[k aval]] (is (= (vec aval) (-> aval nippy/freeze nippy/thaw vec)) (name k)))
|
||||
(get-in (nippy/stress-data {}) [:non-comparable :arrays]))))
|
||||
|
||||
(is (gen-test 1600 [gen-data] (= gen-data (thaw (freeze gen-data)))) "Generative")])
|
||||
|
||||
;;;; Custom types & records
|
||||
|
|
@ -104,7 +120,7 @@
|
|||
(deftest _types
|
||||
[(testing "Extend to custom type"
|
||||
[(is
|
||||
(enc/throws? Exception ; No thaw extension yet
|
||||
(throws? Exception ; No thaw extension yet
|
||||
(do
|
||||
(alter-var-root #'nippy/*custom-readers* (constantly {}))
|
||||
(nippy/extend-freeze MyType 1 [x s]
|
||||
|
|
@ -184,6 +200,19 @@
|
|||
|
||||
(defn ba-hash [^bytes ba] (hash (seq ba)))
|
||||
|
||||
(defn gen-hashes [] (enc/map-vals (fn [v] (ba-hash (freeze v))) test-data))
|
||||
(defn cmp-hashes [new old] (vec (sort (reduce-kv (fn [s k v] (if (= (get old k) v) s (conj s k))) #{} new))))
|
||||
|
||||
(def ref-hashes-v341
|
||||
{:deftype -148586793, :lazy-seq-empty 1277437598, :true -1809580601, :long 598276629, :double -454270428, :lazy-seq -1039619789, :short 1152993378, :meta -858252893, :str-long -1970041891, :instant -1401948864, :many-keywords 665654816, :bigint 2033662230, :sym-ns 769802402, :queue 447747779, :float 603100813, :sorted-set 2005004017, :many-strings 1738215727, :nested -1350538572, :queue-empty 1760934486, :duration -775528642, :false 1506926383, :vector 813550992, :util-date 1326218051, :kw 389651898, :sym -1742024487, :str-short -921330463, :subvec 709331681, :kw-long 852232872, :integer 624865727, :sym-long -1535730190, :list -1207486853, :ratio 1186850097, :byte -1041979678, :bigdec -1846988137, :nil 2005042235, :defrecord -553848560, :sorted-map -1160380145, :sql-date 80018667, :map-entry 1219306839, :false-boxed 1506926383, :uri 870148616, :period -2043530540, :many-longs -1109794519, :uuid -338331115, :set 1649942133, :kw-ns 1050084331, :map 1989337680, :many-doubles -827569787, :char 858269588})
|
||||
|
||||
(def ref-hashes-v340
|
||||
{:deftype 1529147805, :lazy-seq-empty 1277437598, :true -1809580601, :long 219451189, :double -454270428, :lazy-seq -1039619789, :short 1152993378, :meta 352218350, :str-long -1970041891, :instant -1401948864, :many-keywords 665654816, :bigint 2033662230, :sym-ns 769802402, :queue 447747779, :float 603100813, :sorted-set 1443292905, :many-strings 1777678883, :nested -1590473924, :queue-empty 1760934486, :duration -775528642, :false 1506926383, :vector 89425525, :util-date 1326218051, :kw 389651898, :sym -1742024487, :str-short -1097575232, :subvec -2047667173, :kw-long 852232872, :integer 624865727, :sym-long -1535730190, :list -1113199651, :ratio 1186850097, :byte -1041979678, :bigdec -1846988137, :nil 2005042235, :defrecord 287634761, :sorted-map 1464032648, :sql-date 80018667, :map-entry -1353323498, :false-boxed 1506926383, :uri -1374752165, :period -2043530540, :many-longs 759118414, :uuid -338331115, :set -1515144175, :kw-ns 1050084331, :map 358912619, :many-doubles -827569787, :char 858269588})
|
||||
|
||||
(comment
|
||||
(cmp-hashes ref-hashes-v341 ref-hashes-v340)
|
||||
[:defrecord :deftype :list :long :many-longs :many-strings :map :map-entry :meta :nested :set :sorted-map :sorted-set :str-short :subvec :uri :vector])
|
||||
|
||||
(deftest _stable-serialized-output
|
||||
(testing "Stable serialized output"
|
||||
|
||||
|
|
@ -196,12 +225,9 @@
|
|||
(is (ba= (freeze (sorted-map :a 1 :b 1))
|
||||
(freeze (sorted-map :b 1 :a 1))) "Sorted structures are generally safe")
|
||||
|
||||
;; Track serialized output of stress data so that we can at least be aware of
|
||||
;; (and warn about) unintended changes for common/elementary types, etc. Note that
|
||||
;; reference hashes will need to be recalculated on changes to stress data.
|
||||
(let [reference-hashes ; (enc/map-vals (fn [v] (ba-hash (freeze v))) test-data)
|
||||
{:deftype 1529147805, :lazy-seq-empty 1277437598, :true -1809580601, :long 219451189, :double -454270428, :lazy-seq -1039619789, :short 1152993378, :meta 352218350, :str-long -1970041891, :instant -1401948864, :many-keywords 665654816, :bigint 2033662230, :sym-ns 769802402, :queue 447747779, :float 603100813, :sorted-set 1443292905, :many-strings 1777678883, :nested -1590473924, :queue-empty 1760934486, :duration -775528642, :false 1506926383, :vector 89425525, :util-date 1326218051, :kw 389651898, :sym -1742024487, :str-short -1097575232, :subvec -2047667173, :kw-long 852232872, :integer 624865727, :sym-long -1535730190, :list -1113199651, :ratio 1186850097, :byte -1041979678, :bigdec -1846988137, :nil 2005042235, :defrecord 287634761, :sorted-map 1464032648, :sql-date 80018667, :map-entry -1353323498, :false-boxed 1506926383, :uri -1374752165, :period -2043530540, :many-longs 759118414, :uuid -338331115, :set -1515144175, :kw-ns 1050084331, :map 358912619, :many-doubles -827569787, :char 858269588}
|
||||
|
||||
;; Track serialized output of stress data so that we can detect unintentional changes,
|
||||
;; and warn about intended ones. Hashes will need to be recalculated on changes to stress data.
|
||||
(let [reference-hashes ref-hashes-v341
|
||||
failures ; #{{:keys [k v]}}
|
||||
(reduce-kv
|
||||
(fn [failures k v]
|
||||
|
|
@ -305,7 +331,7 @@
|
|||
[(is (= nippy/*thaw-serializable-allowlist* #{"base.1" "base.2" "add.1" "add.2"})
|
||||
"JVM properties override initial allowlist values")
|
||||
|
||||
(is (enc/throws? Exception (nippy/freeze sem {:serializable-allowlist #{}}))
|
||||
(is (throws? Exception (nippy/freeze sem {:serializable-allowlist #{}}))
|
||||
"Can't freeze Serializable objects unless approved by allowlist")
|
||||
|
||||
(is (sem?
|
||||
|
|
@ -417,22 +443,27 @@
|
|||
|
||||
(is (= (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze []))) []) "rf not run on empty colls")
|
||||
|
||||
(let [ex (enc/throws :default (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze [:a :b]))))]
|
||||
(is (= (-> ex enc/ex-cause enc/ex-cause ex-data :call) '(rf acc in)) "Error thrown via `*thaw-xform*`"))])
|
||||
(let [ex (truss/throws :default (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze [:a :b]))))]
|
||||
(is (= (-> ex ex-cause ex-cause ex-data :call) '(rf acc in)) "Error thrown via `*thaw-xform*`"))])
|
||||
|
||||
;;;; Compressors
|
||||
|
||||
(deftest _compressors
|
||||
(println "\nTesting decompression of random data...")
|
||||
(doseq [c [compr/zstd-compressor
|
||||
compr/lz4-compressor
|
||||
compr/lzo-compressor
|
||||
#_compr/snappy-compressor ; Ref. <https://github.com/airlift/aircompressor/issues/183>
|
||||
compr/snappy-compressor
|
||||
compr/lzma2-compressor]]
|
||||
|
||||
(dotimes [_ 2e4]
|
||||
(is
|
||||
(nil? (enc/catching (compr/decompress c (crypto/rand-bytes 1024))))
|
||||
"Decompression never crashes JVM, even against invalid data"))))
|
||||
(print (str " With " (name (compr/header-id c)))) (flush)
|
||||
(dotimes [_ 5] ; Slow, a few k laps should be sufficient for CI
|
||||
(print ".") (flush)
|
||||
(dotimes [_ 1000]
|
||||
(is
|
||||
(nil? (truss/catching :all (compr/decompress c (crypto/rand-bytes 1024))))
|
||||
"Decompression never crashes JVM, even against invalid data")))
|
||||
(println)))
|
||||
|
||||
;;;; Benchmarks
|
||||
|
||||
|
|
|
|||
|
|
@ -107,7 +107,10 @@ Deserialize it:
|
|||
|
||||
Couldn't be simpler!
|
||||
|
||||
See also the lower-level [`freeze-to-out!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-freeze-to-out.21) and [`thaw-from-in!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-thaw-from-in.21) fns for operating on `DataOutput` and `DataInput` types directly.
|
||||
# Streaming
|
||||
|
||||
- To serialize directly to a `java.io.DataInput`, see [`freeze-to-out!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-freeze-to-out.21).
|
||||
- To deserialize directly from a `java.io.DataOutput`, see [`thaw-from-in!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-thaw-from-in.21).
|
||||
|
||||
# Encryption
|
||||
|
||||
|
|
|
|||
30
wiki/2 Operational-considerations.md
Normal file
30
wiki/2 Operational-considerations.md
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
# Data longevity
|
||||
|
||||
Nippy is widely used to store **long-lived** data and promises (as always) that **data serialized today should be readable by all future versions of Nippy**.
|
||||
|
||||
But please note that the **converse is not generally true**:
|
||||
|
||||
- Nippy `vX` **should** be able to read all data from Nippy `vY<=X` (backwards compatibility)
|
||||
- Nippy `vX` **may/not** be able to read all data from Nippy `vY>X` (forwards compatibility)
|
||||
|
||||
# Rolling updates and rollback
|
||||
|
||||
From time to time, Nippy may introduce:
|
||||
|
||||
- Support for serializing **new types**
|
||||
- Optimizations to the serialization of **pre-existing types**
|
||||
|
||||
To help ease **rolling updates** and to better support **rollback**, Nippy (since version v3.4.1) will always introduce such changes over **two version releases**:
|
||||
|
||||
- Release 1: to add **read support** for the new types
|
||||
- Release 2: to add **write support** for the new types
|
||||
|
||||
Starting from v3.4.1, Nippy's release notes will **always clearly indicate** if a particular update sequence is recommended.
|
||||
|
||||
# Stability of byte output
|
||||
|
||||
It has **never been an objective** of Nippy to offer **predictable byte output**, and I'd generally **recommend against** depending on specific byte output.
|
||||
|
||||
However, I know that a small minority of users *do* have specialized needs in this area.
|
||||
|
||||
So starting with Nippy v3.4, Nippy's release notes will **always clearly indicate** if any changes to byte output are expected.
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
> This article was kindly contributed by a Nippy user (@Outrovurt)
|
||||
> This article is **community content** kindly contributed by a Nippy user (@Outrovurt)
|
||||
|
||||
This article describes a number of use cases where you need to make changes to your code which will have some impact on data you have already frozen using Nippy, and how best to manage each specific case. We will also discuss custom freezing and thawing.
|
||||
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
See the menu to the right for content 👉
|
||||
See the **menu to the right** for content 👉
|
||||
|
||||
# Contributions welcome
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue