Compare commits
No commits in common. "master" and "gh-pages" have entirely different histories.
44 changed files with 1349 additions and 5502 deletions
32
.github/workflows/graal-tests.yml
vendored
32
.github/workflows/graal-tests.yml
vendored
|
|
@ -1,32 +0,0 @@
|
|||
name: Graal tests
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
strategy:
|
||||
matrix:
|
||||
java: ['17']
|
||||
os: [ubuntu-latest, macOS-latest, windows-latest]
|
||||
|
||||
runs-on: ${{ matrix.os }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: graalvm/setup-graalvm@v1
|
||||
with:
|
||||
version: 'latest'
|
||||
java-version: ${{ matrix.java }}
|
||||
components: 'native-image'
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- uses: DeLaGuardo/setup-clojure@12.5
|
||||
with:
|
||||
lein: latest
|
||||
bb: latest
|
||||
|
||||
- uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.m2/repository
|
||||
key: deps-${{ hashFiles('deps.edn') }}
|
||||
restore-keys: deps-
|
||||
|
||||
- run: bb graal-tests
|
||||
30
.github/workflows/main-tests.yml
vendored
30
.github/workflows/main-tests.yml
vendored
|
|
@ -1,30 +0,0 @@
|
|||
name: Main tests
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
tests:
|
||||
strategy:
|
||||
matrix:
|
||||
java: ['17', '19', '21']
|
||||
os: [ubuntu-latest]
|
||||
|
||||
runs-on: ${{ matrix.os }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-java@v4
|
||||
with:
|
||||
distribution: 'corretto'
|
||||
java-version: ${{ matrix.java }}
|
||||
|
||||
- uses: DeLaGuardo/setup-clojure@12.5
|
||||
with:
|
||||
lein: latest
|
||||
|
||||
- uses: actions/cache@v4
|
||||
id: cache-deps
|
||||
with:
|
||||
path: ~/.m2/repository
|
||||
key: deps-${{ hashFiles('project.clj') }}
|
||||
restore-keys: deps-
|
||||
|
||||
- run: lein test-all
|
||||
20
.gitignore
vendored
20
.gitignore
vendored
|
|
@ -1,13 +1,13 @@
|
|||
pom.xml*
|
||||
.lein*
|
||||
.nrepl-port
|
||||
/target
|
||||
/lib
|
||||
/classes
|
||||
/checkouts
|
||||
/logs
|
||||
/docs
|
||||
/doc
|
||||
pom.xml
|
||||
*.jar
|
||||
*.class
|
||||
*.sh
|
||||
.lein*
|
||||
.env
|
||||
.DS_Store
|
||||
/lib/
|
||||
/classes/
|
||||
/target/
|
||||
/checkouts/
|
||||
/logs/
|
||||
/wiki/.git
|
||||
|
|
|
|||
558
CHANGELOG.md
558
CHANGELOG.md
|
|
@ -1,558 +0,0 @@
|
|||
This project uses [**Break Versioning**](https://www.taoensso.com/break-versioning).
|
||||
|
||||
---
|
||||
|
||||
# `v3.5.0` (2025-04-15)
|
||||
|
||||
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.5.0)
|
||||
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
|
||||
|
||||
This is a **general maintenance release** focused on updating dependencies and laying groundwork (read support) for new array types coming in Nippy v3.6.
|
||||
|
||||
It **drops support for Clojure v1.9** but should otherwise be a safe update from (at least) all recent versions of Nippy.
|
||||
|
||||
## Since `v3.5.0-RC1` (2024-10-28)
|
||||
|
||||
- **\[mod]** Drop official Clojure v1.9 support \[da57206]
|
||||
- \[new] Add string array type to default [thaw-serializable-allowlist](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*thaw-serializable-allowlist*) \[bfba594]
|
||||
- \[new] Use [Truss exceptions](https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#ex-info) on errors \[8d62dc2]
|
||||
- \[new] [#184] Incl. cause on non-native freeze failures \[8d10765]
|
||||
- \[doc] Clarify `*freeze-fallback*` docstring \[1026ea0]
|
||||
|
||||
## Since `v3.4.2` (2024-05-26)
|
||||
|
||||
- **\[mod]** Drop official Clojure v1.9 support \[da57206]
|
||||
- \[new] Add string array type to default [thaw-serializable-allowlist](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*thaw-serializable-allowlist*) \[bfba594]
|
||||
- \[new] Use [Truss exceptions](https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#ex-info) on errors \[8d62dc2]
|
||||
- \[new] [#184] Incl. cause on non-native freeze failures \[8d10765]
|
||||
- \[new] [#175] Mark [cache](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#cache) feature as stable \[b217db5]
|
||||
- \[doc] Clarify `*freeze-fallback*` docstring \[1026ea0]
|
||||
|
||||
## Migration info
|
||||
|
||||
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
|
||||
| :------------------------ | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
|
||||
| `v3.5.0-RC1` (2024-10-28) | - | - | - |
|
||||
| `v3.4.2` (2024-05-26) | - | - | - |
|
||||
| `v3.4.1` (2024-05-02) | - | - | - |
|
||||
| `v3.4.0` (2024-04-30) | - | **Yes** | - |
|
||||
| `v3.3.0` (2023-10-11) | - | - | - |
|
||||
| `v3.2.0` (2022-07-18) | - | - | - |
|
||||
| `v3.1.3` (2022-06-23) | - | - | - |
|
||||
|
||||
> [1] Relevant only when Nippy introduces support for new types **AND** you plan to update Nippy with a **rolling update** (coexisting new and old versions).
|
||||
|
||||
If updating from older versions of Nippy, please see the relevant release notes.
|
||||
|
||||
As always:
|
||||
|
||||
- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
|
||||
- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
|
||||
- **Please report any unexpected problems** 🙏
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
||||
---
|
||||
|
||||
# `v3.5.0-RC1` (2024-10-28)
|
||||
|
||||
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.5.0-RC1)
|
||||
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
|
||||
|
||||
This is a **non-breaking maintenance release** that updates dependencies and includes read support for more native array types to be introduced in a future v3.6 release.
|
||||
|
||||
It should be safe to update from (at least) all recent versions of Nippy.
|
||||
|
||||
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
|
||||
| :-------------------- | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
|
||||
| `v3.4.2` (2024-05-26) | - | - | - |
|
||||
| `v3.4.1` (2024-05-02) | - | - | - |
|
||||
| `v3.4.0` (2024-04-30) | - | Yes | - |
|
||||
| `v3.3.0` (2023-10-11) | - | - | - |
|
||||
| `v3.2.0` (2022-07-18) | - | - | - |
|
||||
| `v3.1.3` (2022-06-23) | - | - | - |
|
||||
|
||||
> [1] Relevant only when introducing support for new types, and for users that do rolling updates
|
||||
|
||||
If updating from older versions of Nippy, please see the relevant release notes.
|
||||
|
||||
As always:
|
||||
|
||||
- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
|
||||
- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
|
||||
- **Please report any unexpected problems** 🙏
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
||||
---
|
||||
|
||||
# `v3.4.2` (2024-05-26)
|
||||
|
||||
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.2)
|
||||
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
|
||||
|
||||
⚠️ This release addresses a [**security vulnerability**](https://github.com/taoensso/nippy/security/advisories/GHSA-vw78-267v-588h) in Nippy's upstream compression library and is **recommended for all existing users**.
|
||||
|
||||
It should be a **straight-forward and non-breaking update** for almost everyone:
|
||||
|
||||
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
|
||||
| :-------------------- | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
|
||||
| `v3.4.1` (2024-05-02) | - | - | - |
|
||||
| `v3.4.0` (2024-04-30) | - | Yes | - |
|
||||
| `v3.3.0` (2023-10-11) | - | - | - |
|
||||
| `v3.2.0` (2022-07-18) | - | - | - |
|
||||
| `v3.1.3` (2022-06-23) | - | - | - |
|
||||
|
||||
> [1] Relevant only when introducing support for new types, and for users that do rolling updates
|
||||
|
||||
If updating from older versions of Nippy, please see the relevant release notes.
|
||||
|
||||
As always:
|
||||
|
||||
- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
|
||||
- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
|
||||
- **Please report any unexpected problems** 🙏
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
||||
---
|
||||
|
||||
# `v3.4.1` (2024-05-02)
|
||||
|
||||
> **Dep**: Nippy is [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.1).
|
||||
> **Versioning**: Nippy uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
|
||||
Like [`v3.4.0`](https://github.com/taoensso/nippy/releases/tag/v3.4.0) but introduces an internal mechanism to help make it easier for some users that do **rolling updates** from earlier versions of Nippy.
|
||||
|
||||
Still, the usual warning applies: data **frozen by Nippy version X** should ideally be **thawed by version >= X**, otherwise you run the risk of the thaw throwing when unfamiliar types are encountered. Please note that this can affect **rolling updates**, and can limit your ability to **revert a Nippy update**. Please ensure adequate testing in your environment before updating against production data!
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
||||
## Changes since `v3.4.0`
|
||||
|
||||
* [mod] Due to some internal format changes, Nippy `v3.4.1` may produce **different serialized output** to `v3.4.0` and earlier versions of Nippy. Most users won't care about this, but you could be affected if you depend on specific serialized byte values (for example by comparing serialized output between different versions of Nippy).
|
||||
|
||||
---
|
||||
|
||||
# `v3.4.0` (2024-04-30)
|
||||
|
||||
> **Dep**: Nippy is [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0).
|
||||
> **Versioning**: Nippy uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
|
||||
This is a non-breaking **feature and maintenance** release and should be a safe update for existing users. But as always, please **test carefully and report any unexpected problems**, thank you! 🙏
|
||||
|
||||
**IMPORTANT**: data **frozen by Nippy version X** should always be **thawed by version >= X**, otherwise you run the risk of the thaw throwing when encountering unfamiliar types. Please note that this can affect **rolling updates**, and can limit your ability to **revert a Nippy update**. Please ensure adequate testing in your environment before updating against production data.
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
||||
## Changes since `v3.3.0` (2023-10-11)
|
||||
|
||||
* 82a050b [mod] Don't attach empty metadata (meta will now be `nil` rather than `{}`)
|
||||
|
||||
## Fixes since `v3.3.0` (2023-10-11)
|
||||
|
||||
* 92c4a83 [fix] Broken `*final-freeze-fallback*` default val
|
||||
|
||||
## New since `v3.3.0` (2023-10-11)
|
||||
|
||||
* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
|
||||
* 6ad5aeb [new] Add `:zstd` compressor, new compressor backend
|
||||
* 9db09e1 [new] [#163] Track serialized output in tests
|
||||
* dcc6b08 [new] [#164] Update benchmarks
|
||||
* f3ff7ae [new] Add native `MapEntry` freezer
|
||||
* 37cf415 [new] [#171] Auto strip metadata protocol extensions
|
||||
* Misc internal improvements
|
||||
|
||||
## Everything since `v3.4.0-RC3` (2024-04-10)
|
||||
|
||||
* Update dependencies
|
||||
|
||||
---
|
||||
|
||||
# `v3.4.0-RC3` (2024-04-10)
|
||||
|
||||
> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0-RC3), this project uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
|
||||
This is a non-breaking **feature and maintenance** pre-release.
|
||||
Please **test carefully and report any unexpected problems**, thank you! 🙏
|
||||
|
||||
## New since `v3.3.0`
|
||||
|
||||
* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
|
||||
* 6ad5aeb [new] Add `:zstd` compressor, new compressor backend
|
||||
* 9db09e1 [new] [#163] Track serialized output in tests
|
||||
* dcc6b08 [new] [#164] Update benchmarks
|
||||
* f3ff7ae [new] Add native `MapEntry` freezer
|
||||
* 37cf415 [new] [#171] Auto strip metadata protocol extensions
|
||||
* Misc internal improvements
|
||||
|
||||
## Everything since `v3.4.0-RC2`
|
||||
|
||||
* 82a050b [mod] Don't attach empty metadata
|
||||
* 92c4a83 [fix] Broken `*final-freeze-fallback*` default val
|
||||
* 37cf415 [new] [#171] Auto strip metadata protocol extensions
|
||||
* Update dependencies
|
||||
|
||||
|
||||
# `v3.4.0-RC2` (2024-02-26)
|
||||
|
||||
> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0-RC2), this project uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
|
||||
This is a non-breaking **feature and maintenance** pre-release.
|
||||
Please **test carefully and report any unexpected problems**, thank you! 🙏
|
||||
|
||||
## New since `v3.3.0`
|
||||
|
||||
* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
|
||||
* 6ad5aeb [new] Add `:zstd` compressor, new compressor backend
|
||||
* 9db09e1 [new] [#163] Track serialized output in tests
|
||||
* dcc6b08 [new] [#164] Update benchmarks
|
||||
* f3ff7ae [new] Add native `MapEntry` freezer
|
||||
* Misc internal improvements
|
||||
|
||||
## Everything since `v3.4.0-RC1`
|
||||
|
||||
* cb5b7cf [fix] [#169] Can't auto-identify `:zstd` compressor when decompressing
|
||||
* cb0b871 Revert [mod] 578c585 (upstream fix now available)
|
||||
* Update dependencies
|
||||
|
||||
---
|
||||
|
||||
# `v3.4.0-RC1` (2024-02-06)
|
||||
|
||||
> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0-RC1), this project uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
|
||||
This is a non-breaking **feature and maintenance** pre-release.
|
||||
Please **test carefully and report any unexpected problems**, thank you! 🙏
|
||||
|
||||
## Changes since `v3.3.0`
|
||||
|
||||
* 578c585 [mod] Remove `nippy/snappy-compressor`
|
||||
|
||||
## New since `v3.3.0`
|
||||
|
||||
* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
|
||||
* 6ad5aeb [new] Add `:zstd` compressor, new compressor backend
|
||||
* 9db09e1 [new] [#163] Track serialized output in tests
|
||||
* dcc6b08 [new] [#164] Update benchmarks
|
||||
* f3ff7ae [new] Add native `MapEntry` freezer
|
||||
* Misc internal improvements
|
||||
|
||||
---
|
||||
|
||||
# `v3.3.0` (2023-10-11)
|
||||
|
||||
> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.3.3), this project uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
|
||||
Identical to `v3.3.0-RC2`.
|
||||
|
||||
This is a non-breaking **feature and maintenance** release.
|
||||
Please test carefully and report any unexpected problems, thank you! 🙏
|
||||
|
||||
## Changes since `v3.2.0`
|
||||
|
||||
* [mod] Due to micro-optimizations of some elementary types, Nippy v3.3 may produce **different serialized output** to earlier versions of Nippy. Most users won't care about this, but you could be affected if you depend on specific serialized byte values (for example by comparing serialized output between different versions of Nippy).
|
||||
|
||||
## Fixes since `v3.2.0`
|
||||
|
||||
* fa1cc66 [fix] [#143] Don't freeze meta info for types that don't support `with-meta`
|
||||
|
||||
## New since `v3.2.0`
|
||||
|
||||
* 89f98b4 [new] [#153] PoC: transducer support on thaw
|
||||
* 60bc4e9 [new] [Storage efficiency] PoC: unsigned counts for small core colls
|
||||
* 0a9d670 [new] [Storage efficiency] PoC: separate signed long types
|
||||
* 8778fa4 [new] Include `:bindings` in ex-data of thaw failures
|
||||
* aba153e [new] [#159] Add native impln for `java.sql.Date` (@philomates)
|
||||
* d8b1825 [new] [#158] Add `java.lang.ClassCastException` to default thaw allow list (@carlosgeos)
|
||||
* 8b7186a [new] Update [benchmark results](https://github.com/taoensso/nippy#performance)
|
||||
* 3ac06b6 [new] Refactor tools ns, embed dynamic `*freeze-opts*` in wrappers
|
||||
* 129ce95 [new] [#151] [#140] Add experimental `public-types-spec`
|
||||
|
||||
## Other improvements since `v3.2.0`
|
||||
|
||||
* Improved some docstrings
|
||||
* Improved generative unit tests
|
||||
* Updated internal dependencies
|
||||
|
||||
---
|
||||
|
||||
# `v3.4.0-beta1` (2023-09-26)
|
||||
|
||||
> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.0-beta1)
|
||||
|
||||
This is a non-breaking **feature** pre-release.
|
||||
Please **test carefully and report any unexpected problems**, thank you! 🙏
|
||||
|
||||
## New since `v3.3.0-RC2`
|
||||
|
||||
* 6ad5aeb [new] Add `:zstd` compressor, new (faster) compressor backend, better docstrings
|
||||
* fb6f75e [new] Smarter, faster, protocol-based `freezable?` util
|
||||
* f3ff7ae [new] Add native `MapEntry` freezer
|
||||
* fef079d [new] Add subvec to stress data
|
||||
* Misc internal improvements
|
||||
|
||||
## Other improvements since `v3.3.0-RC2`
|
||||
|
||||
* e0cd003 [nop] Update docs
|
||||
* 99970d5 [nop] Update benchmark results
|
||||
* bcf7673 [nop] Move benchmarks ns under tests dir
|
||||
|
||||
---
|
||||
|
||||
# `v3.3.0-RC2` (2023-09-25)
|
||||
|
||||
> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.3.0-RC2)
|
||||
|
||||
Identical to `v3.3.0-RC1` except:
|
||||
|
||||
* Improves some docstrings
|
||||
* Improves generative unit tests
|
||||
* Updates internal dependencies
|
||||
|
||||
If no unexpected problems come up, `v3.3.0` final is planned for release by the end of September.
|
||||
|
||||
---
|
||||
|
||||
# `v3.3.0-RC1` (2023-08-02)
|
||||
|
||||
> 📦 [Available on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.3.0-RC1)
|
||||
|
||||
This is a non-breaking **feature and maintenance** pre-release.
|
||||
Please **test carefully and report any unexpected problems**, thank you! 🙏
|
||||
|
||||
## Fixes since `v3.2.0`
|
||||
|
||||
* fa1cc66 [fix] [#143] Don't freeze meta info for types that don't support `with-meta`
|
||||
|
||||
## New since `v3.2.0`
|
||||
|
||||
* 89f98b4 [new] [#153] PoC: transducer support on thaw
|
||||
* 60bc4e9 [new] [Storage efficiency] PoC: unsigned counts for small core colls
|
||||
* 0a9d670 [new] [Storage efficiency] PoC: separate signed long types
|
||||
* 8778fa4 [new] Include `:bindings` in ex-data of thaw failures
|
||||
* aba153e [new] [#159] Add native impln for `java.sql.Date` (@philomates)
|
||||
* d8b1825 [new] [#158] Add `java.lang.ClassCastException` to default thaw allow list (@carlosgeos)
|
||||
* 8b7186a [new] Update [benchmark results](https://github.com/taoensso/nippy#performance)
|
||||
* 3ac06b6 [new] Refactor tools ns, embed dynamic `*freeze-opts*` in wrappers
|
||||
* 129ce95 [new] [#151] [#140] Add experimental `public-types-spec`
|
||||
|
||||
---
|
||||
|
||||
# `v3.2.0` (2022-07-18)
|
||||
|
||||
> Identical to `v3.2.0-RC3` (2022 Jun 27)
|
||||
|
||||
```clojure
|
||||
[com.taoensso/nippy "3.2.0"]
|
||||
```
|
||||
|
||||
> This is a non-breaking maintenance release.
|
||||
> See [here](https://github.com/taoensso/encore#recommended-steps-after-any-significant-dependency-update) for recommended steps when updating any Clojure/Script dependencies.
|
||||
|
||||
## New since `v3.1.3`
|
||||
|
||||
* [#144] [New] Add `org.joda.time.DateTime` to `default-thaw-serializable-allowlist` (@slipset)
|
||||
* [#146] [New] Add Graal native configurations (@FieryCod)
|
||||
|
||||
## Changes since `v3.1.3`
|
||||
|
||||
* Bump dependencies, incl. minimum Encore version
|
||||
|
||||
## Fixes since `v3.1.1`
|
||||
|
||||
* [#89 #150] [Fix] Boxed Booleans incorrectly freezing to primitive `true` (@RolT)
|
||||
* [#148] [Fix] `tools/freeze` should use `*freeze-opts*` even for unwrapped vals
|
||||
* [#145] [Fix] Freezing custom types with munged field names
|
||||
|
||||
The boxed Boolean bug has been around since the first version of Nippy and is mostly
|
||||
relevant to users doing Java interop. For more info see: https://github.com/taoensso/nippy/commit/8909a32bdd654a136da385e0e09c9cc44416f964
|
||||
|
||||
---
|
||||
|
||||
# `v3.2.0-RC3` (2022-06-27)
|
||||
|
||||
```clojure
|
||||
[com.taoensso/nippy "3.2.0-RC3"]
|
||||
```
|
||||
|
||||
> This is a non-breaking maintenance release.
|
||||
> See [here](https://github.com/taoensso/encore#recommended-steps-after-any-significant-dependency-update) for recommended steps when updating any Clojure/Script dependencies.
|
||||
|
||||
## New since `v3.1.3`
|
||||
|
||||
* [#144] [New] Add `org.joda.time.DateTime` to `default-thaw-serializable-allowlist` (@slipset)
|
||||
* [#146] [New] Add Graal native configurations (@FieryCod)
|
||||
|
||||
## Changes since `v3.1.3`
|
||||
|
||||
* Bump dependencies, incl. minimum Encore version
|
||||
|
||||
## Fixes since `v3.1.1`
|
||||
|
||||
* [#89 #150] [Fix] Boxed Booleans incorrectly freezing to primitive `true` (@RolT)
|
||||
* [#148] [Fix] `tools/freeze` should use `*freeze-opts*` even for unwrapped vals
|
||||
* [#145] [Fix] Freezing custom types with munged field names
|
||||
|
||||
The boxed Boolean bug has been around since the first version of Nippy and is mostly
|
||||
relevant to users doing Java interop. For more info see: https://github.com/taoensso/nippy/commit/8909a32bdd654a136da385e0e09c9cc44416f964
|
||||
|
||||
---
|
||||
|
||||
# `v3.2.0-RC2` (2022-06-23)
|
||||
|
||||
```clojure
|
||||
[com.taoensso/nippy "3.2.0-RC2"]
|
||||
```
|
||||
|
||||
> This is a non-breaking maintenance release.
|
||||
> See [here](https://github.com/taoensso/encore#recommended-steps-after-any-significant-dependency-update) for recommended steps when updating any Clojure/Script dependencies.
|
||||
|
||||
## New since `v3.1.3`
|
||||
|
||||
* [#144] [New] Add `org.joda.time.DateTime` to `default-thaw-serializable-allowlist` (@slipset)
|
||||
* [#146] [New] Add Graal native configurations (@FieryCod)
|
||||
|
||||
## Changes since `v3.1.3`
|
||||
|
||||
* Bump dependencies, incl. minimum Encore version
|
||||
|
||||
## Fixes since `v3.1.1`
|
||||
|
||||
* [#148] [Fix] `tools/freeze` should use `*freeze-opts*` even for unwrapped vals
|
||||
* [#89 #150] [Fix] Boxed Booleans incorrectly freezing to primitive `true` (@RolT)
|
||||
|
||||
The boxed Boolean bug has been around since the first version of Nippy and is mostly
|
||||
relevant to users doing Java interop. For more info see: https://github.com/taoensso/nippy/commit/8909a32bdd654a136da385e0e09c9cc44416f964
|
||||
|
||||
---
|
||||
|
||||
# `v3.1.3` (2022-06-23)
|
||||
|
||||
```clojure
|
||||
[com.taoensso/nippy "3.1.3"]
|
||||
```
|
||||
|
||||
> This is a non-breaking, bugfix release.
|
||||
> See [here](https://github.com/taoensso/encore#recommended-steps-after-any-significant-dependency-update) for recommended steps when updating any Clojure/Script dependencies.
|
||||
|
||||
## Fixes since `v3.1.1`
|
||||
|
||||
* [#148] [Fix] `tools/freeze` should use `*freeze-opts*` even for unwrapped vals
|
||||
* [#89 #150] [Fix] Boxed Booleans incorrectly freezing to primitive `true` (@RolT)
|
||||
|
||||
The boxed Boolean bug has been around since the first version of Nippy and is mostly
|
||||
relevant to users doing Java interop. For more info see: https://github.com/taoensso/nippy/commit/8909a32bdd654a136da385e0e09c9cc44416f964
|
||||
|
||||
---
|
||||
|
||||
# `v3.1.1` (2020-11-18)
|
||||
|
||||
```clojure
|
||||
[com.taoensso/nippy "3.1.1"]
|
||||
```
|
||||
|
||||
> This is a non-breaking, bugfix release. But please note that large keywords or symbols (with >127 characters) frozen with >=`v3.1.1` will need >=`v3.1.1` to thaw.
|
||||
|
||||
## Fixes since `v3.1.0`
|
||||
|
||||
* Large keywords and symbols (with >127 characters) can now thaw without throwing (@danmason).
|
||||
|
||||
|
||||
[1] Keywords or symbols with >127 characters in their name
|
||||
|
||||
---
|
||||
|
||||
# `v3.1.0` (2020-11-06)
|
||||
|
||||
```clojure
|
||||
[com.taoensso/nippy "3.1.0"]
|
||||
```
|
||||
|
||||
> This is a non-breaking, minor feature release.
|
||||
|
||||
## New since `v3.0.0`
|
||||
|
||||
* [#135 #128] Added native `freeze/thaw` support for `java.time` classes on JVM 8+: `Instant`, `Duration`, `Period`.
|
||||
* [#137] Add `thaw-from-resource` convenience util.
|
||||
* Add (DEPRECATED) `swap-serializable-whitelist!` for backwards compatibility.
|
||||
|
||||
## Changes since `v3.0.0`
|
||||
|
||||
* Add several standard `java.time` classes to default `*thaw-serializable-whitelist*`.
|
||||
|
||||
---
|
||||
|
||||
# `v3.1.0-RC1` (2020-10-24)
|
||||
|
||||
```clojure
|
||||
[com.taoensso/nippy "3.1.0-RC1"]
|
||||
```
|
||||
|
||||
> This is a non-breaking, minor feature release.
|
||||
|
||||
## New since `v3.0.0`
|
||||
|
||||
* [#135 #128] Added native `freeze/thaw` support for `java.time.Instant` on JVM 8+ (@cnuernber).
|
||||
|
||||
---
|
||||
|
||||
# `v3.0.0` (2020-09-20)
|
||||
|
||||
```clojure
|
||||
[com.taoensso/nippy "3.0.0"]
|
||||
```
|
||||
|
||||
> This release is focused on smoothing out rough edges left by `CVE-2020-24164` [#130], and to **ease transition** from versions of Nippy < `v2.15.0 final`.
|
||||
|
||||
> See [here](https://github.com/taoensso/encore#recommended-steps-after-any-significant-dependency-update) for recommended steps when updating any Clojure/Script dependencies.
|
||||
|
||||
Note that there's **separate details** below for upgrading from `v2.15` vs `v2.14`:
|
||||
|
||||
## Upgrading from `v2.15` (usually non-breaking)
|
||||
|
||||
Usually a non-breaking drop-in replacement, but there's some changes you might like to take advantage of. See [#130] for **detailed upgrade instructions**.
|
||||
|
||||
### Changes
|
||||
|
||||
- **[BREAKING]** Bumped minimum Clojure version from `v1.5` to `v1.7`.
|
||||
- **[BREAKING]** `:nippy/unthawable` responses now have a standardized form: `{:nippy/unthawable {:type _ :cause _ ...}`. Most folks won't care about this change unless they have code specifically to deal with `:nippy/unthawable` responses.
|
||||
- [Deprecated] `*serializable-whitelist*` has been split into two separate vars: `*freeze-serializable-allowlist*`, `*thaw-serializable-allowlist`*. See [#130] for details.
|
||||
- By default, `freeze` now **allows** the use of Java's Serializable for **any** class. `thaw` continues to be restrictive by default, and will quarantine any objects not on the class allowlist. See [#130] for details.
|
||||
|
||||
### New
|
||||
|
||||
- [#122] Option to disable freezing and/or thawing of metadata.
|
||||
- `freeze` and `thaw` now support opts: `:serializable-allowlist`, `:incl-metadata?`.
|
||||
- New `read-quarantined-serializable-object-unsafe!` util to read quarantined Serializable objects. See [API docs](http://taoensso.github.io/nippy/taoensso.nippy.html#var-read-quarantined-serializable-object-unsafe.21) and/or [#130] for details.
|
||||
- Add `allow-and-record-any-serializable-class-unsafe` util. See [API docs](http://taoensso.github.io/nippy/taoensso.nippy.html#var-allow-and-record-any-serializable-class-unsafe) and/or [#130] for details.
|
||||
|
||||
|
||||
## Upgrading from `v2.14` (may be BREAKING)
|
||||
|
||||
Likely breaking. Please see [#130] for **detailed upgrade instructions**.
|
||||
|
||||
### Changes
|
||||
|
||||
- **[BREAKING]** Bumped minimum Clojure version from `v1.5` to `v1.7`.
|
||||
- **[BREAKING]** [#130] `thaw` will now quarantine Serializable objects whose class is not allowed by `*thaw-serializable-allowlist*`. See [#130] for details.
|
||||
- **[BREAKING]** `:nippy/unthawable` responses now have a standardized form: `{:nippy/unthawable {:type _ :cause _ ...}`. Most folks won't care about this change unless you have code specifically to deal with `:nippy/unthawable` responses.
|
||||
- [#101] Switch default encryptor from `AES-CBC` to `AES-GCM` (faster, includes integrity check)
|
||||
|
||||
### New
|
||||
|
||||
- [#127] Add utils: `freeze-to-string`, `thaw-from-string` (@piotr-yuxuan)
|
||||
- [#113 #114] Add support for object arrays (@isaksky)
|
||||
- [#83 #112] Add support for deftype (@isaksky)
|
||||
- [#83 #113] Add support for URIs (@isaksky)
|
||||
- [#126] `extend-freeze`: include id collision odds in docstring
|
||||
|
||||
### Fixes
|
||||
|
||||
- [#120] Update `freezable?` to cover `nil`
|
||||
|
||||
---
|
||||
|
||||
# Earlier releases
|
||||
|
||||
See [here](https://github.com/taoensso/nippy/releases) for earlier releases.
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
github: ptaoussanis
|
||||
custom: "https://www.taoensso.com/clojure/backers"
|
||||
203
LICENSE.txt
203
LICENSE.txt
|
|
@ -1,203 +0,0 @@
|
|||
Eclipse Public License - v 1.0
|
||||
|
||||
THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC
|
||||
LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM
|
||||
CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT.
|
||||
|
||||
1. DEFINITIONS
|
||||
|
||||
"Contribution" means:
|
||||
|
||||
a) in the case of the initial Contributor, the initial code and documentation
|
||||
distributed under this Agreement, and
|
||||
b) in the case of each subsequent Contributor:
|
||||
i) changes to the Program, and
|
||||
ii) additions to the Program;
|
||||
|
||||
where such changes and/or additions to the Program originate from and are
|
||||
distributed by that particular Contributor. A Contribution 'originates'
|
||||
from a Contributor if it was added to the Program by such Contributor
|
||||
itself or anyone acting on such Contributor's behalf. Contributions do not
|
||||
include additions to the Program which: (i) are separate modules of
|
||||
software distributed in conjunction with the Program under their own
|
||||
license agreement, and (ii) are not derivative works of the Program.
|
||||
|
||||
"Contributor" means any person or entity that distributes the Program.
|
||||
|
||||
"Licensed Patents" mean patent claims licensable by a Contributor which are
|
||||
necessarily infringed by the use or sale of its Contribution alone or when
|
||||
combined with the Program.
|
||||
|
||||
"Program" means the Contributions distributed in accordance with this
|
||||
Agreement.
|
||||
|
||||
"Recipient" means anyone who receives the Program under this Agreement,
|
||||
including all Contributors.
|
||||
|
||||
2. GRANT OF RIGHTS
|
||||
a) Subject to the terms of this Agreement, each Contributor hereby grants
|
||||
Recipient a non-exclusive, worldwide, royalty-free copyright license to
|
||||
reproduce, prepare derivative works of, publicly display, publicly
|
||||
perform, distribute and sublicense the Contribution of such Contributor,
|
||||
if any, and such derivative works, in source code and object code form.
|
||||
b) Subject to the terms of this Agreement, each Contributor hereby grants
|
||||
Recipient a non-exclusive, worldwide, royalty-free patent license under
|
||||
Licensed Patents to make, use, sell, offer to sell, import and otherwise
|
||||
transfer the Contribution of such Contributor, if any, in source code and
|
||||
object code form. This patent license shall apply to the combination of
|
||||
the Contribution and the Program if, at the time the Contribution is
|
||||
added by the Contributor, such addition of the Contribution causes such
|
||||
combination to be covered by the Licensed Patents. The patent license
|
||||
shall not apply to any other combinations which include the Contribution.
|
||||
No hardware per se is licensed hereunder.
|
||||
c) Recipient understands that although each Contributor grants the licenses
|
||||
to its Contributions set forth herein, no assurances are provided by any
|
||||
Contributor that the Program does not infringe the patent or other
|
||||
intellectual property rights of any other entity. Each Contributor
|
||||
disclaims any liability to Recipient for claims brought by any other
|
||||
entity based on infringement of intellectual property rights or
|
||||
otherwise. As a condition to exercising the rights and licenses granted
|
||||
hereunder, each Recipient hereby assumes sole responsibility to secure
|
||||
any other intellectual property rights needed, if any. For example, if a
|
||||
third party patent license is required to allow Recipient to distribute
|
||||
the Program, it is Recipient's responsibility to acquire that license
|
||||
before distributing the Program.
|
||||
d) Each Contributor represents that to its knowledge it has sufficient
|
||||
copyright rights in its Contribution, if any, to grant the copyright
|
||||
license set forth in this Agreement.
|
||||
|
||||
3. REQUIREMENTS
|
||||
|
||||
A Contributor may choose to distribute the Program in object code form under
|
||||
its own license agreement, provided that:
|
||||
|
||||
a) it complies with the terms and conditions of this Agreement; and
|
||||
b) its license agreement:
|
||||
i) effectively disclaims on behalf of all Contributors all warranties
|
||||
and conditions, express and implied, including warranties or
|
||||
conditions of title and non-infringement, and implied warranties or
|
||||
conditions of merchantability and fitness for a particular purpose;
|
||||
ii) effectively excludes on behalf of all Contributors all liability for
|
||||
damages, including direct, indirect, special, incidental and
|
||||
consequential damages, such as lost profits;
|
||||
iii) states that any provisions which differ from this Agreement are
|
||||
offered by that Contributor alone and not by any other party; and
|
||||
iv) states that source code for the Program is available from such
|
||||
Contributor, and informs licensees how to obtain it in a reasonable
|
||||
manner on or through a medium customarily used for software exchange.
|
||||
|
||||
When the Program is made available in source code form:
|
||||
|
||||
a) it must be made available under this Agreement; and
|
||||
b) a copy of this Agreement must be included with each copy of the Program.
|
||||
Contributors may not remove or alter any copyright notices contained
|
||||
within the Program.
|
||||
|
||||
Each Contributor must identify itself as the originator of its Contribution,
|
||||
if
|
||||
any, in a manner that reasonably allows subsequent Recipients to identify the
|
||||
originator of the Contribution.
|
||||
|
||||
4. COMMERCIAL DISTRIBUTION
|
||||
|
||||
Commercial distributors of software may accept certain responsibilities with
|
||||
respect to end users, business partners and the like. While this license is
|
||||
intended to facilitate the commercial use of the Program, the Contributor who
|
||||
includes the Program in a commercial product offering should do so in a manner
|
||||
which does not create potential liability for other Contributors. Therefore,
|
||||
if a Contributor includes the Program in a commercial product offering, such
|
||||
Contributor ("Commercial Contributor") hereby agrees to defend and indemnify
|
||||
every other Contributor ("Indemnified Contributor") against any losses,
|
||||
damages and costs (collectively "Losses") arising from claims, lawsuits and
|
||||
other legal actions brought by a third party against the Indemnified
|
||||
Contributor to the extent caused by the acts or omissions of such Commercial
|
||||
Contributor in connection with its distribution of the Program in a commercial
|
||||
product offering. The obligations in this section do not apply to any claims
|
||||
or Losses relating to any actual or alleged intellectual property
|
||||
infringement. In order to qualify, an Indemnified Contributor must:
|
||||
a) promptly notify the Commercial Contributor in writing of such claim, and
|
||||
b) allow the Commercial Contributor to control, and cooperate with the
|
||||
Commercial Contributor in, the defense and any related settlement
|
||||
negotiations. The Indemnified Contributor may participate in any such claim at
|
||||
its own expense.
|
||||
|
||||
For example, a Contributor might include the Program in a commercial product
|
||||
offering, Product X. That Contributor is then a Commercial Contributor. If
|
||||
that Commercial Contributor then makes performance claims, or offers
|
||||
warranties related to Product X, those performance claims and warranties are
|
||||
such Commercial Contributor's responsibility alone. Under this section, the
|
||||
Commercial Contributor would have to defend claims against the other
|
||||
Contributors related to those performance claims and warranties, and if a
|
||||
court requires any other Contributor to pay any damages as a result, the
|
||||
Commercial Contributor must pay those damages.
|
||||
|
||||
5. NO WARRANTY
|
||||
|
||||
EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON AN
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR
|
||||
IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE,
|
||||
NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Each
|
||||
Recipient is solely responsible for determining the appropriateness of using
|
||||
and distributing the Program and assumes all risks associated with its
|
||||
exercise of rights under this Agreement , including but not limited to the
|
||||
risks and costs of program errors, compliance with applicable laws, damage to
|
||||
or loss of data, programs or equipment, and unavailability or interruption of
|
||||
operations.
|
||||
|
||||
6. DISCLAIMER OF LIABILITY
|
||||
|
||||
EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY
|
||||
CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION
|
||||
LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
|
||||
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
|
||||
ARISING IN ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE
|
||||
EXERCISE OF ANY RIGHTS GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY
|
||||
OF SUCH DAMAGES.
|
||||
|
||||
7. GENERAL
|
||||
|
||||
If any provision of this Agreement is invalid or unenforceable under
|
||||
applicable law, it shall not affect the validity or enforceability of the
|
||||
remainder of the terms of this Agreement, and without further action by the
|
||||
parties hereto, such provision shall be reformed to the minimum extent
|
||||
necessary to make such provision valid and enforceable.
|
||||
|
||||
If Recipient institutes patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Program itself
|
||||
(excluding combinations of the Program with other software or hardware)
|
||||
infringes such Recipient's patent(s), then such Recipient's rights granted
|
||||
under Section 2(b) shall terminate as of the date such litigation is filed.
|
||||
|
||||
All Recipient's rights under this Agreement shall terminate if it fails to
|
||||
comply with any of the material terms or conditions of this Agreement and does
|
||||
not cure such failure in a reasonable period of time after becoming aware of
|
||||
such noncompliance. If all Recipient's rights under this Agreement terminate,
|
||||
Recipient agrees to cease use and distribution of the Program as soon as
|
||||
reasonably practicable. However, Recipient's obligations under this Agreement
|
||||
and any licenses granted by Recipient relating to the Program shall continue
|
||||
and survive.
|
||||
|
||||
Everyone is permitted to copy and distribute copies of this Agreement, but in
|
||||
order to avoid inconsistency the Agreement is copyrighted and may only be
|
||||
modified in the following manner. The Agreement Steward reserves the right to
|
||||
publish new versions (including revisions) of this Agreement from time to
|
||||
time. No one other than the Agreement Steward has the right to modify this
|
||||
Agreement. The Eclipse Foundation is the initial Agreement Steward. The
|
||||
Eclipse Foundation may assign the responsibility to serve as the Agreement
|
||||
Steward to a suitable separate entity. Each new version of the Agreement will
|
||||
be given a distinguishing version number. The Program (including
|
||||
Contributions) may always be distributed subject to the version of the
|
||||
Agreement under which it was received. In addition, after a new version of the
|
||||
Agreement is published, Contributor may elect to distribute the Program
|
||||
(including its Contributions) under the new version. Except as expressly
|
||||
stated in Sections 2(a) and 2(b) above, Recipient receives no rights or
|
||||
licenses to the intellectual property of any Contributor under this Agreement,
|
||||
whether expressly, by implication, estoppel or otherwise. All rights in the
|
||||
Program not expressly granted under this Agreement are reserved.
|
||||
|
||||
This Agreement is governed by the laws of the State of New York and the
|
||||
intellectual property laws of the United States of America. No party to this
|
||||
Agreement will bring a legal action under this Agreement more than one year
|
||||
after the cause of action arose. Each party waives its rights to a jury trial in
|
||||
any resulting litigation.
|
||||
128
README.md
128
README.md
|
|
@ -1,128 +0,0 @@
|
|||
<a href="https://www.taoensso.com/clojure" title="More stuff by @ptaoussanis at www.taoensso.com"><img src="https://www.taoensso.com/open-source.png" alt="Taoensso open source" width="340"/></a>
|
||||
[**API**][cljdoc] | [**Wiki**][GitHub wiki] | [Latest releases](#latest-releases) | [Get support][GitHub issues]
|
||||
|
||||
# Nippy
|
||||
|
||||
### Fast serialization library for Clojure
|
||||
|
||||
Clojure's rich data types are awesome. And its [reader](https://clojure.org/reference/reader) allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).
|
||||
|
||||
Nippy is a mature, high-performance **drop-in alternative to the reader**.
|
||||
|
||||
It is used at scale by [Carmine](https://www.taoensso.com/carmine), [Faraday](https://www.taoensso.com/faraday), [PigPen](https://github.com/Netflix/PigPen), [Onyx](https://github.com/onyx-platform/onyx), [XTDB](https://github.com/xtdb/xtdb), [Datalevin](https://github.com/juji-io/datalevin), and others.
|
||||
|
||||
## Latest release/s
|
||||
|
||||
- `2025-04-15` `v3.5.0`: [release info](../../releases/tag/v3.5.0)
|
||||
|
||||
[![Main tests][Main tests SVG]][Main tests URL]
|
||||
[![Graal tests][Graal tests SVG]][Graal tests URL]
|
||||
|
||||
See [here][GitHub releases] for earlier releases.
|
||||
|
||||
## Why Nippy?
|
||||
|
||||
- Small, simple **pure-Clojure** library
|
||||
- **Terrific performance**: the [best](#performance) for Clojure that I'm aware of
|
||||
- Comprehensive support for [all standard data types](../../wiki/1-Getting-started#deserializing)
|
||||
- Easily extendable to [custom data types](../../wiki/1-Getting-started#custom-types)
|
||||
- **Robust test suite** incl. coverage of every supported type
|
||||
- **Mature** and widely used in production for 12+ years
|
||||
- Optional auto fallback to [Java Serializable](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*freeze-serializable-allowlist*) for [safe](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*freeze-serializable-allowlist*) types
|
||||
- Optional auto fallback to Clojure Reader (including tagged literals)
|
||||
- Optional smart **compression** with [LZ4](https://code.google.com/p/lz4/) or [Zstandard](https://facebook.github.io/zstd/)
|
||||
- Optional [encryption](../../wiki/1-Getting-started#encryption) with AES128
|
||||
- [Tools](https://taoensso.github.io/nippy/taoensso.nippy.tools.html) for easy + robust **integration into 3rd-party libraries**, etc.
|
||||
- Powerful [thaw transducer](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*thaw-xform*) for flexible data inspection and transformation
|
||||
|
||||
## Quick example
|
||||
|
||||
Nippy's super easy to use:
|
||||
|
||||
```clojure
|
||||
(require '[taoensso.nippy :as nippy])
|
||||
|
||||
;; Freeze any Clojure value
|
||||
(nippy/freeze <my-value>) ; => Serialized byte[]
|
||||
|
||||
;; Thaw the byte[] to get back the original value:
|
||||
(nippy/thaw (nippy/freeze <my-value>)) ; => <my-value>
|
||||
```
|
||||
|
||||
See the [wiki](https://github.com/taoensso/nippy/wiki/1-Getting-started#deserializing) for more.
|
||||
|
||||
## Operational considerations
|
||||
|
||||
### Data longevity
|
||||
|
||||
Nippy is widely used to store **long-lived** data and promises (as always) that **data serialized today should be readable by all future versions of Nippy**.
|
||||
|
||||
But please note that the **converse is not generally true**:
|
||||
|
||||
- Nippy `vX` **should** be able to read all data from Nippy `vY<=X` (backwards compatibility)
|
||||
- Nippy `vX` **may/not** be able to read all data from Nippy `vY>X` (forwards compatibility)
|
||||
|
||||
### Rolling updates and rollback
|
||||
|
||||
From time to time, Nippy may introduce:
|
||||
|
||||
- Support for serializing **new types**
|
||||
- Optimizations to the serialization of **pre-existing types**
|
||||
|
||||
To help ease **rolling updates** and to better support **rollback**, Nippy (since version v3.4) will always introduce such changes over **two version releases**:
|
||||
|
||||
- Release 1: to add **read support** for the new types
|
||||
- Release 2: to add **write support** for the new types
|
||||
|
||||
Starting from v3.4, Nippy's release notes will **always clearly indicate** if a particular update sequence is recommended.
|
||||
|
||||
### Stability of byte output
|
||||
|
||||
It has **never been an objective** of Nippy to offer **predictable byte output**, and I'd generally **recommend against** depending on specific byte output.
|
||||
|
||||
However, I know that a small minority of users *do* have specialized needs in this area.
|
||||
|
||||
So starting with Nippy v3.4, Nippy's release notes will **always clearly indicate** if any changes to byte output are expected.
|
||||
|
||||
## Performance
|
||||
|
||||
Nippy is fast! Latest [benchmark](../../blob/master/test/taoensso/nippy_benchmarks.clj) results:
|
||||
|
||||
|
||||
|
||||
PRs welcome to include other alternatives in the bench suite!
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Wiki][GitHub wiki] (getting started, usage, etc.)
|
||||
- API reference via [cljdoc][cljdoc]
|
||||
|
||||
## Funding
|
||||
|
||||
You can [help support][sponsor] continued work on this project, thank you!! 🙏
|
||||
|
||||
## License
|
||||
|
||||
Copyright © 2012-2025 [Peter Taoussanis][].
|
||||
Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).
|
||||
|
||||
<!-- Common -->
|
||||
|
||||
[GitHub releases]: ../../releases
|
||||
[GitHub issues]: ../../issues
|
||||
[GitHub wiki]: ../../wiki
|
||||
|
||||
[Peter Taoussanis]: https://www.taoensso.com
|
||||
[sponsor]: https://www.taoensso.com/sponsor
|
||||
|
||||
<!-- Project -->
|
||||
|
||||
[cljdoc]: https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy
|
||||
|
||||
[Clojars SVG]: https://img.shields.io/clojars/v/com.taoensso/nippy.svg
|
||||
[Clojars URL]: https://clojars.org/com.taoensso/nippy
|
||||
|
||||
[Main tests SVG]: https://github.com/taoensso/nippy/actions/workflows/main-tests.yml/badge.svg
|
||||
[Main tests URL]: https://github.com/taoensso/nippy/actions/workflows/main-tests.yml
|
||||
[Graal tests SVG]: https://github.com/taoensso/nippy/actions/workflows/graal-tests.yml/badge.svg
|
||||
[Graal tests URL]: https://github.com/taoensso/nippy/actions/workflows/graal-tests.yml
|
||||
13
SECURITY.md
13
SECURITY.md
|
|
@ -1,13 +0,0 @@
|
|||
# Security policy
|
||||
|
||||
## Advisories
|
||||
|
||||
All security advisories will be posted [on GitHub](https://github.com/taoensso/nippy/security/advisories).
|
||||
|
||||
## Reporting a vulnerability
|
||||
|
||||
Please report possible security vulnerabilities [via GitHub](https://github.com/taoensso/nippy/security/advisories), or by emailing me at `my first name at taoensso.com`. You may encrypt emails with [my public PGP/GPG key](https://www.taoensso.com/pgp).
|
||||
|
||||
Thank you!
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
10
bb.edn
10
bb.edn
|
|
@ -1,10 +0,0 @@
|
|||
{:paths ["bb"]
|
||||
:tasks
|
||||
{:requires ([graal-tests])
|
||||
graal-tests
|
||||
{:doc "Run Graal native-image tests"
|
||||
:task
|
||||
(do
|
||||
(graal-tests/uberjar)
|
||||
(graal-tests/native-image)
|
||||
(graal-tests/run-tests))}}}
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
#!/usr/bin/env bb
|
||||
|
||||
(ns graal-tests
|
||||
(:require
|
||||
[clojure.string :as str]
|
||||
[babashka.fs :as fs]
|
||||
[babashka.process :refer [shell]]))
|
||||
|
||||
(defn uberjar []
|
||||
(let [command "lein with-profiles +graal-tests uberjar"
|
||||
command
|
||||
(if (fs/windows?)
|
||||
(if (fs/which "lein")
|
||||
command
|
||||
;; Assume PowerShell powershell module
|
||||
(str "powershell.exe -command " (pr-str command)))
|
||||
command)]
|
||||
|
||||
(shell command)))
|
||||
|
||||
(defn executable [dir name]
|
||||
(-> (fs/glob dir (if (fs/windows?) (str name ".{exe,bat,cmd}") name))
|
||||
first
|
||||
fs/canonicalize
|
||||
str))
|
||||
|
||||
(defn native-image []
|
||||
(let [graalvm-home (System/getenv "GRAALVM_HOME")
|
||||
bin-dir (str (fs/file graalvm-home "bin"))]
|
||||
(shell (executable bin-dir "gu") "install" "native-image")
|
||||
(shell (executable bin-dir "native-image")
|
||||
"--features=clj_easy.graal_build_time.InitClojureClasses"
|
||||
"--no-fallback" "-jar" "target/graal-tests.jar" "graal_tests")))
|
||||
|
||||
(defn run-tests []
|
||||
(let [{:keys [out]} (shell {:out :string} (executable "." "graal_tests"))]
|
||||
(assert (str/includes? out "loaded") out)
|
||||
(println "Native image works!")))
|
||||
BIN
benchmarks.png
BIN
benchmarks.png
Binary file not shown.
|
Before Width: | Height: | Size: 68 KiB |
585
css/default.css
Normal file
585
css/default.css
Normal file
|
|
@ -0,0 +1,585 @@
|
|||
body {
|
||||
font-family: Helvetica, Arial, sans-serif;
|
||||
font-size: 15px;
|
||||
}
|
||||
|
||||
pre, code {
|
||||
font-family: Monaco, DejaVu Sans Mono, Consolas, monospace;
|
||||
font-size: 9pt;
|
||||
margin: 15px 0;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-weight: normal;
|
||||
font-size: 29px;
|
||||
margin: 10px 0 2px 0;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
h2 {
|
||||
font-weight: normal;
|
||||
font-size: 25px;
|
||||
}
|
||||
|
||||
h5.license {
|
||||
margin: 9px 0 22px 0;
|
||||
color: #555;
|
||||
font-weight: normal;
|
||||
font-size: 12px;
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
.document h1, .namespace-index h1 {
|
||||
font-size: 32px;
|
||||
margin-top: 12px;
|
||||
}
|
||||
|
||||
#header, #content, .sidebar {
|
||||
position: fixed;
|
||||
}
|
||||
|
||||
#header {
|
||||
top: 0;
|
||||
left: 0;
|
||||
right: 0;
|
||||
height: 22px;
|
||||
color: #f5f5f5;
|
||||
padding: 7px 10px;
|
||||
}
|
||||
|
||||
#content {
|
||||
top: 32px;
|
||||
right: 0;
|
||||
bottom: 0;
|
||||
overflow: auto;
|
||||
background: #fff;
|
||||
color: #333;
|
||||
padding: 0 18px;
|
||||
}
|
||||
|
||||
.sidebar {
|
||||
position: fixed;
|
||||
top: 32px;
|
||||
bottom: 0;
|
||||
overflow: auto;
|
||||
}
|
||||
|
||||
.sidebar.primary {
|
||||
background: #e2e2e2;
|
||||
border-right: solid 1px #cccccc;
|
||||
left: 0;
|
||||
width: 250px;
|
||||
}
|
||||
|
||||
.sidebar.secondary {
|
||||
background: #f2f2f2;
|
||||
border-right: solid 1px #d7d7d7;
|
||||
left: 251px;
|
||||
width: 200px;
|
||||
}
|
||||
|
||||
#content.namespace-index, #content.document {
|
||||
left: 251px;
|
||||
}
|
||||
|
||||
#content.namespace-docs {
|
||||
left: 452px;
|
||||
}
|
||||
|
||||
#content.document {
|
||||
padding-bottom: 10%;
|
||||
}
|
||||
|
||||
#header {
|
||||
background: #3f3f3f;
|
||||
border-bottom: 1px solid #cccccc;
|
||||
z-index: 100;
|
||||
}
|
||||
|
||||
#header h1 {
|
||||
display: block;
|
||||
float: left;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
font-size: 18px;
|
||||
font-weight: lighter;
|
||||
}
|
||||
|
||||
#header h1 .project-version {
|
||||
font-weight: normal;
|
||||
}
|
||||
|
||||
.project-version {
|
||||
padding-left: 0.15em;
|
||||
}
|
||||
|
||||
#header a, .sidebar a {
|
||||
display: block;
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
#header a {
|
||||
color: #f5f5f5;
|
||||
}
|
||||
|
||||
#langs {
|
||||
display: block;
|
||||
float: left;
|
||||
font-size: 16px;
|
||||
margin: 0 10px;
|
||||
}
|
||||
|
||||
#langs .lang {
|
||||
display: inline-block;
|
||||
margin: 0 2px;
|
||||
padding: 0 7px;
|
||||
background-color: #717171;
|
||||
border-radius: 6px;
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
#langs .lang.current {
|
||||
background-color: #a33;
|
||||
}
|
||||
|
||||
.sidebar a {
|
||||
color: #333;
|
||||
}
|
||||
|
||||
#header h2 {
|
||||
float: right;
|
||||
font-size: 9pt;
|
||||
font-weight: normal;
|
||||
margin: 4px 3px;
|
||||
padding: 0;
|
||||
color: #bbb;
|
||||
}
|
||||
|
||||
#header h2 a {
|
||||
display: inline;
|
||||
}
|
||||
|
||||
.sidebar h3 {
|
||||
margin: 0;
|
||||
padding: 10px 13px 0 13px;
|
||||
font-size: 19px;
|
||||
font-weight: lighter;
|
||||
}
|
||||
|
||||
.sidebar h3 a {
|
||||
color: #444;
|
||||
}
|
||||
|
||||
.sidebar h3.no-link {
|
||||
color: #636363;
|
||||
}
|
||||
|
||||
.sidebar ul {
|
||||
padding: 7px 0 6px 0;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.sidebar ul.index-link {
|
||||
padding-bottom: 4px;
|
||||
}
|
||||
|
||||
.sidebar li {
|
||||
display: block;
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.sidebar li a, .sidebar li .no-link {
|
||||
border-left: 3px solid transparent;
|
||||
padding: 0 10px;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.sidebar li .no-link {
|
||||
display: block;
|
||||
color: #777;
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
.sidebar li .inner {
|
||||
display: inline-block;
|
||||
padding-top: 7px;
|
||||
height: 24px;
|
||||
}
|
||||
|
||||
.sidebar li a, .sidebar li .tree {
|
||||
height: 31px;
|
||||
}
|
||||
|
||||
.depth-1 .inner { padding-left: 2px; }
|
||||
.depth-2 .inner { padding-left: 6px; }
|
||||
.depth-3 .inner { padding-left: 20px; }
|
||||
.depth-4 .inner { padding-left: 34px; }
|
||||
.depth-5 .inner { padding-left: 48px; }
|
||||
.depth-6 .inner { padding-left: 62px; }
|
||||
|
||||
.sidebar li .tree {
|
||||
display: block;
|
||||
float: left;
|
||||
position: relative;
|
||||
top: -10px;
|
||||
margin: 0 4px 0 0;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.sidebar li.depth-1 .tree {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.sidebar li .tree .top, .sidebar li .tree .bottom {
|
||||
display: block;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
width: 7px;
|
||||
}
|
||||
|
||||
.sidebar li .tree .top {
|
||||
border-left: 1px solid #aaa;
|
||||
border-bottom: 1px solid #aaa;
|
||||
height: 19px;
|
||||
}
|
||||
|
||||
.sidebar li .tree .bottom {
|
||||
height: 22px;
|
||||
}
|
||||
|
||||
.sidebar li.branch .tree .bottom {
|
||||
border-left: 1px solid #aaa;
|
||||
}
|
||||
|
||||
.sidebar.primary li.current a {
|
||||
border-left: 3px solid #a33;
|
||||
color: #a33;
|
||||
}
|
||||
|
||||
.sidebar.secondary li.current a {
|
||||
border-left: 3px solid #33a;
|
||||
color: #33a;
|
||||
}
|
||||
|
||||
.namespace-index h2 {
|
||||
margin: 30px 0 0 0;
|
||||
}
|
||||
|
||||
.namespace-index h3 {
|
||||
font-size: 16px;
|
||||
font-weight: bold;
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
.namespace-index .topics {
|
||||
padding-left: 30px;
|
||||
margin: 11px 0 0 0;
|
||||
}
|
||||
|
||||
.namespace-index .topics li {
|
||||
padding: 5px 0;
|
||||
}
|
||||
|
||||
.namespace-docs h3 {
|
||||
font-size: 18px;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.public h3 {
|
||||
margin: 0;
|
||||
float: left;
|
||||
}
|
||||
|
||||
.usage {
|
||||
clear: both;
|
||||
}
|
||||
|
||||
.public {
|
||||
margin: 0;
|
||||
border-top: 1px solid #e0e0e0;
|
||||
padding-top: 14px;
|
||||
padding-bottom: 6px;
|
||||
}
|
||||
|
||||
.public:last-child {
|
||||
margin-bottom: 20%;
|
||||
}
|
||||
|
||||
.members .public:last-child {
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
.members {
|
||||
margin: 15px 0;
|
||||
}
|
||||
|
||||
.members h4 {
|
||||
color: #555;
|
||||
font-weight: normal;
|
||||
font-variant: small-caps;
|
||||
margin: 0 0 5px 0;
|
||||
}
|
||||
|
||||
.members .inner {
|
||||
padding-top: 5px;
|
||||
padding-left: 12px;
|
||||
margin-top: 2px;
|
||||
margin-left: 7px;
|
||||
border-left: 1px solid #bbb;
|
||||
}
|
||||
|
||||
#content .members .inner h3 {
|
||||
font-size: 12pt;
|
||||
}
|
||||
|
||||
.members .public {
|
||||
border-top: none;
|
||||
margin-top: 0;
|
||||
padding-top: 6px;
|
||||
padding-bottom: 0;
|
||||
}
|
||||
|
||||
.members .public:first-child {
|
||||
padding-top: 0;
|
||||
}
|
||||
|
||||
h4.type,
|
||||
h4.dynamic,
|
||||
h4.added,
|
||||
h4.deprecated,
|
||||
h4.lang {
|
||||
float: left;
|
||||
margin: 3px 10px 15px 0;
|
||||
font-size: 15px;
|
||||
font-weight: bold;
|
||||
font-variant: small-caps;
|
||||
}
|
||||
|
||||
.public h4.type,
|
||||
.public h4.dynamic,
|
||||
.public h4.added,
|
||||
.public h4.deprecated,
|
||||
.public h4.lang {
|
||||
font-size: 13px;
|
||||
font-weight: bold;
|
||||
margin: 3px 0 0 10px;
|
||||
}
|
||||
|
||||
.members h4.type,
|
||||
.members h4.added,
|
||||
.members h4.deprecated,
|
||||
.members h4.lang {
|
||||
margin-top: 1px;
|
||||
}
|
||||
|
||||
h4.type {
|
||||
color: #717171;
|
||||
}
|
||||
|
||||
h4.dynamic {
|
||||
color: #9933aa;
|
||||
}
|
||||
|
||||
h4.added {
|
||||
color: #508820;
|
||||
}
|
||||
|
||||
h4.deprecated {
|
||||
color: #880000;
|
||||
}
|
||||
|
||||
.public h4.lang, .public h4.lang a {
|
||||
font-weight: normal;
|
||||
font-variant: normal;
|
||||
color: #717171;
|
||||
}
|
||||
|
||||
.public h4.lang.current {
|
||||
color: #a33;
|
||||
}
|
||||
|
||||
.namespace {
|
||||
margin-bottom: 30px;
|
||||
}
|
||||
|
||||
.namespace:last-child {
|
||||
margin-bottom: 10%;
|
||||
}
|
||||
|
||||
.index {
|
||||
padding: 0;
|
||||
font-size: 80%;
|
||||
margin: 15px 0;
|
||||
line-height: 16px;
|
||||
}
|
||||
|
||||
.index * {
|
||||
display: inline;
|
||||
}
|
||||
|
||||
.index p {
|
||||
padding-right: 3px;
|
||||
}
|
||||
|
||||
.index li {
|
||||
padding-right: 5px;
|
||||
}
|
||||
|
||||
.index ul {
|
||||
padding-left: 0;
|
||||
}
|
||||
|
||||
.type-sig {
|
||||
clear: both;
|
||||
color: #088;
|
||||
}
|
||||
|
||||
.type-sig pre {
|
||||
padding-top: 10px;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.usage code {
|
||||
display: block;
|
||||
color: #008;
|
||||
margin: 2px 0;
|
||||
}
|
||||
|
||||
.usage code:first-child {
|
||||
padding-top: 10px;
|
||||
}
|
||||
|
||||
p {
|
||||
margin: 15px 0;
|
||||
}
|
||||
|
||||
.public p:first-child, .public pre.plaintext {
|
||||
margin-top: 12px;
|
||||
}
|
||||
|
||||
.doc {
|
||||
margin: 0 0 26px 0;
|
||||
clear: both;
|
||||
}
|
||||
|
||||
.public .doc {
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.namespace-index .doc {
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
.namespace-index .namespace .doc {
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
.markdown p, .markdown li, .markdown dt, .markdown dd, .markdown td {
|
||||
line-height: 22px;
|
||||
}
|
||||
|
||||
.markdown li {
|
||||
padding: 2px 0;
|
||||
}
|
||||
|
||||
.markdown h2 {
|
||||
font-weight: normal;
|
||||
font-size: 25px;
|
||||
margin: 30px 0 10px 0;
|
||||
}
|
||||
|
||||
.markdown h3 {
|
||||
font-weight: normal;
|
||||
font-size: 20px;
|
||||
margin: 30px 0 0 0;
|
||||
}
|
||||
|
||||
.markdown h4 {
|
||||
font-size: 15px;
|
||||
margin: 22px 0 -4px 0;
|
||||
}
|
||||
|
||||
.doc, .public, .namespace .index {
|
||||
max-width: 680px;
|
||||
overflow-x: visible;
|
||||
}
|
||||
|
||||
.markdown pre > code {
|
||||
display: block;
|
||||
padding: 10px;
|
||||
}
|
||||
|
||||
.markdown pre > code, .src-link a {
|
||||
border: 1px solid #e4e4e4;
|
||||
border-radius: 2px;
|
||||
}
|
||||
|
||||
.markdown code:not(.hljs), .src-link a {
|
||||
background: #f6f6f6;
|
||||
}
|
||||
|
||||
pre.deps {
|
||||
display: inline-block;
|
||||
margin: 0 10px;
|
||||
border: 1px solid #e4e4e4;
|
||||
border-radius: 2px;
|
||||
padding: 10px;
|
||||
background-color: #f6f6f6;
|
||||
}
|
||||
|
||||
.markdown hr {
|
||||
border-style: solid;
|
||||
border-top: none;
|
||||
color: #ccc;
|
||||
}
|
||||
|
||||
.doc ul, .doc ol {
|
||||
padding-left: 30px;
|
||||
}
|
||||
|
||||
.doc table {
|
||||
border-collapse: collapse;
|
||||
margin: 0 10px;
|
||||
}
|
||||
|
||||
.doc table td, .doc table th {
|
||||
border: 1px solid #dddddd;
|
||||
padding: 4px 6px;
|
||||
}
|
||||
|
||||
.doc table th {
|
||||
background: #f2f2f2;
|
||||
}
|
||||
|
||||
.doc dl {
|
||||
margin: 0 10px 20px 10px;
|
||||
}
|
||||
|
||||
.doc dl dt {
|
||||
font-weight: bold;
|
||||
margin: 0;
|
||||
padding: 3px 0;
|
||||
border-bottom: 1px solid #ddd;
|
||||
}
|
||||
|
||||
.doc dl dd {
|
||||
padding: 5px 0;
|
||||
margin: 0 0 5px 10px;
|
||||
}
|
||||
|
||||
.doc abbr {
|
||||
border-bottom: 1px dotted #333;
|
||||
font-variant: none;
|
||||
cursor: help;
|
||||
}
|
||||
|
||||
.src-link {
|
||||
margin-bottom: 15px;
|
||||
}
|
||||
|
||||
.src-link a {
|
||||
font-size: 70%;
|
||||
padding: 1px 4px;
|
||||
text-decoration: none;
|
||||
color: #5555bb;
|
||||
}
|
||||
97
css/highlight.css
Normal file
97
css/highlight.css
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
/*
|
||||
github.com style (c) Vasily Polovnyov <vast@whiteants.net>
|
||||
*/
|
||||
|
||||
.hljs {
|
||||
display: block;
|
||||
overflow-x: auto;
|
||||
padding: 0.5em;
|
||||
color: #333;
|
||||
background: #f8f8f8;
|
||||
}
|
||||
|
||||
.hljs-comment,
|
||||
.hljs-quote {
|
||||
color: #998;
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
.hljs-keyword,
|
||||
.hljs-selector-tag,
|
||||
.hljs-subst {
|
||||
color: #333;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.hljs-number,
|
||||
.hljs-literal,
|
||||
.hljs-variable,
|
||||
.hljs-template-variable,
|
||||
.hljs-tag .hljs-attr {
|
||||
color: #008080;
|
||||
}
|
||||
|
||||
.hljs-string,
|
||||
.hljs-doctag {
|
||||
color: #d14;
|
||||
}
|
||||
|
||||
.hljs-title,
|
||||
.hljs-section,
|
||||
.hljs-selector-id {
|
||||
color: #900;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.hljs-subst {
|
||||
font-weight: normal;
|
||||
}
|
||||
|
||||
.hljs-type,
|
||||
.hljs-class .hljs-title {
|
||||
color: #458;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.hljs-tag,
|
||||
.hljs-name,
|
||||
.hljs-attribute {
|
||||
color: #000080;
|
||||
font-weight: normal;
|
||||
}
|
||||
|
||||
.hljs-regexp,
|
||||
.hljs-link {
|
||||
color: #009926;
|
||||
}
|
||||
|
||||
.hljs-symbol,
|
||||
.hljs-bullet {
|
||||
color: #990073;
|
||||
}
|
||||
|
||||
.hljs-built_in,
|
||||
.hljs-builtin-name {
|
||||
color: #0086b3;
|
||||
}
|
||||
|
||||
.hljs-meta {
|
||||
color: #999;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.hljs-deletion {
|
||||
background: #fdd;
|
||||
}
|
||||
|
||||
.hljs-addition {
|
||||
background: #dfd;
|
||||
}
|
||||
|
||||
.hljs-emphasis {
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
.hljs-strong {
|
||||
font-weight: bold;
|
||||
}
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
{:cljdoc/docstring-format :plaintext}
|
||||
|
||||
4
index.html
Normal file
4
index.html
Normal file
File diff suppressed because one or more lines are too long
2
js/highlight.min.js
vendored
Normal file
2
js/highlight.min.js
vendored
Normal file
File diff suppressed because one or more lines are too long
4
js/jquery.min.js
vendored
Normal file
4
js/jquery.min.js
vendored
Normal file
File diff suppressed because one or more lines are too long
112
js/page_effects.js
Normal file
112
js/page_effects.js
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
function visibleInParent(element) {
|
||||
var position = $(element).position().top
|
||||
return position > -50 && position < ($(element).offsetParent().height() - 50)
|
||||
}
|
||||
|
||||
function hasFragment(link, fragment) {
|
||||
return $(link).attr("href").indexOf("#" + fragment) != -1
|
||||
}
|
||||
|
||||
function findLinkByFragment(elements, fragment) {
|
||||
return $(elements).filter(function(i, e) { return hasFragment(e, fragment)}).first()
|
||||
}
|
||||
|
||||
function scrollToCurrentVarLink(elements) {
|
||||
var elements = $(elements);
|
||||
var parent = elements.offsetParent();
|
||||
|
||||
if (elements.length == 0) return;
|
||||
|
||||
var top = elements.first().position().top;
|
||||
var bottom = elements.last().position().top + elements.last().height();
|
||||
|
||||
if (top >= 0 && bottom <= parent.height()) return;
|
||||
|
||||
if (top < 0) {
|
||||
parent.scrollTop(parent.scrollTop() + top);
|
||||
}
|
||||
else if (bottom > parent.height()) {
|
||||
parent.scrollTop(parent.scrollTop() + bottom - parent.height());
|
||||
}
|
||||
}
|
||||
|
||||
function setCurrentVarLink() {
|
||||
$('.secondary a').parent().removeClass('current')
|
||||
$('.anchor').
|
||||
filter(function(index) { return visibleInParent(this) }).
|
||||
each(function(index, element) {
|
||||
findLinkByFragment(".secondary a", element.id).
|
||||
parent().
|
||||
addClass('current')
|
||||
});
|
||||
scrollToCurrentVarLink('.secondary .current');
|
||||
}
|
||||
|
||||
var hasStorage = (function() { try { return localStorage.getItem } catch(e) {} }())
|
||||
|
||||
function scrollPositionId(element) {
|
||||
var directory = window.location.href.replace(/[^\/]+\.html$/, '')
|
||||
return 'scroll::' + $(element).attr('id') + '::' + directory
|
||||
}
|
||||
|
||||
function storeScrollPosition(element) {
|
||||
if (!hasStorage) return;
|
||||
localStorage.setItem(scrollPositionId(element) + "::x", $(element).scrollLeft())
|
||||
localStorage.setItem(scrollPositionId(element) + "::y", $(element).scrollTop())
|
||||
}
|
||||
|
||||
function recallScrollPosition(element) {
|
||||
if (!hasStorage) return;
|
||||
$(element).scrollLeft(localStorage.getItem(scrollPositionId(element) + "::x"))
|
||||
$(element).scrollTop(localStorage.getItem(scrollPositionId(element) + "::y"))
|
||||
}
|
||||
|
||||
function persistScrollPosition(element) {
|
||||
recallScrollPosition(element)
|
||||
$(element).scroll(function() { storeScrollPosition(element) })
|
||||
}
|
||||
|
||||
function sidebarContentWidth(element) {
|
||||
var widths = $(element).find('.inner').map(function() { return $(this).innerWidth() })
|
||||
return Math.max.apply(Math, widths)
|
||||
}
|
||||
|
||||
function calculateSize(width, snap, margin, minimum) {
|
||||
if (width == 0) {
|
||||
return 0
|
||||
}
|
||||
else {
|
||||
return Math.max(minimum, (Math.ceil(width / snap) * snap) + (margin * 2))
|
||||
}
|
||||
}
|
||||
|
||||
function resizeSidebars() {
|
||||
var primaryWidth = sidebarContentWidth('.primary')
|
||||
var secondaryWidth = 0
|
||||
|
||||
if ($('.secondary').length != 0) {
|
||||
secondaryWidth = sidebarContentWidth('.secondary')
|
||||
}
|
||||
|
||||
// snap to grid
|
||||
primaryWidth = calculateSize(primaryWidth, 32, 13, 160)
|
||||
secondaryWidth = calculateSize(secondaryWidth, 32, 13, 160)
|
||||
|
||||
$('.primary').css('width', primaryWidth)
|
||||
$('.secondary').css('width', secondaryWidth).css('left', primaryWidth + 1)
|
||||
|
||||
if (secondaryWidth > 0) {
|
||||
$('#content').css('left', primaryWidth + secondaryWidth + 2)
|
||||
}
|
||||
else {
|
||||
$('#content').css('left', primaryWidth + 1)
|
||||
}
|
||||
}
|
||||
|
||||
$(window).ready(resizeSidebars)
|
||||
$(window).ready(setCurrentVarLink)
|
||||
$(window).ready(function() { persistScrollPosition('.primary')})
|
||||
$(window).ready(function() {
|
||||
$('#content').scroll(setCurrentVarLink)
|
||||
$(window).resize(setCurrentVarLink)
|
||||
})
|
||||
64
project.clj
64
project.clj
|
|
@ -1,64 +0,0 @@
|
|||
(defproject com.taoensso/nippy "3.5.0"
|
||||
:author "Peter Taoussanis <https://www.taoensso.com>"
|
||||
:description "Fast serialization library for Clojure"
|
||||
:url "https://www.taoensso.com/nippy"
|
||||
|
||||
:license
|
||||
{:name "Eclipse Public License - v 1.0"
|
||||
:url "https://www.eclipse.org/legal/epl-v10.html"}
|
||||
|
||||
:dependencies
|
||||
[[org.clojure/tools.reader "1.5.2"]
|
||||
[com.taoensso/encore "3.142.0"]
|
||||
[org.tukaani/xz "1.10"]
|
||||
[io.airlift/aircompressor "2.0.2"]]
|
||||
|
||||
:test-paths ["test" #_"src"]
|
||||
|
||||
:profiles
|
||||
{;; :default [:base :system :user :provided :dev]
|
||||
:provided {:dependencies [[org.clojure/clojure "1.11.4"]]}
|
||||
:c1.12 {:dependencies [[org.clojure/clojure "1.12.0"]]}
|
||||
:c1.11 {:dependencies [[org.clojure/clojure "1.11.4"]]}
|
||||
:c1.10 {:dependencies [[org.clojure/clojure "1.10.3"]]}
|
||||
|
||||
:graal-tests
|
||||
{:source-paths ["test"]
|
||||
:main taoensso.graal-tests
|
||||
:aot [taoensso.graal-tests]
|
||||
:uberjar-name "graal-tests.jar"
|
||||
:dependencies
|
||||
[[org.clojure/clojure "1.11.3"]
|
||||
[com.github.clj-easy/graal-build-time "1.0.5"]]}
|
||||
|
||||
:dev
|
||||
{:jvm-opts
|
||||
["-server"
|
||||
"-Xms1024m" "-Xmx2048m"
|
||||
"-Dtaoensso.elide-deprecated=true"
|
||||
"-Dtaoensso.nippy.thaw-serializable-allowlist-base=base.1, base.2"
|
||||
"-Dtaoensso.nippy.thaw-serializable-allowlist-add=add.1 , add.2"
|
||||
#_"-Dtaoensso.nippy.target-release=320"
|
||||
#_"-Dtaoensso.nippy.target-release=350"]
|
||||
|
||||
:global-vars
|
||||
{*warn-on-reflection* true
|
||||
*assert* true
|
||||
*unchecked-math* false #_:warn-on-boxed}
|
||||
|
||||
:dependencies
|
||||
[[org.clojure/test.check "1.1.1"]
|
||||
[org.clojure/data.fressian "1.1.0"]]
|
||||
|
||||
:plugins
|
||||
[[lein-pprint "1.3.2"]
|
||||
[lein-ancient "0.7.0"]]}}
|
||||
|
||||
:aliases
|
||||
{"start-dev" ["with-profile" "+dev" "repl" ":headless"]
|
||||
;; "build-once" ["do" ["clean"] ["cljsbuild" "once"]]
|
||||
"deploy-lib" ["do" #_["build-once"] ["deploy" "clojars"] ["install"]]
|
||||
|
||||
"test-clj" ["with-profile" "+c1.12:+c1.11:+c1.10" "test"]
|
||||
;; "test-cljs" ["with-profile" "+c1.12" "cljsbuild" "test"]
|
||||
"test-all" ["do" ["clean"] ["test-clj"] #_["test-cljs"]]})
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
Args=--initialize-at-build-time=clojure,taoensso \
|
||||
-H:SerializationConfigurationResources=${.}/serialization-config.json \
|
||||
-H:ReflectionConfigurationResources=${.}/reflection-config.json
|
||||
|
|
@ -1,82 +0,0 @@
|
|||
[
|
||||
{
|
||||
"name":"net.jpountz.lz4.LZ4HCJNICompressor",
|
||||
"fields":[
|
||||
{
|
||||
"name":"INSTANCE"
|
||||
}
|
||||
],
|
||||
"methods":[
|
||||
{
|
||||
"name":"<init>",
|
||||
"parameterTypes":[
|
||||
"int"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name":"net.jpountz.lz4.LZ4JNICompressor",
|
||||
"fields":[
|
||||
{
|
||||
"name":"INSTANCE"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name":"net.jpountz.lz4.LZ4JNIFastDecompressor",
|
||||
"fields":[
|
||||
{
|
||||
"name":"INSTANCE"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name":"net.jpountz.lz4.LZ4JNISafeDecompressor",
|
||||
"fields":[
|
||||
{
|
||||
"name":"INSTANCE"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name":"net.jpountz.lz4.LZ4JavaSafeCompressor",
|
||||
"fields":[
|
||||
{
|
||||
"name":"INSTANCE"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name":"net.jpountz.lz4.LZ4HCJavaSafeCompressor",
|
||||
"fields":[
|
||||
{
|
||||
"name":"INSTANCE"
|
||||
}
|
||||
],
|
||||
"methods":[
|
||||
{
|
||||
"name":"<init>",
|
||||
"parameterTypes":[
|
||||
"int"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name":"net.jpountz.lz4.LZ4JavaSafeFastDecompressor",
|
||||
"fields":[
|
||||
{
|
||||
"name":"INSTANCE"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name":"net.jpountz.lz4.LZ4JavaSafeSafeDecompressor",
|
||||
"fields":[
|
||||
{
|
||||
"name":"INSTANCE"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
|
@ -1,96 +0,0 @@
|
|||
[
|
||||
{
|
||||
"name":"clojure.lang.APersistentMap"
|
||||
},
|
||||
{
|
||||
"name":"clojure.lang.ExceptionInfo"
|
||||
},
|
||||
{
|
||||
"name":"clojure.lang.ExceptionInfo",
|
||||
"customTargetConstructorClass":"java.lang.Object"
|
||||
},
|
||||
{
|
||||
"name":"clojure.lang.Keyword"
|
||||
},
|
||||
{
|
||||
"name":"clojure.lang.Keyword",
|
||||
"customTargetConstructorClass":"java.lang.Object"
|
||||
},
|
||||
{
|
||||
"name":"clojure.lang.PersistentArrayMap"
|
||||
},
|
||||
{
|
||||
"name":"clojure.lang.PersistentArrayMap",
|
||||
"customTargetConstructorClass":"clojure.lang.AFn"
|
||||
},
|
||||
{
|
||||
"name":"clojure.lang.Symbol"
|
||||
},
|
||||
{
|
||||
"name":"clojure.lang.Symbol",
|
||||
"customTargetConstructorClass":"clojure.lang.AFn"
|
||||
},
|
||||
{
|
||||
"name":"java.lang.ArithmeticException"
|
||||
},
|
||||
{
|
||||
"name":"java.lang.ArithmeticException",
|
||||
"customTargetConstructorClass":"java.lang.Object"
|
||||
},
|
||||
{
|
||||
"name":"java.lang.Exception"
|
||||
},
|
||||
{
|
||||
"name":"java.lang.Object[]"
|
||||
},
|
||||
{
|
||||
"name":"java.lang.RuntimeException"
|
||||
},
|
||||
{
|
||||
"name":"java.lang.StackTraceElement"
|
||||
},
|
||||
{
|
||||
"name":"java.lang.StackTraceElement",
|
||||
"customTargetConstructorClass":"java.lang.Object"
|
||||
},
|
||||
{
|
||||
"name":"java.lang.StackTraceElement[]"
|
||||
},
|
||||
{
|
||||
"name":"java.lang.String"
|
||||
},
|
||||
{
|
||||
"name":"java.lang.String",
|
||||
"customTargetConstructorClass":"java.lang.Object"
|
||||
},
|
||||
{
|
||||
"name":"java.lang.Throwable"
|
||||
},
|
||||
{
|
||||
"name":"java.util.ArrayList"
|
||||
},
|
||||
{
|
||||
"name":"java.util.ArrayList",
|
||||
"customTargetConstructorClass":"java.util.AbstractList"
|
||||
},
|
||||
{
|
||||
"name":"java.util.Collections$UnmodifiableCollection"
|
||||
},
|
||||
{
|
||||
"name":"java.util.Collections$UnmodifiableList"
|
||||
},
|
||||
{
|
||||
"name":"java.util.Collections$UnmodifiableRandomAccessList"
|
||||
},
|
||||
{
|
||||
"name":"java.util.Collections$UnmodifiableRandomAccessList",
|
||||
"customTargetConstructorClass":"java.lang.Object"
|
||||
},
|
||||
{
|
||||
"name":"java.util.Collections$EmptyList",
|
||||
"customTargetConstructorClass":"java.util.AbstractList"
|
||||
},
|
||||
{
|
||||
"name":"clojure.lang.APersistentMap"
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,218 +0,0 @@
|
|||
(ns ^:no-doc taoensso.nippy.compression
|
||||
"Private, implementation detail."
|
||||
(:require
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc])
|
||||
(:import
|
||||
[java.nio ByteBuffer]
|
||||
[java.io
|
||||
ByteArrayInputStream ByteArrayOutputStream
|
||||
DataInputStream DataOutputStream]))
|
||||
|
||||
;;;; Interface
|
||||
|
||||
(defprotocol ICompressor
|
||||
(header-id [compressor])
|
||||
(compress ^bytes [compressor ba])
|
||||
(decompress ^bytes [compressor ba]))
|
||||
|
||||
(def ^:const standard-header-ids
|
||||
"These support `:auto` thaw."
|
||||
#{:zstd :lz4 #_:lzo :lzma2 :snappy})
|
||||
|
||||
;;;; Misc utils
|
||||
|
||||
(defn- int-size->ba ^bytes [size]
|
||||
(let [ba (byte-array 4)
|
||||
baos (ByteArrayOutputStream. 4)
|
||||
dos (DataOutputStream. baos)]
|
||||
(.writeInt dos (int size))
|
||||
(.toByteArray baos)))
|
||||
|
||||
(defn- ba->int-size [ba]
|
||||
(let [bais (ByteArrayInputStream. ba)
|
||||
dis (DataInputStream. bais)]
|
||||
(.readInt dis)))
|
||||
|
||||
(comment (ba->int-size (int-size->ba 3737)))
|
||||
|
||||
;;;; Airlift
|
||||
|
||||
(defn- airlift-compress
|
||||
^bytes [^io.airlift.compress.Compressor c ^bytes ba prepend-size?]
|
||||
(let [in-len (alength ba)
|
||||
max-out-len (.maxCompressedLength c in-len)]
|
||||
|
||||
(if prepend-size?
|
||||
(let [ba-max-out (byte-array (int (+ 4 max-out-len)))
|
||||
int-size-ba (int-size->ba in-len)
|
||||
_ (System/arraycopy int-size-ba 0 ba-max-out 0 4)
|
||||
out-len
|
||||
(.compress c
|
||||
ba 0 in-len
|
||||
ba-max-out 4 max-out-len)]
|
||||
|
||||
(if (== out-len max-out-len)
|
||||
(do ba-max-out)
|
||||
(java.util.Arrays/copyOfRange ba-max-out 0 (+ 4 out-len))))
|
||||
|
||||
(let [ba-max-out (byte-array max-out-len)
|
||||
out-len
|
||||
(.compress c
|
||||
ba 0 in-len
|
||||
ba-max-out 0 max-out-len)]
|
||||
|
||||
(if (== out-len max-out-len)
|
||||
(do ba-max-out)
|
||||
(java.util.Arrays/copyOfRange ba-max-out 0 out-len))))))
|
||||
|
||||
(defn- airlift-decompress
|
||||
^bytes [^io.airlift.compress.Decompressor d ^bytes ba max-out-len]
|
||||
(if max-out-len
|
||||
(let [max-out-len (int max-out-len)
|
||||
ba-max-out (byte-array max-out-len)
|
||||
out-len
|
||||
(.decompress d
|
||||
ba 0 (alength ba)
|
||||
ba-max-out 0 max-out-len)]
|
||||
|
||||
(if (== out-len max-out-len)
|
||||
(do ba-max-out)
|
||||
(java.util.Arrays/copyOfRange ba-max-out 0 out-len)))
|
||||
|
||||
;; Prepended size
|
||||
(let [out-len (ba->int-size ba)
|
||||
ba-out (byte-array (int out-len))]
|
||||
(.decompress d
|
||||
ba 4 (- (alength ba) 4)
|
||||
ba-out 0 out-len)
|
||||
ba-out)))
|
||||
|
||||
(do
|
||||
(enc/def* ^:private airlift-zstd-compressor_ (enc/thread-local (io.airlift.compress.zstd.ZstdCompressor.)))
|
||||
(enc/def* ^:private airlift-zstd-decompressor_ (enc/thread-local (io.airlift.compress.zstd.ZstdDecompressor.)))
|
||||
(deftype ZstdCompressor [prepend-size?]
|
||||
ICompressor
|
||||
(header-id [_] :zstd)
|
||||
(compress [_ ba] (airlift-compress @airlift-zstd-compressor_ ba prepend-size?))
|
||||
(decompress [_ ba] (airlift-decompress @airlift-zstd-decompressor_ ba
|
||||
(when-not prepend-size?
|
||||
(io.airlift.compress.zstd.ZstdDecompressor/getDecompressedSize ba
|
||||
0 (alength ^bytes ba)))))))
|
||||
|
||||
(do
|
||||
(enc/def* ^:private airlift-lz4-compressor_ (enc/thread-local (io.airlift.compress.lz4.Lz4Compressor.)))
|
||||
(enc/def* ^:private airlift-lz4-decompressor_ (enc/thread-local (io.airlift.compress.lz4.Lz4Decompressor.)))
|
||||
(deftype LZ4Compressor []
|
||||
ICompressor
|
||||
(header-id [_] :lz4)
|
||||
(compress [_ ba] (airlift-compress @airlift-lz4-compressor_ ba true))
|
||||
(decompress [_ ba] (airlift-decompress @airlift-lz4-decompressor_ ba nil))))
|
||||
|
||||
(do
|
||||
(enc/def* ^:private airlift-lzo-compressor_ (enc/thread-local (io.airlift.compress.lzo.LzoCompressor.)))
|
||||
(enc/def* ^:private airlift-lzo-decompressor_ (enc/thread-local (io.airlift.compress.lzo.LzoDecompressor.)))
|
||||
(deftype LZOCompressor []
|
||||
ICompressor
|
||||
(header-id [_] :lzo)
|
||||
(compress [_ ba] (airlift-compress @airlift-lzo-compressor_ ba true))
|
||||
(decompress [_ ba] (airlift-decompress @airlift-lzo-decompressor_ ba nil))))
|
||||
|
||||
(do
|
||||
(enc/def* ^:private airlift-snappy-compressor_ (enc/thread-local (io.airlift.compress.snappy.SnappyCompressor.)))
|
||||
(enc/def* ^:private airlift-snappy-decompressor_ (enc/thread-local (io.airlift.compress.snappy.SnappyDecompressor.)))
|
||||
(deftype SnappyCompressor [prepend-size?]
|
||||
ICompressor
|
||||
(header-id [_] :snappy)
|
||||
(compress [_ ba] (airlift-compress @airlift-snappy-compressor_ ba prepend-size?))
|
||||
(decompress [_ ba] (airlift-decompress @airlift-snappy-decompressor_ ba
|
||||
(when-not prepend-size?
|
||||
(io.airlift.compress.snappy.SnappyDecompressor/getUncompressedLength ba 0))))))
|
||||
|
||||
;;;; LZMA2
|
||||
|
||||
(deftype LZMA2Compressor [compression-level]
|
||||
;; Compression level ∈ℕ[0,9] (low->high) with 6 LZMA2 default (we use 0)
|
||||
ICompressor
|
||||
(header-id [_] :lzma2)
|
||||
(compress [_ ba]
|
||||
(let [baos (ByteArrayOutputStream.)
|
||||
dos (DataOutputStream. baos)
|
||||
;;
|
||||
len-decomp (alength ^bytes ba)
|
||||
;; Prefix with uncompressed length:
|
||||
_ (.writeInt dos len-decomp)
|
||||
xzs (org.tukaani.xz.XZOutputStream. baos
|
||||
(org.tukaani.xz.LZMA2Options. compression-level))]
|
||||
(.write xzs ^bytes ba)
|
||||
(.close xzs)
|
||||
(.toByteArray baos)))
|
||||
|
||||
(decompress [_ ba]
|
||||
(let [bais (ByteArrayInputStream. ba)
|
||||
dis (DataInputStream. bais)
|
||||
;;
|
||||
len-decomp (.readInt dis)
|
||||
ba (byte-array len-decomp)
|
||||
xzs (org.tukaani.xz.XZInputStream. bais)]
|
||||
(.read xzs ba 0 len-decomp)
|
||||
(if (== -1 (.read xzs)) ; Good practice as extra safety measure
|
||||
nil
|
||||
(truss/ex-info! "LZMA2 Decompress failed: corrupt data?" {:ba ba}))
|
||||
ba)))
|
||||
|
||||
;;;; Public API
|
||||
|
||||
(def zstd-compressor
|
||||
"Default `Zstd` (`Zstandard`) compressor:
|
||||
- Compression ratio: `B` (0.53 on reference benchmark).
|
||||
- Compression speed: `C` (1300 msecs on reference benchmark).
|
||||
- Decompression speed: `B` (400 msecs on reference benchmark).
|
||||
|
||||
Good general-purpose compressor, balances ratio & speed.
|
||||
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
|
||||
(ZstdCompressor. false))
|
||||
|
||||
(def lz4-compressor
|
||||
"Default `LZ4` compressor:
|
||||
- Compression ratio: `C` (0.58 on reference benchmark).
|
||||
- Compression speed: `A` (240 msecs on reference benchmark).
|
||||
- Decompression speed: `A+` (30 msecs on reference benchmark).
|
||||
|
||||
Good general-purpose compressor, favours speed.
|
||||
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
|
||||
(LZ4Compressor.))
|
||||
|
||||
(def lzo-compressor
|
||||
"Default `LZO` compressor:
|
||||
- Compression ratio: `C` (0.58 on reference benchmark).
|
||||
- Compression speed: `A` (220 msecs on reference benchmark).
|
||||
- Decompression speed: `A` (40 msecs on reference benchmark).
|
||||
|
||||
Good general-purpose compressor, favours speed.
|
||||
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
|
||||
(LZOCompressor.))
|
||||
|
||||
(def lzma2-compressor
|
||||
"Default `LZMA2` compressor:
|
||||
- Compression ratio: `A+` (0.4 on reference benchmark).
|
||||
- Compression speed: `E` (18.5 secs on reference benchmark).
|
||||
- Decompression speed: `D` (12 secs on reference benchmark).
|
||||
|
||||
Specialized compressor, strongly favours ratio.
|
||||
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
|
||||
(LZMA2Compressor. 0))
|
||||
|
||||
(enc/def* snappy-compressor
|
||||
"Default `Snappy` compressor:
|
||||
- Compression ratio: `C` (0.58 on reference benchmark).
|
||||
- Compression speed: `A+` (210 msecs on reference benchmark).
|
||||
- Decompression speed: `B` (130 msecs on reference benchmark).
|
||||
Good general-purpose compressor, favours speed.
|
||||
See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
|
||||
(SnappyCompressor. false))
|
||||
|
||||
(enc/def* ^:no-doc lz4hc-compressor
|
||||
"Different LZ4 modes no longer supported, prefer `lz4-compressor`."
|
||||
{:deprecated "v3.4.0-RC1 (2024-02-06)"}
|
||||
(LZ4Compressor.))
|
||||
|
|
@ -1,155 +0,0 @@
|
|||
(ns ^:no-doc taoensso.nippy.crypto
|
||||
"Low-level crypto utils.
|
||||
Private & alpha, very likely to change!"
|
||||
(:refer-clojure :exclude [rand-nth])
|
||||
(:require
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc]))
|
||||
|
||||
;; Note that AES128 may be preferable to AES256 due to known attack
|
||||
;; vectors specific to AES256, Ref. <https://goo.gl/qU4CCV>
|
||||
;; or for a counter argument, Ref. <https://goo.gl/9LA9Yb>
|
||||
|
||||
;;;; Randomness
|
||||
|
||||
(do
|
||||
(defn rand-nth [coll] (nth coll (int (* (.nextDouble (enc/secure-rng)) (count coll)))))
|
||||
(defn rand-bytes ^bytes [size] (let [ba (byte-array size)] (.nextBytes (enc/secure-rng) ba) ba))
|
||||
(defn rand-double ^double [] (.nextDouble (enc/secure-rng)))
|
||||
(defn rand-gauss ^double [] (.nextGaussian (enc/secure-rng)))
|
||||
(defn rand-bool [] (.nextBoolean (enc/secure-rng)))
|
||||
(defn rand-long
|
||||
(^long [ ] (.nextLong (enc/secure-rng)))
|
||||
(^long [n] (long (* (long n) (.nextDouble (enc/secure-rng)))))))
|
||||
|
||||
(comment
|
||||
(seq (rand-bytes 16))
|
||||
(rand-nth [:a :b :c :d])
|
||||
(rand-long 100))
|
||||
|
||||
;;;; Hashing
|
||||
|
||||
(def ^:private sha256-md* (enc/thread-local-proxy (java.security.MessageDigest/getInstance "SHA-256")))
|
||||
(def ^:private sha512-md* (enc/thread-local-proxy (java.security.MessageDigest/getInstance "SHA-512")))
|
||||
(defn sha256-md ^java.security.MessageDigest [] (.get ^ThreadLocal sha256-md*))
|
||||
(defn sha512-md ^java.security.MessageDigest [] (.get ^ThreadLocal sha512-md*))
|
||||
(defn sha256-ba ^bytes [ba] (.digest (sha256-md) ba))
|
||||
(defn sha512-ba ^bytes [ba] (.digest (sha512-md) ba))
|
||||
|
||||
(enc/compile-if clojure.lang.Murmur3
|
||||
(defn murmur3 [^String s] (clojure.lang.Murmur3/hashUnencodedChars s))
|
||||
nil)
|
||||
|
||||
;;;; Key derivation (salt+password -> key / hash)
|
||||
;; (fn [salt-ba utf8]) -> bytes
|
||||
|
||||
;; (defn ba->hex [^bytes ba] (org.apache.commons.codec.binary.Hex/encodeHexString ba))
|
||||
(defn take-ba ^bytes [n ^bytes ba] (java.util.Arrays/copyOf ba ^int n)) ; Pads if ba too small
|
||||
(defn utf8->ba ^bytes [^String s] (.getBytes s "UTF-8"))
|
||||
(defn- add-salt ^bytes [?salt-ba ba] (if ?salt-ba (enc/ba-concat ?salt-ba ba) ba))
|
||||
(defn pwd-as-ba ^bytes [utf8-or-ba] (if (string? utf8-or-ba) (utf8->ba utf8-or-ba) (truss/have enc/bytes? utf8-or-ba)))
|
||||
|
||||
(comment (seq (pwd-as-ba "foo")))
|
||||
|
||||
(defn sha512-key-ba
|
||||
"SHA512-based key generator. Good JVM availability without extra dependencies
|
||||
(PBKDF2, bcrypt, scrypt, etc.). Decent security when using many rounds."
|
||||
(^bytes [?salt-ba utf8-or-ba ] (sha512-key-ba ?salt-ba utf8-or-ba 163835 #_(* Short/MAX_VALUE 5)))
|
||||
(^bytes [?salt-ba utf8-or-ba ^long n-rounds]
|
||||
(let [ba (add-salt ?salt-ba (pwd-as-ba utf8-or-ba))
|
||||
md (sha512-md)]
|
||||
(enc/reduce-n (fn [acc in] (.digest md acc)) ba n-rounds))))
|
||||
|
||||
(comment
|
||||
(count (seq (sha512-key-ba (utf8->ba "salt") "password" 1)))
|
||||
(count (seq (sha512-key-ba nil "password" 1))))
|
||||
|
||||
;;;; Crypto
|
||||
|
||||
(defprotocol ICipherKit
|
||||
(get-cipher ^javax.crypto.Cipher [_] "Returns a thread-safe `javax.crypto.Cipher` instance.")
|
||||
(get-iv-size [_] "Returns necessary iv-ba length.")
|
||||
(get-key-spec ^javax.crypto.spec.SecretKeySpec [_ ba] "Returns a `javax.crypto.spec.SecretKeySpec`.")
|
||||
(get-param-spec ^java.security.spec.AlgorithmParameterSpec [_ iv-ba] "Returns a `java.security.spec.AlgorithmParameters`."))
|
||||
|
||||
;; Prefer GCM > CBC, Ref. <https://goo.gl/jpZoj8>
|
||||
(def ^:private gcm-cipher* (enc/thread-local-proxy (javax.crypto.Cipher/getInstance "AES/GCM/NoPadding")))
|
||||
(def ^:private cbc-cipher* (enc/thread-local-proxy (javax.crypto.Cipher/getInstance "AES/CBC/PKCS5Padding")))
|
||||
|
||||
(defn gcm-cipher ^javax.crypto.Cipher [] (.get ^ThreadLocal gcm-cipher*))
|
||||
(defn cbc-cipher ^javax.crypto.Cipher [] (.get ^ThreadLocal cbc-cipher*))
|
||||
;
|
||||
(deftype CipherKit-AES-GCM []
|
||||
ICipherKit
|
||||
(get-cipher [_] (gcm-cipher))
|
||||
(get-iv-size [_] 12)
|
||||
(get-key-spec [_ ba] (javax.crypto.spec.SecretKeySpec. ba "AES"))
|
||||
(get-param-spec [_ iv-ba] (javax.crypto.spec.GCMParameterSpec. 128 iv-ba)))
|
||||
|
||||
(deftype CipherKit-AES-CBC []
|
||||
ICipherKit
|
||||
(get-cipher [_] (cbc-cipher))
|
||||
(get-iv-size [_] 16)
|
||||
(get-key-spec [_ ba] (javax.crypto.spec.SecretKeySpec. ba "AES"))
|
||||
(get-param-spec [_ iv-ba] (javax.crypto.spec.IvParameterSpec. iv-ba)))
|
||||
|
||||
(def cipher-kit-aes-gcm "Default CipherKit for AES GCM" (CipherKit-AES-GCM.))
|
||||
(def cipher-kit-aes-cbc "Default CipherKit for AES CBC" (CipherKit-AES-CBC.))
|
||||
|
||||
;; Output bytes: [ <iv> <?salt> <encrypted>]
|
||||
;; Could also do: [<iv-len> <iv> <salt-len> <?salt> <encrypted>]
|
||||
(defn encrypt
|
||||
[{:keys [cipher-kit ?salt-ba key-ba plain-ba rand-bytes-fn]
|
||||
:or {cipher-kit cipher-kit-aes-gcm
|
||||
rand-bytes-fn rand-bytes}}]
|
||||
|
||||
(let [iv-size (long (get-iv-size cipher-kit))
|
||||
iv-ba (rand-bytes-fn iv-size)
|
||||
prefix-ba (if ?salt-ba (enc/ba-concat iv-ba ?salt-ba) iv-ba)
|
||||
key-spec (get-key-spec cipher-kit key-ba)
|
||||
param-spec (get-param-spec cipher-kit iv-ba)
|
||||
cipher (get-cipher cipher-kit)]
|
||||
|
||||
(.init cipher javax.crypto.Cipher/ENCRYPT_MODE key-spec param-spec)
|
||||
(enc/ba-concat prefix-ba (.doFinal cipher plain-ba))))
|
||||
|
||||
(comment (encrypt {:?salt-ba nil :key-ba (take-ba 16 (sha512-key-ba nil "pwd")) :plain-ba (utf8->ba "data")}))
|
||||
|
||||
(defn decrypt
|
||||
[{:keys [cipher-kit salt-size salt->key-fn enc-ba]
|
||||
:or {cipher-kit cipher-kit-aes-gcm}}]
|
||||
(let [salt-size (long salt-size)
|
||||
iv-size (long (get-iv-size cipher-kit))
|
||||
prefix-size (+ iv-size salt-size)
|
||||
[prefix-ba enc-ba] (enc/ba-split enc-ba prefix-size)
|
||||
[iv-ba salt-ba] (if (pos? salt-size)
|
||||
(enc/ba-split prefix-ba iv-size)
|
||||
[prefix-ba nil])
|
||||
|
||||
key-ba (salt->key-fn salt-ba)
|
||||
key-spec (get-key-spec cipher-kit key-ba)
|
||||
param-spec (get-param-spec cipher-kit iv-ba)
|
||||
cipher (get-cipher cipher-kit)]
|
||||
|
||||
(.init cipher javax.crypto.Cipher/DECRYPT_MODE key-spec param-spec)
|
||||
(.doFinal cipher enc-ba)))
|
||||
|
||||
(comment
|
||||
(do
|
||||
(defn sha512-k16 [?salt-ba pwd] (take-ba 16 (sha512-key-ba ?salt-ba pwd)))
|
||||
(defn roundtrip [kit ?salt-ba key-ba key-fn]
|
||||
(let [salt-size (count ?salt-ba)
|
||||
encr (encrypt {:cipher-kit kit :?salt-ba ?salt-ba :key-ba key-ba :plain-ba (utf8->ba "data")})
|
||||
decr (decrypt {:cipher-kit kit :salt-size salt-size :salt->key-fn key-fn :enc-ba encr})]
|
||||
(String. ^bytes decr "UTF-8")))
|
||||
|
||||
[(let [s (rand-bytes 16)] (roundtrip cipher-kit-aes-gcm s (sha512-k16 s "pwd") #(sha512-k16 % "pwd")))
|
||||
(let [s nil] (roundtrip cipher-kit-aes-gcm s (sha512-k16 s "pwd") #(sha512-k16 % "pwd")))
|
||||
(let [s (rand-bytes 16)] (roundtrip cipher-kit-aes-cbc s (sha512-k16 s "pwd") #(sha512-k16 % "pwd")))
|
||||
(let [s nil] (roundtrip cipher-kit-aes-cbc s (sha512-k16 s "pwd") #(sha512-k16 % "pwd")))])
|
||||
|
||||
(enc/qb 10
|
||||
(let [s (rand-bytes 16)]
|
||||
(roundtrip cipher-kit-aes-gcm s (sha512-k16 s "pwd") #(sha512-k16 % "pwd"))))
|
||||
;; 2394.89
|
||||
)
|
||||
|
|
@ -1,132 +0,0 @@
|
|||
(ns ^:no-doc taoensso.nippy.encryption
|
||||
"Private, implementation detail."
|
||||
(:require
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc]
|
||||
[taoensso.nippy.crypto :as crypto]))
|
||||
|
||||
(def ^:const standard-header-ids
|
||||
"These support `:auto` thaw."
|
||||
#{:aes128-cbc-sha512 :aes128-gcm-sha512})
|
||||
|
||||
(defprotocol IEncryptor
|
||||
(header-id [encryptor])
|
||||
(encrypt ^bytes [encryptor pwd ba])
|
||||
(decrypt ^bytes [encryptor pwd ba]))
|
||||
|
||||
(defn- throw-destructure-ex [typed-password]
|
||||
(truss/ex-info!
|
||||
(str "Expected password form: "
|
||||
"[<#{:salted :cached}> <password-string>].\n "
|
||||
"See `aes128-encryptor` docstring for details!")
|
||||
{:typed-password typed-password}))
|
||||
|
||||
(defn- destructure-typed-pwd [typed-password]
|
||||
(if (vector? typed-password)
|
||||
(let [[type password] typed-password]
|
||||
(if (#{:salted :cached} type)
|
||||
[type password]
|
||||
(throw-destructure-ex typed-password)))
|
||||
(throw-destructure-ex typed-password)))
|
||||
|
||||
(comment (destructure-typed-pwd [:salted "foo"]))
|
||||
|
||||
(deftype AES128Encryptor [header-id cipher-kit salted-key-fn cached-key-fn]
|
||||
IEncryptor
|
||||
(header-id [_] header-id)
|
||||
(encrypt [_ typed-pwd plain-ba]
|
||||
(let [[type pwd] (destructure-typed-pwd typed-pwd)
|
||||
salt? (identical? type :salted)
|
||||
?salt-ba (when salt? (crypto/rand-bytes 16))
|
||||
key-ba
|
||||
(crypto/take-ba 16 ; 128 bit AES
|
||||
(if-let [salt-ba ?salt-ba]
|
||||
(salted-key-fn salt-ba pwd)
|
||||
(cached-key-fn nil pwd)))]
|
||||
|
||||
(crypto/encrypt
|
||||
{:cipher-kit cipher-kit
|
||||
:?salt-ba ?salt-ba
|
||||
:key-ba key-ba
|
||||
:plain-ba plain-ba})))
|
||||
|
||||
(decrypt [_ typed-pwd enc-ba]
|
||||
(let [[type pwd] (destructure-typed-pwd typed-pwd)
|
||||
salt? (identical? type :salted)
|
||||
salt->key-fn
|
||||
(if salt?
|
||||
#(salted-key-fn % pwd)
|
||||
#(cached-key-fn % pwd))]
|
||||
|
||||
(crypto/decrypt
|
||||
{:cipher-kit cipher-kit
|
||||
:salt-size (if salt? 16 0)
|
||||
:salt->key-fn salt->key-fn
|
||||
:enc-ba enc-ba}))))
|
||||
|
||||
(def aes128-gcm-encryptor
|
||||
"Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
|
||||
|
||||
Password form [:salted \"my-password\"]
|
||||
---------------------------------------
|
||||
USE CASE: You want more than a small, finite number of passwords (e.g. each
|
||||
item encrypted will use a unique user-provided password).
|
||||
|
||||
IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
|
||||
every key.
|
||||
|
||||
PROS: Each key is independent so would need to be attacked independently.
|
||||
CONS: Key caching impossible, so there's an inherent trade-off between
|
||||
encryption/decryption speed and the difficulty of attacking any
|
||||
particular key.
|
||||
|
||||
Slower than `aes128-cached`, and easier to attack any particular key - but
|
||||
keys are independent.
|
||||
|
||||
Password form [:cached \"my-password\"]
|
||||
---------------------------------------
|
||||
USE CASE: You want only a small, finite number of passwords (e.g. a limited
|
||||
number of staff/admins, or you'll be using a single password to
|
||||
encrypt many items).
|
||||
|
||||
IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.
|
||||
|
||||
PROS: Great amortized encryption/decryption speed. Expensive key hash makes
|
||||
attacking any particular key very difficult.
|
||||
CONS: Using a small number of keys for many encrypted items means that if any
|
||||
key _is_ somehow compromised, _all_ items encrypted with that key are
|
||||
compromised.
|
||||
|
||||
Faster than `aes128-salted`, and harder to attack any particular key - but
|
||||
increased danger if a key is somehow compromised."
|
||||
|
||||
(AES128Encryptor. :aes128-gcm-sha512
|
||||
crypto/cipher-kit-aes-gcm
|
||||
(do (fn [ salt-ba pwd] (crypto/take-ba 16 (crypto/sha512-key-ba salt-ba pwd (* Short/MAX_VALUE 5)))))
|
||||
(enc/fmemoize (fn [_salt-ba pwd] (crypto/take-ba 16 (crypto/sha512-key-ba nil pwd (* Short/MAX_VALUE 64)))))))
|
||||
|
||||
(def aes128-cbc-encryptor
|
||||
"Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen.
|
||||
See also `aes-128-cbc-encryptor`."
|
||||
(AES128Encryptor. :aes128-cbc-sha512
|
||||
crypto/cipher-kit-aes-cbc
|
||||
(do (fn [ salt-ba pwd] (crypto/take-ba 16 (crypto/sha512-key-ba salt-ba pwd (* Short/MAX_VALUE 5)))))
|
||||
(enc/fmemoize (fn [_salt-ba pwd] (crypto/take-ba 16 (crypto/sha512-key-ba nil pwd (* Short/MAX_VALUE 64)))))))
|
||||
|
||||
;;;; Default implementation
|
||||
|
||||
(comment
|
||||
(def dae aes128-encryptor)
|
||||
(def secret-ba (.getBytes "Secret message" "UTF-8"))
|
||||
(encrypt dae "p" secret-ba) ; Malformed
|
||||
(enc/qb 10
|
||||
(encrypt dae [:salted "p"] secret-ba)
|
||||
(encrypt dae [:cached "p"] secret-ba))
|
||||
|
||||
(enc/qb 10
|
||||
(->> secret-ba
|
||||
(encrypt dae [:salted "p"])
|
||||
(encrypt dae [:cached "p"])
|
||||
(decrypt dae [:cached "p"])
|
||||
(decrypt dae [:salted "p"])
|
||||
(String.))))
|
||||
|
|
@ -1,273 +0,0 @@
|
|||
(ns ^:no-doc taoensso.nippy.impl
|
||||
"Private, implementation detail."
|
||||
(:require
|
||||
[clojure.string :as str]
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc]))
|
||||
|
||||
;;;; Fallback type tests
|
||||
|
||||
(defn cache-by-type [f]
|
||||
(let [cache_ (enc/latom {})] ; {<type> <result_>}
|
||||
(fn [x]
|
||||
(let [t (if (fn? x) ::fn (type x))]
|
||||
(if-let [result_ (get (cache_) t)]
|
||||
@result_
|
||||
(if-let [uncacheable-type? (re-find #"\d" (str t))]
|
||||
(do (f x))
|
||||
@(cache_ t #(or % (delay (f x))))))))))
|
||||
|
||||
(def seems-readable?
|
||||
(cache-by-type
|
||||
(fn [x]
|
||||
(try
|
||||
(enc/read-edn (enc/pr-edn x))
|
||||
true
|
||||
(catch Throwable _ false)))))
|
||||
|
||||
(def seems-serializable?
|
||||
(cache-by-type
|
||||
(fn [x]
|
||||
(enc/cond
|
||||
(fn? x) false ; Falsely reports as Serializable
|
||||
|
||||
(instance? java.io.Serializable x)
|
||||
(try
|
||||
(let [c (Class/forName (.getName (class x))) ; Try 1st (fail fast)
|
||||
bas (java.io.ByteArrayOutputStream.)
|
||||
_ (.writeObject (java.io.ObjectOutputStream. bas) x)
|
||||
ba (.toByteArray bas)]
|
||||
#_
|
||||
(cast c
|
||||
(.readObject ; Unsafe + usu. unnecessary to check
|
||||
(ObjectInputStream. (ByteArrayInputStream. ba))))
|
||||
true)
|
||||
(catch Throwable _ false))
|
||||
|
||||
:else false))))
|
||||
|
||||
(comment
|
||||
(enc/qb 1e6 ; [60.83 61.16 59.86 57.37]
|
||||
(seems-readable? "Hello world")
|
||||
(seems-serializable? "Hello world")
|
||||
(seems-readable? (fn []))
|
||||
(seems-serializable? (fn []))))
|
||||
|
||||
;;;; Java Serializable
|
||||
|
||||
(def ^:const ^:private allow-and-record "allow-and-record")
|
||||
(defn- allow-and-record? [x] (= x allow-and-record))
|
||||
|
||||
(defn- classname-set
|
||||
"Returns ?#{<classname>}."
|
||||
[x]
|
||||
(when x
|
||||
(if (string? x)
|
||||
(if (= x "") #{} (set (mapv str/trim (str/split x #"[,:]"))))
|
||||
(truss/have set? x))))
|
||||
|
||||
(comment
|
||||
(mapv classname-set [nil #{"foo"} "" "foo, bar:baz"])
|
||||
(.getName (.getSuperclass (.getClass (java.util.concurrent.TimeoutException.)))))
|
||||
|
||||
(defn parse-allowlist
|
||||
"Returns #{<classname>}, or `allow-and-record`."
|
||||
[default base add]
|
||||
(if (or
|
||||
(allow-and-record? base)
|
||||
(allow-and-record? add))
|
||||
allow-and-record
|
||||
(into
|
||||
(or (classname-set base) default)
|
||||
(do (classname-set add)))))
|
||||
|
||||
(comment (parse-allowlist #{"default"} "base1,base2" "add1"))
|
||||
|
||||
(let [nmax 1000
|
||||
ngc 16000
|
||||
state_ (enc/latom {}) ; {<class-name> <frequency>}
|
||||
lock_ (enc/latom nil) ; ?promise
|
||||
trim
|
||||
(fn [nmax state]
|
||||
(persistent!
|
||||
(enc/reduce-top nmax val enc/rcompare conj!
|
||||
(transient {}) state)))]
|
||||
|
||||
;; Note: trim strategy isn't perfect: it can be tough for new
|
||||
;; classes to break into the top set since frequencies are being
|
||||
;; reset only for classes outside the top set.
|
||||
;;
|
||||
;; In practice this is probably good enough since the main objective
|
||||
;; is to discard one-off anonymous classes to protect state from
|
||||
;; endlessly growing. Also `gc-rate` allows state to temporarily grow
|
||||
;; significantly beyond `nmax` size, which helps to give new classes
|
||||
;; some chance to accumulate a competitive frequency before next GC.
|
||||
|
||||
(defn ^{:-state_ state_} ; Undocumented
|
||||
allow-and-record-any-serializable-class-unsafe
|
||||
"A predicate (fn allow-class? [class-name]) fn that can be assigned
|
||||
to `*freeze-serializable-allowlist*` and/or
|
||||
`*thaw-serializable-allowlist*` that:
|
||||
|
||||
- Will allow ANY class to use Nippy's `Serializable` support (unsafe).
|
||||
- And will record {<class-name> <frequency-allowed>} for the <=1000
|
||||
classes that ~most frequently made use of this support.
|
||||
|
||||
`get-recorded-serializable-classes` returns the recorded state.
|
||||
|
||||
This predicate is provided as a convenience for users upgrading from
|
||||
previous versions of Nippy that allowed the use of `Serializable` for all
|
||||
classes by default.
|
||||
|
||||
While transitioning from an unsafe->safe configuration, you can use
|
||||
this predicate (unsafe) to record information about which classes have
|
||||
been using Nippy's `Serializable` support in your environment.
|
||||
|
||||
Once some time has passed, you can check the recorded state. If you're
|
||||
satisfied that all recorded classes are safely `Serializable`, you can
|
||||
then merge the recorded classes into Nippy's default allowlist/s, e.g.:
|
||||
|
||||
(alter-var-root #'thaw-serializable-allowlist*
|
||||
(fn [_] (into default-thaw-serializable-allowlist
|
||||
(keys (get-recorded-serializable-classes)))))"
|
||||
|
||||
[class-name]
|
||||
(when-let [p (lock_)] @p)
|
||||
(let [n (count (state_ #(assoc % class-name (inc (long (or (get % class-name) 0))))))]
|
||||
;; Garbage collection (GC): may be serializing anonymous classes, etc.
|
||||
;; so input domain could be infinite
|
||||
(when (> n ngc) ; Too many classes recorded, uncommon
|
||||
(let [p (promise)]
|
||||
(when (compare-and-set! lock_ nil p) ; Acquired GC lock
|
||||
(try
|
||||
(do (reset! state_ (trim nmax (state_)))) ; GC state
|
||||
(finally (reset! lock_ nil) (deliver p nil))))))
|
||||
n))
|
||||
|
||||
(defn get-recorded-serializable-classes
|
||||
"Returns {<class-name> <frequency>} of the <=1000 classes that ~most
|
||||
frequently made use of Nippy's `Serializable` support via
|
||||
`allow-and-record-any-serializable-class-unsafe`.
|
||||
|
||||
See that function's docstring for more info."
|
||||
[] (trim nmax (state_))))
|
||||
|
||||
(comment
|
||||
(count (get-recorded-serializable-classes))
|
||||
(enc/reduce-n
|
||||
(fn [_ n] (allow-and-record-any-serializable-class-unsafe (str n)))
|
||||
nil 0 1e5))
|
||||
|
||||
(let [compile
|
||||
(enc/fmemoize
|
||||
(fn [x]
|
||||
(if (allow-and-record? x)
|
||||
allow-and-record-any-serializable-class-unsafe
|
||||
(enc/name-filter x))))
|
||||
|
||||
fn? fn?
|
||||
conform?
|
||||
(fn [x cn]
|
||||
(if (fn? x)
|
||||
(x cn) ; Intentionally uncached, can be handy
|
||||
((compile x) cn)))]
|
||||
|
||||
(defn serializable-allowed? [allow-list class-name]
|
||||
(conform? allow-list class-name)))
|
||||
|
||||
;;;; Release targeting
|
||||
|
||||
(comment
|
||||
(set! *print-length* nil)
|
||||
(vec (sort (keys taoensso.nippy/public-types-spec)))
|
||||
|
||||
;; To help support release targeting, we track new type ids added over time
|
||||
(let [id-history ; {<release> #{type-ids}}
|
||||
{350 ; v3.5.0 (2025-04-15), added 5x
|
||||
;; #{int-array-lg long-array-lg float-array-lg double-array-lg string-array-lg}
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
|
||||
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104
|
||||
105 106 107 108 109 110 111 112 113 114 115 116 117}
|
||||
|
||||
340 ; v3.4.0 (2024-04-30), added 2x
|
||||
;; #{map-entry meta-protocol-key}
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
|
||||
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104
|
||||
105 106 110 111 112 113 114 115}
|
||||
|
||||
330 ; v3.3.0 (2023-10-11), added 11x
|
||||
;; #{long-pos-sm long-pos-md long-pos-lg long-neg-sm long-neg-md long-neg-lg
|
||||
;; str-sm* vec-sm* set-sm* map-sm* sql-date}
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
|
||||
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 105 106
|
||||
110 111 112 113 114 115}
|
||||
|
||||
320 ; v3.2.0 (2022-07-18), added none
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
|
||||
81 82 83 84 85 86 90 91 100 101 102 105 106 110 111 112 113 114 115}
|
||||
|
||||
313 ; v3.1.3 (2022-06-23), added 5x
|
||||
;; #{time-instant time-duration time-period kw-md sym-md}
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
|
||||
81 82 83 84 85 86 90 91 100 101 102 105 106 110 111 112 113 114 115}
|
||||
|
||||
300 ; v3.0.0 (2020-09-20), baseline
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 80
|
||||
81 82 90 91 100 101 102 105 106 110 111 112 113 114 115}}
|
||||
|
||||
diff
|
||||
(fn [new-release old-release]
|
||||
(vec (sort (clojure.set/difference (id-history new-release) (id-history old-release)))))]
|
||||
|
||||
(diff 350 340)))
|
||||
|
||||
(let [target-release
|
||||
(enc/get-env {:as :edn, :default 320}
|
||||
:taoensso.nippy.target-release)
|
||||
|
||||
target>=
|
||||
(fn [min-release]
|
||||
(if target-release
|
||||
(>= (long target-release) (long min-release))
|
||||
true))]
|
||||
|
||||
(defmacro target-release< [min-release] (not (target>= min-release)))
|
||||
(defmacro target-release>=
|
||||
"Returns true iff `target-release` is nil or >= given `min-release`.
|
||||
Used to help ease data migration for changes to core data types.
|
||||
|
||||
When support is added for a new type in Nippy version X, it necessarily means
|
||||
that data containing that new type and frozen with Nippy version X is unthawable
|
||||
with Nippy versions < X.
|
||||
|
||||
Earlier versions of Nippy will throw an exception on thawing affected data:
|
||||
\"Unrecognized type id (<n>). Data frozen with newer Nippy version?\"
|
||||
|
||||
This can present a challenge when updating to new versions of Nippy, e.g.:
|
||||
|
||||
- Rolling updates could lead to old and new versions of Nippy temporarily co-existing.
|
||||
- Data written with new types could limit your ability to revert a Nippy update.
|
||||
|
||||
There's no easy solution to this in GENERAL, but we CAN at least help reduce the
|
||||
burden related to CHANGES in core data types by introducing changes over 2 phases:
|
||||
|
||||
1. Nippy vX reads new (changed) type, writes old type
|
||||
2. Nippy vX+1 writes new (changed) type
|
||||
|
||||
When relevant, we can then warn users in the CHANGELOG to not leapfrog
|
||||
(e.g. Nippy vX -> Nippy vX+2) when doing rolling updates."
|
||||
[min-release] (target>= min-release)))
|
||||
|
||||
(comment (macroexpand '(target-release>= 340)))
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
(ns taoensso.nippy.tools
|
||||
"Utils for community tools that want to add user-configurable Nippy support.
|
||||
Used by Carmine, Faraday, etc."
|
||||
(:require
|
||||
[taoensso.encore :as enc]
|
||||
[taoensso.nippy :as nippy]))
|
||||
|
||||
(def ^:dynamic *freeze-opts* nil)
|
||||
(def ^:dynamic *thaw-opts* nil)
|
||||
|
||||
(do
|
||||
(defmacro with-freeze-opts [opts & body] `(binding [*freeze-opts* ~opts ] ~@body))
|
||||
(defmacro with-freeze-opts+ [opts & body] `(binding [*freeze-opts* (enc/merge *freeze-opts* ~opts)] ~@body))
|
||||
(defmacro with-thaw-opts [opts & body] `(binding [*thaw-opts* ~opts ] ~@body))
|
||||
(defmacro with-thaw-opts+ [opts & body] `(binding [*thaw-opts* (enc/merge *thaw-opts* ~opts)] ~@body)))
|
||||
|
||||
(deftype WrappedForFreezing [val opts])
|
||||
(defn wrapped-for-freezing? [x] (instance? WrappedForFreezing x))
|
||||
|
||||
(defn wrap-for-freezing
|
||||
"Captures (merge `tools/*thaw-opts*` `wrap-opts`), and returns
|
||||
the given argument in a wrapped form so that `tools/freeze` will
|
||||
use the captured options when freezing the wrapper argument.
|
||||
|
||||
See also `tools/freeze`."
|
||||
([x ] (wrap-for-freezing x nil))
|
||||
([x wrap-opts]
|
||||
(let [captured-opts (enc/merge *freeze-opts* wrap-opts)] ; wrap > dynamic
|
||||
(if (instance? WrappedForFreezing x)
|
||||
(let [^WrappedForFreezing x x]
|
||||
(if (= (.-opts x) captured-opts)
|
||||
x
|
||||
(WrappedForFreezing. (.-val x) captured-opts)))
|
||||
(WrappedForFreezing. x captured-opts)))))
|
||||
|
||||
(defn freeze
|
||||
"Like `nippy/freeze` but uses as options the following, merged in
|
||||
order of ascending preference:
|
||||
|
||||
1. `default-opts` given to this fn (default nil).
|
||||
2. `tools/*freeze-opts*` dynamic value (default nil).
|
||||
3. Opts captured by `tools/wrap-for-freezing` (default nil).
|
||||
|
||||
See also `tools/wrap-for-freezing`."
|
||||
([x ] (freeze x nil))
|
||||
([x default-opts]
|
||||
(let [default-opts (get default-opts :default-opts default-opts) ; Back compatibility
|
||||
active-opts (enc/merge default-opts *freeze-opts*)] ; dynamic > default
|
||||
|
||||
(if (instance? WrappedForFreezing x)
|
||||
(let [^WrappedForFreezing x x]
|
||||
(nippy/freeze (.-val x) (enc/merge active-opts (.-opts x)))) ; captured > active!
|
||||
(nippy/freeze x active-opts)))))
|
||||
|
||||
(defn thaw
|
||||
"Like `nippy/thaw` but uses as options the following, merged in
|
||||
order of ascending preference:
|
||||
|
||||
1. `default-opts` given to this fn (default nil).
|
||||
2. `tools/*thaw-opts*` dynamic value (default nil)."
|
||||
([ba ] (thaw ba nil))
|
||||
([ba default-opts]
|
||||
(let [default-opts (get default-opts :default-opts default-opts) ; Back compatibility
|
||||
active-opts (enc/merge default-opts *thaw-opts*)] ; dynamic > default
|
||||
|
||||
(nippy/thaw ba active-opts))))
|
||||
|
||||
(comment (thaw (freeze (wrap-for-freezing "wrapped"))))
|
||||
5
taoensso.nippy.benchmarks.html
Normal file
5
taoensso.nippy.benchmarks.html
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
<!DOCTYPE html PUBLIC ""
|
||||
"">
|
||||
<html><head><meta charset="UTF-8" /><title>taoensso.nippy.benchmarks documentation</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">Nippy</span> <span class="project-version">3.3.0</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>taoensso</span></div></div></li><li class="depth-2"><a href="taoensso.nippy.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>nippy</span></div></a></li><li class="depth-3 branch current"><a href="taoensso.nippy.benchmarks.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>benchmarks</span></div></a></li><li class="depth-3"><a href="taoensso.nippy.tools.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>tools</span></div></a></li></ul></div><div class="sidebar secondary"><h3><a href="#top"><span class="inner">Public Vars</span></a></h3><ul><li class="depth-1"><a href="taoensso.nippy.benchmarks.html#var-bench"><div class="inner"><span>bench</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.benchmarks.html#var-data"><div class="inner"><span>data</span></div></a></li></ul></div><div class="namespace-docs" id="content"><h1 class="anchor" id="top">taoensso.nippy.benchmarks</h1><div class="doc"><pre class="plaintext">Nippy benchmarks.
|
||||
</pre></div><div class="public anchor" id="var-bench"><h3>bench</h3><div class="usage"><code>(bench {:keys [all? reader? fressian? fressian? lzma2? laps warmup], :or {laps 10000.0, warmup 25000.0}})</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-data"><h3>data</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Map of data suitable for benching, a subset of
|
||||
`nippy/stress-data-comparable`.</pre></div></div></div></body></html>
|
||||
24
taoensso.nippy.compression.html
Normal file
24
taoensso.nippy.compression.html
Normal file
File diff suppressed because one or more lines are too long
11
taoensso.nippy.crypto.html
Normal file
11
taoensso.nippy.crypto.html
Normal file
File diff suppressed because one or more lines are too long
39
taoensso.nippy.encryption.html
Normal file
39
taoensso.nippy.encryption.html
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
<!DOCTYPE html PUBLIC ""
|
||||
"">
|
||||
<html><head><meta charset="UTF-8" /><title>taoensso.nippy.encryption documentation</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">Nippy</span> <span class="project-version">3.2.0</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>taoensso</span></div></div></li><li class="depth-2"><a href="taoensso.nippy.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>nippy</span></div></a></li><li class="depth-3 branch"><a href="taoensso.nippy.benchmarks.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>benchmarks</span></div></a></li><li class="depth-3 branch"><a href="taoensso.nippy.compression.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>compression</span></div></a></li><li class="depth-3 branch"><a href="taoensso.nippy.crypto.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>crypto</span></div></a></li><li class="depth-3 branch current"><a href="taoensso.nippy.encryption.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>encryption</span></div></a></li><li class="depth-3 branch"><a href="taoensso.nippy.tools.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>tools</span></div></a></li><li class="depth-3"><a href="taoensso.nippy.utils.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>utils</span></div></a></li></ul></div><div class="sidebar secondary"><h3><a href="#top"><span class="inner">Public Vars</span></a></h3><ul><li class="depth-1"><a href="taoensso.nippy.encryption.html#var-aes128-cbc-encryptor"><div class="inner"><span>aes128-cbc-encryptor</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.encryption.html#var-aes128-gcm-encryptor"><div class="inner"><span>aes128-gcm-encryptor</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.encryption.html#var-IEncryptor"><div class="inner"><span>IEncryptor</span></div></a></li><li class="depth-2 branch"><a href="taoensso.nippy.encryption.html#var-decrypt"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>decrypt</span></div></a></li><li class="depth-2 branch"><a href="taoensso.nippy.encryption.html#var-encrypt"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>encrypt</span></div></a></li><li class="depth-2"><a href="taoensso.nippy.encryption.html#var-header-id"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>header-id</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.encryption.html#var-standard-header-ids"><div class="inner"><span>standard-header-ids</span></div></a></li></ul></div><div class="namespace-docs" id="content"><h1 class="anchor" id="top">taoensso.nippy.encryption</h1><div class="doc"><pre class="plaintext">Simple no-nonsense crypto with reasonable defaults
|
||||
</pre></div><div class="public anchor" id="var-aes128-cbc-encryptor"><h3>aes128-cbc-encryptor</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Default 128bit AES-CBC encryptor with many-round SHA-512 key-gen.
|
||||
See also `aes-128-cbc-encryptor`.</pre></div></div><div class="public anchor" id="var-aes128-gcm-encryptor"><h3>aes128-gcm-encryptor</h3><div class="usage"></div><div class="doc"><pre class="plaintext">Default 128bit AES-GCM encryptor with many-round SHA-512 key-gen.
|
||||
|
||||
Password form [:salted "my-password"]
|
||||
---------------------------------------
|
||||
USE CASE: You want more than a small, finite number of passwords (e.g. each
|
||||
item encrypted will use a unique user-provided password).
|
||||
|
||||
IMPLEMENTATION: Uses a relatively cheap key hash, but automatically salts
|
||||
every key.
|
||||
|
||||
PROS: Each key is independent so would need to be attacked independently.
|
||||
CONS: Key caching impossible, so there's an inherent trade-off between
|
||||
encryption/decryption speed and the difficulty of attacking any
|
||||
particular key.
|
||||
|
||||
Slower than `aes128-cached`, and easier to attack any particular key - but
|
||||
keys are independent.
|
||||
|
||||
Password form [:cached "my-password"]
|
||||
---------------------------------------
|
||||
USE CASE: You want only a small, finite number of passwords (e.g. a limited
|
||||
number of staff/admins, or you'll be using a single password to
|
||||
encrypt many items).
|
||||
|
||||
IMPLEMENTATION: Uses a _very_ expensive (but cached) key hash, and no salt.
|
||||
|
||||
PROS: Great amortized encryption/decryption speed. Expensive key hash makes
|
||||
attacking any particular key very difficult.
|
||||
CONS: Using a small number of keys for many encrypted items means that if any
|
||||
key _is_ somehow compromised, _all_ items encrypted with that key are
|
||||
compromised.
|
||||
|
||||
Faster than `aes128-salted`, and harder to attack any particular key - but
|
||||
increased danger if a key is somehow compromised.</pre></div></div><div class="public anchor" id="var-IEncryptor"><h3>IEncryptor</h3><h4 class="type">protocol</h4><div class="usage"></div><div class="doc"><pre class="plaintext"></pre></div><div class="members"><h4>members</h4><div class="inner"><div class="public anchor" id="var-decrypt"><h3>decrypt</h3><div class="usage"><code>(decrypt encryptor pwd ba)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-encrypt"><h3>encrypt</h3><div class="usage"><code>(encrypt encryptor pwd ba)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-header-id"><h3>header-id</h3><div class="usage"><code>(header-id encryptor)</code></div><div class="doc"><pre class="plaintext"></pre></div></div></div></div></div><div class="public anchor" id="var-standard-header-ids"><h3>standard-header-ids</h3><div class="usage"></div><div class="doc"><pre class="plaintext">These'll support :auto thaw
|
||||
</pre></div></div></div></body></html>
|
||||
427
taoensso.nippy.html
Normal file
427
taoensso.nippy.html
Normal file
File diff suppressed because one or more lines are too long
19
taoensso.nippy.tools.html
Normal file
19
taoensso.nippy.tools.html
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
<!DOCTYPE html PUBLIC ""
|
||||
"">
|
||||
<html><head><meta charset="UTF-8" /><title>taoensso.nippy.tools documentation</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">Nippy</span> <span class="project-version">3.5.0-RC1</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>taoensso</span></div></div></li><li class="depth-2"><a href="taoensso.nippy.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>nippy</span></div></a></li><li class="depth-3 current"><a href="taoensso.nippy.tools.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>tools</span></div></a></li></ul></div><div class="sidebar secondary"><h3><a href="#top"><span class="inner">Public Vars</span></a></h3><ul><li class="depth-1"><a href="taoensso.nippy.tools.html#var-*freeze-opts*"><div class="inner"><span>*freeze-opts*</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.tools.html#var-*thaw-opts*"><div class="inner"><span>*thaw-opts*</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.tools.html#var-freeze"><div class="inner"><span>freeze</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.tools.html#var-thaw"><div class="inner"><span>thaw</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.tools.html#var-with-freeze-opts"><div class="inner"><span>with-freeze-opts</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.tools.html#var-with-freeze-opts.2B"><div class="inner"><span>with-freeze-opts+</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.tools.html#var-with-thaw-opts"><div class="inner"><span>with-thaw-opts</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.tools.html#var-with-thaw-opts.2B"><div class="inner"><span>with-thaw-opts+</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.tools.html#var-wrap-for-freezing"><div class="inner"><span>wrap-for-freezing</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.tools.html#var-wrapped-for-freezing.3F"><div class="inner"><span>wrapped-for-freezing?</span></div></a></li></ul></div><div class="namespace-docs" id="content"><h1 class="anchor" id="top">taoensso.nippy.tools</h1><div class="doc"><pre class="plaintext">Utils for community tools that want to add user-configurable Nippy support.
|
||||
Used by Carmine, Faraday, etc.</pre></div><div class="public anchor" id="var-*freeze-opts*"><h3>*freeze-opts*</h3><h4 class="dynamic">dynamic</h4><div class="usage"></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-*thaw-opts*"><h3>*thaw-opts*</h3><h4 class="dynamic">dynamic</h4><div class="usage"></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-freeze"><h3>freeze</h3><div class="usage"><code>(freeze x)</code><code>(freeze x default-opts)</code></div><div class="doc"><pre class="plaintext">Like `nippy/freeze` but uses as options the following, merged in
|
||||
order of ascending preference:
|
||||
|
||||
1. `default-opts` given to this fn (default nil).
|
||||
2. `tools/*freeze-opts*` dynamic value (default nil).
|
||||
3. Opts captured by `tools/wrap-for-freezing` (default nil).
|
||||
|
||||
See also `tools/wrap-for-freezing`.</pre></div></div><div class="public anchor" id="var-thaw"><h3>thaw</h3><div class="usage"><code>(thaw ba)</code><code>(thaw ba default-opts)</code></div><div class="doc"><pre class="plaintext">Like `nippy/thaw` but uses as options the following, merged in
|
||||
order of ascending preference:
|
||||
|
||||
1. `default-opts` given to this fn (default nil).
|
||||
2. `tools/*thaw-opts*` dynamic value (default nil).</pre></div></div><div class="public anchor" id="var-with-freeze-opts"><h3>with-freeze-opts</h3><h4 class="type">macro</h4><div class="usage"><code>(with-freeze-opts opts & body)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-with-freeze-opts.2B"><h3>with-freeze-opts+</h3><h4 class="type">macro</h4><div class="usage"><code>(with-freeze-opts+ opts & body)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-with-thaw-opts"><h3>with-thaw-opts</h3><h4 class="type">macro</h4><div class="usage"><code>(with-thaw-opts opts & body)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-with-thaw-opts.2B"><h3>with-thaw-opts+</h3><h4 class="type">macro</h4><div class="usage"><code>(with-thaw-opts+ opts & body)</code></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-wrap-for-freezing"><h3>wrap-for-freezing</h3><div class="usage"><code>(wrap-for-freezing x)</code><code>(wrap-for-freezing x wrap-opts)</code></div><div class="doc"><pre class="plaintext">Captures (merge `tools/*thaw-opts*` `wrap-opts`), and returns
|
||||
the given argument in a wrapped form so that `tools/freeze` will
|
||||
use the captured options when freezing the wrapper argument.
|
||||
|
||||
See also `tools/freeze`.</pre></div></div><div class="public anchor" id="var-wrapped-for-freezing.3F"><h3>wrapped-for-freezing?</h3><div class="usage"><code>(wrapped-for-freezing? x)</code></div><div class="doc"><pre class="plaintext"></pre></div></div></div></body></html>
|
||||
10
taoensso.nippy.utils.html
Normal file
10
taoensso.nippy.utils.html
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
<!DOCTYPE html PUBLIC ""
|
||||
"">
|
||||
<html><head><meta charset="UTF-8" /><title>taoensso.nippy.utils documentation</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">Nippy</span> <span class="project-version">3.2.0</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>taoensso</span></div></div></li><li class="depth-2"><a href="taoensso.nippy.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>nippy</span></div></a></li><li class="depth-3 branch"><a href="taoensso.nippy.benchmarks.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>benchmarks</span></div></a></li><li class="depth-3 branch"><a href="taoensso.nippy.compression.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>compression</span></div></a></li><li class="depth-3 branch"><a href="taoensso.nippy.crypto.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>crypto</span></div></a></li><li class="depth-3 branch"><a href="taoensso.nippy.encryption.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>encryption</span></div></a></li><li class="depth-3 branch"><a href="taoensso.nippy.tools.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>tools</span></div></a></li><li class="depth-3 current"><a href="taoensso.nippy.utils.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>utils</span></div></a></li></ul></div><div class="sidebar secondary"><h3><a href="#top"><span class="inner">Public Vars</span></a></h3><ul><li class="depth-1"><a href="taoensso.nippy.utils.html#var-freezable.3F"><div class="inner"><span>freezable?</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.utils.html#var-readable.3F"><div class="inner"><span>readable?</span></div></a></li><li class="depth-1"><a href="taoensso.nippy.utils.html#var-serializable.3F"><div class="inner"><span>serializable?</span></div></a></li></ul></div><div class="namespace-docs" id="content"><h1 class="anchor" id="top">taoensso.nippy.utils</h1><div class="doc"><pre class="plaintext"></pre></div><div class="public anchor" id="var-freezable.3F"><h3>freezable?</h3><div class="usage"><code>(freezable? x)</code><code>(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})</code></div><div class="doc"><pre class="plaintext">Alpha - subject to change.
|
||||
Returns truthy iff Nippy *appears* to support freezing the given argument.
|
||||
|
||||
`:allow-clojure-reader?` and `:allow-java-serializable?` options may be
|
||||
used to enable the relevant roundtrip fallback test(s). These tests are
|
||||
only **moderately reliable** since they're cached by arg type and don't
|
||||
test for pre/post serialization value equality (there's no good general
|
||||
way of doing so).</pre></div></div><div class="public anchor" id="var-readable.3F"><h3>readable?</h3><div class="usage"></div><div class="doc"><pre class="plaintext"></pre></div></div><div class="public anchor" id="var-serializable.3F"><h3>serializable?</h3><div class="usage"></div><div class="doc"><pre class="plaintext"></pre></div></div></div></body></html>
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
(ns taoensso.graal-tests
|
||||
(:require [taoensso.nippy :as nippy])
|
||||
(:gen-class))
|
||||
|
||||
(defn -main [& args] (println "Namespace loaded successfully"))
|
||||
|
|
@ -1,207 +0,0 @@
|
|||
(ns taoensso.nippy-benchmarks
|
||||
(:require
|
||||
[clojure.data.fressian :as fress]
|
||||
[taoensso.encore :as enc]
|
||||
[taoensso.nippy :as nippy]
|
||||
[taoensso.nippy.compression :as compr]))
|
||||
|
||||
;;;; Reader
|
||||
|
||||
(defn- freeze-reader [x] (enc/pr-edn x))
|
||||
(defn- thaw-reader [edn] (enc/read-edn edn))
|
||||
|
||||
;;;; Fressian
|
||||
|
||||
(defn- freeze-fress [x]
|
||||
(let [^java.nio.ByteBuffer bb (fress/write x)
|
||||
len (.remaining bb)
|
||||
ba (byte-array len)]
|
||||
(.get bb ba 0 len)
|
||||
(do ba)))
|
||||
|
||||
(defn- thaw-fress [^bytes ba]
|
||||
(let [bb (java.nio.ByteBuffer/wrap ba)]
|
||||
(fress/read bb)))
|
||||
|
||||
(comment (-> data freeze-fress thaw-fress))
|
||||
|
||||
;;;; Bench data
|
||||
|
||||
(def default-bench-data
|
||||
"Subset of stress data suitable for benching."
|
||||
(let [sd (nippy/stress-data {:comparable? true})]
|
||||
(reduce-kv
|
||||
(fn [m k v]
|
||||
(try
|
||||
(-> v freeze-reader thaw-reader)
|
||||
(-> v freeze-fress thaw-fress)
|
||||
m
|
||||
(catch Throwable _ (dissoc m k))))
|
||||
sd sd)))
|
||||
|
||||
(comment
|
||||
(clojure.set/difference
|
||||
(set (keys (nippy/stress-data {:comparable? true})))
|
||||
(set (keys default-bench-data))))
|
||||
|
||||
;;;; Serialization
|
||||
|
||||
(defn- bench1-serialization
|
||||
[freezer thawer sizer
|
||||
{:keys [laps warmup bench-data]
|
||||
:or
|
||||
{laps 1e4
|
||||
warmup 25e3
|
||||
bench-data default-bench-data}}]
|
||||
|
||||
(let [data-frozen (freezer bench-data)
|
||||
time-freeze (enc/bench laps {:warmup-laps warmup} (freezer bench-data))
|
||||
time-thaw (enc/bench laps {:warmup-laps warmup} (thawer data-frozen))
|
||||
data-size (sizer data-frozen)]
|
||||
|
||||
{:round (+ time-freeze time-thaw)
|
||||
:freeze time-freeze
|
||||
:thaw time-thaw
|
||||
:size data-size}))
|
||||
|
||||
(comment (bench1-serialization nippy/freeze nippy/thaw count {}))
|
||||
|
||||
(defn- printed-results [results]
|
||||
(println "\nBenchmark results:")
|
||||
(doseq [[k v] results] (println k " " v))
|
||||
(do results))
|
||||
|
||||
(defn bench-serialization
|
||||
[{:keys [all? reader? fressian? fressian? lzma2? laps warmup bench-data]
|
||||
:as opts
|
||||
:or
|
||||
{laps 1e4
|
||||
warmup 25e3}}]
|
||||
|
||||
(println "\nRunning benchmarks...")
|
||||
|
||||
(let [results_ (atom {})]
|
||||
(when (or all? reader?)
|
||||
(println " With Reader...")
|
||||
(swap! results_ assoc :reader
|
||||
(bench1-serialization freeze-reader thaw-reader
|
||||
(fn [^String s] (count (.getBytes s "UTF-8")))
|
||||
(assoc opts :laps laps, :warmup warmup))))
|
||||
|
||||
(when (or all? fressian?)
|
||||
(println " With Fressian...")
|
||||
(swap! results_ assoc :fressian
|
||||
(bench1-serialization freeze-fress thaw-fress count
|
||||
(assoc opts :laps laps, :warmup warmup))))
|
||||
|
||||
(when (or all? lzma2?)
|
||||
(println " With Nippy/LZMA2...")
|
||||
(swap! results_ assoc :nippy/lzma2
|
||||
(bench1-serialization
|
||||
#(nippy/freeze % {:compressor nippy/lzma2-compressor})
|
||||
#(nippy/thaw % {:compressor nippy/lzma2-compressor})
|
||||
count
|
||||
(assoc opts :laps laps, :warmup warmup))))
|
||||
|
||||
(println " With Nippy/encrypted...")
|
||||
(swap! results_ assoc :nippy/encrypted
|
||||
(bench1-serialization
|
||||
#(nippy/freeze % {:password [:cached "p"]})
|
||||
#(nippy/thaw % {:password [:cached "p"]})
|
||||
count
|
||||
(assoc opts :laps laps, :warmup warmup)))
|
||||
|
||||
(println " With Nippy/default...")
|
||||
(swap! results_ assoc :nippy/default
|
||||
(bench1-serialization nippy/freeze nippy/thaw count
|
||||
(assoc opts :laps laps, :warmup warmup)))
|
||||
|
||||
(println " With Nippy/fast...")
|
||||
(swap! results_ assoc :nippy/fast
|
||||
(bench1-serialization nippy/fast-freeze nippy/fast-thaw count
|
||||
(assoc opts :laps laps, :warmup warmup)))
|
||||
|
||||
(println "\nBenchmarks done:")
|
||||
(printed-results @results_)))
|
||||
|
||||
;;;; Compression
|
||||
|
||||
(defn- bench1-compressor
|
||||
[compressor
|
||||
{:keys [laps warmup bench-data]
|
||||
:or
|
||||
{laps 1e4
|
||||
warmup 2e4
|
||||
bench-data default-bench-data}}]
|
||||
|
||||
(let [data-frozen (nippy/freeze bench-data {:compressor nil})
|
||||
data-compressed (compr/compress compressor data-frozen)
|
||||
time-compress (enc/bench laps {:warmup-laps warmup} (compr/compress compressor data-frozen))
|
||||
time-decompress (enc/bench laps {:warmup-laps warmup} (compr/decompress compressor data-compressed))]
|
||||
|
||||
{:round (+ time-compress time-decompress)
|
||||
:compress time-compress
|
||||
:decompress time-decompress
|
||||
:ratio (enc/round2 (/ (count data-compressed) (count data-frozen)))}))
|
||||
|
||||
(defn bench-compressors [opts lzma-opts]
|
||||
(printed-results
|
||||
(merge
|
||||
(let [bench1 #(bench1-compressor % opts)]
|
||||
{:zstd/prepended (bench1 (compr/->ZstdCompressor true))
|
||||
:zstd/unprepended (bench1 (compr/->ZstdCompressor false))
|
||||
:lz4 (bench1 (compr/->LZ4Compressor))
|
||||
:lzo (bench1 (compr/->LZOCompressor))
|
||||
:snappy/prepended (bench1 (compr/->SnappyCompressor true))
|
||||
:snappy/unprepended (bench1 (compr/->SnappyCompressor false))})
|
||||
|
||||
(let [bench1 #(bench1-compressor % (merge opts lzma-opts))]
|
||||
{:lzma2/level0 (bench1 (compr/->LZMA2Compressor 0))
|
||||
:lzma2/level3 (bench1 (compr/->LZMA2Compressor 3))
|
||||
:lzma2/level6 (bench1 (compr/->LZMA2Compressor 6))
|
||||
:lzma2/level9 (bench1 (compr/->LZMA2Compressor 9))}))))
|
||||
|
||||
;;;; Results
|
||||
|
||||
(comment
|
||||
{:last-updated "2024-01-16"
|
||||
:system "2020 Macbook Pro M1, 16 GB memory"
|
||||
:clojure-version "1.11.1"
|
||||
:java-version "OpenJDK 21"
|
||||
:deps
|
||||
'[[com.taoensso/nippy "3.4.0-beta1"]
|
||||
[org.clojure/tools.reader "1.3.7"]
|
||||
[org.clojure/data.fressian "1.0.0"]
|
||||
[org.tukaani/xz "1.9"]
|
||||
[io.airlift/aircompressor "0.25"]]}
|
||||
|
||||
(bench-serialization {:all? true})
|
||||
|
||||
{:reader {:round 13496, :freeze 5088, :thaw 8408, :size 15880}
|
||||
:fressian {:round 3898, :freeze 2350, :thaw 1548, :size 12222}
|
||||
:nippy/lzma2 {:round 12341, :freeze 7809, :thaw 4532, :size 3916}
|
||||
:nippy/encrypted {:round 2939, :freeze 1505, :thaw 1434, :size 8547}
|
||||
:nippy/default {:round 2704, :freeze 1330, :thaw 1374, :size 8519}
|
||||
:nippy/fast {:round 2425, :freeze 1117, :thaw 1308, :size 17088}}
|
||||
|
||||
(enc/round2 (/ 2704 13496)) ; 0.20 of reader roundtrip time
|
||||
(enc/round2 (/ 2704 3898)) ; 0.69 of fressian roundtrip time
|
||||
|
||||
(enc/round2 (/ 8519 15880)) ; 0.54 of reader output size
|
||||
(enc/round2 (/ 8519 12222)) ; 0.70 of fressian output size
|
||||
|
||||
(bench-compressors
|
||||
{:laps 1e4 :warmup 2e4}
|
||||
{:laps 1e2 :warmup 2e2})
|
||||
|
||||
;; Note that ratio depends on compressibility of stress data
|
||||
{:lz4 {:round 293, :compress 234, :decompress 59, :ratio 0.5}
|
||||
:lzo {:round 483, :compress 349, :decompress 134, :ratio 0.46}
|
||||
:snappy/prepended {:round 472, :compress 296, :decompress 176, :ratio 0.43}
|
||||
:snappy/unprepended {:round 420, :compress 260, :decompress 160, :ratio 0.43}
|
||||
:zstd/prepended {:round 2105, :compress 1419, :decompress 686, :ratio 0.3}
|
||||
:zstd/unprepended {:round 1261, :compress 921, :decompress 340, :ratio 0.3}
|
||||
:lzma2/level0 {:round 158, :compress 121, :decompress 37, :ratio 0.24}
|
||||
:lzma2/level3 {:round 536, :compress 436, :decompress 100, :ratio 0.22}
|
||||
:lzma2/level6 {:round 1136, :compress 1075, :decompress 61, :ratio 0.21}
|
||||
:lzma2/level9 {:round 2391, :compress 2096, :decompress 295, :ratio 0.21}})
|
||||
|
|
@ -1,471 +0,0 @@
|
|||
(ns taoensso.nippy-tests
|
||||
(:require
|
||||
[clojure.test :as test :refer [deftest testing is]]
|
||||
[clojure.test.check :as tc]
|
||||
[clojure.test.check.generators :as tc-gens]
|
||||
[clojure.test.check.properties :as tc-props]
|
||||
[taoensso.truss :as truss :refer [throws?]]
|
||||
[taoensso.encore :as enc :refer [ba=]]
|
||||
[taoensso.nippy :as nippy :refer [freeze thaw]]
|
||||
[taoensso.nippy.impl :as impl]
|
||||
[taoensso.nippy.tools :as tools]
|
||||
[taoensso.nippy.compression :as compr]
|
||||
[taoensso.nippy.crypto :as crypto]
|
||||
[taoensso.nippy-benchmarks :as benchmarks]))
|
||||
|
||||
(comment
|
||||
(remove-ns 'taoensso.nippy-tests)
|
||||
(test/run-tests 'taoensso.nippy-tests))
|
||||
|
||||
;;;; Config, etc.
|
||||
|
||||
(def test-data (nippy/stress-data {:comparable? true}))
|
||||
(def tc-gen-recursive-any-equatable
|
||||
(tc-gens/recursive-gen tc-gens/container-type
|
||||
tc-gens/any-equatable))
|
||||
|
||||
(defmacro gen-test [num-tests [data-sym] & body]
|
||||
`(let [tc-result#
|
||||
(tc/quick-check ~num-tests
|
||||
(tc-props/for-all [~data-sym tc-gen-recursive-any-equatable]
|
||||
~@body))]
|
||||
(true? (:pass? tc-result#))))
|
||||
|
||||
(comment
|
||||
(tc-gens/sample tc-gen-recursive-any-equatable 10)
|
||||
(gen-test 10 [gen-data] true))
|
||||
|
||||
;;;; Core
|
||||
|
||||
(deftest _core
|
||||
(println (str "Clojure version: " *clojure-version*))
|
||||
[(is (= test-data test-data) "Test data is comparable")
|
||||
(is (=
|
||||
(nippy/stress-data {:comparable? true})
|
||||
(nippy/stress-data {:comparable? true}))
|
||||
"Stress data is deterministic")
|
||||
|
||||
(is (= test-data ((comp thaw freeze) test-data)))
|
||||
(is (= test-data ((comp #(thaw % {:no-header? true
|
||||
:compressor nippy/lz4-compressor
|
||||
:encryptor nil})
|
||||
#(freeze % {:no-header? true}))
|
||||
test-data)))
|
||||
|
||||
(is (= test-data ((comp #(thaw % {:password [:salted "p"]})
|
||||
#(freeze % {:password [:salted "p"]}))
|
||||
test-data)))
|
||||
|
||||
(is (= test-data ((comp #(thaw % {:compressor nippy/lzma2-compressor})
|
||||
#(freeze % {:compressor nippy/lzma2-compressor}))
|
||||
test-data)))
|
||||
|
||||
(is (= test-data ((comp #(thaw % {:compressor nippy/lzma2-compressor
|
||||
:password [:salted "p"]})
|
||||
#(freeze % {:compressor nippy/lzma2-compressor
|
||||
:password [:salted "p"]}))
|
||||
test-data)))
|
||||
|
||||
(is (= test-data ((comp #(thaw % {:compressor nippy/lz4-compressor})
|
||||
#(freeze % {:compressor nippy/lz4-compressor}))
|
||||
test-data)))
|
||||
|
||||
(is (= test-data ((comp #(thaw % {:compressor nippy/zstd-compressor})
|
||||
#(freeze % {:compressor nippy/zstd-compressor}))
|
||||
test-data)))
|
||||
|
||||
(is (throws? Exception (thaw (freeze test-data {:password "malformed"}))))
|
||||
(is (throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
|
||||
(is (throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
|
||||
|
||||
(is
|
||||
(= "payload"
|
||||
(thaw (freeze "payload" {:password [:salted "pwd"] :encryptor nippy/aes128-cbc-encryptor})
|
||||
(do {:password [:salted "pwd"]})))
|
||||
"CBC auto-encryptor compatibility")
|
||||
|
||||
(testing "Unsigned long types"
|
||||
(let [range-ushort+ (+ (long @#'nippy/range-ushort) 128)
|
||||
range-uint+ (+ (long @#'nippy/range-uint) 128)]
|
||||
|
||||
[(let [r (range (long -2.5e6) (long 2.5e6))] (= (thaw (freeze r)) r))
|
||||
(let [r (range (- range-ushort+) range-ushort+)] (= (thaw (freeze r)) r))
|
||||
(let [n range-uint+] (= (thaw (freeze n)) n))
|
||||
(let [n (- range-uint+)] (= (thaw (freeze n)) n))]))
|
||||
|
||||
(is (throws? :ex-info "Failed to freeze type" (nippy/freeze (fn []))))
|
||||
|
||||
(testing "Clojure v1.10+ metadata protocol extensions"
|
||||
[(is (throws? :ex-info "Failed to freeze type" (nippy/freeze (with-meta [] {:a :A, 'b (fn [])}))))
|
||||
(is (= {:a :A} (meta (nippy/thaw (nippy/freeze (with-meta [] {:a :A, 'b/c (fn [])}))))))
|
||||
(is (= nil (meta (nippy/thaw (nippy/freeze (with-meta [] { 'b/c (fn [])})))))
|
||||
"Don't attach empty metadata")])
|
||||
|
||||
(let [d (nippy/stress-data {})]
|
||||
[(is (= (vec (:bytes d)) ((comp vec thaw freeze) (:bytes d))))
|
||||
(is (= (vec (:objects d)) ((comp vec thaw freeze) (:objects d))))])
|
||||
|
||||
(testing "Arrays"
|
||||
(binding [nippy/*thaw-serializable-allowlist* nippy/default-freeze-serializable-allowlist]
|
||||
(mapv (fn [[k aval]] (is (= (vec aval) (-> aval nippy/freeze nippy/thaw vec)) (name k)))
|
||||
(get-in (nippy/stress-data {}) [:non-comparable :arrays]))))
|
||||
|
||||
(is (gen-test 1600 [gen-data] (= gen-data (thaw (freeze gen-data)))) "Generative")])
|
||||
|
||||
;;;; Custom types & records
|
||||
|
||||
(deftype MyType [basic_field fancy-field!]) ; Note `fancy-field!` field name will be munged
|
||||
(defrecord MyRec [basic_field fancy-field!])
|
||||
|
||||
(deftest _types
|
||||
[(testing "Extend to custom type"
|
||||
[(is
|
||||
(throws? Exception ; No thaw extension yet
|
||||
(do
|
||||
(alter-var-root #'nippy/*custom-readers* (constantly {}))
|
||||
(nippy/extend-freeze MyType 1 [x s]
|
||||
(.writeUTF s (.basic_field x))
|
||||
(.writeUTF s (.fancy-field! x)))
|
||||
|
||||
(thaw (freeze (MyType. "basic" "fancy"))))))
|
||||
|
||||
(is
|
||||
(do
|
||||
(nippy/extend-thaw 1 [s] (MyType. (.readUTF s) (.readUTF s)))
|
||||
(let [mt1 (MyType. "basic" "fancy")
|
||||
^MyType mt2 (thaw (freeze mt1))]
|
||||
(=
|
||||
[(.basic_field mt1) (.fancy-field! mt1)]
|
||||
[(.basic_field mt2) (.fancy-field! mt2)]))))])
|
||||
|
||||
(testing "Extend to custom Record"
|
||||
(is
|
||||
(do
|
||||
(nippy/extend-freeze MyRec 2 [x s]
|
||||
(.writeUTF s (str "foo-" (:basic_field x)))
|
||||
(.writeUTF s (str "foo-" (:fancy-field! x))))
|
||||
|
||||
(nippy/extend-thaw 2 [s] (MyRec. (.readUTF s) (.readUTF s)))
|
||||
(=
|
||||
(do (MyRec. "foo-basic" "foo-fancy"))
|
||||
(thaw (freeze (MyRec. "basic" "fancy")))))))
|
||||
|
||||
(testing "Keyword (prefixed) extensions"
|
||||
(is
|
||||
(do
|
||||
(nippy/extend-freeze MyRec :nippy-tests/MyRec [x s]
|
||||
(.writeUTF s (:basic_field x))
|
||||
(.writeUTF s (:fancy-field! x)))
|
||||
|
||||
(nippy/extend-thaw :nippy-tests/MyRec [s] (MyRec. (.readUTF s) (.readUTF s)))
|
||||
(let [mr (MyRec. "basic" "fancy")]
|
||||
(= mr (thaw (freeze mr)))))))])
|
||||
|
||||
;;;; Caching
|
||||
|
||||
(deftest _caching
|
||||
(let [test-data* [test-data test-data test-data test-data] ; Data with duplicates
|
||||
cached (mapv nippy/cache test-data*)
|
||||
cached (mapv nippy/cache test-data*) ; <=1 wrap auto-enforced
|
||||
]
|
||||
|
||||
[(is (= test-data* (thaw (freeze test-data* {:compressor nil}))))
|
||||
(is (= test-data* (thaw (freeze cached {:compressor nil}))))
|
||||
(let [size-stress (count (freeze test-data* {:compressor nil}))
|
||||
size-cached (count (freeze cached {:compressor nil}))]
|
||||
(is (>= size-stress (* 3 size-cached)))
|
||||
(is (< size-stress (* 4 size-cached))))]))
|
||||
|
||||
(deftest _caching-metadata
|
||||
(let [v1 (with-meta [] {:id :v1})
|
||||
v2 (with-meta [] {:id :v2})
|
||||
|
||||
frozen-without-caching (freeze [v1 v2 v1 v2])
|
||||
frozen-with-caching
|
||||
(freeze [(nippy/cache v1)
|
||||
(nippy/cache v2)
|
||||
(nippy/cache v1)
|
||||
(nippy/cache v2)])]
|
||||
|
||||
[(is (> (count frozen-without-caching)
|
||||
(count frozen-with-caching)))
|
||||
|
||||
(is (= (thaw frozen-without-caching)
|
||||
(thaw frozen-with-caching)))
|
||||
|
||||
(is (= (mapv meta (thaw frozen-with-caching))
|
||||
[{:id :v1} {:id :v2} {:id :v1} {:id :v2}]))]))
|
||||
|
||||
;;;; Serialized output
|
||||
|
||||
(defn ba-hash [^bytes ba] (hash (seq ba)))
|
||||
|
||||
(defn gen-hashes [] (enc/map-vals (fn [v] (ba-hash (freeze v))) test-data))
|
||||
(defn cmp-hashes [new old] (vec (sort (reduce-kv (fn [s k v] (if (= (get old k) v) s (conj s k))) #{} new))))
|
||||
|
||||
(def ref-hashes-v341
|
||||
{:deftype -148586793, :lazy-seq-empty 1277437598, :true -1809580601, :long 598276629, :double -454270428, :lazy-seq -1039619789, :short 1152993378, :meta -858252893, :str-long -1970041891, :instant -1401948864, :many-keywords 665654816, :bigint 2033662230, :sym-ns 769802402, :queue 447747779, :float 603100813, :sorted-set 2005004017, :many-strings 1738215727, :nested -1350538572, :queue-empty 1760934486, :duration -775528642, :false 1506926383, :vector 813550992, :util-date 1326218051, :kw 389651898, :sym -1742024487, :str-short -921330463, :subvec 709331681, :kw-long 852232872, :integer 624865727, :sym-long -1535730190, :list -1207486853, :ratio 1186850097, :byte -1041979678, :bigdec -1846988137, :nil 2005042235, :defrecord -553848560, :sorted-map -1160380145, :sql-date 80018667, :map-entry 1219306839, :false-boxed 1506926383, :uri 870148616, :period -2043530540, :many-longs -1109794519, :uuid -338331115, :set 1649942133, :kw-ns 1050084331, :map 1989337680, :many-doubles -827569787, :char 858269588})
|
||||
|
||||
(def ref-hashes-v340
|
||||
{:deftype 1529147805, :lazy-seq-empty 1277437598, :true -1809580601, :long 219451189, :double -454270428, :lazy-seq -1039619789, :short 1152993378, :meta 352218350, :str-long -1970041891, :instant -1401948864, :many-keywords 665654816, :bigint 2033662230, :sym-ns 769802402, :queue 447747779, :float 603100813, :sorted-set 1443292905, :many-strings 1777678883, :nested -1590473924, :queue-empty 1760934486, :duration -775528642, :false 1506926383, :vector 89425525, :util-date 1326218051, :kw 389651898, :sym -1742024487, :str-short -1097575232, :subvec -2047667173, :kw-long 852232872, :integer 624865727, :sym-long -1535730190, :list -1113199651, :ratio 1186850097, :byte -1041979678, :bigdec -1846988137, :nil 2005042235, :defrecord 287634761, :sorted-map 1464032648, :sql-date 80018667, :map-entry -1353323498, :false-boxed 1506926383, :uri -1374752165, :period -2043530540, :many-longs 759118414, :uuid -338331115, :set -1515144175, :kw-ns 1050084331, :map 358912619, :many-doubles -827569787, :char 858269588})
|
||||
|
||||
(comment
|
||||
(cmp-hashes ref-hashes-v341 ref-hashes-v340)
|
||||
[:defrecord :deftype :list :long :many-longs :many-strings :map :map-entry :meta :nested :set :sorted-map :sorted-set :str-short :subvec :uri :vector])
|
||||
|
||||
(deftest _stable-serialized-output
|
||||
(testing "Stable serialized output"
|
||||
|
||||
(testing "x=y => f(x)=f(y) for SOME inputs, SOMETIMES"
|
||||
;; `x=y => f(x)=f(y)` is unfortunately NOT true in general, and NOT something we
|
||||
;; promise. Still, we do unofficially try our best to maintain this property when
|
||||
;; possible - and to warn when it'll be violated for common/elementary types.
|
||||
[(is (not (ba= (freeze {:a 1 :b 1}) (freeze {:b 1 :a 1}))) "Small (array) map (not= (seq {:a 1 :b 1}) (seq {:b 1 :a 1}))")
|
||||
(is (not (ba= (freeze [[]]) (freeze ['()]))) "(= [] '()) is true")
|
||||
(is (ba= (freeze (sorted-map :a 1 :b 1))
|
||||
(freeze (sorted-map :b 1 :a 1))) "Sorted structures are generally safe")
|
||||
|
||||
;; Track serialized output of stress data so that we can detect unintentional changes,
|
||||
;; and warn about intended ones. Hashes will need to be recalculated on changes to stress data.
|
||||
(let [reference-hashes ref-hashes-v341
|
||||
failures ; #{{:keys [k v]}}
|
||||
(reduce-kv
|
||||
(fn [failures k v]
|
||||
(or
|
||||
(when (not= v :taoensso.nippy/skip)
|
||||
(let [frozen (freeze v)
|
||||
actual (ba-hash frozen)
|
||||
ref (get reference-hashes k)]
|
||||
(when (not= actual ref)
|
||||
(conj failures
|
||||
{:k k,
|
||||
:v {:type (type v), :value v}
|
||||
:actual actual
|
||||
:ref ref
|
||||
:frozen (vec frozen)}))))
|
||||
failures))
|
||||
#{}
|
||||
test-data)]
|
||||
|
||||
(is (empty? failures)))])
|
||||
|
||||
(testing "x==y => f(x)=f(y)"
|
||||
;; This weaker version of `x=y => f(x)=f(y)` does hold
|
||||
[(is (ba= (freeze test-data)
|
||||
(freeze test-data)))
|
||||
|
||||
(is (every? true? (repeatedly 1000 (fn [] (ba= (freeze test-data)
|
||||
(freeze test-data)))))
|
||||
"Try repeatedly to catch possible protocol interface races")
|
||||
|
||||
(is (gen-test 400 [gen-data]
|
||||
(ba= (freeze gen-data)
|
||||
(freeze gen-data))) "Generative")])
|
||||
|
||||
(testing "f(x)=f(f-1(f(x)))"
|
||||
[(is (ba= (-> test-data freeze)
|
||||
(-> test-data freeze thaw freeze)))
|
||||
|
||||
(is (ba= (freeze test-data)
|
||||
(reduce (fn [frozen _] (freeze (thaw frozen))) (freeze test-data) (range 1000)))
|
||||
"Try repeatedly to catch possible protocol interface races")
|
||||
|
||||
(is (gen-test 400 [gen-data]
|
||||
(ba= (-> gen-data freeze)
|
||||
(-> gen-data freeze thaw freeze))) "Generative")])
|
||||
|
||||
(testing "f(x)=f(y) => x=y"
|
||||
(let [vals_ (atom {})]
|
||||
(gen-test 400 [gen-data]
|
||||
(let [out (freeze gen-data)
|
||||
ref (get @vals_ gen-data ::nx)]
|
||||
(swap! vals_ assoc out gen-data)
|
||||
(or (= ref ::nx) (= ref out))))))))
|
||||
|
||||
;;;; Thread safety
|
||||
|
||||
(deftest _thread-safe
|
||||
[(is
|
||||
(let [futures (mapv (fn [_] (future (= (thaw (freeze test-data)) test-data)))
|
||||
(range 50))]
|
||||
(every? deref futures)))
|
||||
|
||||
(is
|
||||
(let [futures
|
||||
(mapv
|
||||
(fn [_]
|
||||
(future
|
||||
(= (thaw (freeze test-data {:password [:salted "password"]})
|
||||
{:password [:salted "password"]})
|
||||
test-data)))
|
||||
(range 50))]
|
||||
(every? deref futures)))
|
||||
|
||||
(is
|
||||
(let [futures
|
||||
(mapv
|
||||
(fn [_]
|
||||
(future
|
||||
(= (thaw (freeze test-data {:password [:cached "password"]})
|
||||
{:password [:cached "password"]})
|
||||
test-data)))
|
||||
(range 50))]
|
||||
(every? deref futures)))])
|
||||
|
||||
;;;; Redefs
|
||||
|
||||
(defrecord MyFoo [] Object (toString [_] "v1"))
|
||||
(defrecord MyFoo [] Object (toString [_] "v2"))
|
||||
|
||||
(deftest _redefs
|
||||
(is (= (str (thaw (freeze (MyFoo.)))) "v2")))
|
||||
|
||||
;;;; Serializable
|
||||
|
||||
(do
|
||||
(def ^:private semcn "java.util.concurrent.Semaphore")
|
||||
(def ^:private sem (java.util.concurrent.Semaphore. 1))
|
||||
(defn- sem? [x] (instance? java.util.concurrent.Semaphore x)))
|
||||
|
||||
(deftest _serializable
|
||||
[(is (= nippy/*thaw-serializable-allowlist* #{"base.1" "base.2" "add.1" "add.2"})
|
||||
"JVM properties override initial allowlist values")
|
||||
|
||||
(is (throws? Exception (nippy/freeze sem {:serializable-allowlist #{}}))
|
||||
"Can't freeze Serializable objects unless approved by allowlist")
|
||||
|
||||
(is (sem?
|
||||
(nippy/thaw
|
||||
(nippy/freeze sem {:serializable-allowlist #{semcn}})
|
||||
{:serializable-allowlist #{semcn}}))
|
||||
|
||||
"Can freeze and thaw Serializable objects if approved by allowlist")
|
||||
|
||||
(is (sem?
|
||||
(nippy/thaw
|
||||
(nippy/freeze sem {:serializable-allowlist #{"java.util.concurrent.*"}})
|
||||
{:serializable-allowlist #{"java.util.concurrent.*"}}))
|
||||
|
||||
"Strings in allowlist sets may contain \"*\" wildcards")
|
||||
|
||||
(let [ba (nippy/freeze sem #_{:serializable-allowlist "*"})
|
||||
thawed (nippy/thaw ba {:serializable-allowlist #{}})]
|
||||
|
||||
[(is (= :quarantined (get-in thawed [:nippy/unthawable :cause]))
|
||||
"Serializable objects will be quarantined when approved for freezing but not thawing.")
|
||||
|
||||
(is (sem? (nippy/read-quarantined-serializable-object-unsafe! thawed))
|
||||
"Quarantined Serializable objects can still be manually force-read.")
|
||||
|
||||
(is (sem? (nippy/read-quarantined-serializable-object-unsafe!
|
||||
(nippy/thaw (nippy/freeze thawed))))
|
||||
"Quarantined Serializable objects are themselves safely transportable.")])
|
||||
|
||||
(let [obj
|
||||
(nippy/thaw
|
||||
(nippy/freeze sem)
|
||||
{:serializable-allowlist "allow-and-record"})]
|
||||
|
||||
[(is (sem? obj)
|
||||
"Special \"allow-and-record\" allowlist permits any class")
|
||||
|
||||
(is
|
||||
(contains? (nippy/get-recorded-serializable-classes) semcn)
|
||||
"Special \"allow-and-record\" allowlist records classes")])])
|
||||
|
||||
;;;; Metadata
|
||||
|
||||
(def my-var "Just a string")
|
||||
|
||||
(deftest _metadata
|
||||
[(is
|
||||
(:has-meta?
|
||||
(meta
|
||||
(nippy/thaw
|
||||
(nippy/freeze (with-meta [] {:has-meta? true}) {:incl-metadata? true})
|
||||
{:incl-metadata? true}
|
||||
)))
|
||||
|
||||
"Metadata successfully included")
|
||||
|
||||
(is
|
||||
(nil?
|
||||
(meta
|
||||
(nippy/thaw
|
||||
(nippy/freeze (with-meta [] {:has-meta? true}) {:incl-metadata? true})
|
||||
{:incl-metadata? false}
|
||||
)))
|
||||
|
||||
"Metadata successfully excluded by thaw")
|
||||
|
||||
(is
|
||||
(nil?
|
||||
(meta
|
||||
(nippy/thaw
|
||||
(nippy/freeze (with-meta [] {:has-meta? true}) {:incl-metadata? false})
|
||||
{:incl-metadata? true}
|
||||
)))
|
||||
|
||||
"Metadata successfully excluded by freeze")
|
||||
|
||||
(is (var? (nippy/read-quarantined-serializable-object-unsafe!
|
||||
(nippy/thaw (nippy/freeze #'my-var))))
|
||||
|
||||
"Don't try to preserve metadata on vars")])
|
||||
|
||||
;;;; Freezable?
|
||||
|
||||
(deftest _freezable?
|
||||
[(is (= (nippy/freezable? :foo) :native))
|
||||
(is (= (nippy/freezable? [:a :b]) :native))
|
||||
(is (= (nippy/freezable? [:a (fn [])]) nil))
|
||||
(is (= (nippy/freezable? [:a (byte-array [1 2 3])]) :native))
|
||||
(is (= (nippy/freezable? [:a (java.util.Date.)]) :native))
|
||||
(is (= (nippy/freezable? (Exception.)) nil))
|
||||
(is (= (nippy/freezable? (MyType. "a" "b")) :native))
|
||||
(is (= (nippy/freezable? (MyRec. "a" "b")) :native))
|
||||
(is (= (nippy/freezable? (Exception.) {:allow-java-serializable? true})
|
||||
:maybe-java-serializable))])
|
||||
|
||||
;;;; thaw-xform
|
||||
|
||||
(deftest _thaw-xform
|
||||
[(is (= (binding [nippy/*thaw-xform* nil] (thaw (freeze [1 2 :secret 3 4]))) [1 2 :secret 3 4]))
|
||||
(is (= (binding [nippy/*thaw-xform* (map (fn [x] (if (= x :secret) :redacted x)))] (thaw (freeze [1 2 :secret 3 4]))) [1 2 :redacted 3 4]))
|
||||
|
||||
(is (= (binding [nippy/*thaw-xform* (remove (fn [x] (and (map-entry? x) (and (= (key x) :x) (val x)))))]
|
||||
(thaw (freeze {:a :A, :b :B, :x :X, :c {:x :X}, :d #{:d1 :d2 {:d3 :D3, :x :X}}})))
|
||||
{:a :A, :b :B, :c {}, :d #{:d1 :d2 {:d3 :D3}}}))
|
||||
|
||||
(is (= (binding [nippy/*thaw-xform* (remove (fn [x] (and (map? x) (contains? x :x))))]
|
||||
(thaw (freeze {:a :A, :b :B, :x :X, :c {:x :X}, :d #{:d1 :d2 {:d3 :D3, :x :X}}})))
|
||||
{:a :A, :b :B, :x :X, :c {:x :X}, :d #{:d1 :d2}}))
|
||||
|
||||
(is (= (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze []))) []) "rf not run on empty colls")
|
||||
|
||||
(let [ex (truss/throws :default (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze [:a :b]))))]
|
||||
(is (= (-> ex ex-cause ex-cause ex-data :call) '(rf acc in)) "Error thrown via `*thaw-xform*`"))])
|
||||
|
||||
;;;; Compressors
|
||||
|
||||
(deftest _compressors
|
||||
(println "\nTesting decompression of random data...")
|
||||
(doseq [c [compr/zstd-compressor
|
||||
compr/lz4-compressor
|
||||
compr/lzo-compressor
|
||||
compr/snappy-compressor
|
||||
compr/lzma2-compressor]]
|
||||
|
||||
(print (str " With " (name (compr/header-id c)))) (flush)
|
||||
(dotimes [_ 5] ; Slow, a few k laps should be sufficient for CI
|
||||
(print ".") (flush)
|
||||
(dotimes [_ 1000]
|
||||
(is
|
||||
(nil? (truss/catching :all (compr/decompress c (crypto/rand-bytes 1024))))
|
||||
"Decompression never crashes JVM, even against invalid data")))
|
||||
(println)))
|
||||
|
||||
;;;; Benchmarks
|
||||
|
||||
(deftest _benchmarks
|
||||
(is (benchmarks/bench-serialization {:all? true})))
|
||||
1
wiki/.gitignore
vendored
1
wiki/.gitignore
vendored
|
|
@ -1 +0,0 @@
|
|||
README.md
|
||||
|
|
@ -1,145 +0,0 @@
|
|||
# Setup
|
||||
|
||||
Add the [relevant dependency](../#latest-releases) to your project:
|
||||
|
||||
```clojure
|
||||
Leiningen: [com.taoensso/nippy "x-y-z"] ; or
|
||||
deps.edn: com.taoensso/nippy {:mvn/version "x-y-z"}
|
||||
```
|
||||
|
||||
And setup your namespace imports:
|
||||
|
||||
```clojure
|
||||
(ns my-app (:require [taoensso.nippy :as nippy]))
|
||||
```
|
||||
|
||||
# De/serializing
|
||||
|
||||
As an example of what it can do, let's take a look at Nippy's own reference [stress data](https://taoensso.github.io/nippy/taoensso.nippy.html#var-stress-data):
|
||||
|
||||
```clojure
|
||||
{:nil nil
|
||||
:true true
|
||||
:false false
|
||||
:false-boxed (Boolean. false)
|
||||
|
||||
:char \ಬ
|
||||
:str-short "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"
|
||||
:str-long (reduce str (range 1024))
|
||||
:kw :keyword
|
||||
:kw-ns ::keyword
|
||||
:sym 'foo
|
||||
:sym-ns 'foo/bar
|
||||
:kw-long (keyword (reduce str "_" (range 128)) (reduce str "_" (range 128)))
|
||||
:sym-long (symbol (reduce str "_" (range 128)) (reduce str "_" (range 128)))
|
||||
|
||||
:byte (byte 16)
|
||||
:short (short 42)
|
||||
:integer (int 3)
|
||||
:long (long 3)
|
||||
:float (float 3.1415926535897932384626433832795)
|
||||
:double (double 3.1415926535897932384626433832795)
|
||||
:bigdec (bigdec 3.1415926535897932384626433832795)
|
||||
:bigint (bigint 31415926535897932384626433832795)
|
||||
:ratio 22/7
|
||||
|
||||
:list (list 1 2 3 4 5 (list 6 7 8 (list 9 10 (list) ())))
|
||||
:vector [1 2 3 4 5 [6 7 8 [9 10 [[]]]]]
|
||||
:subvec (subvec [1 2 3 4 5 6 7 8] 2 8)
|
||||
:map {:a 1 :b 2 :c 3 :d {:e 4 :f {:g 5 :h 6 :i 7 :j {{} {}}}}}
|
||||
:map-entry (clojure.lang.MapEntry/create "key" "val")
|
||||
:set #{1 2 3 4 5 #{6 7 8 #{9 10 #{#{}}}}}
|
||||
:meta (with-meta {:a :A} {:metakey :metaval})
|
||||
:nested [#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{[] ()}}] #{:a :b}}
|
||||
#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{[] ()}}] #{:a :b}}
|
||||
[1 [1 2 [1 2 3 [1 2 3 4 [1 2 3 4 5 "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"] {} #{} [] ()]]]]]
|
||||
|
||||
:regex #"^(https?:)?//(www\?|\?)?"
|
||||
:sorted-set (sorted-set 1 2 3 4 5)
|
||||
:sorted-map (sorted-map :b 2 :a 1 :d 4 :c 3)
|
||||
:lazy-seq-empty (map identity ())
|
||||
:lazy-seq (repeatedly 64 #(do nil))
|
||||
:queue-empty (into clojure.lang.PersistentQueue/EMPTY [:a :b :c :d :e :f :g])
|
||||
:queue clojure.lang.PersistentQueue/EMPTY
|
||||
|
||||
:uuid (java.util.UUID. 7232453380187312026 -7067939076204274491)
|
||||
:uri (java.net.URI. "https://clojure.org")
|
||||
:defrecord (nippy/StressRecord. "data")
|
||||
:deftype (nippy/StressType. "data")
|
||||
:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
|
||||
:objects (object-array [1 "two" {:data "data"}])
|
||||
|
||||
:util-date (java.util.Date. 1577884455500)
|
||||
:sql-date (java.sql.Date. 1577884455500)
|
||||
:instant (java.time.Instant/parse "2020-01-01T13:14:15.50Z")
|
||||
:duration (java.time.Duration/ofSeconds 100 100)
|
||||
:period (java.time.Period/of 1 1 1)
|
||||
|
||||
:throwable (Throwable. "Msg")
|
||||
:exception (Exception. "Msg")
|
||||
:ex-info (ex-info "Msg" {:data "data"})
|
||||
|
||||
:many-longs (vec (repeatedly 512 #(rand-nth (range 10))))
|
||||
:many-doubles (vec (repeatedly 512 #(double (rand-nth (range 10)))))
|
||||
:many-strings (vec (repeatedly 512 #(rand-nth ["foo" "bar" "baz" "qux"])))
|
||||
:many-keywords (vec (repeatedly 512
|
||||
#(keyword
|
||||
(rand-nth ["foo" "bar" "baz" "qux" nil])
|
||||
(rand-nth ["foo" "bar" "baz" "qux" ]))))}
|
||||
```
|
||||
|
||||
Serialize it:
|
||||
|
||||
```clojure
|
||||
(def frozen-stress-data (nippy/freeze (nippy/stress-data {})))
|
||||
=> #<byte[] [B@3253bcf3>
|
||||
```
|
||||
|
||||
Deserialize it:
|
||||
|
||||
```clojure
|
||||
(nippy/thaw frozen-stress-data)
|
||||
=> {:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
|
||||
:nil nil
|
||||
:boolean true
|
||||
<...> }
|
||||
```
|
||||
|
||||
Couldn't be simpler!
|
||||
|
||||
# Streaming
|
||||
|
||||
- To serialize directly to a `java.io.DataInput`, see [`freeze-to-out!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-freeze-to-out.21).
|
||||
- To deserialize directly from a `java.io.DataOutput`, see [`thaw-from-in!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-thaw-from-in.21).
|
||||
|
||||
# Encryption
|
||||
|
||||
> You may want to consider using Nippy with [Tempel](https://www.taoensso.com/tempel) for more comprehensive encryption options.
|
||||
|
||||
Nippy also gives you **dead simple data encryption**.
|
||||
Add a single option to your usual freeze/thaw calls like so:
|
||||
|
||||
```clojure
|
||||
(nippy/freeze (nippy/stress-data {}) {:password [:salted "my-password"]}) ; Encrypt
|
||||
(nippy/thaw <encrypted-data> {:password [:salted "my-password"]}) ; Decrypt
|
||||
```
|
||||
|
||||
There's two default forms of encryption on offer: `:salted` and `:cached`. Each of these makes carefully-chosen trade-offs and is suited to one of two common use cases. See [`aes128-encryptor`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-aes128-encryptor) for a detailed explanation of why/when you'd want one or the other.
|
||||
|
||||
# Custom types
|
||||
|
||||
It's easy to extend Nippy to your own data types:
|
||||
|
||||
```clojure
|
||||
(defrecord MyType [data])
|
||||
|
||||
(nippy/extend-freeze MyType :my-type/foo ; A unique (namespaced) type identifier
|
||||
[x data-output]
|
||||
(.writeUTF data-output (:data x)))
|
||||
|
||||
(nippy/extend-thaw :my-type/foo ; Same type id
|
||||
[data-input]
|
||||
(MyType. (.readUTF data-input)))
|
||||
|
||||
(nippy/thaw (nippy/freeze (MyType. "Joe"))) => #taoensso.nippy.MyType{:data "Joe"}
|
||||
```
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
# Data longevity
|
||||
|
||||
Nippy is widely used to store **long-lived** data and promises (as always) that **data serialized today should be readable by all future versions of Nippy**.
|
||||
|
||||
But please note that the **converse is not generally true**:
|
||||
|
||||
- Nippy `vX` **should** be able to read all data from Nippy `vY<=X` (backwards compatibility)
|
||||
- Nippy `vX` **may/not** be able to read all data from Nippy `vY>X` (forwards compatibility)
|
||||
|
||||
# Rolling updates and rollback
|
||||
|
||||
From time to time, Nippy may introduce:
|
||||
|
||||
- Support for serializing **new types**
|
||||
- Optimizations to the serialization of **pre-existing types**
|
||||
|
||||
To help ease **rolling updates** and to better support **rollback**, Nippy (since version v3.4.1) will always introduce such changes over **two version releases**:
|
||||
|
||||
- Release 1: to add **read support** for the new types
|
||||
- Release 2: to add **write support** for the new types
|
||||
|
||||
Starting from v3.4.1, Nippy's release notes will **always clearly indicate** if a particular update sequence is recommended.
|
||||
|
||||
# Stability of byte output
|
||||
|
||||
It has **never been an objective** of Nippy to offer **predictable byte output**, and I'd generally **recommend against** depending on specific byte output.
|
||||
|
||||
However, I know that a small minority of users *do* have specialized needs in this area.
|
||||
|
||||
So starting with Nippy v3.4, Nippy's release notes will **always clearly indicate** if any changes to byte output are expected.
|
||||
|
|
@ -1,278 +0,0 @@
|
|||
> This article is **community content** kindly contributed by a Nippy user (@Outrovurt)
|
||||
|
||||
This article describes a number of use cases where you need to make changes to your code which will have some impact on data you have already frozen using Nippy, and how best to manage each specific case. We will also discuss custom freezing and thawing.
|
||||
|
||||
It is assumed you have working knowledge of a good editor (e.g. emacs) and know how to start a Clojure REPL (e.g. via CIDER).
|
||||
|
||||
Throughout this article we will refer to serialization as *freezing*, and deserialization as *thawing*, which are the terms used by Nippy. (The word "thawer" will be used throughout the article, it's obviously made up, but please bear with it!)
|
||||
|
||||
# Project setup
|
||||
|
||||
If you want to follow along, you can create a fresh project using whichever Clojure project management tool you are currently most comfortable with (lein, Clojure deps), and [include Nippy in the dependencies](https://github.com/taoensso/nippy/wiki#setup). Create the following namespaces:
|
||||
|
||||
* nippy.evolve
|
||||
* nippy.other
|
||||
|
||||
and delete any default code which is generated for you by your project management tool.
|
||||
|
||||
We will start in `nippy.evolve`. Ensure your `(ns ...)` looks like this:
|
||||
|
||||
```clojure
|
||||
(ns nippy.evolve
|
||||
(:require
|
||||
[taoensso.nippy
|
||||
:refer (freeze
|
||||
thaw)]))
|
||||
```
|
||||
|
||||
Start a REPL and switch to `nippy.evolve` if you aren't already there.
|
||||
|
||||
# Freezing a record
|
||||
|
||||
Create the following record, either in your source code or in the REPL, and create a new instance of it:
|
||||
|
||||
```clojure
|
||||
(defrecord FirstRec [])
|
||||
(def x1 (FirstRec.))
|
||||
```
|
||||
|
||||
Now freeze it:
|
||||
|
||||
```clojure
|
||||
(def f1 (freeze x1))
|
||||
```
|
||||
|
||||
The first thing you'll notice is that you don't have to set anything up at all, Nippy will freeze the record instance out of the box. It does this by determining that `x1` is a record, and uses a built-in freezer to freeze it.
|
||||
|
||||
## A frozen record
|
||||
|
||||
Let's take a quick look at `f1` at the REPL:
|
||||
|
||||
```clojure
|
||||
nippy.evolve> (type f1)
|
||||
[B
|
||||
nippy.evolve> f1
|
||||
[78, 80, 89, 0, 48, 21, 110, 105, 112, 112, 121, 46, 101, 118, 111,
|
||||
108, 118, 101, 46, 70, 105, 114, 115, 116, 82, 101, 99, 19]
|
||||
```
|
||||
|
||||
The "[B" indicates that this is a Java Byte array, which you can also create using the Clojure function `byte-array`.
|
||||
|
||||
The value of `f1` is an array of bytes, consisting of two parts:
|
||||
|
||||
* the envelope
|
||||
- a 4-byte header, followed by
|
||||
- a 1-byte type id
|
||||
* the packet itself, whatever you are encoding
|
||||
|
||||
The header provides sanity checking, the Nippy version number that created it, and information related to any compression or encryption employed. We won't go into detail here as to how the header is composed as it is not relevant to our discussion.
|
||||
|
||||
The type id tells us what type of data we have frozen. A positive value indicates a built-in type. In this case we have 48, which refers to a record, specifically a "small record" (small referring to the total number of bytes used to encode the body of the record.) This is all explained in the Nippy source code, where you can find a list of all built-in types that Nippy recognizes together with the type id used to represent it. In the latest version of Nippy this list can be found in the var `public-types-spec`.
|
||||
|
||||
So when we froze an instance of `FirstRec`, we used a built-in freezer for records to produce the above byte array.
|
||||
|
||||
What about the rest of the array? There are a total of 28 bytes, 5 of which are for the envelope, so the remaining 23 bytes are for the packet itself. But wait a minute. If we are basically freezing an empty record with no fields, why do we need 23 bytes to freeze it? The answer is that while the envelope encodes the fact that the remaining data is for a record, it also needs to encode the type of record that we have frozen. Otherwise there is no way to thaw it back to a record of type `FirstRec`. Since all envelopes contain 5 bytes, this information can only be encoded in one place: the packet itself.
|
||||
|
||||
So if you type the following at the REPL, you'll get a clue as to what is happening:
|
||||
|
||||
```clojure
|
||||
nippy.evolve> (freeze "nippy.evolve.FirstRec")
|
||||
[78, 80, 89, 0, 105, 21, 110, 105, 112, 112, 121, 46, 101, 118, 111,
|
||||
108, 118, 101, 46, 70, 105, 114, 115, 116, 82, 101, 99]
|
||||
```
|
||||
|
||||
Once again the first 4-bytes represent the header. The fifth byte, 105, tells us that this is a string (specifically a "small" string of under 128 bytes), and the remaining packet is a 22-byte array:
|
||||
|
||||
`[21, 110, 105, 112, 112, 121, 46, 101, 118, 111,
|
||||
108, 118, 101, 46, 70, 105, 114, 115, 116, 82, 101, 99]`
|
||||
|
||||
If you look back at `f1` above, you will see that of the 23 bytes of the packet, the first 22 bytes are identical to the 22 bytes of the frozen string above. In other words, Nippy encodes the fully-qualified record name at the very start of the packet itself.
|
||||
|
||||
That just leaves a single byte, 19, at the end of `f1`. You can probably guess what this represents, so let's try the following:
|
||||
|
||||
```clojure
|
||||
nippy.evolve> (freeze {})
|
||||
[78, 80, 89, 0, 19]
|
||||
```
|
||||
|
||||
Here we have tried to freeze just an empty map, and that has produced what appears to be a byte array with just an envelope and no packet. Ignoring the header, the fifth byte is 19, which corresponds to an empty map (:map-0 in the code), so there is no need for any packet.
|
||||
|
||||
To summarise, when Nippy freezes an empty record, it encodes it with:
|
||||
|
||||
* a 4-byte header
|
||||
* a 1-byte type-id of 48 indicated a small record
|
||||
* a 23-byte packet, the first 22 bytes of which represent the string "nippy.evolve.FirstRec", and the final byte which represents an empty map
|
||||
|
||||
This of course contains all the information Nippy requires to thaw the data back to an instance of `FirstRec`.
|
||||
|
||||
## Thawing a record
|
||||
|
||||
Now let us turn to thawing. Enter the following code into your source file or into the REPL:
|
||||
|
||||
```clojure
|
||||
(def t1 (thaw f1))
|
||||
```
|
||||
|
||||
```clojure
|
||||
nippy.evolve> t1
|
||||
{}
|
||||
nippy.evolve> (type t1)
|
||||
nippy.evolve.FirstRec
|
||||
```
|
||||
|
||||
Exactly as we expected, `t1` returns what appears to be an empty map (though this depends on how your REPL is set up), but when we examine its type, we find that is has correctly been thawed as a `nippy.evolve.FirstRec`. This is entirely due to the way Nippy has interpreted all the information provided in the envelope and packet, described in the previous section.
|
||||
|
||||
# Evolving your code
|
||||
|
||||
So without setting anything up at all in your project, you can see how simple it is just to use Nippy's `freeze` and `thaw` functions to serialize instances of any record you care to create. However, if you have been following the above discussion, you will probably have noted that there are a number of problems here, one or more of which you might even have run into at some stage:
|
||||
|
||||
* a number of bytes are used to encode the name of the record in the packet; in our example 22 bytes are used
|
||||
* since the name of the record is encoded in the packet, this means that if we change the name of the record or move it to another namespace, then try to thaw a previously frozen byte array, the operation will since Nippy will be unable to match up the previously encoded with the now renamed or moved record
|
||||
|
||||
We will look at moving and renaming first, and then consider how we can reduce the number of bytes in a packet afterwards.
|
||||
|
||||
## Moving or renaming a record
|
||||
|
||||
If we want to move or rename a record, for all data previously frozen using the record before renaming/moving it, Nippy will no longer be able to thaw that data since it can no longer match the record name encoded in the packet with a class generated by the record. To be exact, if you try this within the same session, while the REPL is open, then even if you have moved/renamed a record, the old record and the compiled class associated with it will still be available. This is a consequence of the way Clojure compiles records, and even if you try to do an `(ns-unamp 'nippy.evolve 'FirstRec)`, it will still be there. So to better understand this issue, we are going to first save the frozen byte array `f1` to a file, as follows.
|
||||
|
||||
Update the (ns) form to include the following:
|
||||
|
||||
```clojure
|
||||
(ns nippy.evolve
|
||||
(:require
|
||||
...
|
||||
[clojure.java.io
|
||||
:refer (file
|
||||
output-stream
|
||||
input-stream)]
|
||||
))
|
||||
```
|
||||
|
||||
Now enter the following code into the REPL:
|
||||
|
||||
```clojure
|
||||
nippy.evolve> (with-open [out (output-stream (file "./frozen-first-rec"))]
|
||||
(.write out f1))
|
||||
nil
|
||||
```
|
||||
|
||||
This will result in the file `./frozen-first-rec` being created in the top-level of your project. We will come back to this file subsequently.
|
||||
|
||||
Next, move `FirstRec` to the namespace `nippy.other`, and delete any code which references it within `nippy.evolve`:
|
||||
|
||||
```clojure
|
||||
(ns nippy.other)
|
||||
|
||||
(defrecord FirstRec [])
|
||||
```
|
||||
|
||||
Now quit the REPL, and start a new one. If not already there, change to namespace `nippy.evolve` and type the following:
|
||||
|
||||
```clojure
|
||||
nippy.evolve> FirstRec
|
||||
Syntax error compiling at (*cider-repl clojure/evolve:localhost:36735(clj)*:0:0).
|
||||
Unable to resolve symbol: FirstRec in this context
|
||||
```
|
||||
|
||||
You should see the above error, which shows that FirstRec is no longer defined in `nippy.evolve`.
|
||||
|
||||
Still from within `nippy.evolve`, type the following:
|
||||
|
||||
```clojure
|
||||
nippy.evolve> (with-open [ina (input-stream (file "./frozen-first-rec"))]
|
||||
(let [buf (byte-array 28)
|
||||
n (.read in buf)]
|
||||
(thaw buf)))
|
||||
```
|
||||
|
||||
This code attempts to open the file `./frozen-first-rec` and read it into `buf`, a byte array. If you have been following everything exactly thus far, running the above should result in the following being returned:
|
||||
|
||||
```clojure
|
||||
#:nippy{:unthawable
|
||||
{:type :record,
|
||||
:cause :exception,
|
||||
:class-name "nippy.evolve.FirstRec",
|
||||
:content {},
|
||||
:exception #error {...}}}
|
||||
```
|
||||
|
||||
Once again, this is to be expected. We have tried to thaw a byte array corresponding to a record with a class-name of "nippy.evolve.FirstRec", which of course no longer exists as we have moved it to `nippy.other`.
|
||||
|
||||
Whenever you encounter a `:nippy/unthawable` as a result of thawing, one approach is to write custom code to fix it. For example in the above situation, you could parse the map, and for `:type :record`, `:class-name "nippy.evolve.FirstRec"`, you could then look at the `:content` and create that as a `nippy.other.FirstRec` record. If the `:nippy/unthawable` appears deeply nested within the returned structure, you could call `clojure.walk/prewalk` as a more general solution, and provide a mapping table of {<old-class-name-str> <new-class-name>}, in this case {"nippy.evolve.FirstRec" nippy.other.FirstRec}, and use that to create FirstRec records of the new type. In any cases where you don't have access to the frozen files containing the old format, for example where you have created a desktop application which saves files which contain a frozen data structure, this will be your only option. However, in cases where you do have access to the frozen data, there is an alternative, better approach.
|
||||
|
||||
## Custom freeze and thaw
|
||||
|
||||
Before we talk about custom freeze and thaw, it's worth taking a step back and looking at how each of these processes work from a high-level.
|
||||
|
||||
* **freezing** takes as its input a piece of data, and the process is driven entirely by the *type* of that data
|
||||
* **thawing** takes as its input a piece of previously frozen data, and that process is driven by the *type-id* in the envelope and where appropriate some additional data in the packet, such as the name of the record's class for records
|
||||
|
||||
In general, any data we freeze, we want to be able to thaw back to its original form. In other words, the following should always hold true:
|
||||
|
||||
```clojure
|
||||
(= data
|
||||
(-> data freeze thaw))
|
||||
```
|
||||
|
||||
More accurately, at any given time we want to be able to restore any frozen data to its original state when we thaw it. Although this appears to be described by the above condition, there is a subtle but important distinction in that the above assumes that we are freezing and then thawing the data an instant later, whereas in reality *the thawing process can happen at any future time*. What this means is that when writing custom freeze and thaw code, it is important that only the thaw code matches the frozen data at any given time. This will become important soon.
|
||||
|
||||
In situations where we have access to previously frozen data, if we want to rename or move a record, we have an additional option to parsing the result of thaw and looking for the occurrence of any `:nippy/unthawable` maps nested in the resultant data (described above): custom freezing and thawing. Even if we have already frozen data using the built-in record freezer, we can still deal with this situation fairly easily. The trick is to understand that there is a sequence to be followed when it comes to implementing custom freezers and thawers.
|
||||
|
||||
In our above example, to avoid receiving the `:nippy/unthawable` result, we can start by moving the `defrecord` code back from `nippy.other` to `nippy.evolve`. Now if we evaluate the `nippy.evolve` namespace, then attempt to thaw the file we saved, we should get back our original data that we froze before, an instance of a record of type `nippy.evolve.FirstRec`. So far so good. Now the next step is to break the dependency between the frozen data and the name of the original record used to freeze it. To do this, we can write a custom freezer and thawer, and the best part is that we can have these active within nippy at the same time as the built-in record freezer/thawer. Here is how:
|
||||
|
||||
First, within `nippy.evolve`, update the `(ns)` form to include `extend-freeze` and `extend-thaw`:
|
||||
|
||||
```clojure
|
||||
(ns nippy.evolve
|
||||
(:require
|
||||
[taoensso.nippy
|
||||
:refer (...
|
||||
extend-freeze
|
||||
extend-thaw
|
||||
freeze-to-out!
|
||||
thaw-from-in!)]
|
||||
...
|
||||
))
|
||||
```
|
||||
|
||||
Ensure that the the following code is included in `nippy.evolve`:
|
||||
|
||||
```clojure
|
||||
(defrecord FirstRec [])
|
||||
```
|
||||
|
||||
Then add the following code:
|
||||
|
||||
```clojure
|
||||
(extend-freeze FirstRec 1
|
||||
[x data-out]
|
||||
(freeze-to-out!
|
||||
data-out
|
||||
(into {}
|
||||
(:data x))))
|
||||
|
||||
(extend-thaw 1 [data-input] (map->FirstRec (thaw-from-in! data-input)))
|
||||
```
|
||||
|
||||
These two blocks create respectively a custom freezer for FirstRec, which writes out an envelope with custom id 1, and a custom thawer which is used only for packets with custom id 1. We can use any id in the range `1 <= id < = 127`. As stated at the start of this section, this code shows that the freezer is driven by the type of data being frozen, and the thawer is driven only by the custom id of the data being thawed.
|
||||
|
||||
Now evaluate the whole `nippy.evolve` namespace again. This will extend Nippy by providing it with a custom freeze and thaw for any instances of `FirstRec`:
|
||||
|
||||
* from this point onwards, any new data we create and freeze will be frozen using the custom freezer
|
||||
* any newly frozen data will be thawed by the above custom thawer
|
||||
* any previously frozen data, with type-id 48 for records (see above), will still be thawed by the built-in record thawer
|
||||
|
||||
This last point is important in that it allows us to simultaneously deal with legacy data while also being able to process new data.
|
||||
|
||||
Let's try freezing a new record:
|
||||
|
||||
```clojure
|
||||
nippy.evolve> (freeze (FirstRec.))
|
||||
[78, 80, 89, 0, -1, 19]
|
||||
```
|
||||
|
||||
Once again we have our 4-byte header, but this time we have a negative number as our type-id. This is actually the negative of the custom-id we specified in our call to `extend-freeze`, and is how Nippy stores custom ids. We could have also used a (preferably namespaced) keyword, but that would have taken an extra 16-bits (a hash of the keyword) in the packet itself, and arguably doesn't provide any benefits over using an integer id. As long as we maintain a mapping between custom id and type in our code, and don't use the same custom id for a completely different type in the future, we shouldn't run into any issues with using a custom id over a keyword.
|
||||
|
||||
The packet in this case is just a single byte, 19, which refers to an empty map in nippy.clj type-ids. This map is then used by the thawer to reconstruct our original instance a `FirstRec`.
|
||||
|
||||
The first thing that should be evident is that this is much shorter, by 22-bytes, than the output produced by the built-in record freezer. This is because our custom freezer only stores the record as a map, with no string corresponding to the name of the record. This has resulted in decoupling the frozen data from any concrete record type (e.g. `nippy.evolve.FirstRec`), by instead coupling it to only an arbitrary custom id of our choosing (e.g. 1), and leaving us to provide the mapping between the custom id and some type within the custom thawer, which from now on we can do by updating the thawer with custom id 1.
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
See the **menu to the right** for content 👉
|
||||
|
||||
# Contributions welcome
|
||||
|
||||
**PRs very welcome** to help improve this documentation!
|
||||
See the [wiki](../tree/master/wiki) folder in the main repo for the relevant files.
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
# Attention!
|
||||
|
||||
This wiki is designed for viewing from [here](../../../wiki)!
|
||||
|
||||
Viewing from GitHub's file browser will result in **broken links**.
|
||||
Loading…
Reference in a new issue