Compare commits
11 commits
v3.5.0-RC1
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ae4463a85c | ||
|
|
1b99516177 | ||
|
|
e68c2d56fe | ||
|
|
bfba59483a | ||
|
|
f7bb2824ac | ||
|
|
8d62dc2826 | ||
|
|
da57206b0d | ||
|
|
e0f49ced5a | ||
|
|
8d107650cd | ||
|
|
1026ea0ae7 | ||
|
|
c92457025f |
10 changed files with 305 additions and 265 deletions
92
CHANGELOG.md
92
CHANGELOG.md
|
|
@ -2,25 +2,77 @@ This project uses [**Break Versioning**](https://www.taoensso.com/break-versioni
|
|||
|
||||
---
|
||||
|
||||
# `v3.5.0` (2025-04-15)
|
||||
|
||||
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.5.0)
|
||||
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
|
||||
|
||||
This is a **general maintenance release** focused on updating dependencies and laying groundwork (read support) for new array types coming in Nippy v3.6.
|
||||
|
||||
It **drops support for Clojure v1.9** but should otherwise be a safe update from (at least) all recent versions of Nippy.
|
||||
|
||||
## Since `v3.5.0-RC1` (2024-10-28)
|
||||
|
||||
- **\[mod]** Drop official Clojure v1.9 support \[da57206]
|
||||
- \[new] Add string array type to default [thaw-serializable-allowlist](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*thaw-serializable-allowlist*) \[bfba594]
|
||||
- \[new] Use [Truss exceptions](https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#ex-info) on errors \[8d62dc2]
|
||||
- \[new] [#184] Incl. cause on non-native freeze failures \[8d10765]
|
||||
- \[doc] Clarify `*freeze-fallback*` docstring \[1026ea0]
|
||||
|
||||
## Since `v3.4.2` (2024-05-26)
|
||||
|
||||
- **\[mod]** Drop official Clojure v1.9 support \[da57206]
|
||||
- \[new] Add string array type to default [thaw-serializable-allowlist](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*thaw-serializable-allowlist*) \[bfba594]
|
||||
- \[new] Use [Truss exceptions](https://cljdoc.org/d/com.taoensso/truss/CURRENT/api/taoensso.truss#ex-info) on errors \[8d62dc2]
|
||||
- \[new] [#184] Incl. cause on non-native freeze failures \[8d10765]
|
||||
- \[new] [#175] Mark [cache](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#cache) feature as stable \[b217db5]
|
||||
- \[doc] Clarify `*freeze-fallback*` docstring \[1026ea0]
|
||||
|
||||
## Migration info
|
||||
|
||||
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
|
||||
| :------------------------ | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
|
||||
| `v3.5.0-RC1` (2024-10-28) | - | - | - |
|
||||
| `v3.4.2` (2024-05-26) | - | - | - |
|
||||
| `v3.4.1` (2024-05-02) | - | - | - |
|
||||
| `v3.4.0` (2024-04-30) | - | **Yes** | - |
|
||||
| `v3.3.0` (2023-10-11) | - | - | - |
|
||||
| `v3.2.0` (2022-07-18) | - | - | - |
|
||||
| `v3.1.3` (2022-06-23) | - | - | - |
|
||||
|
||||
> [1] Relevant only when Nippy introduces support for new types **AND** you plan to update Nippy with a **rolling update** (coexisting new and old versions).
|
||||
|
||||
If updating from older versions of Nippy, please see the relevant release notes.
|
||||
|
||||
As always:
|
||||
|
||||
- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
|
||||
- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
|
||||
- **Please report any unexpected problems** 🙏
|
||||
|
||||
\- [Peter Taoussanis](https://www.taoensso.com)
|
||||
|
||||
---
|
||||
|
||||
# `v3.5.0-RC1` (2024-10-28)
|
||||
|
||||
- 📦 **Dependency**: available [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.5.0-RC1)
|
||||
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.5.0-RC1)
|
||||
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
|
||||
|
||||
This is a **non-breaking maintenance release** that updates dependencies and includes read support for more native array types to be introduced in a future v3.6 release.
|
||||
|
||||
It should be safe to update from (at least) all recent versions of Nippy.
|
||||
|
||||
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Recommended update sequence [1]
|
||||
| :-- | :-- | :-- | :--
|
||||
| `v3.4.2` (2024-05-26) | - | - | -
|
||||
| `v3.4.1` (2024-05-02) | - | - | -
|
||||
| `v3.4.0` (2024-04-30) | - | Yes | -
|
||||
| `v3.3.0` (2023-10-11) | - | - | -
|
||||
| `v3.2.0` (2022-07-18) | - | - | -
|
||||
| `v3.1.3` (2022-06-23) | - | - | -
|
||||
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
|
||||
| :-------------------- | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
|
||||
| `v3.4.2` (2024-05-26) | - | - | - |
|
||||
| `v3.4.1` (2024-05-02) | - | - | - |
|
||||
| `v3.4.0` (2024-04-30) | - | Yes | - |
|
||||
| `v3.3.0` (2023-10-11) | - | - | - |
|
||||
| `v3.2.0` (2022-07-18) | - | - | - |
|
||||
| `v3.1.3` (2022-06-23) | - | - | - |
|
||||
|
||||
> [1] Relevant only when introducing support for new types, to help with rolling updates
|
||||
> [1] Relevant only when introducing support for new types, and for users that do rolling updates
|
||||
|
||||
If updating from older versions of Nippy, please see the relevant release notes.
|
||||
|
||||
|
|
@ -36,22 +88,22 @@ As always:
|
|||
|
||||
# `v3.4.2` (2024-05-26)
|
||||
|
||||
> **Dep**: Nippy is [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.2).
|
||||
> **Versioning**: Nippy uses [Break Versioning](https://www.taoensso.com/break-versioning).
|
||||
- **Dependency**: [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.2)
|
||||
- **Versioning**: [Break Versioning](https://www.taoensso.com/break-versioning)
|
||||
|
||||
⚠️ This release addresses a [**security vulnerability**](https://github.com/taoensso/nippy/security/advisories/GHSA-vw78-267v-588h) in Nippy's upstream compression library and is **recommended for all existing users**.
|
||||
|
||||
It should be a **straight-forward and non-breaking update** for almost everyone:
|
||||
|
||||
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Recommended update sequence [1]
|
||||
| :-- | :-- | :-- | :--
|
||||
| `v3.4.1` (2024-05-02) | - | - | -
|
||||
| `v3.4.0` (2024-04-30) | - | Yes | -
|
||||
| `v3.3.0` (2023-10-11) | - | - | -
|
||||
| `v3.2.0` (2022-07-18) | - | - | -
|
||||
| `v3.1.3` (2022-06-23) | - | - | -
|
||||
| Updating from Nippy | Changes to API? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | Rolling update sequence [1] |
|
||||
| :-------------------- | :-------------- | :---------------------------------------------------------------------------------------------------------------------- | :-------------------------- |
|
||||
| `v3.4.1` (2024-05-02) | - | - | - |
|
||||
| `v3.4.0` (2024-04-30) | - | Yes | - |
|
||||
| `v3.3.0` (2023-10-11) | - | - | - |
|
||||
| `v3.2.0` (2022-07-18) | - | - | - |
|
||||
| `v3.1.3` (2022-06-23) | - | - | - |
|
||||
|
||||
> [1] Relevant only when introducing support for new types, to help with rolling updates
|
||||
> [1] Relevant only when introducing support for new types, and for users that do rolling updates
|
||||
|
||||
If updating from older versions of Nippy, please see the relevant release notes.
|
||||
|
||||
|
|
|
|||
18
README.md
18
README.md
|
|
@ -1,9 +1,9 @@
|
|||
<a href="https://www.taoensso.com/clojure" title="More stuff by @ptaoussanis at www.taoensso.com"><img src="https://www.taoensso.com/open-source.png" alt="Taoensso open source" width="340"/></a>
|
||||
[**API**][cljdoc docs] | [**Wiki**][GitHub wiki] | [Latest releases](#latest-releases) | [Get support][GitHub issues]
|
||||
[**API**][cljdoc] | [**Wiki**][GitHub wiki] | [Latest releases](#latest-releases) | [Get support][GitHub issues]
|
||||
|
||||
# Nippy
|
||||
|
||||
### The fastest serialization library for Clojure
|
||||
### Fast serialization library for Clojure
|
||||
|
||||
Clojure's rich data types are awesome. And its [reader](https://clojure.org/reference/reader) allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).
|
||||
|
||||
|
|
@ -13,8 +13,7 @@ It is used at scale by [Carmine](https://www.taoensso.com/carmine), [Faraday](ht
|
|||
|
||||
## Latest release/s
|
||||
|
||||
- `2024-05-26` `v3.4.2` (stable): [release info](../../releases/tag/v3.4.2)
|
||||
- `2024-10-28` `v3.5.0-RC1` (dev): [release info](../../releases/tag/v3.5.0-RC1)
|
||||
- `2025-04-15` `v3.5.0`: [release info](../../releases/tag/v3.5.0)
|
||||
|
||||
[![Main tests][Main tests SVG]][Main tests URL]
|
||||
[![Graal tests][Graal tests SVG]][Graal tests URL]
|
||||
|
|
@ -87,16 +86,16 @@ So starting with Nippy v3.4, Nippy's release notes will **always clearly indicat
|
|||
|
||||
## Performance
|
||||
|
||||
Since its earliest versions, Nippy has consistently been the **fastest serialization library for Clojure** that I'm aware of. Latest results:
|
||||
Nippy is fast! Latest [benchmark](../../blob/master/test/taoensso/nippy_benchmarks.clj) results:
|
||||
|
||||

|
||||
|
||||
PRs welcome to include other alternatives in the [benchmark suite](../../blob/master/test/taoensso/nippy_benchmarks.clj)!
|
||||
PRs welcome to include other alternatives in the bench suite!
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Wiki][GitHub wiki] (getting started, usage, etc.)
|
||||
- API reference: [cljdoc][cljdoc docs], [Codox][Codox docs]
|
||||
- API reference via [cljdoc][cljdoc]
|
||||
|
||||
## Funding
|
||||
|
||||
|
|
@ -104,7 +103,7 @@ You can [help support][sponsor] continued work on this project, thank you!! 🙏
|
|||
|
||||
## License
|
||||
|
||||
Copyright © 2012-2024 [Peter Taoussanis][].
|
||||
Copyright © 2012-2025 [Peter Taoussanis][].
|
||||
Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).
|
||||
|
||||
<!-- Common -->
|
||||
|
|
@ -118,8 +117,7 @@ Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).
|
|||
|
||||
<!-- Project -->
|
||||
|
||||
[Codox docs]: https://taoensso.github.io/nippy/
|
||||
[cljdoc docs]: https://cljdoc.org/d/com.taoensso/nippy/
|
||||
[cljdoc]: https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy
|
||||
|
||||
[Clojars SVG]: https://img.shields.io/clojars/v/com.taoensso/nippy.svg
|
||||
[Clojars URL]: https://clojars.org/com.taoensso/nippy
|
||||
|
|
|
|||
32
project.clj
32
project.clj
|
|
@ -1,6 +1,6 @@
|
|||
(defproject com.taoensso/nippy "3.5.0-RC1"
|
||||
(defproject com.taoensso/nippy "3.5.0"
|
||||
:author "Peter Taoussanis <https://www.taoensso.com>"
|
||||
:description "The fastest serialization library for Clojure"
|
||||
:description "Fast serialization library for Clojure"
|
||||
:url "https://www.taoensso.com/nippy"
|
||||
|
||||
:license
|
||||
|
|
@ -8,8 +8,8 @@
|
|||
:url "https://www.eclipse.org/legal/epl-v10.html"}
|
||||
|
||||
:dependencies
|
||||
[[org.clojure/tools.reader "1.5.0"]
|
||||
[com.taoensso/encore "3.127.0"]
|
||||
[[org.clojure/tools.reader "1.5.2"]
|
||||
[com.taoensso/encore "3.142.0"]
|
||||
[org.tukaani/xz "1.10"]
|
||||
[io.airlift/aircompressor "2.0.2"]]
|
||||
|
||||
|
|
@ -17,11 +17,10 @@
|
|||
|
||||
:profiles
|
||||
{;; :default [:base :system :user :provided :dev]
|
||||
:provided {:dependencies [[org.clojure/clojure "1.11.3"]]}
|
||||
:c1.12 {:dependencies [[org.clojure/clojure "1.12.0-alpha12"]]}
|
||||
:c1.11 {:dependencies [[org.clojure/clojure "1.11.3"]]}
|
||||
:c1.10 {:dependencies [[org.clojure/clojure "1.10.1"]]}
|
||||
:c1.9 {:dependencies [[org.clojure/clojure "1.9.0"]]}
|
||||
:provided {:dependencies [[org.clojure/clojure "1.11.4"]]}
|
||||
:c1.12 {:dependencies [[org.clojure/clojure "1.12.0"]]}
|
||||
:c1.11 {:dependencies [[org.clojure/clojure "1.11.4"]]}
|
||||
:c1.10 {:dependencies [[org.clojure/clojure "1.10.3"]]}
|
||||
|
||||
:graal-tests
|
||||
{:source-paths ["test"]
|
||||
|
|
@ -38,7 +37,9 @@
|
|||
"-Xms1024m" "-Xmx2048m"
|
||||
"-Dtaoensso.elide-deprecated=true"
|
||||
"-Dtaoensso.nippy.thaw-serializable-allowlist-base=base.1, base.2"
|
||||
"-Dtaoensso.nippy.thaw-serializable-allowlist-add=add.1 , add.2"]
|
||||
"-Dtaoensso.nippy.thaw-serializable-allowlist-add=add.1 , add.2"
|
||||
#_"-Dtaoensso.nippy.target-release=320"
|
||||
#_"-Dtaoensso.nippy.target-release=350"]
|
||||
|
||||
:global-vars
|
||||
{*warn-on-reflection* true
|
||||
|
|
@ -51,18 +52,13 @@
|
|||
|
||||
:plugins
|
||||
[[lein-pprint "1.3.2"]
|
||||
[lein-ancient "0.7.0"]
|
||||
[com.taoensso.forks/lein-codox "0.10.11"]]
|
||||
|
||||
:codox
|
||||
{:language #{:clojure #_:clojurescript}
|
||||
:base-language :clojure}}}
|
||||
[lein-ancient "0.7.0"]]}}
|
||||
|
||||
:aliases
|
||||
{"start-dev" ["with-profile" "+dev" "repl" ":headless"]
|
||||
;; "build-once" ["do" ["clean"] ["cljsbuild" "once"]]
|
||||
"deploy-lib" ["do" #_["build-once"] ["deploy" "clojars"] ["install"]]
|
||||
|
||||
"test-clj" ["with-profile" "+c1.12:+c1.11:+c1.10:+c1.9" "test"]
|
||||
;; "test-cljs" ["with-profile" "+c1.12" "cljsbuild" "test"]
|
||||
"test-clj" ["with-profile" "+c1.12:+c1.11:+c1.10" "test"]
|
||||
;; "test-cljs" ["with-profile" "+c1.12" "cljsbuild" "test"]
|
||||
"test-all" ["do" ["clean"] ["test-clj"] #_["test-cljs"]]})
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@
|
|||
(:require
|
||||
[clojure.string :as str]
|
||||
[clojure.java.io :as jio]
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc]
|
||||
[taoensso.nippy
|
||||
[impl :as impl]
|
||||
|
|
@ -25,7 +26,7 @@
|
|||
PersistentQueue PersistentTreeMap PersistentTreeSet PersistentList
|
||||
MapEntry LazySeq IRecord ISeq IType]))
|
||||
|
||||
(enc/assert-min-encore-version [3 127 0])
|
||||
(enc/assert-min-encore-version [3 142 0])
|
||||
|
||||
(comment
|
||||
(set! *unchecked-math* :warn-on-boxed)
|
||||
|
|
@ -196,13 +197,13 @@
|
|||
15 [:byte-array-md [[:elements {:read 2}]]]
|
||||
2 [:byte-array-lg [[:elements {:read 4}]]]
|
||||
|
||||
108 [:long-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (YYYY-MM-DD)
|
||||
109 [:int-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (YYYY-MM-DD)
|
||||
109 [:int-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
|
||||
108 [:long-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
|
||||
|
||||
116 [:double-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (YYYY-MM-DD)
|
||||
117 [:float-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (YYYY-MM-DD)
|
||||
117 [:float-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
|
||||
116 [:double-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
|
||||
|
||||
107 [:string-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (YYYY-MM-DD)
|
||||
107 [:string-array-lg [[:elements {:read 4}]]] ; Added v3.5.0 (2025-04-15)
|
||||
115 [:object-array-lg [[:elements {:read 4}]]]
|
||||
|
||||
;; Serializable
|
||||
|
|
@ -365,25 +366,26 @@
|
|||
;; For back compatibility (incl. Timbre's Carmine appender)
|
||||
(enc/defonce ^:dynamic ^:no-doc ^:deprecated *final-freeze-fallback* "Prefer `*freeze-fallback`." nil)
|
||||
(enc/defonce ^:dynamic *freeze-fallback*
|
||||
"Controls Nippy's behaviour when trying to freeze an item for which Nippy
|
||||
doesn't currently have a native freeze/thaw implementation.
|
||||
"Controls Nippy's behaviour when trying to freeze an object with a type for
|
||||
which Nippy doesn't currently have a native (protocol) implementation.
|
||||
|
||||
Possible values:
|
||||
|
||||
1. `nil` (no freeze-fallback, default)
|
||||
1. `nil` (no fallback, default)
|
||||
Tries the following in order:
|
||||
- Freeze with Java's `Serializable` interface if possible
|
||||
- Freeze with Clojure's reader if possible
|
||||
- Freeze with Java's `Serializable` interface if this seems possible
|
||||
- Freeze with Clojure's reader if this seems possible
|
||||
- Throw
|
||||
|
||||
2. `:write-unfreezable` keyword
|
||||
Tries the following in order:
|
||||
- Freeze with Java's `Serializable` interface if possible
|
||||
- Freeze with Clojure's reader if possible
|
||||
- Freeze with Java's `Serializable` interface if this seems possible
|
||||
- Freeze with Clojure's reader if this seems possible
|
||||
- Freeze a {:nippy/unfreezable {:type _}} placeholder value
|
||||
|
||||
3. [Advanced] Custom (fn [^java.io.DataOutput out item]) that must
|
||||
write exactly one value to the given `DataOutput` stream"
|
||||
3. [Advanced] Custom (fn [^java.io.DataOutput out obj]) that must
|
||||
write an appropriate object type id and payload to the given
|
||||
`DataOutput` stream."
|
||||
|
||||
nil)
|
||||
|
||||
|
|
@ -440,16 +442,16 @@
|
|||
;; Unfortunately quite a bit of complexity to do this safely
|
||||
|
||||
(def default-freeze-serializable-allowlist
|
||||
"Allows *any* class-name to be frozen using Java's `Serializable` interface.
|
||||
"Allows *any* class name to be frozen using Java's `Serializable` interface.
|
||||
This is generally safe since RCE risk is present only when thawing.
|
||||
See also `*freeze-serializable-allowlist*`."
|
||||
#{"*"})
|
||||
|
||||
(def default-thaw-serializable-allowlist
|
||||
"A set of common safe class-names to allow to be frozen using Java's
|
||||
"A set of common safe class names to allow to be frozen using Java's
|
||||
`Serializable` interface. PRs welcome for additions.
|
||||
See also `*thaw-serializable-allowlist*`."
|
||||
#{"[I" "[F" "[Z" "[B" "[C" "[D" "[S" "[J"
|
||||
#{"[Z" "[B" "[S" "[I" "[J" "[F" "[D" "[C" "[Ljava.lang.String;"
|
||||
|
||||
"java.lang.Throwable"
|
||||
"java.lang.Exception"
|
||||
|
|
@ -513,10 +515,10 @@
|
|||
|
||||
Example allowlist values:
|
||||
- `(fn allow-class? [class-name] true)` ; Arbitrary predicate fn
|
||||
- `#{\"java.lang.Throwable\", \"clojure.lang.*\"}` ; Set of class-names
|
||||
- `#{\"java.lang.Throwable\", \"clojure.lang.*\"}` ; Set of class names
|
||||
- `\"allow-and-record\"` ; Special value, see [2]
|
||||
|
||||
Note that class-names in sets may contain \"*\" wildcards.
|
||||
Note that class names in sets may contain \"*\" wildcards.
|
||||
|
||||
Default allowlist values are:
|
||||
- default-freeze-serializable-allowlist ; `{\"*\"}` => allow any class
|
||||
|
|
@ -694,7 +696,7 @@
|
|||
(sm-count? len) (do (write-id out id-kw-sm) (write-sm-count out len))
|
||||
(md-count? len) (do (write-id out id-kw-md) (write-md-count out len))
|
||||
;; :else (do (write-id out id-kw-lg) (write-lg-count out len)) ; Unrealistic
|
||||
:else (throw (ex-info "Keyword too long" {:name s})))
|
||||
:else (truss/ex-info! "Keyword too long" {:name s}))
|
||||
|
||||
(.write out ba 0 len)))
|
||||
|
||||
|
|
@ -706,7 +708,7 @@
|
|||
(sm-count? len) (do (write-id out id-sym-sm) (write-sm-count out len))
|
||||
(md-count? len) (do (write-id out id-sym-md) (write-md-count out len))
|
||||
;; :else (do (write-id out id-sym-lg) (write-lg-count out len)) ; Unrealistic
|
||||
:else (throw (ex-info "Symbol too long" {:name s})))
|
||||
:else (truss/ex-info! "Symbol too long" {:name s}))
|
||||
|
||||
(.write out ba 0 len)))
|
||||
|
||||
|
|
@ -893,51 +895,45 @@
|
|||
|
||||
(-run! (fn [in] (-freeze-with-meta! in out)) s)))))
|
||||
|
||||
(defn- write-serializable [^DataOutput out x ^String class-name]
|
||||
(defn- write-serializable [^DataOutput out x]
|
||||
(when-debug (println (str "write-serializable: " (type x))))
|
||||
(let [class-name-ba (.getBytes class-name StandardCharsets/UTF_8)
|
||||
len (alength class-name-ba)]
|
||||
(when (and (instance? Serializable x) (not (fn? x)))
|
||||
(let [class-name (.getName (class x))] ; Reflect
|
||||
(when (freeze-serializable-allowed? class-name)
|
||||
(let [class-name-ba (.getBytes class-name StandardCharsets/UTF_8)
|
||||
len (alength class-name-ba)]
|
||||
|
||||
(enc/cond
|
||||
(sm-count? len) (do (write-id out id-sz-quarantined-sm) (write-bytes-sm out class-name-ba))
|
||||
(md-count? len) (do (write-id out id-sz-quarantined-md) (write-bytes-md out class-name-ba))
|
||||
;; :else (do (write-id out id-sz-quarantined-lg) (write-bytes-md out class-name-ba)) ; Unrealistic
|
||||
:else (throw (ex-info "Serializable class name too long" {:name class-name})))
|
||||
(enc/cond
|
||||
(sm-count? len) (do (write-id out id-sz-quarantined-sm) (write-bytes-sm out class-name-ba))
|
||||
(md-count? len) (do (write-id out id-sz-quarantined-md) (write-bytes-md out class-name-ba))
|
||||
;; :else (do (write-id out id-sz-quarantined-lg) (write-bytes-md out class-name-ba)) ; Unrealistic
|
||||
:else (truss/ex-info! "Serializable class name too long" {:name class-name}))
|
||||
|
||||
;; Legacy: write object directly to out.
|
||||
;; (.writeObject (ObjectOutputStream. out) x)
|
||||
;; Legacy: write object directly to out.
|
||||
;; (.writeObject (ObjectOutputStream. out) x)
|
||||
|
||||
;; Quarantined: write object to ba, then ba to out.
|
||||
;; We'll have object length during thaw, allowing us to skip readObject.
|
||||
(let [quarantined-ba (ByteArrayOutputStream.)]
|
||||
(.writeObject (ObjectOutputStream. (DataOutputStream. quarantined-ba)) x)
|
||||
(write-bytes out (.toByteArray quarantined-ba)))))
|
||||
;; Quarantined: write object to ba, then ba to out.
|
||||
;; We'll have object length during thaw, allowing us to skip readObject.
|
||||
(let [quarantined-ba (ByteArrayOutputStream.)]
|
||||
(.writeObject (ObjectOutputStream. (DataOutputStream. quarantined-ba)) x)
|
||||
(write-bytes out (.toByteArray quarantined-ba)))
|
||||
|
||||
true)))))
|
||||
|
||||
(defn- write-readable [^DataOutput out x]
|
||||
(when-debug (println (str "write-readable: " (type x))))
|
||||
(let [edn (enc/pr-edn x)
|
||||
edn-ba (.getBytes ^String edn StandardCharsets/UTF_8)
|
||||
len (alength edn-ba)]
|
||||
(enc/cond
|
||||
(sm-count? len) (do (write-id out id-reader-sm) (write-bytes-sm out edn-ba))
|
||||
(md-count? len) (do (write-id out id-reader-md) (write-bytes-md out edn-ba))
|
||||
:else (do (write-id out id-reader-lg) (write-bytes-lg out edn-ba)))))
|
||||
|
||||
(defn try-write-serializable [out x]
|
||||
(when (and (instance? Serializable x) (not (fn? x)))
|
||||
(try
|
||||
(let [class-name (.getName (class x))] ; Reflect
|
||||
(when (freeze-serializable-allowed? class-name)
|
||||
(write-serializable out x class-name)
|
||||
true))
|
||||
(catch Throwable _ nil))))
|
||||
|
||||
(defn try-write-readable [out x]
|
||||
(when (impl/seems-readable? x)
|
||||
(try
|
||||
(write-readable out x)
|
||||
true
|
||||
(catch Throwable _ nil))))
|
||||
(let [edn (enc/pr-edn x)
|
||||
edn-ba (.getBytes ^String edn StandardCharsets/UTF_8)
|
||||
len (alength edn-ba)]
|
||||
(enc/cond
|
||||
(sm-count? len) (do (write-id out id-reader-sm) (write-bytes-sm out edn-ba))
|
||||
(md-count? len) (do (write-id out id-reader-md) (write-bytes-md out edn-ba))
|
||||
:else (do (write-id out id-reader-lg) (write-bytes-lg out edn-ba)))
|
||||
true)))
|
||||
|
||||
(defn ^:deprecated try-write-serializable [out x] (truss/catching :all (write-serializable out x)))
|
||||
(defn ^:deprecated try-write-readable [out x] (truss/catching :all (write-readable out x)))
|
||||
|
||||
(defn- try-pr-edn [x]
|
||||
(try
|
||||
|
|
@ -955,13 +951,6 @@
|
|||
:content (try-pr-edn x)}}
|
||||
out))
|
||||
|
||||
(defn throw-unfreezable [x]
|
||||
(let [t (type x)]
|
||||
(throw
|
||||
(ex-info (str "Unfreezable type: " t)
|
||||
{:type t
|
||||
:as-str (try-pr-edn x)}))))
|
||||
|
||||
;; Public `-freeze-with-meta!` with different arg order
|
||||
(defn freeze-to-out!
|
||||
"Serializes arg (any Clojure data type) to a DataOutput.
|
||||
|
|
@ -1040,7 +1029,7 @@
|
|||
(when first-occurance? (-freeze-with-meta! x-val out)))
|
||||
|
||||
:else
|
||||
;; (throw (ex-info "Max cache size exceeded" {:idx idx}))
|
||||
;; (truss/ex-info! "Max cache size exceeded" {:idx idx})
|
||||
(-freeze-with-meta! x-val out) ; Just freeze uncached
|
||||
))
|
||||
|
||||
|
|
@ -1057,7 +1046,7 @@
|
|||
(vswap! cache_ assoc idx x)
|
||||
x)
|
||||
v))
|
||||
(throw (ex-info "Can't thaw without cache available. See `with-cache`." {}))))))
|
||||
(truss/ex-info! "Can't thaw without cache available. See `with-cache`." {})))))
|
||||
|
||||
(comment
|
||||
(thaw (freeze [(cache "foo") (cache "foo") (cache "foo")]))
|
||||
|
|
@ -1116,12 +1105,10 @@
|
|||
(freezer (Class/forName "[Ljava.lang.Object;") nil true (write-array-lg out x (alength ^"[Ljava.lang.Object;" x) id-object-array-lg))
|
||||
|
||||
(when (impl/target-release>= 350)
|
||||
(freezer (Class/forName "[J") nil true (write-array-lg out x (alength ^"[J" x) id-long-array-lg))
|
||||
(freezer (Class/forName "[I") nil true (write-array-lg out x (alength ^"[I" x) id-int-array-lg))
|
||||
|
||||
(freezer (Class/forName "[D") nil true (write-array-lg out x (alength ^"[D" x) id-double-array-lg))
|
||||
(freezer (Class/forName "[J") nil true (write-array-lg out x (alength ^"[J" x) id-long-array-lg))
|
||||
(freezer (Class/forName "[F") nil true (write-array-lg out x (alength ^"[F" x) id-float-array-lg))
|
||||
|
||||
(freezer (Class/forName "[D") nil true (write-array-lg out x (alength ^"[D" x) id-double-array-lg))
|
||||
(freezer (Class/forName "[Ljava.lang.String;") nil true (write-array-lg out x (alength ^"[Ljava.lang.String;" x) id-string-array-lg)))
|
||||
|
||||
(freezer PersistentQueue nil true (write-counted-coll out id-queue-lg x))
|
||||
|
|
@ -1141,7 +1128,7 @@
|
|||
(sm-count? len) (do (write-id out id-record-sm) (write-bytes-sm out class-name-ba))
|
||||
(md-count? len) (do (write-id out id-record-md) (write-bytes-md out class-name-ba))
|
||||
;; :else (do (write-id out id-record-lg) (write-bytes-md out class-name-ba)) ; Unrealistic
|
||||
:else (throw (ex-info "Record class name too long" {:name class-name})))
|
||||
:else (truss/ex-info! "Record class name too long" {:name class-name}))
|
||||
|
||||
(-freeze-without-meta! (into {} x) out)))
|
||||
|
||||
|
|
@ -1196,13 +1183,20 @@
|
|||
(write-unfreezable out x)))
|
||||
|
||||
;; Without ff
|
||||
(or
|
||||
(try-write-serializable out x)
|
||||
(try-write-readable out x)
|
||||
(enc/cond
|
||||
:let [[r1 e1] (try [(write-serializable out x)] (catch Throwable t [nil t]))], r1 r1
|
||||
:let [[r2 e2] (try [(write-readable out x)] (catch Throwable t [nil t]))], r2 r2
|
||||
|
||||
(when-let [fff *final-freeze-fallback*] (fff out x) true) ; Deprecated
|
||||
|
||||
(throw-unfreezable x)))))
|
||||
:if-let [fff *final-freeze-fallback*] (fff out x) ; Deprecated
|
||||
:else
|
||||
(let [t (type x)]
|
||||
(truss/ex-info! (str "Failed to freeze type: " t)
|
||||
(enc/assoc-some
|
||||
{:type t
|
||||
:as-str (try-pr-edn x)}
|
||||
{:serializable-error e1
|
||||
:readable-error e2})
|
||||
(or e1 e2)))))))
|
||||
|
||||
;;;;
|
||||
|
||||
|
|
@ -1216,9 +1210,8 @@
|
|||
(defn- wrap-header [data-ba head-meta]
|
||||
(if-let [head-ba (get-head-ba head-meta)]
|
||||
(enc/ba-concat head-ba data-ba)
|
||||
(throw
|
||||
(ex-info (str "Unrecognized header meta: " head-meta)
|
||||
{:head-meta head-meta}))))
|
||||
(truss/ex-info! (str "Unrecognized header meta: " head-meta)
|
||||
{:head-meta head-meta})))
|
||||
|
||||
(comment (wrap-header (.getBytes "foo") {:compressor-id :lz4
|
||||
:encryptor-id nil}))
|
||||
|
|
@ -1352,7 +1345,7 @@
|
|||
id-byte-array-md (read-bytes in (read-md-count in))
|
||||
id-byte-array-lg (read-bytes in (read-lg-count in)))))
|
||||
|
||||
(defmacro ^:private read-array [in thaw-type array array-type]
|
||||
(defmacro ^:private read-array [in thaw-type array-type array]
|
||||
(let [thawed-sym (with-meta 'thawed-sym {:tag thaw-type})
|
||||
array-sym (with-meta 'array-sym {:tag array-type})]
|
||||
`(let [~array-sym ~array]
|
||||
|
|
@ -1381,7 +1374,7 @@
|
|||
|
||||
(defmacro ^:private editable? [coll] `(instance? clojure.lang.IEditableCollection ~coll))
|
||||
|
||||
(defn- xform* [xform] (enc/catching-xform {:error/msg "Error thrown via `*thaw-xform*`"} xform))
|
||||
(defn- xform* [xform] (truss/catching-xform {:error/msg "Error thrown via `*thaw-xform*`"} xform))
|
||||
|
||||
(let [rf! (fn rf! ([x] (persistent! x)) ([acc x] (conj! acc x)))
|
||||
rf* (fn rf* ([x] x) ([acc x] (conj acc x)))]
|
||||
|
|
@ -1418,14 +1411,12 @@
|
|||
(try
|
||||
(custom-reader in)
|
||||
(catch Exception e
|
||||
(throw
|
||||
(ex-info
|
||||
(str "Reader exception for custom type id: " type-id)
|
||||
{:type-id type-id, :prefixed? prefixed?} e))))
|
||||
(throw
|
||||
(ex-info
|
||||
(str "No reader provided for custom type id: " type-id)
|
||||
{:type-id type-id, :prefixed? prefixed?}))))
|
||||
(truss/ex-info!
|
||||
(str "Reader exception for custom type id: " type-id)
|
||||
{:type-id type-id, :prefixed? prefixed?} e)))
|
||||
(truss/ex-info!
|
||||
(str "No reader provided for custom type id: " type-id)
|
||||
{:type-id type-id, :prefixed? prefixed?})))
|
||||
|
||||
(defn- read-edn [edn]
|
||||
(try
|
||||
|
|
@ -1502,9 +1493,9 @@
|
|||
[^DataInput in class-name]
|
||||
(if (thaw-serializable-allowed? class-name)
|
||||
(read-object in class-name)
|
||||
(throw ; No way to skip bytes, so best we can do is throw
|
||||
(ex-info "Cannot thaw object: `taoensso.nippy/*thaw-serializable-allowlist*` check failed. This is a security feature. See `*thaw-serializable-allowlist*` docstring or https://github.com/ptaoussanis/nippy/issues/130 for details!"
|
||||
{:class-name class-name}))))
|
||||
(truss/ex-info! ; No way to skip bytes, so best we can do is throw
|
||||
"Cannot thaw object: `taoensso.nippy/*thaw-serializable-allowlist*` check failed. This is a security feature. See `*thaw-serializable-allowlist*` docstring or https://github.com/ptaoussanis/nippy/issues/130 for details!"
|
||||
{:class-name class-name})))
|
||||
|
||||
(defn- read-record [in class-name]
|
||||
(let [content (thaw-from-in! in)]
|
||||
|
|
@ -1607,14 +1598,14 @@
|
|||
id-byte-array-md (read-bytes in (read-md-count in))
|
||||
id-byte-array-lg (read-bytes in (read-lg-count in))
|
||||
|
||||
id-long-array-lg (read-array in long (long-array (read-lg-count in)) "[J")
|
||||
id-int-array-lg (read-array in int (int-array (read-lg-count in)) "[I")
|
||||
id-long-array-lg (read-array in long "[J" (long-array (read-lg-count in)))
|
||||
id-int-array-lg (read-array in int "[I" (int-array (read-lg-count in)))
|
||||
|
||||
id-double-array-lg (read-array in double (double-array (read-lg-count in)) "[D")
|
||||
id-float-array-lg (read-array in float (float-array (read-lg-count in)) "[F")
|
||||
id-double-array-lg (read-array in double "[D" (double-array (read-lg-count in)))
|
||||
id-float-array-lg (read-array in float "[F" (float-array (read-lg-count in)))
|
||||
|
||||
id-string-array-lg (read-array in String (make-array String (read-lg-count in)) "[Ljava.lang.String;")
|
||||
id-object-array-lg (read-array in Object (object-array (read-lg-count in)) "[Ljava.lang.Object;")
|
||||
id-string-array-lg (read-array in String "[Ljava.lang.String;" (make-array String (read-lg-count in)))
|
||||
id-object-array-lg (read-array in Object "[Ljava.lang.Object;" (object-array (read-lg-count in)))
|
||||
|
||||
id-str-0 ""
|
||||
id-str-sm* (read-str in (read-sm-count* in))
|
||||
|
|
@ -1757,15 +1748,13 @@
|
|||
|
||||
(if (neg? type-id)
|
||||
(read-custom! in nil type-id) ; Unprefixed custom type
|
||||
(throw
|
||||
(ex-info
|
||||
(str "Unrecognized type id (" type-id "). Data frozen with newer Nippy version?")
|
||||
{:type-id type-id}))))
|
||||
(truss/ex-info!
|
||||
(str "Unrecognized type id (" type-id "). Data frozen with newer Nippy version?")
|
||||
{:type-id type-id})))
|
||||
|
||||
(catch Throwable t
|
||||
(throw
|
||||
(ex-info (str "Thaw failed against type-id: " type-id)
|
||||
{:type-id type-id} t))))))
|
||||
(truss/ex-info! (str "Thaw failed against type-id: " type-id)
|
||||
{:type-id type-id} t)))))
|
||||
|
||||
(let [head-sig head-sig] ; Not ^:const
|
||||
(defn- try-parse-header [^bytes ba]
|
||||
|
|
@ -1785,20 +1774,20 @@
|
|||
:lzma2 lzma2-compressor
|
||||
:lz4 lz4-compressor
|
||||
:zstd zstd-compressor
|
||||
:no-header (throw (ex-info ":auto not supported on headerless data." {}))
|
||||
:else (throw (ex-info ":auto not supported for non-standard compressors." {}))
|
||||
(do (throw (ex-info (str "Unrecognized :auto compressor id: " compressor-id)
|
||||
{:compressor-id compressor-id})))))
|
||||
:no-header (truss/ex-info! ":auto not supported on headerless data." {})
|
||||
:else (truss/ex-info! ":auto not supported for non-standard compressors." {})
|
||||
(do (truss/ex-info! (str "Unrecognized :auto compressor id: " compressor-id)
|
||||
{:compressor-id compressor-id}))))
|
||||
|
||||
(defn- get-auto-encryptor [encryptor-id]
|
||||
(case encryptor-id
|
||||
nil nil
|
||||
:aes128-gcm-sha512 aes128-gcm-encryptor
|
||||
:aes128-cbc-sha512 aes128-cbc-encryptor
|
||||
:no-header (throw (ex-info ":auto not supported on headerless data." {}))
|
||||
:else (throw (ex-info ":auto not supported for non-standard encryptors." {}))
|
||||
(do (throw (ex-info (str "Unrecognized :auto encryptor id: " encryptor-id)
|
||||
{:encryptor-id encryptor-id})))))
|
||||
:no-header (truss/ex-info! ":auto not supported on headerless data." {})
|
||||
:else (truss/ex-info! ":auto not supported for non-standard encryptors." {})
|
||||
(do (truss/ex-info! (str "Unrecognized :auto encryptor id: " encryptor-id)
|
||||
{:encryptor-id encryptor-id}))))
|
||||
|
||||
(def ^:private err-msg-unknown-thaw-failure "Possible decryption/decompression error, unfrozen/damaged data, etc.")
|
||||
(def ^:private err-msg-unrecognized-header "Unrecognized (but apparently well-formed) header. Data frozen with newer Nippy version?")
|
||||
|
|
@ -1845,24 +1834,23 @@
|
|||
(fn ex
|
||||
([ msg] (ex nil msg))
|
||||
([e msg]
|
||||
(throw
|
||||
(ex-info (str "Thaw failed. " msg)
|
||||
{:opts
|
||||
(assoc opts
|
||||
:compressor compressor
|
||||
:encryptor encryptor)
|
||||
(truss/ex-info! (str "Thaw failed. " msg)
|
||||
{:opts
|
||||
(assoc opts
|
||||
:compressor compressor
|
||||
:encryptor encryptor)
|
||||
|
||||
:bindings
|
||||
(enc/assoc-some {}
|
||||
'*freeze-fallback* *freeze-fallback*
|
||||
'*final-freeze-fallback* *final-freeze-fallback*
|
||||
'*auto-freeze-compressor* *auto-freeze-compressor*
|
||||
'*custom-readers* *custom-readers*
|
||||
'*incl-metadata?* *incl-metadata?*
|
||||
'*thaw-serializable-allowlist* *thaw-serializable-allowlist*
|
||||
'*thaw-xform* *thaw-xform*)}
|
||||
:bindings
|
||||
(enc/assoc-some {}
|
||||
'*freeze-fallback* *freeze-fallback*
|
||||
'*final-freeze-fallback* *final-freeze-fallback*
|
||||
'*auto-freeze-compressor* *auto-freeze-compressor*
|
||||
'*custom-readers* *custom-readers*
|
||||
'*incl-metadata?* *incl-metadata?*
|
||||
'*thaw-serializable-allowlist* *thaw-serializable-allowlist*
|
||||
'*thaw-xform* *thaw-xform*)}
|
||||
|
||||
e))))
|
||||
e)))
|
||||
|
||||
thaw-data
|
||||
(fn [data-ba compressor-id encryptor-id ex-fn]
|
||||
|
|
@ -2019,7 +2007,7 @@
|
|||
[{:keys [comparable?] :as opts}]
|
||||
(let [rng (java.util.Random. 123456) ; Seeded for determinism
|
||||
rand-nth (fn [coll] (nth coll (.nextInt rng (count coll))))
|
||||
all
|
||||
base
|
||||
{:nil nil
|
||||
:true true
|
||||
:false false
|
||||
|
|
@ -2056,7 +2044,6 @@
|
|||
#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{[] ()}}] #{:a :b}}
|
||||
[1 [1 2 [1 2 3 [1 2 3 4 [1 2 3 4 5 "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"] {} #{} [] ()]]]]]
|
||||
|
||||
:regex #"^(https?:)?//(www\?|\?)?"
|
||||
:sorted-set (sorted-set 1 2 3 4 5)
|
||||
:sorted-map (sorted-map :b 2 :a 1 :d 4 :c 3)
|
||||
:lazy-seq-empty (map identity ())
|
||||
|
|
@ -2064,25 +2051,10 @@
|
|||
:queue (into clojure.lang.PersistentQueue/EMPTY [:a :b :c :d :e :f :g])
|
||||
:queue-empty clojure.lang.PersistentQueue/EMPTY
|
||||
|
||||
:uuid (java.util.UUID. 7232453380187312026 -7067939076204274491)
|
||||
:uri (java.net.URI. "https://clojure.org")
|
||||
:defrecord (StressRecord. "data")
|
||||
:deftype (StressType. "data")
|
||||
|
||||
:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
|
||||
:objects (object-array [1 "two" {:data "data"}])
|
||||
|
||||
;; TODO (target-release>= 350)
|
||||
;; :byte-array (byte-array [(byte 1) (byte 2) (byte 3) (byte 4)])
|
||||
|
||||
;; :long-array (long-array [1 2 3 4])
|
||||
;; :int-array (int-array [1 2 3 4])
|
||||
|
||||
;; :double-array (double-array [1.5 2.5 3.5 4.5])
|
||||
;; :float-array (float-array [1.5 2.5 3.5 4.5])
|
||||
|
||||
;; :object-array (object-array [1 "two" {:data "data"}])
|
||||
;; :string-array (into-array String ["a" "b" "c"])
|
||||
:uuid (java.util.UUID. 7232453380187312026 -7067939076204274491)
|
||||
:uri (java.net.URI. "https://clojure.org")
|
||||
:defrecord (StressRecord. "data")
|
||||
:deftype (StressType. "data")
|
||||
|
||||
:util-date (java.util.Date. 1577884455500)
|
||||
:sql-date (java.sql.Date. 1577884455500)
|
||||
|
|
@ -2090,10 +2062,6 @@
|
|||
:duration (enc/compile-if java.time.Duration (java.time.Duration/ofSeconds 100 100) ::skip)
|
||||
:period (enc/compile-if java.time.Period (java.time.Period/of 1 1 1) ::skip)
|
||||
|
||||
:throwable (Throwable. "Msg")
|
||||
:exception (Exception. "Msg")
|
||||
:ex-info (ex-info "Msg" {:data "data"})
|
||||
|
||||
:many-longs (vec (repeatedly 512 #(rand-nth (range 10))))
|
||||
:many-doubles (vec (repeatedly 512 #(double (rand-nth (range 10)))))
|
||||
:many-strings (vec (repeatedly 512 #(rand-nth ["foo" "bar" "baz" "qux"])))
|
||||
|
|
@ -2103,8 +2071,24 @@
|
|||
(rand-nth ["foo" "bar" "baz" "qux" ]))))}]
|
||||
|
||||
(if comparable?
|
||||
(dissoc all :bytes :objects :throwable :exception :ex-info :regex)
|
||||
(do all))))
|
||||
base
|
||||
(assoc base
|
||||
:non-comparable
|
||||
{:regex #"^(https?:)?//(www\?|\?)?"
|
||||
:throwable (Throwable. "Msg")
|
||||
:exception (Exception. "Msg")
|
||||
:ex-info (ex-info "Msg" {:data "data"})
|
||||
:arrays
|
||||
{:boolean (boolean-array (mapv even? (range 32)))
|
||||
:byte (byte-array (mapv byte (range 32)))
|
||||
:short (short-array (mapv short (range 32)))
|
||||
:int (int-array (mapv int (range 32)))
|
||||
:long (long-array (mapv long (range 32)))
|
||||
:float (float-array (mapv float (range 32)))
|
||||
:double (double-array (mapv double (range 32)))
|
||||
:char (char-array (mapv char (range 32)))
|
||||
:str (into-array String (mapv str (range 32)))
|
||||
:object (object-array (mapv vector (range 32)))}}))))
|
||||
|
||||
(comment
|
||||
[(= (stress-data {:comparable? true}) (stress-data {:comparable? true}))
|
||||
|
|
@ -2247,5 +2231,5 @@
|
|||
(alter-var-root *thaw-serializable-allowlist* f) and/or
|
||||
(alter-var-root *freeze-serializable-allow-list* f) instead."
|
||||
[f]
|
||||
(alter-var-root *freeze-serializable-allowlist* (fn [old] (f (enc/have set? old))))
|
||||
(alter-var-root *thaw-serializable-allowlist* (fn [old] (f (enc/have set? old))))))
|
||||
(alter-var-root *freeze-serializable-allowlist* (fn [old] (f (truss/have set? old))))
|
||||
(alter-var-root *thaw-serializable-allowlist* (fn [old] (f (truss/have set? old))))))
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
(ns ^:no-doc taoensso.nippy.compression
|
||||
"Private, implementation detail."
|
||||
(:require [taoensso.encore :as enc])
|
||||
(:require
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc])
|
||||
(:import
|
||||
[java.nio ByteBuffer]
|
||||
[java.io
|
||||
|
|
@ -156,7 +158,7 @@
|
|||
(.read xzs ba 0 len-decomp)
|
||||
(if (== -1 (.read xzs)) ; Good practice as extra safety measure
|
||||
nil
|
||||
(throw (ex-info "LZMA2 Decompress failed: corrupt data?" {:ba ba})))
|
||||
(truss/ex-info! "LZMA2 Decompress failed: corrupt data?" {:ba ba}))
|
||||
ba)))
|
||||
|
||||
;;;; Public API
|
||||
|
|
|
|||
|
|
@ -2,7 +2,9 @@
|
|||
"Low-level crypto utils.
|
||||
Private & alpha, very likely to change!"
|
||||
(:refer-clojure :exclude [rand-nth])
|
||||
(:require [taoensso.encore :as enc]))
|
||||
(:require
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc]))
|
||||
|
||||
;; Note that AES128 may be preferable to AES256 due to known attack
|
||||
;; vectors specific to AES256, Ref. <https://goo.gl/qU4CCV>
|
||||
|
|
@ -45,7 +47,7 @@
|
|||
(defn take-ba ^bytes [n ^bytes ba] (java.util.Arrays/copyOf ba ^int n)) ; Pads if ba too small
|
||||
(defn utf8->ba ^bytes [^String s] (.getBytes s "UTF-8"))
|
||||
(defn- add-salt ^bytes [?salt-ba ba] (if ?salt-ba (enc/ba-concat ?salt-ba ba) ba))
|
||||
(defn pwd-as-ba ^bytes [utf8-or-ba] (if (string? utf8-or-ba) (utf8->ba utf8-or-ba) (enc/have enc/bytes? utf8-or-ba)))
|
||||
(defn pwd-as-ba ^bytes [utf8-or-ba] (if (string? utf8-or-ba) (utf8->ba utf8-or-ba) (truss/have enc/bytes? utf8-or-ba)))
|
||||
|
||||
(comment (seq (pwd-as-ba "foo")))
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
(ns ^:no-doc taoensso.nippy.encryption
|
||||
"Private, implementation detail."
|
||||
(:require
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc]
|
||||
[taoensso.nippy.crypto :as crypto]))
|
||||
|
||||
|
|
@ -14,11 +15,11 @@
|
|||
(decrypt ^bytes [encryptor pwd ba]))
|
||||
|
||||
(defn- throw-destructure-ex [typed-password]
|
||||
(throw (ex-info
|
||||
(str "Expected password form: "
|
||||
"[<#{:salted :cached}> <password-string>].\n "
|
||||
"See `aes128-encryptor` docstring for details!")
|
||||
{:typed-password typed-password})))
|
||||
(truss/ex-info!
|
||||
(str "Expected password form: "
|
||||
"[<#{:salted :cached}> <password-string>].\n "
|
||||
"See `aes128-encryptor` docstring for details!")
|
||||
{:typed-password typed-password}))
|
||||
|
||||
(defn- destructure-typed-pwd [typed-password]
|
||||
(if (vector? typed-password)
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@
|
|||
"Private, implementation detail."
|
||||
(:require
|
||||
[clojure.string :as str]
|
||||
[taoensso.truss :as truss]
|
||||
[taoensso.encore :as enc]))
|
||||
|
||||
;;;; Fallback type tests
|
||||
|
|
@ -63,7 +64,7 @@
|
|||
(when x
|
||||
(if (string? x)
|
||||
(if (= x "") #{} (set (mapv str/trim (str/split x #"[,:]"))))
|
||||
(enc/have set? x))))
|
||||
(truss/have set? x))))
|
||||
|
||||
(comment
|
||||
(mapv classname-set [nil #{"foo"} "" "foo, bar:baz"])
|
||||
|
|
@ -151,6 +152,29 @@
|
|||
See that function's docstring for more info."
|
||||
[] (trim nmax (state_))))
|
||||
|
||||
(comment
|
||||
(count (get-recorded-serializable-classes))
|
||||
(enc/reduce-n
|
||||
(fn [_ n] (allow-and-record-any-serializable-class-unsafe (str n)))
|
||||
nil 0 1e5))
|
||||
|
||||
(let [compile
|
||||
(enc/fmemoize
|
||||
(fn [x]
|
||||
(if (allow-and-record? x)
|
||||
allow-and-record-any-serializable-class-unsafe
|
||||
(enc/name-filter x))))
|
||||
|
||||
fn? fn?
|
||||
conform?
|
||||
(fn [x cn]
|
||||
(if (fn? x)
|
||||
(x cn) ; Intentionally uncached, can be handy
|
||||
((compile x) cn)))]
|
||||
|
||||
(defn serializable-allowed? [allow-list class-name]
|
||||
(conform? allow-list class-name)))
|
||||
|
||||
;;;; Release targeting
|
||||
|
||||
(comment
|
||||
|
|
@ -159,8 +183,8 @@
|
|||
|
||||
;; To help support release targeting, we track new type ids added over time
|
||||
(let [id-history ; {<release> #{type-ids}}
|
||||
{350 ; v3.5.0 (YYYY-MM-DD), added 5x
|
||||
;; #{string-array-lg long-array-lg int-array-lg double-array-lg float-array-lg}
|
||||
{350 ; v3.5.0 (2025-04-15), added 5x
|
||||
;; #{int-array-lg long-array-lg float-array-lg double-array-lg string-array-lg}
|
||||
#{0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
|
||||
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
|
||||
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
|
||||
|
|
@ -247,28 +271,3 @@
|
|||
[min-release] (target>= min-release)))
|
||||
|
||||
(comment (macroexpand '(target-release>= 340)))
|
||||
|
||||
;;;
|
||||
|
||||
(comment
|
||||
(count (get-recorded-serializable-classes))
|
||||
(enc/reduce-n
|
||||
(fn [_ n] (allow-and-record-any-serializable-class-unsafe (str n)))
|
||||
nil 0 1e5))
|
||||
|
||||
(let [compile
|
||||
(enc/fmemoize
|
||||
(fn [x]
|
||||
(if (allow-and-record? x)
|
||||
allow-and-record-any-serializable-class-unsafe
|
||||
(enc/name-filter x))))
|
||||
|
||||
fn? fn?
|
||||
conform?
|
||||
(fn [x cn]
|
||||
(if (fn? x)
|
||||
(x cn) ; Intentionally uncached, can be handy
|
||||
((compile x) cn)))]
|
||||
|
||||
(defn serializable-allowed? [allow-list class-name]
|
||||
(conform? allow-list class-name)))
|
||||
|
|
|
|||
|
|
@ -1,10 +1,8 @@
|
|||
(ns taoensso.nippy.tools
|
||||
"Utils for community tools that want to add user-configurable Nippy support.
|
||||
Used by Carmine, Faraday, etc."
|
||||
|
||||
(:refer-clojure :exclude [binding])
|
||||
(:require
|
||||
[taoensso.encore :as enc :refer [binding]]
|
||||
[taoensso.encore :as enc]
|
||||
[taoensso.nippy :as nippy]))
|
||||
|
||||
(def ^:dynamic *freeze-opts* nil)
|
||||
|
|
|
|||
|
|
@ -4,8 +4,11 @@
|
|||
[clojure.test.check :as tc]
|
||||
[clojure.test.check.generators :as tc-gens]
|
||||
[clojure.test.check.properties :as tc-props]
|
||||
[taoensso.truss :as truss :refer [throws?]]
|
||||
[taoensso.encore :as enc :refer [ba=]]
|
||||
[taoensso.nippy :as nippy :refer [freeze thaw]]
|
||||
[taoensso.nippy.impl :as impl]
|
||||
[taoensso.nippy.tools :as tools]
|
||||
[taoensso.nippy.compression :as compr]
|
||||
[taoensso.nippy.crypto :as crypto]
|
||||
[taoensso.nippy-benchmarks :as benchmarks]))
|
||||
|
|
@ -53,10 +56,6 @@
|
|||
#(freeze % {:password [:salted "p"]}))
|
||||
test-data)))
|
||||
|
||||
(let [d (nippy/stress-data {})]
|
||||
[(is (= (vec (:bytes d)) ((comp vec thaw freeze) (:bytes d))))
|
||||
(is (= (vec (:objects d)) ((comp vec thaw freeze) (:objects d))))])
|
||||
|
||||
(is (= test-data ((comp #(thaw % {:compressor nippy/lzma2-compressor})
|
||||
#(freeze % {:compressor nippy/lzma2-compressor}))
|
||||
test-data)))
|
||||
|
|
@ -75,9 +74,9 @@
|
|||
#(freeze % {:compressor nippy/zstd-compressor}))
|
||||
test-data)))
|
||||
|
||||
(is (enc/throws? Exception (thaw (freeze test-data {:password "malformed"}))))
|
||||
(is (enc/throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
|
||||
(is (enc/throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
|
||||
(is (throws? Exception (thaw (freeze test-data {:password "malformed"}))))
|
||||
(is (throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
|
||||
(is (throws? Exception (thaw (freeze test-data {:password [:salted "p"]}))))
|
||||
|
||||
(is
|
||||
(= "payload"
|
||||
|
|
@ -94,14 +93,23 @@
|
|||
(let [n range-uint+] (= (thaw (freeze n)) n))
|
||||
(let [n (- range-uint+)] (= (thaw (freeze n)) n))]))
|
||||
|
||||
(is (enc/throws? :ex-info "Unfreezable type" (nippy/freeze (fn []))))
|
||||
(is (throws? :ex-info "Failed to freeze type" (nippy/freeze (fn []))))
|
||||
|
||||
(testing "Clojure v1.10+ metadata protocol extensions"
|
||||
[(is (enc/throws? :ex-info "Unfreezable type" (nippy/freeze (with-meta [] {:a :A, 'b (fn [])}))))
|
||||
(is (= {:a :A} (meta (nippy/thaw (nippy/freeze (with-meta [] {:a :A, 'b/c (fn [])}))))))
|
||||
(is (= nil (meta (nippy/thaw (nippy/freeze (with-meta [] { 'b/c (fn [])})))))
|
||||
[(is (throws? :ex-info "Failed to freeze type" (nippy/freeze (with-meta [] {:a :A, 'b (fn [])}))))
|
||||
(is (= {:a :A} (meta (nippy/thaw (nippy/freeze (with-meta [] {:a :A, 'b/c (fn [])}))))))
|
||||
(is (= nil (meta (nippy/thaw (nippy/freeze (with-meta [] { 'b/c (fn [])})))))
|
||||
"Don't attach empty metadata")])
|
||||
|
||||
(let [d (nippy/stress-data {})]
|
||||
[(is (= (vec (:bytes d)) ((comp vec thaw freeze) (:bytes d))))
|
||||
(is (= (vec (:objects d)) ((comp vec thaw freeze) (:objects d))))])
|
||||
|
||||
(testing "Arrays"
|
||||
(binding [nippy/*thaw-serializable-allowlist* nippy/default-freeze-serializable-allowlist]
|
||||
(mapv (fn [[k aval]] (is (= (vec aval) (-> aval nippy/freeze nippy/thaw vec)) (name k)))
|
||||
(get-in (nippy/stress-data {}) [:non-comparable :arrays]))))
|
||||
|
||||
(is (gen-test 1600 [gen-data] (= gen-data (thaw (freeze gen-data)))) "Generative")])
|
||||
|
||||
;;;; Custom types & records
|
||||
|
|
@ -112,7 +120,7 @@
|
|||
(deftest _types
|
||||
[(testing "Extend to custom type"
|
||||
[(is
|
||||
(enc/throws? Exception ; No thaw extension yet
|
||||
(throws? Exception ; No thaw extension yet
|
||||
(do
|
||||
(alter-var-root #'nippy/*custom-readers* (constantly {}))
|
||||
(nippy/extend-freeze MyType 1 [x s]
|
||||
|
|
@ -323,7 +331,7 @@
|
|||
[(is (= nippy/*thaw-serializable-allowlist* #{"base.1" "base.2" "add.1" "add.2"})
|
||||
"JVM properties override initial allowlist values")
|
||||
|
||||
(is (enc/throws? Exception (nippy/freeze sem {:serializable-allowlist #{}}))
|
||||
(is (throws? Exception (nippy/freeze sem {:serializable-allowlist #{}}))
|
||||
"Can't freeze Serializable objects unless approved by allowlist")
|
||||
|
||||
(is (sem?
|
||||
|
|
@ -435,8 +443,8 @@
|
|||
|
||||
(is (= (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze []))) []) "rf not run on empty colls")
|
||||
|
||||
(let [ex (enc/throws :default (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze [:a :b]))))]
|
||||
(is (= (-> ex enc/ex-cause enc/ex-cause ex-data :call) '(rf acc in)) "Error thrown via `*thaw-xform*`"))])
|
||||
(let [ex (truss/throws :default (binding [nippy/*thaw-xform* (map (fn [x] (/ 1 0)))] (thaw (freeze [:a :b]))))]
|
||||
(is (= (-> ex ex-cause ex-cause ex-data :call) '(rf acc in)) "Error thrown via `*thaw-xform*`"))])
|
||||
|
||||
;;;; Compressors
|
||||
|
||||
|
|
@ -453,7 +461,7 @@
|
|||
(print ".") (flush)
|
||||
(dotimes [_ 1000]
|
||||
(is
|
||||
(nil? (enc/catching (compr/decompress c (crypto/rand-bytes 1024))))
|
||||
(nil? (truss/catching :all (compr/decompress c (crypto/rand-bytes 1024))))
|
||||
"Decompression never crashes JVM, even against invalid data")))
|
||||
(println)))
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue