[doc] Misc doc improvements

CHANGELOG.md

@@ -7,15 +7,25 @@ This project uses [**Break Versioning**](https://www.taoensso.com/break-versioni
 > **Dep**: Nippy is [on Clojars](https://clojars.org/com.taoensso/nippy/versions/3.4.2).  
 > **Versioning**: Nippy uses [Break Versioning](https://www.taoensso.com/break-versioning).
 
-This release updates some internal dependencies and is **recommended for all existing users**.
+⚠️ This release addresses a [**security vulnerability**](https://github.com/taoensso/nippy/security/advisories/GHSA-vw78-267v-588h) in Nippy's upstream compression library and is **recommended for all existing users**.
 
-It should be a **non-breaking update** for almost all users of Nippy `v3.4.x`, `v3.3.x`, and `v3.2.x`.
+It should be a **straight-forward and non-breaking update** for almost everyone:
 
-Notes:
+| Updating from Nippy version | API changes? | Changes to [byte output](https://github.com/taoensso/nippy/wiki/2-Operational-considerations#stability-of-byte-output)? | New types                                       |
+| :-------------------------- | :----------- | :---------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------- |
+| `v3.4.1` (2024-05-02)       | -            | -                                                                                                                       | -                                               |
+| `v3.4.0` (2024-04-30)       | -            | Yes                                                                                                                     | `clojure.lang.MapEntry`                         |
+| `v3.3.0` (2023-10-11)       | -            | -                                                                                                                       | `java.lang.ClassCastException`, `java.sql.Date` |
+| `v3.2.0` (2022-07-18)       | -            | -                                                                                                                       | `org.joda.time.DateTime`                        |
+| `v3.1.3` (2022-06-23)       | -            | -                                                                                                                       | Several `java.time.X` types                     |
 
-- May produce **different serialized output** to `v3.4.0` and `v3.3.0`. Most users won't care about this, but you could be affected if you depend on specific serialized byte values (for example by comparing serialized output between different versions of Nippy).
+If updating from older versions of Nippy, please see the relevant release notes.
-- When using Nippy version **X** to thaw data frozen by Nippy version **Y>X**, there is necessarily a risk of the thaw throwing when encountering unfamiliar types. This **can affect rolling updates** and/or **limit your ability to revert** a Nippy update - **so please ensure adequate testing** in your environment before updating against production data!
+
-- As always, **please report any unexpected problems** 🙏
+As always:
+
+- See [operational considerations](https://github.com/taoensso/nippy/wiki/2-Operational-considerations) for info on: **data compatibility**, **rolling updates**, **rollback support**, etc.
+- It's always a good idea to **ensure adequate testing** in your environment before updating against production data!
+- **Please report any unexpected problems** 🙏
 
 \- [Peter Taoussanis](https://www.taoensso.com)
 

README.md

@@ -7,14 +7,13 @@
 
 Clojure's rich data types are awesome. And its [reader](https://clojure.org/reference/reader) allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).
 
-Nippy is an attempt to provide a reliable, high-performance **drop-in alternative to the reader**.
+Nippy is a mature, high-performance **drop-in alternative to the reader**.
 
-Used by [Carmine](https://www.taoensso.com/carmine), [Faraday](https://www.taoensso.com/faraday), [PigPen](https://github.com/Netflix/PigPen), [Onyx](https://github.com/onyx-platform/onyx), 
+It is used at scale by [Carmine](https://www.taoensso.com/carmine), [Faraday](https://www.taoensso.com/faraday), [PigPen](https://github.com/Netflix/PigPen), [Onyx](https://github.com/onyx-platform/onyx), [XTDB](https://github.com/xtdb/xtdb), [Datalevin](https://github.com/juji-io/datalevin), and others.
-[XTDB](https://github.com/xtdb/xtdb), [Datalevin](https://github.com/juji-io/datalevin), and others.
 
 ## Latest release/s
 
-- `2024-05-26` `v3.4.2`: [release info](../../releases/tag/v3.4.2)
+- `2024-05-26` `v3.4.2`: [release info](../../releases/tag/v3.4.2) (⚠️ contains [**security fix**](https://github.com/taoensso/nippy/security/advisories/GHSA-vw78-267v-588h))
 
 [![Main tests][Main tests SVG]][Main tests URL]
 [![Graal tests][Graal tests SVG]][Graal tests URL]
@@ -23,24 +22,76 @@ See [here][GitHub releases] for earlier releases.
 
 ## Why Nippy?
 
-- Small, simple **all-Clojure** library
+- Small, simple **pure-Clojure** library
 - **Terrific performance**: the [best](#performance) for Clojure that I'm aware of
 - Comprehensive support for [all standard data types](../../wiki/1-Getting-started#deserializing)
 - Easily extendable to [custom data types](../../wiki/1-Getting-started#custom-types)
-- **Robust test suite**, incl. full coverage for every supported type
+- **Robust test suite** incl. coverage of every supported type
-- Auto fallback to [Java Serializable](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*freeze-serializable-allowlist*) when available
+- **Mature** and widely used in production for 12+ years
-- Auto fallback to Clojure Reader for all other types (including tagged literals)
+- Optional auto fallback to [Java Serializable](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*freeze-serializable-allowlist*) for [safe](https://cljdoc.org/d/com.taoensso/nippy/CURRENT/api/taoensso.nippy#*freeze-serializable-allowlist*) types
-- Pluggable **compression** with built-in [LZ4](https://code.google.com/p/lz4/), [Zstandard](https://facebook.github.io/zstd/), etc.
+- Optional auto fallback to Clojure Reader (including tagged literals)
-- Pluggable [encryption](../../wiki/1-Getting-started#encryption) with built-in AES128
+- Optional smart **compression** with [LZ4](https://code.google.com/p/lz4/) or [Zstandard](https://facebook.github.io/zstd/)
+- Optional [encryption](../../wiki/1-Getting-started#encryption) with AES128
 - [Tools](https://taoensso.github.io/nippy/taoensso.nippy.tools.html) for easy + robust **integration into 3rd-party libraries**, etc.
 - Powerful [thaw transducer](https://taoensso.github.io/nippy/taoensso.nippy.html#var-*thaw-xform*) for flexible data inspection and transformation
 
+## Quick example
+
+Nippy's super easy to use:
+
+```clojure
+(require '[taoensso.nippy :as nippy])
+
+;; Freeze any Clojure value
+(nippy/freeze <my-value>) ; => Serialized byte[]
+
+;; Thaw the byte[] to get back the original value:
+(nippy/thaw (nippy/freeze <my-value>)) ; => <my-value>
+```
+
+See the [wiki](https://github.com/taoensso/nippy/wiki/1-Getting-started#deserializing) for more.
+
+## Operational considerations
+
+### Data longevity
+
+Nippy is widely used to store **long-lived** data and promises (as always) that **data serialized today should be readable by all future versions of Nippy**.
+
+But please note that the **converse is not generally true**:
+
+- Nippy `vX` **should** be able to read all data from Nippy `vY<=X` (backwards compatibility)
+- Nippy `vX` **may/not** be able to read all data from Nippy `vY>X` (forwards compatibility)
+
+### Rolling updates and rollback
+
+From time to time, Nippy may introduce:
+
+- Support for serializing **new types**
+- Optimizations to the serialization of **pre-existing types**
+
+To help ease **rolling updates** and to better support **rollback**, Nippy (since version v3.4) will always introduce such changes over **two version releases**:
+
+- Release 1: to add **read support** for the new types
+- Release 2: to add **write support** for the new types
+
+Starting from v3.4, Nippy's release notes will **always clearly indicate** if a particular update sequence is recommended.
+
+### Stability of byte output
+
+It has **never been an objective** of Nippy to offer **predictable byte output**, and I'd generally **recommend against** depending on specific byte output.
+
+However, I know that a small minority of users *do* have specialized needs in this area.
+
+So starting with Nippy v3.4, Nippy's release notes will **always clearly indicate** if any changes to byte output are expected.
+
 ## Performance
 
-Since its earliest versions, Nippy has consistently been the **fastest serialization library for Clojure** that I'm aware of. Latest [benchmark](../../blob/master/test/taoensso/nippy_benchmarks.clj) results:
+Since its earliest versions, Nippy has consistently been the **fastest serialization library for Clojure** that I'm aware of. Latest results:
 
 ![benchmarks-png](../../raw/master/benchmarks.png)
 
+PRs welcome to include other alternatives in the [benchmark suite](../../blob/master/test/taoensso/nippy_benchmarks.clj)!
+
 ## Documentation
 
 - [Wiki][GitHub wiki] (getting started, usage, etc.)

src/taoensso/nippy.clj

@@ -1250,10 +1250,15 @@
     - Drops all support for compression and encryption
     - Must be thawed with `fast-thaw`
 
-  Equivalent to (but a little faster than) `freeze` with opts:
+  Equivalent to (but a little faster than) `freeze` with opts
-    - :compressor nil
+    {:no-header? true, :compressor nil, :encryptor nil}.
-    - :encryptor  nil
+
-    - :no-header? true"
+  Intended for use only by advanced users that clearly understand the tradeoffs.
+  I STRONGLY recommend that most users prefer the standard `freeze` since:
+    - The Nippy header is useful for data portability and preservation
+    - Compression is often benefitial at little/no cost
+    - The performance difference between `freeze` and `fast-freeze` is
+      often negligible in practice."
   ^bytes [x]
   (let [baos (ByteArrayOutputStream. 64)
         dos  (DataOutputStream. baos)]
@@ -1771,13 +1776,11 @@
 
 (defn fast-thaw
   "Like `thaw` but:
-    - Drops all support for compression and encryption
     - Supports only data frozen with `fast-freeze`
+    - Drops all support for compression and encryption
 
   Equivalent to (but a little faster than) `thaw` with opts:
-    - :compressor nil
+    {:no-header? true, :compressor nil, :encryptor nil}."
-    - :encryptor  nil
-    - :no-header? true"
   [^bytes ba]
   (let [dis (DataInputStream. (ByteArrayInputStream. ba))]
     (with-cache (thaw-from-in! dis))))

src/taoensso/nippy/compression.clj

@@ -174,8 +174,8 @@
 (def lz4-compressor
   "Default `LZ4` compressor:
     -   Compression ratio: `C`  (0.58      on reference benchmark).
-    -   Compression speed: `A`  (238 msecs on reference benchmark).
+    -   Compression speed: `A`  (240 msecs on reference benchmark).
-    - Decompression speed: `A+` (31  msecs on reference benchmark).
+    - Decompression speed: `A+` (30  msecs on reference benchmark).
 
   Good general-purpose compressor, favours speed.
   See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
@@ -184,8 +184,8 @@
 (def lzo-compressor
   "Default `LZO` compressor:
     -   Compression ratio: `C` (0.58      on reference benchmark).
-    -   Compression speed: `A` (216 msecs on reference benchmark).
+    -   Compression speed: `A` (220 msecs on reference benchmark).
-    - Decompression speed: `A` (43  msecs on reference benchmark).
+    - Decompression speed: `A` (40  msecs on reference benchmark).
 
   Good general-purpose compressor, favours speed.
   See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
@@ -195,7 +195,7 @@
   "Default `LZMA2` compressor:
     -   Compression ratio: `A+` (0.4       on reference benchmark).
     -   Compression speed: `E`  (18.5 secs on reference benchmark).
-    - Decompression speed: `D`  (11.8 secs on reference benchmark).
+    - Decompression speed: `D`  (12   secs on reference benchmark).
 
   Specialized compressor, strongly favours ratio.
   See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
@@ -204,8 +204,8 @@
 (enc/def* snappy-compressor
   "Default `Snappy` compressor:
     -   Compression ratio: `C`  (0.58      on reference benchmark).
-    -   Compression speed: `A+` (206 msecs on reference benchmark).
+    -   Compression speed: `A+` (210 msecs on reference benchmark).
-    - Decompression speed: `B`  (134 msecs on reference benchmark).
+    - Decompression speed: `B`  (130 msecs on reference benchmark).
   Good general-purpose compressor, favours speed.
   See `taoensso.nippy-benchmarks` for detailed comparative benchmarks."
   (SnappyCompressor. false))

wiki/1 Getting-started.md

@@ -107,7 +107,10 @@ Deserialize it:
 
 Couldn't be simpler!
 
-See also the lower-level [`freeze-to-out!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-freeze-to-out.21) and [`thaw-from-in!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-thaw-from-in.21) fns for operating on `DataOutput` and `DataInput` types directly.
+# Streaming
+
+- To serialize directly to a `java.io.DataInput`, see [`freeze-to-out!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-freeze-to-out.21).
+- To deserialize directly from a `java.io.DataOutput`, see [`thaw-from-in!`](https://taoensso.github.io/nippy/taoensso.nippy.html#var-thaw-from-in.21).
 
 # Encryption
 

wiki/2 Operational-considerations.md

@@ -0,0 +1,30 @@
+# Data longevity
+
+Nippy is widely used to store **long-lived** data and promises (as always) that **data serialized today should be readable by all future versions of Nippy**.
+
+But please note that the **converse is not generally true**:
+
+- Nippy `vX` **should** be able to read all data from Nippy `vY<=X` (backwards compatibility)
+- Nippy `vX` **may/not** be able to read all data from Nippy `vY>X` (forwards compatibility)
+
+# Rolling updates and rollback
+
+From time to time, Nippy may introduce:
+
+- Support for serializing **new types**
+- Optimizations to the serialization of **pre-existing types**
+
+To help ease **rolling updates** and to better support **rollback**, Nippy (since version v3.4.1) will always introduce such changes over **two version releases**:
+
+- Release 1: to add **read support** for the new types
+- Release 2: to add **write support** for the new types
+
+Starting from v3.4.1, Nippy's release notes will **always clearly indicate** if a particular update sequence is recommended.
+
+# Stability of byte output
+
+It has **never been an objective** of Nippy to offer **predictable byte output**, and I'd generally **recommend against** depending on specific byte output.
+
+However, I know that a small minority of users *do* have specialized needs in this area.
+
+So starting with Nippy v3.4, Nippy's release notes will **always clearly indicate** if any changes to byte output are expected.

wiki/3 Evolving-your-Nippy-data.md

@@ -1,4 +1,4 @@
-> This article was kindly contributed by a Nippy user (@Outrovurt)
+> This article is **community content** kindly contributed by a Nippy user (@Outrovurt)
 
 This article describes a number of use cases where you need to make changes to your code which will have some impact on data you have already frozen using Nippy, and how best to manage each specific case. We will also discuss custom freezing and thawing.