luciano/nippy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
618 commits 2 branches 109 tags 2 MiB
Commit graph

333 commits

Author SHA1 Message Date
Peter Taoussanis
7aa6425159 [#127] Add utils: freeze-to-string, thaw-from-string (@piotr-yuxuan) 2020-07-23 12:22:27 +02:00
Peter Taoussanis
f1c71b58d8 [Crypto] Use enc/srng 2020-07-23 12:22:27 +02:00
Peter Taoussanis
649e140889 [Crypto] Add rand-long [n] arity 2019-03-30 13:42:46 +01:00
Peter Taoussanis
23276ac910 [#101] NB Change default encryption from AES-CBC to AES-GCM 
Why?

  - AES-GCM is faster and can be more secure, Ref. https://goo.gl/Dsc9mL, etc.
  - AES-GCM is an authenticated[1] encryption mechanism, providing
    automatic integrity checks. This is relevant to [#101].

What's the issue with #101?

  - We    compress then encrypt    on freeze ; Reverse would make compression useless
  - So we decrypt  then decompress on thaw

Attempting CBC decryption with the wrong password will often but not
*always* throw. Meaning it's possible for decompression could be
attempted with a junk ba. And this can cause some decompressors to
fail in a destructive way, including large allocations (DDoS) or even
taking down the JVM in extreme cases.

Possible solutions?

  - We could add our own HMAC, etc.
  - And/or we could use something like AES-GCM which offers built-in
    integrity and will throw an AEADBadTagException on failure.

There may indeed be reasons [2,3,4] to consider adding a custom HMAC -
and that's still on the cards for later.

But in the meantime, the overall balance of pros/cons seems to lean
in the direction of choosing AES-GCM as a reasonable default.

Note that the change in this commit is done in a backward-compatible
way using Nippy's versioned header: new payloads will be written using
AES-GCM by default. But old payloads already written using AES-CBC will
continue to be read using that scheme.

References
  [1] https://en.wikipedia.org/wiki/Authenticated_encryption
  [2] https://www.daemonology.net/blog/2009-06-24-encrypt-then-mac.html
  [3] https://blog.cryptographyengineering.com/2011/12/04/matt-green-smackdown-watch-are-aead/
  [4] HMAC vs AEAD integrity,           https://crypto.stackexchange.com/q/24379
  [5] AES-GCM vs HMAC-SHA256 integrity, https://crypto.stackexchange.com/q/30627
 2019-01-06 14:13:34 +01:00
Peter Taoussanis
ae8baa639d [Crypto] Misc housekeeping, prep for next release 2019-01-06 12:24:29 +01:00
Peter Taoussanis
b0c7a0f8c7 [Crypto] Rename some arguments for extra clarity 2019-01-06 12:24:29 +01:00
Peter Taoussanis
7f9b075ba7 [#114] PR housekeeping 2019-01-06 12:24:29 +01:00
Isak Sky
abb55da29e [#113 #114] Support object arrays (@isaksky) 2019-01-06 12:24:29 +01:00
Peter Taoussanis
cfc904799b [Crypto] Rename prng->srng 
Better reflects the fact that the source of randomness is now
actually conditional (e.g. via InstanceStrong).
 2018-10-07 09:38:48 +02:00
Peter Taoussanis
90f0ff9315 [Crypto] sha512-key-ba: support utf8 *or* ba keys 
Minor convenience.
 2018-10-06 13:52:45 +02:00
Peter Taoussanis
c83572f0a8 [#112] PR housekeeping 2018-10-06 10:22:13 +02:00
Isak Sky
67dde8d7bd [#83 #112] Add support for deftype (@isaksky) 2018-10-06 09:57:35 +02:00
Isak Sky
192666c09e [#83 #113] Add URI support (@isaksky) 2018-10-06 09:50:24 +02:00
Peter Taoussanis
e16c64c4f4 [Crypto] Add secure rand-nth fn 2018-09-23 19:38:51 +02:00
Peter Taoussanis
f6c17a7411 [Crypto] Tune prng re-seeding frequency 2018-09-15 22:20:27 +02:00
Peter Taoussanis
d7993bb469 [Crypto] Add more hashing fns 
Specifically:

  - `sha512-ba` (without key salting, etc.)
  - `sha256-ba`
  - `murmur3` (Clojure 1.6+)
 2018-09-15 11:56:40 +02:00
Peter Taoussanis
82bb2f0104 [Crypto] Randomness improvements 
Specifically:

  - Now use blocking `getInstanceStrong` when available (Java 8+)
  - Now auto reseed prng after every ~10k calls (slower but safer)
  - [BREAKING] Support arbitrary random-bytes fn
  - Added new `rand-x` fns (double, long, bool, gauss)
 2018-09-15 11:36:03 +02:00
Peter Taoussanis
868a8f65de [Encryption] Rewrite encryption ns to build off new crypto ns 2018-09-08 19:28:02 +02:00
Peter Taoussanis
2dec26fd95 [Encryption] Add a new (private, alpha) taoensso.nippy.crypto ns with low-level utils 
Specifically:
  - Exposes ability to use arb crypto algorithm
  - Exposes ability to use arb key function
  - Supports explicit salts (incl. variable salt length)
  - Supports arbitrary key length (e.g. AES 256)
  - Defaults to AES/GCM/NoPadding algorithm
 2018-09-08 19:28:02 +02:00
Peter Taoussanis
2812ffa6e5 [Encryption] Housekeeping, no behavioural changes 2018-09-08 15:10:51 +02:00
Peter Taoussanis
2272d5ea57 [#104] Micro-optimization: remove unnecessary runtime 'if' in extend-freeze macro (@scramjet) 2017-12-21 10:47:30 +01:00
Peter Taoussanis
1a8a44286a [#93] Pickup record redefinitions at REPL, etc. (@smee) 2017-12-21 10:12:33 +01:00
Peter Taoussanis
ded6cc034f [#91] Add convenience utils for freeze/thaw to/from files 
Suggested by @Engelberg (thanks Mark!).

Also seems to be a common question online, e.g.:
http://stackoverflow.com/q/23018870
 2017-02-13 17:52:19 +01:00
Peter Taoussanis
bc33489dce Bump 1-byte cache count: 5->8 2016-10-28 16:52:05 +07:00
Peter Taoussanis
bc5f045979 Revert experimental semi-auto key caching 2016-10-28 16:37:54 +07:00
Peter Taoussanis
7c8acfe663 Experimental: optional semi-auto key caching 2016-10-28 16:36:04 +07:00
Peter Taoussanis
4aa1a3b871 ns form housekeeping 2016-10-28 10:25:46 +07:00
Peter Taoussanis
dee62aa0fc [#85] Lazily create LZ4 instance, fixes issue with Google App Engine 2016-09-01 13:57:25 +07:00
Peter Taoussanis
2eb3d25dba Hotfix: deprecated private API typo 2016-08-23 22:33:34 +07:00
Peter Taoussanis
5c94841313 De-deprecate type ids 6, 80 
Conceptually simpler to just retain these as first-class thaw-only
types.
 2016-07-26 12:22:23 +07:00
Peter Taoussanis
c1d48c7ef9 Hotfix: missing thaw routines for deprecated type ids: 6, 80 
Ref. https://github.com/ptaoussanis/faraday/issues/98
 2016-07-26 12:06:23 +07:00
Peter Taoussanis
a8faac734c Sync housekeeping 2016-07-24 15:48:09 +07:00
Peter Taoussanis
e07ec91f41 Misc housekeeping 2016-07-18 11:50:39 +07:00
Peter Taoussanis
3d8bc0eee1 Experimental: add cache metadata support 2016-07-17 15:42:41 +07:00
Peter Taoussanis
773180ef65 Misc minor optimizations, housekeeping 2016-07-17 15:42:41 +07:00
Peter Taoussanis
f94bc79a01 Hotfix: *final-freeze-fallback* back compatibility was broken 2016-06-17 12:25:31 +07:00
Peter Taoussanis
537b39aba2 Hotfix: fn?s were incorrectly reporting true for serializable? 2016-06-17 12:17:53 +07:00
Peter Taoussanis
4e2c24642f Misc housekeeping 2016-06-10 11:18:55 +07:00
Peter Taoussanis
a5a04dc11a v2.12.0-SNAPSHOT 2016-05-09 14:05:02 +07:00
Peter Taoussanis
4c647465f5 Encryption: micro optimizations, housekeeping 2016-05-09 14:05:02 +07:00
Peter Taoussanis
0df6a7b0f3 Misc hk 2016-05-09 14:05:02 +07:00
Peter Taoussanis
cac9123794 Restore backwards compatibility with Timbre v4.x Carmine appender 2016-04-18 13:36:25 +07:00
Peter Taoussanis
460c20d21f NB Fix missing String. charset 2016-04-14 13:19:58 +07:00
Peter Taoussanis
9a354784ae Remove arg type hints (slower) 2016-04-14 12:16:51 +07:00
Peter Taoussanis
c85329fe05 Cache housekeeping (incl. tests, switch to volatiles) 2016-04-14 12:16:51 +07:00
Peter Taoussanis
414b787684 Add fast-freeze, fast-thaw utils 2016-04-14 12:16:51 +07:00
Peter Taoussanis
3ab91763c6 [#82] Make it easier to spot new->old Nippy thaw failures 2016-04-14 12:16:51 +07:00
Peter Taoussanis
8fda27e996 Disable cache 2016-04-14 12:16:51 +07:00
Peter Taoussanis
699bb7cb51 Experimental support for signed counts 2016-04-14 12:16:51 +07:00
Peter Taoussanis
2028f80854 Experimental caching impl. 2016-04-14 12:16:51 +07:00
Peter Taoussanis
b623b4a8cc NB *BREAKING*: refactor type defs, variable-sized types, etc. 
Changes incl:
  - Hid a bunch of undocumented impl. details
  - A number of performance optimizations
 2016-04-14 12:16:27 +07:00
Peter Taoussanis
3f43542adb Tools housekeeping 2016-04-13 11:13:01 +07:00
Peter Taoussanis
892937eb34 [#80] Clarify docstrings for low-level freeze/thaw utils 2016-04-13 11:13:01 +07:00
Peter Taoussanis
d327f0ff38 Types: add dedicated 2 and 3 tuple type 2016-04-13 11:13:01 +07:00
Peter Taoussanis
7adad2240c Perf: optimize coll freezing via new enc/reduce-n 
- Take advantage of clojure.lang.LongRange
  - Avoid unnecessary temp `[k v]` (map entry) constructions
 2016-04-13 11:13:01 +07:00
Peter Taoussanis
28765ac501 Bump deps 2016-01-23 11:10:17 +07:00
Peter Taoussanis
c483e157bd Encryption ns housekeeping 2015-12-01 15:32:22 +07:00
Peter Taoussanis
643d762bbe Fix :auto encryption unit tests 2015-12-01 15:27:26 +07:00
Peter Taoussanis
f59f2f33cb NB fix min-val int-as-long 2015-12-01 14:45:24 +07:00
Peter Taoussanis
4df5446c5b Update benchmarks 2015-10-06 17:56:15 +07:00
Peter Taoussanis
3479ddad00 Clean up thaw fallback behaviour, decrease number of fallback cases 2015-10-06 17:56:15 +07:00
Peter Taoussanis
037cb14739 Misc hk 2015-10-06 15:57:55 +07:00
Peter Taoussanis
0905b96ca6 NB: Refactor thaw v1 compatibility support 2015-10-06 14:39:34 +07:00
Peter Taoussanis
9c8adfe513 **NB BREAKING**: change default :v1-compatibility? thaw option 
Motivation for changing this default:

v1 compatibility requires that in the event of a thaw failure, a fallback
attempt is made using v1 options. This must include an attempt at Snappy
decompression.

But the version of Snappy we're using has a major bug that can segfault +
crash the JVM when attempted against non-Snappy data:
https://github.com/dain/snappy/issues/20

I'd switch to an alternative Snappy implementation, but the only other
implementation I'm aware of uses JNI which can introduce troublesome
compatibility issues even for people who don't want the Snappy support.

Had hoped that the Snappy bug would eventually get fixed, but that's
looking unlikely.

Nippy v2 was released on July 22nd 2013 (2 years, 2 months ago) - so
am hoping that the majority of lib users will no longer have a need
for v1 data thaw support at this point.

For those that do, they can re-enable v1 thaw support with this flag.

If a better alternative solution ever presents (e.g. the Snappy bug
is fixed, an alternative implementation turns up, or we write a util
to reliably identify Snappy compressed data) - we can re-enable this
flag by default.
 2015-10-06 13:12:29 +07:00
Peter Taoussanis
cf38d6f111 Fix final-freeze-fallback arg order 2015-10-06 13:07:42 +07:00
Peter Taoussanis
2df9cb80d6 Add small-bytes type 2015-10-06 13:07:42 +07:00
Peter Taoussanis
c7c0c6fe54 Stop documenting :skip-header? option 
It's almost entirely useless now, and dangerous: folks who absolutely know what
they're doing can keep using it, but don't broadcast its existance.
 2015-10-06 13:07:41 +07:00
Peter Taoussanis
c5901730ea Update ba inspector 2015-10-06 13:07:41 +07:00
Peter Taoussanis
7faaf48ee7 Deprecate Compressable-LZMA2 (was anyway marked as experimental) 2015-10-06 13:07:41 +07:00
Peter Taoussanis
7072f73952 Misc hk 2015-10-06 13:07:41 +07:00
Peter Taoussanis
d61fb06f3b Primitive ided-long checks 2015-09-30 12:29:05 +07:00
Peter Taoussanis
2ebd8ce2ac Fix id typing 2015-09-30 12:29:05 +07:00
Peter Taoussanis
998dabc195 NB: refactor freezing utils for easier use by libs + custom extensions, etc. 2015-09-30 12:29:05 +07:00
Peter Taoussanis
15f0de1658 Simplify stream thaw API, switch from macros->fns 2015-09-30 12:29:04 +07:00
Peter Taoussanis
50ffb78c22 Refer rename: encore->enc 2015-09-30 12:29:04 +07:00
Peter Taoussanis
734e88b20c defonce on all dynamic vars (allow alter-var-root) 2015-09-30 12:29:04 +07:00
Peter Taoussanis
f67f9da64e Remove alpha status on final-freeze-fallback 2015-09-30 12:29:04 +07:00
Peter Taoussanis
9c1e8751c4 Simplify stream freeze API, switch from macros->fns 2015-09-30 12:29:04 +07:00
Peter Taoussanis
da77b3d582 NB: Remove (long-deprecated) freezing legacy mode 2015-09-30 12:29:04 +07:00
Peter Taoussanis
89c9328596 Experimental optimization: zero-copy freeze mode 2015-09-30 12:29:04 +07:00
Peter Taoussanis
fa17eb3a78 Update benchmarks 2015-09-30 12:29:04 +07:00
Peter Taoussanis
327a800d80 Experimental: optimize common case of small maps, sets, vectors 2015-09-30 12:29:04 +07:00
Peter Taoussanis
b298d690c7 Misc hk, reorganize type ids 2015-09-30 12:29:04 +07:00
Peter Taoussanis
885f192f6b Micro optimization: drop unnecessary double kvs count 
Had a vestigial count doubling from an historical implementation
that constructed hash-maps using `(apply hash-map ...)`
 2015-09-30 12:20:53 +07:00
Peter Taoussanis
4765a32e4e Optimize compact long freezer 2015-09-30 11:57:47 +07:00
Peter Taoussanis
1506747e42 Tune buffer size, freeze compressor selector 2015-09-30 11:57:47 +07:00
Peter Taoussanis
9c33f4f5ac Update benchmarks 2015-09-30 11:57:47 +07:00
Peter Taoussanis
ea9286dc90 Micro optimization: kv run is faster still 2015-09-30 11:57:47 +07:00
Peter Taoussanis
1ae8e6c389 Micro optimization: destructure faster than explicit calls here 2015-09-30 11:57:47 +07:00
Peter Taoussanis
956ce7df7e Micro optimization: read-bytes expansion 2015-09-30 11:57:46 +07:00
Peter Taoussanis
7ae954a229 Micro optimization: remove & args 2015-09-30 11:57:46 +07:00
Peter Taoussanis
2f27666d05 Rename deprecated type ids 
Making room for >1 deprecated id per type
 2015-09-30 11:57:46 +07:00
Peter Taoussanis
bbbc12ce30 Misc hk 2015-09-30 11:57:46 +07:00
Peter Taoussanis
db375e7686 Perf: anon fn is faster here 2015-09-30 11:57:46 +07:00
Peter Taoussanis
cea505484a Note re double vestigial kvs length 2015-09-30 11:57:46 +07:00
Peter Taoussanis
36abe07f2b Revert "Experimental/perf: kvs work directly against MapEntry" 
This reverts commit e150775cfe82f8206ddc88034417421e200851fa.
 2015-09-30 11:57:45 +07:00
Peter Taoussanis
aa9a18088f Experimental/perf: kvs work directly against MapEntry 2015-09-30 11:57:45 +07:00
Peter Taoussanis
9d4db3106e Perf: create Ratio's directly 2015-09-30 11:57:45 +07:00
Peter Taoussanis
41d3dcd467 *default-freeze-compressor-selector* should be public 2015-09-17 11:11:06 +07:00
Peter Taoussanis
a3847a4818 Misc housekeeping 2015-09-17 11:11:06 +07:00