luciano/nippy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
410 commits 2 branches 109 tags 2 MiB
Commit graph

211 commits

Author SHA1 Message Date
Peter Taoussanis
da1622f271 Experimental: add Compressable-LZMA2 type & (replaceable) de/serializer 2013-11-06 13:58:55 +07:00
Peter Taoussanis
7e7975910a tools: fix thaw arg destructuring & opt merging 2013-11-05 22:41:51 +07:00
Peter Taoussanis
84e336fe65 Make byte transform on seqs 1-to-1, re-add ISeq as a fallback implementation 2013-10-31 13:29:34 +07:00
Peter Taoussanis
aa78ad6a0b Rename test type: :coll->:lazy-seq 2013-10-31 13:16:26 +07:00
Peter Taoussanis
b1dab1dc4b Add debug-mode flag 2013-10-31 13:15:22 +07:00
Peter Taoussanis
8f8ab99e95 Add some Serializable tests (exceptions) 2013-10-24 13:55:47 +07:00
Peter Taoussanis
9448d2b3ce Add Java's Serializable as a pre-Reader fallback mechanism 2013-10-24 13:55:47 +07:00
Peter Taoussanis
c58bf97af7 Move Record test into stress data 2013-10-24 13:55:47 +07:00
Peter Taoussanis
5c48ba8e26 Reduce amount of (unnecessary) freeze procotol extensions to interfaces 
These can cause confusing, inconsistent behaviour when the protocol dispatch
needs to choose between multiple implementations:
Ref. https://groups.google.com/forum/#!topic/clojure-dev/-zoLA78--Mo

The approach as of this commit is more robust and allows more flexibility in
extension.
 2013-10-24 13:55:47 +07:00
Peter Taoussanis
75dd0d9040 Remove tagged literal labels for uuid & date (both have been baked-in for some time) 2013-10-24 13:55:47 +07:00
Peter Taoussanis
a122e62fe0 Reposition freeze-to-stream! fn for clarity 2013-10-24 13:55:47 +07:00
Peter Taoussanis
26f32c7a95 Housekeeping 2013-10-24 13:34:38 +07:00
Peter Taoussanis
33eee786be Lower default LZMA2 compression level -> 0 (much better perf @ ~same ratio) 2013-10-19 18:17:47 +07:00
Peter Taoussanis
3879cf4d62 Add experimental LZMA2 compressor for higher compression ratios than Snappy 2013-10-19 15:37:38 +07:00
Peter Taoussanis
512aca32aa Housekeeping 2013-10-19 13:51:58 +07:00
Peter Taoussanis
f6d5ebd862 Revert "add fast-path encoding for long, doubles, keywords, and strings" 
This reverts commit a92c493375.

As per https://github.com/ptaoussanis/nippy/pull/31, tests appear to be
inconclusive about the effetcs of this commit. Candidate for future
reconsideration.
 2013-10-17 17:44:54 +07:00
Peter Taoussanis
ae0a2e7907 Update benchmarks 2013-10-12 16:45:19 +07:00
Zach Tellman
a92c493375 add fast-path encoding for long, doubles, keywords, and strings 2013-10-12 16:45:19 +07:00
Zach Tellman
5492c1ea0f don't iterate twice over uncounted seqs, and use explicit key/val accessors for kv-collections, appears to give ~25% improvement in 'freeze' 2013-10-12 16:45:19 +07:00
Peter Taoussanis
1b27c1be7a v2.2.0 2013-10-03 21:21:59 +07:00
Peter Taoussanis
49fbbf270f Deps 2013-09-25 16:11:22 +07:00
Peter Taoussanis
6cce8a2857 Housekeeping: docstring typo 2013-08-08 14:22:46 +07:00
Peter Taoussanis
2061dd0868 Add & _ arg for thaw-from-stream! 2013-08-07 18:43:04 +07:00
Peter Taoussanis
d15a12af76 Update benchmarks 2013-08-07 18:29:36 +07:00
Peter Taoussanis
08f2cb1273 Use *data-readers* value for edn/read-string 2013-08-07 17:52:58 +07:00
Peter Taoussanis
d3120f0246 Remove :print-dup? option (no longer useful since switch to edn/read-string) 2013-08-07 17:52:00 +07:00
Peter Taoussanis
efa56eca73 Use tools.reader.edn in benchmarks 2013-08-07 17:37:22 +07:00
Peter Taoussanis
05907403dc Revert 8e4cc07 (unnecessary, slower) 2013-08-07 17:37:22 +07:00
Peter Taoussanis
3addeed195 Swap IPersistentMap -> APersistentMap 2013-08-07 17:37:22 +07:00
Peter Taoussanis
8e1d649bcb Swap IPersistentCollection->ISeq (the former seems to interfere with IRecord dispatch) 2013-08-07 17:37:22 +07:00
Peter Taoussanis
a76734a25a Housekeeping, add test for records with custom extension 2013-08-07 17:30:13 +07:00
James Reeves
5442216343 Add a fast serialization for UUID objects 2013-08-07 10:21:10 +01:00
James Reeves
8b13ff68b9 Add a fast serialization for Date objects 2013-08-07 10:19:53 +01:00
Peter Taoussanis
e48ccb4c45 Merge pull request #26 from weavejester/fast-records 
Fast serialization for records
 2013-08-06 23:54:33 -07:00
James Reeves
6ab00df42c Add fast serialization for records 2013-08-06 17:56:43 +01:00
James Reeves
6caff3503e Remove read-eval option made unnecessary by EDN reader 2013-08-06 16:59:04 +01:00
James Reeves
6024a9cc31 Replace unsafe Clojure reader with safe EDN reader 2013-08-06 16:45:30 +01:00
Peter Taoussanis
c69bb0ec5f Perf: only set bindings when necessary 2013-08-02 21:28:51 +07:00
Peter Taoussanis
4071d0f3ec Switch to simpler extend-freeze, extend-thaw custom type API (ALPHA) 2013-08-02 21:28:51 +07:00
Peter Taoussanis
c2a964932c Add custom-freezer macro for easier Freezable extension 2013-08-02 15:04:11 +07:00
Peter Taoussanis
99091b0a32 Add support for custom thaw readers 2013-07-29 16:43:29 +07:00
Peter Taoussanis
8e4cc072e3 Move Reader fallback out of Freezable protocol to make protocol extensible 2013-07-29 15:58:51 +07:00
Peter Taoussanis
69611657c7 Expose low-level fns: freeze-to-stream!, thaw-from-stream! 2013-07-29 15:57:36 +07:00
Peter Taoussanis
a8b1686a54 Make head-meta ^:const 2013-07-25 15:41:13 +07:00
Peter Taoussanis
93ef118e80 Housekeeping 2013-07-06 15:55:59 +07:00
Peter Taoussanis
4adb7fa376 Housekeeping 2013-06-24 15:22:54 +07:00
Peter Taoussanis
8a33f15801 Temporarily disable grease dep 2013-06-22 18:38:16 +07:00
Max Penet
fb09ebb89b try grease 0.2.1 2013-06-18 18:32:54 +02:00
Peter Taoussanis
ac3901f626 Reverse use of fast io-streams 
Seemed to be causing issues in multi-JVM auto-tests. The added perf. benefit was
minor for Nippy's use case (~2-3%) - so this isn't an urgent change. Will
re-examine again later.
 2013-06-18 14:08:22 +07:00
Peter Taoussanis
f7a96bd37f Housekeeping 2013-06-18 13:48:34 +07:00
Peter Taoussanis
9a2b0a068b thaw API should throw on first non-legacy error 2013-06-18 09:49:42 +07:00
Peter Taoussanis
4326da0ae5 Add tooling API 2013-06-17 23:20:17 +07:00
Peter Taoussanis
c8c0b99ae2 Update benchmarks (2.0.0-alpha6 w/fast io-streams) 2013-06-17 15:12:18 +07:00
Peter Taoussanis
3e6b12ac11 Housekeeping for 67633d5 2013-06-17 13:54:45 +07:00
Max Penet
67633d5fd6 try non sync version of byte-array iostreams 2013-06-16 22:27:22 +02:00
Peter Taoussanis
559c73abef Fix core dump issue (important), simplify API 
PROBLEM: :legacy-mode :auto/true thawing was resulting in JVM core
dumps when attempting to use Snappy to decompress encrypted data.

CAUSE: The org.iq80.snappy implementation seems to choke on the
random IV byte data being generated by the AES128 encrypter. This
may or may not be a bug (still awaiting feedback from lib's authors).

SOLUTION: We're only susceptible to this issue when trying to
decompress data that is: a) encrypted, b) being thawed in legacy mode.
In particular, we're _not_ susceptible to this issue when thawing
in non-legacy mode because in that case we have a header explicitly
warning us that the data is encrypted.

An obvious work-around, therefore, is just to disable decryption when
attempting to thaw legacy-mode data. In practice this isn't a problem
because older versions of Nippy didn't support encryption anyway.
 2013-06-16 19:33:23 +07:00
Peter Taoussanis
e351fdfc43 Rename default compressor, encryptor 2013-06-16 17:55:20 +07:00
Peter Taoussanis
a2376184dc Auto AES128 block size 2013-06-16 16:20:55 +07:00
Peter Taoussanis
5877dc4df0 Update benchmarks (2.0.0-alpha6) 2013-06-16 14:06:36 +07:00
Peter Taoussanis
4ec950417c Rewrite all stream helpers as macros (performance) 2013-06-16 13:23:32 +07:00
Peter Taoussanis
26ad8dd252 Reorganize low-level stream fns (prepping for perf work) 2013-06-16 13:23:32 +07:00
Peter Taoussanis
c5d039b183 Make coll-thaw, coll-thaw-kvs macros (performance) 2013-06-16 13:23:32 +07:00
Peter Taoussanis
9c207cd6af Housekeeping 2013-06-16 13:23:32 +07:00
Max Penet
63765c8510 move thaw exception handler outside of thaw (it's recreated everytime the function gets called now) 2013-06-16 11:58:58 +07:00
Max Penet
e4cde95d54 repeatedly-into as a macro 2013-06-16 11:47:56 +07:00
Peter Taoussanis
823f2c3c46 Housekeeping 2013-06-15 20:41:43 +07:00
Peter Taoussanis
c42457a48c Housekeeping, new keyword implementation (mpenet) 2013-06-14 17:52:25 +07:00
Peter Taoussanis
f706a51a4d Rework v2 header format 2013-06-14 17:49:53 +07:00
Peter Taoussanis
15dd24ac06 Add :auto legacy mode for _full_, transparent backwards-compatibility 2013-06-13 19:01:57 +07:00
Peter Taoussanis
4bc6dde12b Update benchmarks (2.0.0-alpha1) 2013-06-13 16:41:54 +07:00
Peter Taoussanis
8d48ec9d75 NB: Simpler, more flexible API (backwards-compatible) 2013-06-13 16:41:54 +07:00
Peter Taoussanis
284d11c660 Prep for pluggable compression+encryption 2013-06-13 16:41:54 +07:00
Peter Taoussanis
5a398efd9f Rename repeatedly* -> repeatedly-into 2013-06-13 16:41:54 +07:00
Peter Taoussanis
bfc16ce3ab Optimize collection creation (much faster) 2013-06-13 16:41:54 +07:00
Peter Taoussanis
613c77b8a1 Further prep for new API, mark freeze-to-bytes and thaw-from-bytes as deprecated 2013-06-13 11:45:45 +07:00
Peter Taoussanis
9734e882bb Swap repeatedly -> utils/repeatedly* (faster) 2013-06-13 11:45:45 +07:00
Peter Taoussanis
7705c42142 Housekeeping prep for new API 2013-06-13 11:45:45 +07:00
Peter Taoussanis
ac380eb621 Mark freeze-to-stream! and thaw-from-stream! as private 2013-06-13 11:45:45 +07:00
Peter Taoussanis
da077c6a54 Drop Clojure 1.3 support 2013-06-13 11:45:45 +07:00
Peter Taoussanis
6fe433b579 Update tests, migrate to Expectations lib 2013-06-13 11:45:44 +07:00
Peter Taoussanis
ab3209f2dc Housekeeping 2013-06-13 01:20:18 +07:00
Peter Taoussanis
9a38a12e11 Add helpful thaw exceptions 2013-06-12 15:39:30 +07:00
Peter Taoussanis
611ab56b74 Housekeeping 2013-06-12 15:28:49 +07:00
Peter Taoussanis
4ac2a34d7a Crypto: simplify design, add auto salting 
Have decided to simplify the API even further and bring configuration down to
essentially one decision: do you want auto salting, or key caching?
 2013-06-12 15:09:17 +07:00
Peter Taoussanis
bea3f5e84e Fix crypto encryption ba-concat order 2013-06-12 02:04:36 +07:00
Peter Taoussanis
2dcad8f4bd Housekeeping 2013-06-11 21:15:41 +07:00
Peter Taoussanis
3ecbc420cd Add basic crypto support (alpha) 2013-06-11 21:10:08 +07:00
Peter Taoussanis
749228dde5 Update benchmarks 2013-06-03 16:42:49 +07:00
Max Penet
8305f30499 replace partial with an anonymous fn (performance optimisation) 2013-06-02 17:54:44 +02:00
Peter Taoussanis
b91ab8e031 Fix OOM issue for large lists 2013-04-14 14:47:09 +07:00
Peter Taoussanis
37a2a80a98 Add support for sorted-set and sorted-map types 2013-04-14 14:44:06 +07:00
Peter Taoussanis
eb9e828647 Bump dependency (snappy 0.3), simplify utils/compress-bytes 2013-02-05 20:24:14 +07:00
Peter Taoussanis
3e178dabf3 EXPERIMENTAL: Swap Snappy implementation org.xerial.snappy/snappy-java -> org.iq80.snappy/snappy 
This (native Java) implementation appears to be about as fast as
the old (JNI) implementation, but has better support across more platforms.

Going to evaluate stability on this branch for possible later merging into master.
 2013-01-20 19:39:08 +07:00
Peter Taoussanis
fdff6aacd1 Add autobench to tests 2013-01-20 19:37:08 +07:00
Peter Taoussanis
f056abc0e8 Add optional print-dup? arg to freeze-to-stream! (default to true) 2013-01-20 19:24:12 +07:00
Peter Taoussanis
03684b6d28 Housekeeping 2013-01-20 19:14:59 +07:00
Peter Taoussanis
f5b2b14320 Housekeeping 2012-12-05 13:47:54 +07:00
Peter Taoussanis
c2c46167ec Swap hash-map thaw implementation for perf & to avoid OOM errors 
Thanks to moonranger for pointing out the OOM issue.
 2012-12-04 13:17:50 +07:00
Peter Taoussanis
f20da14cb4 Stop writing version headers 
It's slow, and in practice doesn't offer anything better than deprecated per-type IDs.
 2012-08-15 23:24:44 +07:00
Peter Taoussanis
8bec62767d BREAKING (minor): Change freeze-to-bytes, thaw-from-bytes API when not using defaults. 
Breaks only for users that were overriding `freeze-to-bytes` or `thaw-from-bytes` defaults to disable
compression.

Changed to map-based config to allow extra control (like :read-eval? as suggested by Luc Préfontaine).
This change will also allow further additions in future without the need to break the API again.
 2012-07-24 14:42:19 +07:00
Peter Taoussanis
1bcab7ede2 Add version comparison stuff to utils. Now test tagged-literals when Clojure version is sufficient. 2012-07-21 18:23:48 +07:00
Peter Taoussanis
ba99940784 Fixed 'deprecated as of' comments. 2012-07-21 01:56:30 +07:00
Peter Taoussanis
780e06c693 Support strings (incl. reader strings) of length > 64K. 
Luc Préfontaine kindly pointed out that `.writeUTF` and `.readUTF` have a maximum length of
64k. This commit switches to a custom integer-length prefix for strings and reader types to
boost their max length. Keywords are unchanged.

Old string and reader type IDs are now deprecated but will continue to be supported for
thawing.
 2012-07-21 01:47:54 +07:00
Peter Taoussanis
8bc790cf06 Improve performance: remove unnecessary reflection. 
`write-as-bytes!` was unnecessarily general. Renamed as type-hinted `write-biginteger!`.
Improves write performance of all types that relied on `write-as-bytes`: bigints, bigdecs,
rational.
 2012-07-21 01:12:45 +07:00
Peter Taoussanis
0b3667c0e2 Simplified benchmarking. 2012-07-14 14:28:02 +07:00
Peter Taoussanis
1e92263f36 Housekeeping. 2012-07-08 18:34:47 +07:00
Peter Taoussanis
cddd2eef13 Added support for namespaced keywords. 2012-07-08 13:01:00 +07:00
Peter Taoussanis
8f9bce289b Add notes about tagged literals. 2012-07-08 12:49:22 +07:00
Peter Taoussanis
07e542dc52 Housekeeping. 2012-07-07 20:17:16 +07:00
Peter Taoussanis
146f82e079 Initial source, tests, benchmarks. 2012-07-07 19:03:44 +07:00
Peter Taoussanis
53332644bb Initial structure commit. 2012-07-07 00:29:41 +07:00