luciano/nippy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
412 commits 2 branches 109 tags 2 MiB
Commit graph

172 commits

Author SHA1 Message Date
Peter Taoussanis
4d1a374c8f Typehint typo 2014-11-04 12:31:41 +07:00
Peter Taoussanis
7925982939 Print warning when replacing a custom type reader 2014-08-27 19:20:23 +07:00
Peter Taoussanis
13cdf06a60 Make coerce-custom-type-id private 2014-08-27 19:20:23 +07:00
Peter Taoussanis
9e60939848 EXPERIMENTAL: Support keyword-id extensions (#50) 2014-07-06 13:25:08 +07:00
Peter Taoussanis
20b1c2b1d2 Encode compression type in Nippy header, major refactor/housekeeping 
Housekeeping includes:
  * Importing useful encryption+compression stuff into primary ns
    for lib consumers.
  * Promoting a number of things from Alpha status.
  * Exceptions are now all `ex-info`s.
  * Simplification of `thaw` API: Nippy v1 support is now automatic
    & configuration-free (performance impact in most cases is negligible).
 2014-04-29 23:10:26 +07:00
Peter Taoussanis
b7a454a9c8 EXPERIMENTAL: Make LZ4 the default Nippy compressor (back-compatible for header'ed data) 2014-04-29 23:10:26 +07:00
Peter Taoussanis
3c8661a094 [#48] Fix freeze/thaw identity on empty lazy seqs 2014-04-29 23:08:00 +07:00
Peter Taoussanis
7d84099074 IMPORTANT FIX: small strings weren't getting a proper UTF-8 encoding (xkihzew) 
`(.getBytes <string>)` was being used here instead of
`(.getBytes <string> "UTF-8")` as is correct and done elsewhere.

This means that small UTF-8 strings may have been incorrectly stored
in environments where UTF-8 is not the default JVM character encoding.

Bug was introduced in Nippy v2.6.0, released 9 days ago (2014 Mar 30).

*********************************************************************
Please check for possible errors in Unicode text written using Nippy
v2.6.0 if your JVM uses an alternative character encoding by default
*********************************************************************

Really sorry about this, not sure how this slipped in.

Thanks to @xkihzew for the bug report.
 2014-04-08 17:04:53 +07:00
Peter Taoussanis
6058ac7b54 Fix freezable? fn name typo (mbossenbroek) 2014-03-18 14:23:15 +07:00
Peter Taoussanis
54389cc3a0 NB migrate utils stuff to encore lib 2014-02-23 19:15:55 +07:00
Peter Taoussanis
42b366a432 Experimental freezable? fn 2014-02-16 18:49:45 +07:00
Peter Taoussanis
0b56746a61 Mod (experimental) Compressable-LZMA2 to use new :skip-header? mode, add :headerless-meta assertion 2014-02-09 18:29:16 +07:00
Peter Taoussanis
e2847f348f [#38]: Distinguish between BigInt/BigInteger on thawing (mlacorte) 2014-01-30 16:16:50 +07:00
Peter Taoussanis
3f982e78cd Make public some macros useful for custom extensions, add experimental compact-long type 2014-01-23 23:13:21 +07:00
Peter Taoussanis
cfa0969bea Optimize some small, common types 2014-01-22 16:50:53 +07:00
Peter Taoussanis
1edde2d53b Revert "New compact byte representation, backwards-compatible" 
This reverts commit 93ce024f58.
 2014-01-22 15:38:32 +07:00
Peter Taoussanis
93ce024f58 New compact byte representation, backwards-compatible 2014-01-22 15:38:22 +07:00
Peter Taoussanis
1d2daf206b Clean up new DataInput/Output API 2014-01-22 15:01:12 +07:00
Peter Taoussanis
87fcd3a9c6 Generalise to allow serialising from/to any DataInput/DataOutput. (cmf) 2014-01-22 14:56:07 +07:00
Peter Taoussanis
b413adc62a Make id defs easily IDE-collapsable 2014-01-22 14:56:07 +07:00
Peter Taoussanis
5690a9611e Improve support for headerless freezing (docs, error msgs, etc.) 2014-01-22 14:01:27 +07:00
Peter Taoussanis
d8bbbbdb32 Add Fressian benchmark 2014-01-21 14:55:51 +07:00
Peter Taoussanis
42950ec24a Add stress-data-comparable, stress-data-benchable 2014-01-21 14:55:50 +07:00
Peter Taoussanis
7e1b1bb6b4 Deprecate :legacy-mode, add :skip-header? mode 2014-01-21 14:55:50 +07:00
Peter Taoussanis
32e4e82091 BREAKING: Clean up (experimental) Compressable-LZMA2 2014-01-21 12:56:36 +07:00
Peter Taoussanis
c11d4e36a8 hk 2014-01-21 12:47:46 +07:00
Peter Taoussanis
d112ecc395 Revert "Experimental: write-compact-int for _all_ int-like writing" 
This reverts commit 93afb3820e.
 2014-01-21 12:46:35 +07:00
Peter Taoussanis
93afb3820e Experimental: write-compact-int for _all_ int-like writing 2014-01-21 12:46:23 +07:00
Peter Taoussanis
a5c2f39865 Revert "wip: experimenting with common number optimizations" 
This reverts commit 307a265447.
 2014-01-20 15:12:18 +07:00
Peter Taoussanis
307a265447 wip: experimenting with common number optimizations 2014-01-20 14:44:32 +07:00
Peter Taoussanis
f8239531da v2.5.2 2013-12-07 16:49:34 +07:00
Peter Taoussanis
5ea4af7356 Further work on freezer fallback reliability 2013-12-07 16:44:53 +07:00
Peter Taoussanis
3093eae322 Experimental: add freeze-fallback-as-str fn 2013-12-03 16:46:30 +07:00
Peter Taoussanis
43c85059d7 Throw on freeze (rather than thaw) when trying to freeze an unreadable object with pr-str 2013-12-03 16:22:52 +07:00
Peter Taoussanis
1a7812522c Experimental: add inspect-ba util for inspecting possibly-frozen data 2013-12-03 16:22:51 +07:00
Peter Taoussanis
0fd9275e37 Experimental compressable-lzma2: fix thaw bug 2013-11-07 12:47:36 +07:00
Peter Taoussanis
da1622f271 Experimental: add Compressable-LZMA2 type & (replaceable) de/serializer 2013-11-06 13:58:55 +07:00
Peter Taoussanis
84e336fe65 Make byte transform on seqs 1-to-1, re-add ISeq as a fallback implementation 2013-10-31 13:29:34 +07:00
Peter Taoussanis
aa78ad6a0b Rename test type: :coll->:lazy-seq 2013-10-31 13:16:26 +07:00
Peter Taoussanis
b1dab1dc4b Add debug-mode flag 2013-10-31 13:15:22 +07:00
Peter Taoussanis
8f8ab99e95 Add some Serializable tests (exceptions) 2013-10-24 13:55:47 +07:00
Peter Taoussanis
9448d2b3ce Add Java's Serializable as a pre-Reader fallback mechanism 2013-10-24 13:55:47 +07:00
Peter Taoussanis
c58bf97af7 Move Record test into stress data 2013-10-24 13:55:47 +07:00
Peter Taoussanis
5c48ba8e26 Reduce amount of (unnecessary) freeze procotol extensions to interfaces 
These can cause confusing, inconsistent behaviour when the protocol dispatch
needs to choose between multiple implementations:
Ref. https://groups.google.com/forum/#!topic/clojure-dev/-zoLA78--Mo

The approach as of this commit is more robust and allows more flexibility in
extension.
 2013-10-24 13:55:47 +07:00
Peter Taoussanis
75dd0d9040 Remove tagged literal labels for uuid & date (both have been baked-in for some time) 2013-10-24 13:55:47 +07:00
Peter Taoussanis
a122e62fe0 Reposition freeze-to-stream! fn for clarity 2013-10-24 13:55:47 +07:00
Peter Taoussanis
26f32c7a95 Housekeeping 2013-10-24 13:34:38 +07:00
Peter Taoussanis
512aca32aa Housekeeping 2013-10-19 13:51:58 +07:00
Peter Taoussanis
f6d5ebd862 Revert "add fast-path encoding for long, doubles, keywords, and strings" 
This reverts commit a92c493375.

As per https://github.com/ptaoussanis/nippy/pull/31, tests appear to be
inconclusive about the effetcs of this commit. Candidate for future
reconsideration.
 2013-10-17 17:44:54 +07:00
Zach Tellman
a92c493375 add fast-path encoding for long, doubles, keywords, and strings 2013-10-12 16:45:19 +07:00
Zach Tellman
5492c1ea0f don't iterate twice over uncounted seqs, and use explicit key/val accessors for kv-collections, appears to give ~25% improvement in 'freeze' 2013-10-12 16:45:19 +07:00
Peter Taoussanis
6cce8a2857 Housekeeping: docstring typo 2013-08-08 14:22:46 +07:00
Peter Taoussanis
2061dd0868 Add & _ arg for thaw-from-stream! 2013-08-07 18:43:04 +07:00
Peter Taoussanis
08f2cb1273 Use *data-readers* value for edn/read-string 2013-08-07 17:52:58 +07:00
Peter Taoussanis
d3120f0246 Remove :print-dup? option (no longer useful since switch to edn/read-string) 2013-08-07 17:52:00 +07:00
Peter Taoussanis
05907403dc Revert 8e4cc07 (unnecessary, slower) 2013-08-07 17:37:22 +07:00
Peter Taoussanis
3addeed195 Swap IPersistentMap -> APersistentMap 2013-08-07 17:37:22 +07:00
Peter Taoussanis
8e1d649bcb Swap IPersistentCollection->ISeq (the former seems to interfere with IRecord dispatch) 2013-08-07 17:37:22 +07:00
Peter Taoussanis
a76734a25a Housekeeping, add test for records with custom extension 2013-08-07 17:30:13 +07:00
James Reeves
5442216343 Add a fast serialization for UUID objects 2013-08-07 10:21:10 +01:00
James Reeves
8b13ff68b9 Add a fast serialization for Date objects 2013-08-07 10:19:53 +01:00
Peter Taoussanis
e48ccb4c45 Merge pull request #26 from weavejester/fast-records 
Fast serialization for records
 2013-08-06 23:54:33 -07:00
James Reeves
6ab00df42c Add fast serialization for records 2013-08-06 17:56:43 +01:00
James Reeves
6caff3503e Remove read-eval option made unnecessary by EDN reader 2013-08-06 16:59:04 +01:00
James Reeves
6024a9cc31 Replace unsafe Clojure reader with safe EDN reader 2013-08-06 16:45:30 +01:00
Peter Taoussanis
c69bb0ec5f Perf: only set bindings when necessary 2013-08-02 21:28:51 +07:00
Peter Taoussanis
4071d0f3ec Switch to simpler extend-freeze, extend-thaw custom type API (ALPHA) 2013-08-02 21:28:51 +07:00
Peter Taoussanis
c2a964932c Add custom-freezer macro for easier Freezable extension 2013-08-02 15:04:11 +07:00
Peter Taoussanis
99091b0a32 Add support for custom thaw readers 2013-07-29 16:43:29 +07:00
Peter Taoussanis
8e4cc072e3 Move Reader fallback out of Freezable protocol to make protocol extensible 2013-07-29 15:58:51 +07:00
Peter Taoussanis
69611657c7 Expose low-level fns: freeze-to-stream!, thaw-from-stream! 2013-07-29 15:57:36 +07:00
Peter Taoussanis
a8b1686a54 Make head-meta ^:const 2013-07-25 15:41:13 +07:00
Peter Taoussanis
93ef118e80 Housekeeping 2013-07-06 15:55:59 +07:00
Peter Taoussanis
8a33f15801 Temporarily disable grease dep 2013-06-22 18:38:16 +07:00
Max Penet
fb09ebb89b try grease 0.2.1 2013-06-18 18:32:54 +02:00
Peter Taoussanis
ac3901f626 Reverse use of fast io-streams 
Seemed to be causing issues in multi-JVM auto-tests. The added perf. benefit was
minor for Nippy's use case (~2-3%) - so this isn't an urgent change. Will
re-examine again later.
 2013-06-18 14:08:22 +07:00
Peter Taoussanis
3e6b12ac11 Housekeeping for 67633d5 2013-06-17 13:54:45 +07:00
Max Penet
67633d5fd6 try non sync version of byte-array iostreams 2013-06-16 22:27:22 +02:00
Peter Taoussanis
559c73abef Fix core dump issue (important), simplify API 
PROBLEM: :legacy-mode :auto/true thawing was resulting in JVM core
dumps when attempting to use Snappy to decompress encrypted data.

CAUSE: The org.iq80.snappy implementation seems to choke on the
random IV byte data being generated by the AES128 encrypter. This
may or may not be a bug (still awaiting feedback from lib's authors).

SOLUTION: We're only susceptible to this issue when trying to
decompress data that is: a) encrypted, b) being thawed in legacy mode.
In particular, we're _not_ susceptible to this issue when thawing
in non-legacy mode because in that case we have a header explicitly
warning us that the data is encrypted.

An obvious work-around, therefore, is just to disable decryption when
attempting to thaw legacy-mode data. In practice this isn't a problem
because older versions of Nippy didn't support encryption anyway.
 2013-06-16 19:33:23 +07:00
Peter Taoussanis
e351fdfc43 Rename default compressor, encryptor 2013-06-16 17:55:20 +07:00
Peter Taoussanis
4ec950417c Rewrite all stream helpers as macros (performance) 2013-06-16 13:23:32 +07:00
Peter Taoussanis
26ad8dd252 Reorganize low-level stream fns (prepping for perf work) 2013-06-16 13:23:32 +07:00
Peter Taoussanis
c5d039b183 Make coll-thaw, coll-thaw-kvs macros (performance) 2013-06-16 13:23:32 +07:00
Peter Taoussanis
9c207cd6af Housekeeping 2013-06-16 13:23:32 +07:00
Max Penet
63765c8510 move thaw exception handler outside of thaw (it's recreated everytime the function gets called now) 2013-06-16 11:58:58 +07:00
Max Penet
e4cde95d54 repeatedly-into as a macro 2013-06-16 11:47:56 +07:00
Peter Taoussanis
823f2c3c46 Housekeeping 2013-06-15 20:41:43 +07:00
Peter Taoussanis
c42457a48c Housekeeping, new keyword implementation (mpenet) 2013-06-14 17:52:25 +07:00
Peter Taoussanis
f706a51a4d Rework v2 header format 2013-06-14 17:49:53 +07:00
Peter Taoussanis
15dd24ac06 Add :auto legacy mode for _full_, transparent backwards-compatibility 2013-06-13 19:01:57 +07:00
Peter Taoussanis
8d48ec9d75 NB: Simpler, more flexible API (backwards-compatible) 2013-06-13 16:41:54 +07:00
Peter Taoussanis
5a398efd9f Rename repeatedly* -> repeatedly-into 2013-06-13 16:41:54 +07:00
Peter Taoussanis
bfc16ce3ab Optimize collection creation (much faster) 2013-06-13 16:41:54 +07:00
Peter Taoussanis
613c77b8a1 Further prep for new API, mark freeze-to-bytes and thaw-from-bytes as deprecated 2013-06-13 11:45:45 +07:00
Peter Taoussanis
9734e882bb Swap repeatedly -> utils/repeatedly* (faster) 2013-06-13 11:45:45 +07:00
Peter Taoussanis
7705c42142 Housekeeping prep for new API 2013-06-13 11:45:45 +07:00
Peter Taoussanis
ac380eb621 Mark freeze-to-stream! and thaw-from-stream! as private 2013-06-13 11:45:45 +07:00
Peter Taoussanis
da077c6a54 Drop Clojure 1.3 support 2013-06-13 11:45:45 +07:00
Peter Taoussanis
ab3209f2dc Housekeeping 2013-06-13 01:20:18 +07:00
Peter Taoussanis
9a38a12e11 Add helpful thaw exceptions 2013-06-12 15:39:30 +07:00