Merge branch 'dev': v0.9.0.
This commit is contained in:
Peter Taoussanis
commit
27bd0d2055
11 changed files with 784 additions and 6 deletions
15
.gitignore
vendored
15
.gitignore
vendored
|
|
@ -1,5 +1,14 @@
|
||||||
|
/target
|
||||||
|
/lib
|
||||||
|
/classes
|
||||||
|
/checkouts
|
||||||
|
/logs
|
||||||
|
/docs
|
||||||
pom.xml
|
pom.xml
|
||||||
*jar
|
*.jar
|
||||||
/lib/
|
*.class
|
||||||
/classes/
|
*.sh
|
||||||
.lein-deps-sum
|
.lein-deps-sum
|
||||||
|
.lein-failures
|
||||||
|
.lein-plugins
|
||||||
|
dump.rdb
|
||||||
7
.travis.yml
Normal file
7
.travis.yml
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
language: clojure
|
||||||
|
lein: lein2
|
||||||
|
script: lein2 all test
|
||||||
|
jdk:
|
||||||
|
- openjdk7
|
||||||
|
- openjdk6
|
||||||
|
- oraclejdk7
|
||||||
134
README.md
134
README.md
|
|
@ -1,4 +1,132 @@
|
||||||
nippy
|
Current [semantic](http://semver.org/) version:
|
||||||
=====
|
|
||||||
|
|
||||||
TODO: serialization lib
|
```clojure
|
||||||
|
[com.taoensso/nippy "0.9.0"]
|
||||||
|
```
|
||||||
|
|
||||||
|
# Nippy, a serialization library for Clojure
|
||||||
|
|
||||||
|
Clojure's [rich data types](http://clojure.org/datatypes) are *awesome*. And its [reader](http://clojure.org/reader) allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).
|
||||||
|
|
||||||
|
Nippy is an attempt to provide a drop-in, high-performance alternative to the reader. It's a fork of [Deep-Freeze](https://github.com/halgari/deep-freeze) and is used as the [Carmine Redis client](https://github.com/ptaoussanis/carmine) serializer.
|
||||||
|
|
||||||
|
## What's In The Box?
|
||||||
|
* Simple, **high-performance** all-Clojure de/serializer.
|
||||||
|
* Comprehesive, extensible **support for all major data types**.
|
||||||
|
* **Reader-fallback** for difficult/future types.
|
||||||
|
* **Full test coverage** for every supported type.
|
||||||
|
* [Snappy](http://code.google.com/p/snappy/) **integrated de/compression** for efficient storage and network transfer.
|
||||||
|
|
||||||
|
## Status [](http://travis-ci.org/ptaoussanis/nippy)
|
||||||
|
|
||||||
|
Nippy is still currently *experimental*. It **has not yet been thoroughly tested in production** and its API is subject to change. To run tests against all supported Clojure versions, use:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
lein2 all test
|
||||||
|
```
|
||||||
|
|
||||||
|
### Known issue with Java 7 on OSX
|
||||||
|
|
||||||
|
Nippy uses [Snappy](http://code.google.com/p/snappy-java/) which currently has a minor path issue with Java 7 on OSX. Please see [here](https://github.com/ptaoussanis/carmine/issues/5#issuecomment-6450607) for a workaround until a proper fix is available.
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
### Leiningen
|
||||||
|
|
||||||
|
Depend on Nippy in your `project.clj`:
|
||||||
|
|
||||||
|
```clojure
|
||||||
|
[com.taoensso/nippy "0.9.0"]
|
||||||
|
```
|
||||||
|
|
||||||
|
and `require` the library:
|
||||||
|
|
||||||
|
```clojure
|
||||||
|
(ns my-app (:require [taoensso.nippy :as nippy]))
|
||||||
|
```
|
||||||
|
|
||||||
|
### De/Serializing
|
||||||
|
|
||||||
|
As an example of what Nippy can do, let's take a look at its own reference stress data:
|
||||||
|
|
||||||
|
```clojure
|
||||||
|
nippy/stress-data
|
||||||
|
=>
|
||||||
|
{:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
|
||||||
|
:nil nil
|
||||||
|
:boolean true
|
||||||
|
|
||||||
|
:char-utf8 \ಬ
|
||||||
|
:string-utf8 "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"
|
||||||
|
:string-long (apply str (range 1000))
|
||||||
|
:keyword :keyword
|
||||||
|
|
||||||
|
:list (list 1 2 3 4 5 (list 6 7 8 (list 9 10)))
|
||||||
|
:list-quoted '(1 2 3 4 5 (6 7 8 (9 10)))
|
||||||
|
:list-empty (list)
|
||||||
|
:vector [1 2 3 4 5 [6 7 8 [9 10]]]
|
||||||
|
:vector-empty []
|
||||||
|
:map {:a 1 :b 2 :c 3 :d {:e 4 :f {:g 5 :h 6 :i 7}}}
|
||||||
|
:map-empty {}
|
||||||
|
:set #{1 2 3 4 5 #{6 7 8 #{9 10}}}
|
||||||
|
:set-empty #{}
|
||||||
|
:meta (with-meta {:a :A} {:metakey :metaval})
|
||||||
|
:queue (-> (PersistentQueue/EMPTY) (conj :a :b :c :d :e :f :g))
|
||||||
|
:queue-empty (PersistentQueue/EMPTY)
|
||||||
|
:coll (repeatedly 1000 rand)
|
||||||
|
|
||||||
|
:byte (byte 16)
|
||||||
|
:short (short 42)
|
||||||
|
:integer (int 3)
|
||||||
|
:long (long 3)
|
||||||
|
:bigint (bigint 31415926535897932384626433832795)
|
||||||
|
|
||||||
|
:float (float 3.14)
|
||||||
|
:double (double 3.14)
|
||||||
|
:bigdec (bigdec 3.1415926535897932384626433832795)
|
||||||
|
|
||||||
|
:ratio 22/7}
|
||||||
|
```
|
||||||
|
|
||||||
|
Serialize it:
|
||||||
|
|
||||||
|
```clojure
|
||||||
|
(def frozen-stress-data (nippy/freeze-to-bytes nippy/stress-data))
|
||||||
|
=> #<byte[] [B@3253bcf3>
|
||||||
|
```
|
||||||
|
|
||||||
|
Deserialize it:
|
||||||
|
|
||||||
|
```clojure
|
||||||
|
(nippy/thaw-from-bytes frozen-stress-data)
|
||||||
|
=> {:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
|
||||||
|
:nil nil
|
||||||
|
:boolean true
|
||||||
|
<...> }
|
||||||
|
```
|
||||||
|
|
||||||
|
Couldn't be simpler!
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
[Detailed benchmark information](https://docs.google.com/spreadsheet/ccc?key=0AuSXb68FH4uhdE5kTTlocGZKSXppWG9sRzA5Y2pMVkE&pli=1#gid=0) is available on Google Docs.
|
||||||
|
|
||||||
|
## Nippy supports the ClojureWerkz Project Goals
|
||||||
|
|
||||||
|
ClojureWerkz is a growing collection of open-source, batteries-included [Clojure libraries](http://clojurewerkz.org/) that emphasise modern targets, great documentation, and thorough testing.
|
||||||
|
|
||||||
|
## Contact & Contribution
|
||||||
|
|
||||||
|
Reach me (Peter Taoussanis) at *ptaoussanis at gmail.com* for questions/comments/suggestions/whatever. I'm very open to ideas if you have any!
|
||||||
|
|
||||||
|
I'm also on Twitter: [@ptaoussanis](https://twitter.com/#!/ptaoussanis).
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Copyright © 2012 Peter Taoussanis
|
||||||
|
|
||||||
|
Distributed under the [Eclipse Public License](http://www.eclipse.org/legal/epl-v10.html), the same as Clojure.
|
||||||
48
benchmarks/benchmarks.clj
Normal file
48
benchmarks/benchmarks.clj
Normal file
|
|
@ -0,0 +1,48 @@
|
||||||
|
(ns taoensso.nippy.benchmarks
|
||||||
|
{:author "Peter Taoussanis"}
|
||||||
|
(:use [taoensso.nippy :as nippy :only (freeze-to-bytes thaw-from-bytes)]))
|
||||||
|
|
||||||
|
;; Remove stuff from stress-data that breaks reader
|
||||||
|
(def bench-data (dissoc nippy/stress-data :queue :queue-empty :bytes))
|
||||||
|
|
||||||
|
(defn reader-freeze [x] (binding [*print-dup* false] (pr-str x)))
|
||||||
|
(defn reader-thaw [x] (binding [*read-eval* false] (read-string x)))
|
||||||
|
|
||||||
|
(def roundtrip (comp thaw-from-bytes freeze-to-bytes))
|
||||||
|
(def reader-roundtrip (comp reader-thaw reader-freeze))
|
||||||
|
|
||||||
|
(defmacro time-requests
|
||||||
|
"Warms up, then executes given number of requests and returns total execution
|
||||||
|
times in msecs."
|
||||||
|
[num-requests & body]
|
||||||
|
`(do (dotimes [_# (int (/ ~num-requests 4))] ~@body) ; Warm-up
|
||||||
|
(let [start-time# (System/nanoTime)]
|
||||||
|
(dotimes [_# ~num-requests] ~@body)
|
||||||
|
(Math/round (/ (- (System/nanoTime) start-time#) 1000000.0)))))
|
||||||
|
|
||||||
|
(comment
|
||||||
|
|
||||||
|
;;; Times
|
||||||
|
(println
|
||||||
|
"---\n"
|
||||||
|
(let [num 10000]
|
||||||
|
{:reader {:freeze (time-requests num (reader-freeze bench-data))
|
||||||
|
:thaw (let [frozen (reader-freeze bench-data)]
|
||||||
|
(time-requests num (reader-thaw frozen)))
|
||||||
|
:round (time-requests num (reader-roundtrip bench-data))}
|
||||||
|
|
||||||
|
:nippy {:freeze (time-requests num (freeze-to-bytes bench-data))
|
||||||
|
:thaw (let [frozen (freeze-to-bytes bench-data)]
|
||||||
|
(time-requests num (thaw-from-bytes frozen)))
|
||||||
|
:round (time-requests num (roundtrip bench-data))}}))
|
||||||
|
|
||||||
|
;; Clojure 1.3.0, Nippy 0.9.0
|
||||||
|
;; {:reader {:freeze 23573, :thaw 31923, :round 53253},
|
||||||
|
;; :nippy {:freeze 3805, :thaw 3789, :round 7522}}
|
||||||
|
;; (float (/ 53253 7522)) = 7.079633
|
||||||
|
|
||||||
|
;;; Data size
|
||||||
|
(let [frozen (reader-freeze bench-data)] (count (.getBytes frozen "UTF8")))
|
||||||
|
(let [frozen (freeze-to-bytes bench-data)] (count frozen))
|
||||||
|
;; 22711, 12168
|
||||||
|
)
|
||||||
BIN
benchmarks/chart1.png
Normal file
BIN
benchmarks/chart1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 4.7 KiB |
BIN
benchmarks/chart2.png
Normal file
BIN
benchmarks/chart2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 3.9 KiB |
261
epl-v10.html
Normal file
261
epl-v10.html
Normal file
|
|
@ -0,0 +1,261 @@
|
||||||
|
<?xml version="1.0" encoding="ISO-8859-1" ?>
|
||||||
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
||||||
|
|
||||||
|
<head>
|
||||||
|
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" />
|
||||||
|
<title>Eclipse Public License - Version 1.0</title>
|
||||||
|
<style type="text/css">
|
||||||
|
body {
|
||||||
|
size: 8.5in 11.0in;
|
||||||
|
margin: 0.25in 0.5in 0.25in 0.5in;
|
||||||
|
tab-interval: 0.5in;
|
||||||
|
}
|
||||||
|
p {
|
||||||
|
margin-left: auto;
|
||||||
|
margin-top: 0.5em;
|
||||||
|
margin-bottom: 0.5em;
|
||||||
|
}
|
||||||
|
p.list {
|
||||||
|
margin-left: 0.5in;
|
||||||
|
margin-top: 0.05em;
|
||||||
|
margin-bottom: 0.05em;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
|
||||||
|
</head>
|
||||||
|
|
||||||
|
<body lang="EN-US">
|
||||||
|
|
||||||
|
<p align=center><b>Eclipse Public License - v 1.0</b></p>
|
||||||
|
|
||||||
|
<p>THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE
|
||||||
|
PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR
|
||||||
|
DISTRIBUTION OF THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS
|
||||||
|
AGREEMENT.</p>
|
||||||
|
|
||||||
|
<p><b>1. DEFINITIONS</b></p>
|
||||||
|
|
||||||
|
<p>"Contribution" means:</p>
|
||||||
|
|
||||||
|
<p class="list">a) in the case of the initial Contributor, the initial
|
||||||
|
code and documentation distributed under this Agreement, and</p>
|
||||||
|
<p class="list">b) in the case of each subsequent Contributor:</p>
|
||||||
|
<p class="list">i) changes to the Program, and</p>
|
||||||
|
<p class="list">ii) additions to the Program;</p>
|
||||||
|
<p class="list">where such changes and/or additions to the Program
|
||||||
|
originate from and are distributed by that particular Contributor. A
|
||||||
|
Contribution 'originates' from a Contributor if it was added to the
|
||||||
|
Program by such Contributor itself or anyone acting on such
|
||||||
|
Contributor's behalf. Contributions do not include additions to the
|
||||||
|
Program which: (i) are separate modules of software distributed in
|
||||||
|
conjunction with the Program under their own license agreement, and (ii)
|
||||||
|
are not derivative works of the Program.</p>
|
||||||
|
|
||||||
|
<p>"Contributor" means any person or entity that distributes
|
||||||
|
the Program.</p>
|
||||||
|
|
||||||
|
<p>"Licensed Patents" mean patent claims licensable by a
|
||||||
|
Contributor which are necessarily infringed by the use or sale of its
|
||||||
|
Contribution alone or when combined with the Program.</p>
|
||||||
|
|
||||||
|
<p>"Program" means the Contributions distributed in accordance
|
||||||
|
with this Agreement.</p>
|
||||||
|
|
||||||
|
<p>"Recipient" means anyone who receives the Program under
|
||||||
|
this Agreement, including all Contributors.</p>
|
||||||
|
|
||||||
|
<p><b>2. GRANT OF RIGHTS</b></p>
|
||||||
|
|
||||||
|
<p class="list">a) Subject to the terms of this Agreement, each
|
||||||
|
Contributor hereby grants Recipient a non-exclusive, worldwide,
|
||||||
|
royalty-free copyright license to reproduce, prepare derivative works
|
||||||
|
of, publicly display, publicly perform, distribute and sublicense the
|
||||||
|
Contribution of such Contributor, if any, and such derivative works, in
|
||||||
|
source code and object code form.</p>
|
||||||
|
|
||||||
|
<p class="list">b) Subject to the terms of this Agreement, each
|
||||||
|
Contributor hereby grants Recipient a non-exclusive, worldwide,
|
||||||
|
royalty-free patent license under Licensed Patents to make, use, sell,
|
||||||
|
offer to sell, import and otherwise transfer the Contribution of such
|
||||||
|
Contributor, if any, in source code and object code form. This patent
|
||||||
|
license shall apply to the combination of the Contribution and the
|
||||||
|
Program if, at the time the Contribution is added by the Contributor,
|
||||||
|
such addition of the Contribution causes such combination to be covered
|
||||||
|
by the Licensed Patents. The patent license shall not apply to any other
|
||||||
|
combinations which include the Contribution. No hardware per se is
|
||||||
|
licensed hereunder.</p>
|
||||||
|
|
||||||
|
<p class="list">c) Recipient understands that although each Contributor
|
||||||
|
grants the licenses to its Contributions set forth herein, no assurances
|
||||||
|
are provided by any Contributor that the Program does not infringe the
|
||||||
|
patent or other intellectual property rights of any other entity. Each
|
||||||
|
Contributor disclaims any liability to Recipient for claims brought by
|
||||||
|
any other entity based on infringement of intellectual property rights
|
||||||
|
or otherwise. As a condition to exercising the rights and licenses
|
||||||
|
granted hereunder, each Recipient hereby assumes sole responsibility to
|
||||||
|
secure any other intellectual property rights needed, if any. For
|
||||||
|
example, if a third party patent license is required to allow Recipient
|
||||||
|
to distribute the Program, it is Recipient's responsibility to acquire
|
||||||
|
that license before distributing the Program.</p>
|
||||||
|
|
||||||
|
<p class="list">d) Each Contributor represents that to its knowledge it
|
||||||
|
has sufficient copyright rights in its Contribution, if any, to grant
|
||||||
|
the copyright license set forth in this Agreement.</p>
|
||||||
|
|
||||||
|
<p><b>3. REQUIREMENTS</b></p>
|
||||||
|
|
||||||
|
<p>A Contributor may choose to distribute the Program in object code
|
||||||
|
form under its own license agreement, provided that:</p>
|
||||||
|
|
||||||
|
<p class="list">a) it complies with the terms and conditions of this
|
||||||
|
Agreement; and</p>
|
||||||
|
|
||||||
|
<p class="list">b) its license agreement:</p>
|
||||||
|
|
||||||
|
<p class="list">i) effectively disclaims on behalf of all Contributors
|
||||||
|
all warranties and conditions, express and implied, including warranties
|
||||||
|
or conditions of title and non-infringement, and implied warranties or
|
||||||
|
conditions of merchantability and fitness for a particular purpose;</p>
|
||||||
|
|
||||||
|
<p class="list">ii) effectively excludes on behalf of all Contributors
|
||||||
|
all liability for damages, including direct, indirect, special,
|
||||||
|
incidental and consequential damages, such as lost profits;</p>
|
||||||
|
|
||||||
|
<p class="list">iii) states that any provisions which differ from this
|
||||||
|
Agreement are offered by that Contributor alone and not by any other
|
||||||
|
party; and</p>
|
||||||
|
|
||||||
|
<p class="list">iv) states that source code for the Program is available
|
||||||
|
from such Contributor, and informs licensees how to obtain it in a
|
||||||
|
reasonable manner on or through a medium customarily used for software
|
||||||
|
exchange.</p>
|
||||||
|
|
||||||
|
<p>When the Program is made available in source code form:</p>
|
||||||
|
|
||||||
|
<p class="list">a) it must be made available under this Agreement; and</p>
|
||||||
|
|
||||||
|
<p class="list">b) a copy of this Agreement must be included with each
|
||||||
|
copy of the Program.</p>
|
||||||
|
|
||||||
|
<p>Contributors may not remove or alter any copyright notices contained
|
||||||
|
within the Program.</p>
|
||||||
|
|
||||||
|
<p>Each Contributor must identify itself as the originator of its
|
||||||
|
Contribution, if any, in a manner that reasonably allows subsequent
|
||||||
|
Recipients to identify the originator of the Contribution.</p>
|
||||||
|
|
||||||
|
<p><b>4. COMMERCIAL DISTRIBUTION</b></p>
|
||||||
|
|
||||||
|
<p>Commercial distributors of software may accept certain
|
||||||
|
responsibilities with respect to end users, business partners and the
|
||||||
|
like. While this license is intended to facilitate the commercial use of
|
||||||
|
the Program, the Contributor who includes the Program in a commercial
|
||||||
|
product offering should do so in a manner which does not create
|
||||||
|
potential liability for other Contributors. Therefore, if a Contributor
|
||||||
|
includes the Program in a commercial product offering, such Contributor
|
||||||
|
("Commercial Contributor") hereby agrees to defend and
|
||||||
|
indemnify every other Contributor ("Indemnified Contributor")
|
||||||
|
against any losses, damages and costs (collectively "Losses")
|
||||||
|
arising from claims, lawsuits and other legal actions brought by a third
|
||||||
|
party against the Indemnified Contributor to the extent caused by the
|
||||||
|
acts or omissions of such Commercial Contributor in connection with its
|
||||||
|
distribution of the Program in a commercial product offering. The
|
||||||
|
obligations in this section do not apply to any claims or Losses
|
||||||
|
relating to any actual or alleged intellectual property infringement. In
|
||||||
|
order to qualify, an Indemnified Contributor must: a) promptly notify
|
||||||
|
the Commercial Contributor in writing of such claim, and b) allow the
|
||||||
|
Commercial Contributor to control, and cooperate with the Commercial
|
||||||
|
Contributor in, the defense and any related settlement negotiations. The
|
||||||
|
Indemnified Contributor may participate in any such claim at its own
|
||||||
|
expense.</p>
|
||||||
|
|
||||||
|
<p>For example, a Contributor might include the Program in a commercial
|
||||||
|
product offering, Product X. That Contributor is then a Commercial
|
||||||
|
Contributor. If that Commercial Contributor then makes performance
|
||||||
|
claims, or offers warranties related to Product X, those performance
|
||||||
|
claims and warranties are such Commercial Contributor's responsibility
|
||||||
|
alone. Under this section, the Commercial Contributor would have to
|
||||||
|
defend claims against the other Contributors related to those
|
||||||
|
performance claims and warranties, and if a court requires any other
|
||||||
|
Contributor to pay any damages as a result, the Commercial Contributor
|
||||||
|
must pay those damages.</p>
|
||||||
|
|
||||||
|
<p><b>5. NO WARRANTY</b></p>
|
||||||
|
|
||||||
|
<p>EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS
|
||||||
|
PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
|
||||||
|
OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION,
|
||||||
|
ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY
|
||||||
|
OR FITNESS FOR A PARTICULAR PURPOSE. Each Recipient is solely
|
||||||
|
responsible for determining the appropriateness of using and
|
||||||
|
distributing the Program and assumes all risks associated with its
|
||||||
|
exercise of rights under this Agreement , including but not limited to
|
||||||
|
the risks and costs of program errors, compliance with applicable laws,
|
||||||
|
damage to or loss of data, programs or equipment, and unavailability or
|
||||||
|
interruption of operations.</p>
|
||||||
|
|
||||||
|
<p><b>6. DISCLAIMER OF LIABILITY</b></p>
|
||||||
|
|
||||||
|
<p>EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT
|
||||||
|
NOR ANY CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT,
|
||||||
|
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING
|
||||||
|
WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF
|
||||||
|
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
|
||||||
|
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OR
|
||||||
|
DISTRIBUTION OF THE PROGRAM OR THE EXERCISE OF ANY RIGHTS GRANTED
|
||||||
|
HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.</p>
|
||||||
|
|
||||||
|
<p><b>7. GENERAL</b></p>
|
||||||
|
|
||||||
|
<p>If any provision of this Agreement is invalid or unenforceable under
|
||||||
|
applicable law, it shall not affect the validity or enforceability of
|
||||||
|
the remainder of the terms of this Agreement, and without further action
|
||||||
|
by the parties hereto, such provision shall be reformed to the minimum
|
||||||
|
extent necessary to make such provision valid and enforceable.</p>
|
||||||
|
|
||||||
|
<p>If Recipient institutes patent litigation against any entity
|
||||||
|
(including a cross-claim or counterclaim in a lawsuit) alleging that the
|
||||||
|
Program itself (excluding combinations of the Program with other
|
||||||
|
software or hardware) infringes such Recipient's patent(s), then such
|
||||||
|
Recipient's rights granted under Section 2(b) shall terminate as of the
|
||||||
|
date such litigation is filed.</p>
|
||||||
|
|
||||||
|
<p>All Recipient's rights under this Agreement shall terminate if it
|
||||||
|
fails to comply with any of the material terms or conditions of this
|
||||||
|
Agreement and does not cure such failure in a reasonable period of time
|
||||||
|
after becoming aware of such noncompliance. If all Recipient's rights
|
||||||
|
under this Agreement terminate, Recipient agrees to cease use and
|
||||||
|
distribution of the Program as soon as reasonably practicable. However,
|
||||||
|
Recipient's obligations under this Agreement and any licenses granted by
|
||||||
|
Recipient relating to the Program shall continue and survive.</p>
|
||||||
|
|
||||||
|
<p>Everyone is permitted to copy and distribute copies of this
|
||||||
|
Agreement, but in order to avoid inconsistency the Agreement is
|
||||||
|
copyrighted and may only be modified in the following manner. The
|
||||||
|
Agreement Steward reserves the right to publish new versions (including
|
||||||
|
revisions) of this Agreement from time to time. No one other than the
|
||||||
|
Agreement Steward has the right to modify this Agreement. The Eclipse
|
||||||
|
Foundation is the initial Agreement Steward. The Eclipse Foundation may
|
||||||
|
assign the responsibility to serve as the Agreement Steward to a
|
||||||
|
suitable separate entity. Each new version of the Agreement will be
|
||||||
|
given a distinguishing version number. The Program (including
|
||||||
|
Contributions) may always be distributed subject to the version of the
|
||||||
|
Agreement under which it was received. In addition, after a new version
|
||||||
|
of the Agreement is published, Contributor may elect to distribute the
|
||||||
|
Program (including its Contributions) under the new version. Except as
|
||||||
|
expressly stated in Sections 2(a) and 2(b) above, Recipient receives no
|
||||||
|
rights or licenses to the intellectual property of any Contributor under
|
||||||
|
this Agreement, whether expressly, by implication, estoppel or
|
||||||
|
otherwise. All rights in the Program not expressly granted under this
|
||||||
|
Agreement are reserved.</p>
|
||||||
|
|
||||||
|
<p>This Agreement is governed by the laws of the State of New York and
|
||||||
|
the intellectual property laws of the United States of America. No party
|
||||||
|
to this Agreement will bring a legal action under this Agreement more
|
||||||
|
than one year after the cause of action arose. Each party waives its
|
||||||
|
rights to a jury trial in any resulting litigation.</p>
|
||||||
|
|
||||||
|
</body>
|
||||||
|
|
||||||
|
</html>
|
||||||
19
project.clj
Normal file
19
project.clj
Normal file
|
|
@ -0,0 +1,19 @@
|
||||||
|
(defproject com.taoensso/nippy "0.9.0"
|
||||||
|
:description "Simple, high-performance Clojure serialization library."
|
||||||
|
:url "https://github.com/ptaoussanis/nippy"
|
||||||
|
:license {:name "Eclipse Public License"}
|
||||||
|
:dependencies [[org.clojure/clojure "1.3.0"]
|
||||||
|
[org.xerial.snappy/snappy-java "1.0.4.1"]]
|
||||||
|
:profiles {:1.3 {:dependencies [[org.clojure/clojure "1.3.0"]]}
|
||||||
|
:1.4 {:dependencies [[org.clojure/clojure "1.4.0"]]}
|
||||||
|
:1.5 {:dependencies [[org.clojure/clojure "1.5.0-master-SNAPSHOT"]]}
|
||||||
|
:dev {:dependencies []}}
|
||||||
|
:repositories {"sonatype" {:url "http://oss.sonatype.org/content/repositories/releases"
|
||||||
|
:snapshots false
|
||||||
|
:releases {:checksum :fail :update :always}}
|
||||||
|
"sonatype-snapshots" {:url "http://oss.sonatype.org/content/repositories/snapshots"
|
||||||
|
:snapshots true
|
||||||
|
:releases {:checksum :fail :update :always}}}
|
||||||
|
:aliases {"all" ["with-profile" "1.3:1.4:1.5"]}
|
||||||
|
:min-lein-version "2.0.0"
|
||||||
|
:warn-on-reflection true)
|
||||||
283
src/taoensso/nippy.clj
Normal file
283
src/taoensso/nippy.clj
Normal file
|
|
@ -0,0 +1,283 @@
|
||||||
|
(ns taoensso.nippy
|
||||||
|
"Simple, high-performance Clojure serialization library. Adapted from
|
||||||
|
Deep-Freeze."
|
||||||
|
{:author "Peter Taoussanis"}
|
||||||
|
(:require [taoensso.nippy.utils :as utils])
|
||||||
|
(:import [java.io DataInputStream DataOutputStream ByteArrayOutputStream
|
||||||
|
ByteArrayInputStream]
|
||||||
|
[org.xerial.snappy Snappy]
|
||||||
|
[clojure.lang IPersistentList IPersistentVector IPersistentMap
|
||||||
|
IPersistentSet PersistentQueue IPersistentCollection Keyword
|
||||||
|
BigInt Ratio]))
|
||||||
|
|
||||||
|
;;;; Define type IDs
|
||||||
|
|
||||||
|
(def ^:const schema-header "\u0000~0.9.0")
|
||||||
|
|
||||||
|
(def ^:const id-reader (int 1)) ; Fallback: *print-dup* pr-str output
|
||||||
|
(def ^:const id-bytes (int 2))
|
||||||
|
(def ^:const id-nil (int 3))
|
||||||
|
(def ^:const id-boolean (int 4))
|
||||||
|
|
||||||
|
(def ^:const id-char (int 10))
|
||||||
|
(def ^:const id-string (int 11))
|
||||||
|
(def ^:const id-keyword (int 12))
|
||||||
|
|
||||||
|
(def ^:const id-list (int 20))
|
||||||
|
(def ^:const id-vector (int 21))
|
||||||
|
(def ^:const id-old-map (int 22)) ; DEPRECATED as of 0.9.0
|
||||||
|
(def ^:const id-set (int 23))
|
||||||
|
(def ^:const id-coll (int 24)) ; Fallback: non-specific collection
|
||||||
|
(def ^:const id-meta (int 25))
|
||||||
|
(def ^:const id-queue (int 26))
|
||||||
|
(def ^:const id-map (int 27))
|
||||||
|
|
||||||
|
(def ^:const id-byte (int 40))
|
||||||
|
(def ^:const id-short (int 41))
|
||||||
|
(def ^:const id-integer (int 42))
|
||||||
|
(def ^:const id-long (int 43))
|
||||||
|
(def ^:const id-bigint (int 44))
|
||||||
|
|
||||||
|
(def ^:const id-float (int 60))
|
||||||
|
(def ^:const id-double (int 61))
|
||||||
|
(def ^:const id-bigdec (int 62))
|
||||||
|
|
||||||
|
(def ^:const id-ratio (int 70))
|
||||||
|
|
||||||
|
;;;; Shared low-level stream stuff
|
||||||
|
|
||||||
|
(defn- write-id! [^DataOutputStream stream ^Integer id] (.writeByte stream id))
|
||||||
|
|
||||||
|
(defn- write-bytes!
|
||||||
|
[^DataOutputStream stream ^bytes ba]
|
||||||
|
(let [size (alength ba)]
|
||||||
|
(.writeInt stream size) ; Encode size of byte array
|
||||||
|
(.write stream ba 0 size)))
|
||||||
|
|
||||||
|
(defn- read-bytes!
|
||||||
|
^bytes [^DataInputStream stream]
|
||||||
|
(let [size (.readInt stream)
|
||||||
|
ba (byte-array size)]
|
||||||
|
(.read stream ba 0 size) ba))
|
||||||
|
|
||||||
|
(defn- write-as-bytes!
|
||||||
|
"Write arbitrary object as bytes using reflection."
|
||||||
|
[^DataOutputStream stream obj]
|
||||||
|
(write-bytes! stream (.toByteArray obj)))
|
||||||
|
|
||||||
|
(defn- read-biginteger!
|
||||||
|
"Wrapper around read-bytes! for common case of reading to a BigInteger.
|
||||||
|
Note that as of Clojure 1.3, java.math.BigInteger ≠ clojure.lang.BigInt."
|
||||||
|
^BigInteger [^DataInputStream stream]
|
||||||
|
(BigInteger. (read-bytes! stream)))
|
||||||
|
|
||||||
|
;;;; Freezing
|
||||||
|
|
||||||
|
(defprotocol Freezable (freeze [this stream]))
|
||||||
|
|
||||||
|
(comment (meta '^:DataOutputStream s))
|
||||||
|
|
||||||
|
(defmacro freezer
|
||||||
|
"Helper to extend Freezable protocol."
|
||||||
|
[type id & body]
|
||||||
|
`(extend-type ~type
|
||||||
|
~'Freezable
|
||||||
|
(~'freeze [~'x ~(with-meta 's {:tag 'DataOutputStream})]
|
||||||
|
(write-id! ~'s ~id)
|
||||||
|
~@body)))
|
||||||
|
|
||||||
|
(defmacro coll-freezer
|
||||||
|
"Helper to freeze simple collection types."
|
||||||
|
[type id & body]
|
||||||
|
`(freezer
|
||||||
|
~type ~id
|
||||||
|
(.writeInt ~'s (count ~'x)) ; Encode collection length
|
||||||
|
(doseq [i# ~'x] (freeze-to-stream!* ~'s i#))))
|
||||||
|
|
||||||
|
(freezer (Class/forName "[B") id-bytes (write-bytes! s x))
|
||||||
|
(freezer nil id-nil)
|
||||||
|
(freezer Boolean id-boolean (.writeBoolean s x))
|
||||||
|
|
||||||
|
(freezer Character id-char (.writeChar s (int x)))
|
||||||
|
(freezer String id-string (.writeUTF s x))
|
||||||
|
(freezer Keyword id-keyword (.writeUTF s (name x)))
|
||||||
|
|
||||||
|
(declare freeze-to-stream!*)
|
||||||
|
|
||||||
|
(coll-freezer IPersistentList id-list)
|
||||||
|
(coll-freezer IPersistentVector id-vector)
|
||||||
|
(freezer IPersistentMap id-map
|
||||||
|
(.writeInt s (* 2 (count x))) ; Encode num kvs
|
||||||
|
(doseq [[k v] x]
|
||||||
|
(freeze-to-stream!* s k)
|
||||||
|
(freeze-to-stream!* s v)))
|
||||||
|
(coll-freezer IPersistentSet id-set)
|
||||||
|
(coll-freezer PersistentQueue id-queue)
|
||||||
|
(coll-freezer IPersistentCollection id-coll) ; Must be LAST collection freezer!
|
||||||
|
|
||||||
|
(freezer Byte id-byte (.writeByte s x))
|
||||||
|
(freezer Short id-short (.writeShort s x))
|
||||||
|
(freezer Integer id-integer (.writeInt s x))
|
||||||
|
(freezer Long id-long (.writeLong s x))
|
||||||
|
(freezer BigInt id-bigint (write-as-bytes! s (.toBigInteger x)))
|
||||||
|
(freezer BigInteger id-bigint (write-as-bytes! s x))
|
||||||
|
|
||||||
|
(freezer Float id-float (.writeFloat s x))
|
||||||
|
(freezer Double id-double (.writeDouble s x))
|
||||||
|
(freezer BigDecimal id-bigdec
|
||||||
|
(write-as-bytes! s (.unscaledValue x))
|
||||||
|
(.writeInt s (.scale x)))
|
||||||
|
|
||||||
|
(freezer Ratio id-ratio
|
||||||
|
(write-as-bytes! s (.numerator x))
|
||||||
|
(write-as-bytes! s (.denominator x)))
|
||||||
|
|
||||||
|
;; Use Clojure's own reader as final fallback
|
||||||
|
(freezer Object id-reader (.writeUTF s (pr-str x)))
|
||||||
|
|
||||||
|
(defn- freeze-to-stream!* [^DataOutputStream s x]
|
||||||
|
(if-let [m (meta x)]
|
||||||
|
(do (write-id! s id-meta)
|
||||||
|
(freeze-to-stream!* s m)))
|
||||||
|
(freeze x s))
|
||||||
|
|
||||||
|
(defn freeze-to-stream!
|
||||||
|
"Serializes x to given output stream."
|
||||||
|
[data-output-stream x]
|
||||||
|
(binding [*print-dup* true] ; For `pr-str`
|
||||||
|
(freeze-to-stream!* data-output-stream schema-header)
|
||||||
|
(freeze-to-stream!* data-output-stream x)))
|
||||||
|
|
||||||
|
(defn freeze-to-bytes
|
||||||
|
"Serializes x to a byte array and returns the array."
|
||||||
|
(^bytes [x] (freeze-to-bytes x true))
|
||||||
|
(^bytes [x compress?]
|
||||||
|
(let [ba (ByteArrayOutputStream.)
|
||||||
|
stream (DataOutputStream. ba)]
|
||||||
|
(freeze-to-stream! stream x)
|
||||||
|
(let [ba (.toByteArray ba)]
|
||||||
|
(if compress? (Snappy/compress ba) ba)))))
|
||||||
|
|
||||||
|
;;;; Thawing
|
||||||
|
|
||||||
|
(declare thaw-from-stream!*)
|
||||||
|
|
||||||
|
(defn coll-thaw!
|
||||||
|
"Helper to thaw simple collection types."
|
||||||
|
[^DataInputStream s]
|
||||||
|
(repeatedly (.readInt s) (partial thaw-from-stream!* s)))
|
||||||
|
|
||||||
|
(defn- thaw-from-stream!*
|
||||||
|
[^DataInputStream s]
|
||||||
|
(let [type-id (.readByte s)]
|
||||||
|
(utils/case-eval
|
||||||
|
type-id
|
||||||
|
|
||||||
|
id-reader (read-string (.readUTF s))
|
||||||
|
id-bytes (read-bytes! s)
|
||||||
|
id-nil nil
|
||||||
|
id-boolean (.readBoolean s)
|
||||||
|
|
||||||
|
id-char (.readChar s)
|
||||||
|
id-string (.readUTF s)
|
||||||
|
id-keyword (keyword (.readUTF s))
|
||||||
|
|
||||||
|
id-list (apply list (coll-thaw! s))
|
||||||
|
id-vector (into [] (coll-thaw! s))
|
||||||
|
id-set (into #{} (coll-thaw! s))
|
||||||
|
id-map (apply hash-map (coll-thaw! s))
|
||||||
|
id-coll (doall (coll-thaw! s))
|
||||||
|
id-queue (into (PersistentQueue/EMPTY) (coll-thaw! s))
|
||||||
|
|
||||||
|
;; DEPRECATED as of 0.9.0
|
||||||
|
id-old-map (apply hash-map (repeatedly (* 2 (.readInt s))
|
||||||
|
(partial thaw-from-stream!* s)))
|
||||||
|
|
||||||
|
id-meta (let [m (thaw-from-stream!* s)] (with-meta (thaw-from-stream!* s) m))
|
||||||
|
|
||||||
|
id-byte (.readByte s)
|
||||||
|
id-short (.readShort s)
|
||||||
|
id-integer (.readInt s)
|
||||||
|
id-long (.readLong s)
|
||||||
|
id-bigint (bigint (read-biginteger! s))
|
||||||
|
|
||||||
|
id-float (.readFloat s)
|
||||||
|
id-double (.readDouble s)
|
||||||
|
id-bigdec (BigDecimal. (read-biginteger! s) (.readInt s))
|
||||||
|
|
||||||
|
id-ratio (/ (bigint (read-biginteger! s))
|
||||||
|
(bigint (read-biginteger! s)))
|
||||||
|
|
||||||
|
(throw (Exception. (str "Failed to thaw unknown type ID: " type-id))))))
|
||||||
|
|
||||||
|
;; TODO Scheduled for Carmine version 1.0.0
|
||||||
|
;; (defn thaw-from-stream!
|
||||||
|
;; "Deserializes an object from given input stream."
|
||||||
|
;; [data-input-stream]
|
||||||
|
;; (binding [*read-eval* false] ; For `read-string` injection safety - NB!!!
|
||||||
|
;; (let [schema-header (thaw-from-stream!* data-input-stream)]
|
||||||
|
;; (thaw-from-stream!* data-input-stream))))
|
||||||
|
|
||||||
|
;; DEPRECATED: Includes temporary support for older versions of serialization
|
||||||
|
;; schema that didn't include a version header. This is for people that used
|
||||||
|
;; Carmine < 0.8.3 and haven't yet migrated their databases.
|
||||||
|
(defn thaw-from-stream!
|
||||||
|
"Deserializes an object from given input stream."
|
||||||
|
[data-input-stream]
|
||||||
|
(binding [*read-eval* false] ; For `read-string` injection safety - NB!!!
|
||||||
|
(let [maybe-schema-header (thaw-from-stream!* data-input-stream)]
|
||||||
|
(if (and (string? maybe-schema-header)
|
||||||
|
(.startsWith ^String maybe-schema-header "\u0000~"))
|
||||||
|
(thaw-from-stream!* data-input-stream)
|
||||||
|
maybe-schema-header))))
|
||||||
|
|
||||||
|
(defn thaw-from-bytes
|
||||||
|
"Deserializes an object from given byte array."
|
||||||
|
([ba] (thaw-from-bytes ba true))
|
||||||
|
([ba compressed?]
|
||||||
|
(->> (if compressed? (Snappy/uncompress ba) ba)
|
||||||
|
(ByteArrayInputStream.)
|
||||||
|
(DataInputStream.)
|
||||||
|
(thaw-from-stream!))))
|
||||||
|
|
||||||
|
(def stress-data
|
||||||
|
"Reference data used for tests & benchmarks."
|
||||||
|
{;; Breaks reader, roundtrip equality
|
||||||
|
:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
|
||||||
|
|
||||||
|
:nil nil
|
||||||
|
:boolean true
|
||||||
|
|
||||||
|
:char-utf8 \ಬ
|
||||||
|
:string-utf8 "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"
|
||||||
|
:string-long (apply str (range 1000))
|
||||||
|
:keyword :keyword
|
||||||
|
|
||||||
|
:list (list 1 2 3 4 5 (list 6 7 8 (list 9 10)))
|
||||||
|
:list-quoted '(1 2 3 4 5 (6 7 8 (9 10)))
|
||||||
|
:list-empty (list)
|
||||||
|
:vector [1 2 3 4 5 [6 7 8 [9 10]]]
|
||||||
|
:vector-empty []
|
||||||
|
:map {:a 1 :b 2 :c 3 :d {:e 4 :f {:g 5 :h 6 :i 7}}}
|
||||||
|
:map-empty {}
|
||||||
|
:set #{1 2 3 4 5 #{6 7 8 #{9 10}}}
|
||||||
|
:set-empty #{}
|
||||||
|
:meta (with-meta {:a :A} {:metakey :metaval})
|
||||||
|
|
||||||
|
;; Breaks reader
|
||||||
|
:queue (-> (PersistentQueue/EMPTY) (conj :a :b :c :d :e :f :g))
|
||||||
|
:queue-empty (PersistentQueue/EMPTY)
|
||||||
|
|
||||||
|
:coll (repeatedly 1000 rand)
|
||||||
|
|
||||||
|
:byte (byte 16)
|
||||||
|
:short (short 42)
|
||||||
|
:integer (int 3)
|
||||||
|
:long (long 3)
|
||||||
|
:bigint (bigint 31415926535897932384626433832795)
|
||||||
|
|
||||||
|
:float (float 3.14)
|
||||||
|
:double (double 3.14)
|
||||||
|
:bigdec (bigdec 3.1415926535897932384626433832795)
|
||||||
|
|
||||||
|
:ratio 22/7})
|
||||||
13
src/taoensso/nippy/utils.clj
Normal file
13
src/taoensso/nippy/utils.clj
Normal file
|
|
@ -0,0 +1,13 @@
|
||||||
|
(ns taoensso.nippy.utils
|
||||||
|
{:author "Peter Taoussanis"})
|
||||||
|
|
||||||
|
(defmacro case-eval
|
||||||
|
"Like `case` but evaluates test constants for their compile-time value."
|
||||||
|
[e & clauses]
|
||||||
|
(let [;; Don't evaluate default expression!
|
||||||
|
default (when (odd? (count clauses)) (last clauses))
|
||||||
|
clauses (if default (butlast clauses) clauses)]
|
||||||
|
`(case ~e
|
||||||
|
~@(map-indexed (fn [i# form#] (if (even? i#) (eval form#) form#))
|
||||||
|
clauses)
|
||||||
|
~(when default default))))
|
||||||
10
test/test_nippy/main.clj
Normal file
10
test/test_nippy/main.clj
Normal file
|
|
@ -0,0 +1,10 @@
|
||||||
|
(ns test-nippy.main
|
||||||
|
(:use [clojure.test])
|
||||||
|
(:require [taoensso.nippy :as nippy]))
|
||||||
|
|
||||||
|
;; Remove stuff from stress-data that breaks roundtrip equality
|
||||||
|
(def test-data (dissoc nippy/stress-data :bytes))
|
||||||
|
|
||||||
|
(def roundtrip (comp nippy/thaw-from-bytes nippy/freeze-to-bytes))
|
||||||
|
|
||||||
|
(deftest test-roundtrip (is (= test-data (roundtrip test-data))))
|
||||||
Loading…
Reference in a new issue