Initial commit.
This commit is contained in:
Chas Emerick
commit
cab0f2fd29
6 changed files with 853 additions and 0 deletions
14
.gitignore
vendored
Normal file
14
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
# emacs + vi backup files
|
||||
*~
|
||||
.*.sw*
|
||||
|
||||
# various IDE junk
|
||||
*.ipr
|
||||
*.iml
|
||||
*.iws
|
||||
.project
|
||||
.classpath
|
||||
.settings
|
||||
|
||||
target
|
||||
classes
|
||||
270
README.asciidoc
Normal file
270
README.asciidoc
Normal file
|
|
@ -0,0 +1,270 @@
|
|||
= Bandalore
|
||||
|
||||
http://github.com/cemerick/bandalore[Bandalore] is a Clojure client
|
||||
library for Amazon's http://aws.amazon.com/sqs/[Simple Queue Service (SQS)]. It depends upon
|
||||
the standard http://aws.amazon.com/sdkforjava/[AWS SDK for Java],
|
||||
and provides a Clojure-idiomatic API for the SQS-related functionality
|
||||
therein.
|
||||
|
||||
== "Installation"
|
||||
|
||||
+++<del>Bandalore is available in Maven central.</del>+++
|
||||
|
||||
*I'll get Bandalore releases to central in the next couple of days, promise. :-)*
|
||||
|
||||
Add it to your Maven project's `pom.xml`:
|
||||
|
||||
----
|
||||
<dependency>
|
||||
<groupId>cemerick</groupId>
|
||||
<artifactId>bandalore</artifactId>
|
||||
<version>0.0.1-SNAPSHOT</version>
|
||||
</dependency>
|
||||
----
|
||||
|
||||
or your leiningen project.clj:
|
||||
|
||||
----
|
||||
[cemerick/bandalore "0.0.1"]
|
||||
----
|
||||
|
||||
== Usage
|
||||
|
||||
You should be familiar with http://aws.amazon.com/sqs/[SQS itself]
|
||||
before sensibly using this library. That said, Bandalore's API
|
||||
is well-documented.
|
||||
|
||||
You'll first need to load the library and create a SQS client object
|
||||
to do anything:
|
||||
|
||||
----
|
||||
(require '[cemerick.bandalore :as sqs])
|
||||
(def client (sqs/create-client "your aws id" "your aws secret-key"))
|
||||
----
|
||||
|
||||
You can create, delete, and list queues:
|
||||
|
||||
----
|
||||
=> (sqs/create-queue client "foo")
|
||||
"https://queue.amazonaws.com/499312652346/foo"
|
||||
=> (sqs/list-queues client)
|
||||
("https://queue.amazonaws.com/499312652346/foo")
|
||||
=> (sqs/delete-queue client (first *1))
|
||||
nil
|
||||
=> (list-queues client)
|
||||
nil
|
||||
----
|
||||
|
||||
*Note that SQS is _eventually consistent_. This means that a created
|
||||
queue won't necessarily show up in an immediate listing of queues,
|
||||
messages aren't necessarily immediately available to be received, etc.*
|
||||
|
||||
You can send, receive, and delete messages:
|
||||
|
||||
----
|
||||
=> (def q (sqs/create-queue client "foo"))
|
||||
#'cemerick.bandalore-test/q
|
||||
=> (sqs/send client q "my message body")
|
||||
{:id "75d5d7a1-2274-4163-97b2-aa4c75f209ee", :body-md5 "05d358de00fc63dd2fa2026b77e112f6"}
|
||||
=> (sqs/receive client q)
|
||||
({:attrs #<HashMap {}>, :body "my message body", :body-md5 "05d358de00fc63dd2fa2026b77e112f6",
|
||||
:id "75d5d7a1-2274-4163-97b2-aa4c75f209ee",
|
||||
:receipt-handle "…very long string…"})
|
||||
;;
|
||||
;; …presumably do something with the received message(s)…
|
||||
;;
|
||||
=> (sqs/delete client q (first *1))
|
||||
nil
|
||||
=> (sqs/receive client q)
|
||||
()
|
||||
----
|
||||
|
||||
That's cleaner than having to interop directly with the Java SDK, but it's all
|
||||
pretty pedestrian stuff. You can do more interesting things with some
|
||||
simple higher-order functions and other nifty Clojure facilities.
|
||||
|
||||
=== Sending and receiving Clojure values
|
||||
|
||||
SQS' message bodies are strings, so you can stuff anything in them that you can
|
||||
serialize to a string. That said, `pr-str` and `read-string` are too handy
|
||||
to not use, assuming your consumers are using Clojure as well:
|
||||
|
||||
----
|
||||
=> (sqs/send client q (pr-str {:a 5 :b "blah" :c 6.022e23}))
|
||||
{:id "3756c302-866a-4fcc-a7a3-746e6f531f47", :body-md5 "60052fc2ffb835257c26b9957c6e9ffd"}
|
||||
=> (-?> (sqs/receive client q) first :body read-string)
|
||||
{:a 5, :b "blah", :c 6.022E23}
|
||||
----
|
||||
|
||||
=== Sending seqs of messages
|
||||
|
||||
…with more gratuitous use of `pr-str` and `read-string` to send and receive
|
||||
Clojure values:
|
||||
|
||||
----
|
||||
=> (->> [:foo 'bar ["some vector" 42] #{#"silly place for a regex"}]
|
||||
(map (comp (partial sqs/send client q) pr-str))
|
||||
dorun)
|
||||
nil
|
||||
=> (map (comp read-string :body)
|
||||
(sqs/receive client q :limit 10))
|
||||
(bar ["some vector" 42])
|
||||
=> (map (comp read-string :body)
|
||||
(sqs/receive client q :limit 10))
|
||||
(#{#"silly place for a regex"})
|
||||
=> (map (comp read-string :body)
|
||||
(sqs/receive client q :limit 10))
|
||||
(:foo)
|
||||
----
|
||||
|
||||
=== (Mostly) automatic deletion of consumed messages
|
||||
|
||||
When you're done processing a received message, you need to delete it from its
|
||||
originaing queue:
|
||||
|
||||
----
|
||||
; ensure our queue is empty to start
|
||||
=> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages")
|
||||
"0"
|
||||
=> (dorun (map (partial sqs/send client q) (map str (range 100))))
|
||||
nil
|
||||
=> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages")
|
||||
"100"
|
||||
|
||||
; received messages must be removed from the queue or they will
|
||||
; be delivered again after their visibility timeout expires
|
||||
=> (sqs/receive client q)
|
||||
(…message seq…)
|
||||
=> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages")
|
||||
"100"
|
||||
=> (->> (sqs/receive client q) first (sqs/delete client))
|
||||
nil
|
||||
=> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages")
|
||||
"99"
|
||||
----
|
||||
|
||||
Rather than trying to remember to do this, just use the
|
||||
`deleting-consumer` "middleware" to produce a function that calls
|
||||
the message-processing function you provide to it, and then
|
||||
automatically deletes the processed message from the origining queue:
|
||||
|
||||
----
|
||||
=> (doall (map
|
||||
(sqs/deleting-consumer client (comp println :body))
|
||||
(sqs/receive client q :limit 10)))
|
||||
0
|
||||
4
|
||||
9
|
||||
12
|
||||
26
|
||||
36
|
||||
40
|
||||
44
|
||||
52
|
||||
55
|
||||
(nil nil nil nil nil nil nil nil nil nil)
|
||||
=> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages")
|
||||
"90"
|
||||
----
|
||||
|
||||
=== Consuming queues as seqs
|
||||
|
||||
seqs being the _lingua franca_ of Clojure collections, it would be helpful if we
|
||||
could treat an SQS queue as a seq of messages. While `receive` does return
|
||||
a seq of messages, each `receive` call is limited to receiving a maximum of
|
||||
10 messages, and there is no streaming or push counterpart in the SQS API.
|
||||
|
||||
The solution to this is `polling-receive`, which returns a lazy seq that
|
||||
reaches out to SQS as necessary:
|
||||
|
||||
----
|
||||
=> (map (sqs/deleting-consumer client :body)
|
||||
(sqs/polling-receive client q :limit 10))
|
||||
("3" "5" "7" "8" ... "81" "90" "91")
|
||||
----
|
||||
|
||||
`polling-receive` accepts all of the same optional kwargs as `receive` does,
|
||||
but adds two more to control its usage of `receive`:
|
||||
|
||||
:period - time in ms to wait after an unsuccessful `receive` request (default: 500)
|
||||
:max-wait - maximum time in ms to wait to successfully receive messages before terminating
|
||||
the lazy seq (default 5000ms)
|
||||
|
||||
Often queues are used to direct compute resources, so you'd like to be able to saturate
|
||||
those boxen with as much work as your queue can offer up. The obvious solution
|
||||
is to `pmap` across a seq of incoming messages, which you can do trivially with the seq
|
||||
provided by `polling-receive`. Just make sure you tweak the `:max-wait` time so that,
|
||||
assuming you want to continuously process incoming messages, the seq of messages doesn't
|
||||
terminate because none have been available for a while.
|
||||
|
||||
Here's an example where one thread sends a message once a second for a minute,
|
||||
and another consumes those messages using a lazy seq provided by `polling-receive`:
|
||||
|
||||
----
|
||||
=> (defn send-dummy-messages
|
||||
[client q count]
|
||||
(future (doseq [n (range count)]
|
||||
(Thread/sleep 100)
|
||||
(sqs/send client q (str n)))))
|
||||
#'cemerick.bandalore-test/send-dummy-messages
|
||||
=> (defn consume-dummy-messages
|
||||
[client q]
|
||||
(future (dorun (map (sqs/deleting-consumer client (comp println :body))
|
||||
(sqs/polling-receive client q :max-wait Integer/MAX_VALUE :limit 10)))))
|
||||
#'cemerick.bandalore-test/consume-dummy-messages
|
||||
=> (consume-dummy-messages client q) ;; start the consumer
|
||||
#<core$future_call$reify__5500@a6f00bc: :pending>
|
||||
=> (send-dummy-messages client q 1000) ;; start the sender
|
||||
#<core$future_call$reify__5500@18986032: :pending>
|
||||
3
|
||||
4
|
||||
1
|
||||
0
|
||||
2
|
||||
8
|
||||
5
|
||||
7
|
||||
...
|
||||
----
|
||||
|
||||
You'd presumably want to set up some ways to control your consumer, but hopefully
|
||||
you see that it would be trivial to parallelize the processing function being
|
||||
wrapped by `deleting-consumer` using `pmap`, distribute processing among agents
|
||||
if that's more appropriate, etc.
|
||||
|
||||
== Building Bandalore
|
||||
|
||||
Have maven. From the command line:
|
||||
|
||||
----
|
||||
$ mvn clean install
|
||||
----
|
||||
|
||||
The tests are all live, so you either need to add your AWS credentials to your
|
||||
`~/.m2/settings.xml` file as properties, or specify them on the command line
|
||||
using `-D` switches:
|
||||
|
||||
----
|
||||
$ mvn -Daws.id=XXXXXXX -Daws.secret-key=YYYYYYY clean install
|
||||
----
|
||||
|
||||
Or, you can skip the tests entirely if you're brave:
|
||||
|
||||
----
|
||||
$ mvn -Dmaven.test.skip=true clean install
|
||||
----
|
||||
|
||||
In any case, you'll find a built `.jar` file in the `target` directory, and in
|
||||
its designated spot in `~/.m2/repository` (assuming you ran `install` rather than
|
||||
e.g. `package`).
|
||||
|
||||
== Need Help?
|
||||
|
||||
Ping `cemerick` on freenode irc or twitter if you have questions
|
||||
or would like to contribute patches.
|
||||
|
||||
== License
|
||||
|
||||
Copyright © 2011 Chas Emerick
|
||||
|
||||
Licensed under the EPL. (See the file epl-v10.html.)
|
||||
261
epl-v10.html
Normal file
261
epl-v10.html
Normal file
|
|
@ -0,0 +1,261 @@
|
|||
<?xml version="1.0" encoding="ISO-8859-1" ?>
|
||||
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
||||
<html xmlns="http://www.w3.org/1999/xhtml">
|
||||
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" />
|
||||
<title>Eclipse Public License - Version 1.0</title>
|
||||
<style type="text/css">
|
||||
body {
|
||||
size: 8.5in 11.0in;
|
||||
margin: 0.25in 0.5in 0.25in 0.5in;
|
||||
tab-interval: 0.5in;
|
||||
}
|
||||
p {
|
||||
margin-left: auto;
|
||||
margin-top: 0.5em;
|
||||
margin-bottom: 0.5em;
|
||||
}
|
||||
p.list {
|
||||
margin-left: 0.5in;
|
||||
margin-top: 0.05em;
|
||||
margin-bottom: 0.05em;
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
|
||||
<body lang="EN-US">
|
||||
|
||||
<h2>Eclipse Public License - v 1.0</h2>
|
||||
|
||||
<p>THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE
|
||||
PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR
|
||||
DISTRIBUTION OF THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS
|
||||
AGREEMENT.</p>
|
||||
|
||||
<p><b>1. DEFINITIONS</b></p>
|
||||
|
||||
<p>"Contribution" means:</p>
|
||||
|
||||
<p class="list">a) in the case of the initial Contributor, the initial
|
||||
code and documentation distributed under this Agreement, and</p>
|
||||
<p class="list">b) in the case of each subsequent Contributor:</p>
|
||||
<p class="list">i) changes to the Program, and</p>
|
||||
<p class="list">ii) additions to the Program;</p>
|
||||
<p class="list">where such changes and/or additions to the Program
|
||||
originate from and are distributed by that particular Contributor. A
|
||||
Contribution 'originates' from a Contributor if it was added to the
|
||||
Program by such Contributor itself or anyone acting on such
|
||||
Contributor's behalf. Contributions do not include additions to the
|
||||
Program which: (i) are separate modules of software distributed in
|
||||
conjunction with the Program under their own license agreement, and (ii)
|
||||
are not derivative works of the Program.</p>
|
||||
|
||||
<p>"Contributor" means any person or entity that distributes
|
||||
the Program.</p>
|
||||
|
||||
<p>"Licensed Patents" mean patent claims licensable by a
|
||||
Contributor which are necessarily infringed by the use or sale of its
|
||||
Contribution alone or when combined with the Program.</p>
|
||||
|
||||
<p>"Program" means the Contributions distributed in accordance
|
||||
with this Agreement.</p>
|
||||
|
||||
<p>"Recipient" means anyone who receives the Program under
|
||||
this Agreement, including all Contributors.</p>
|
||||
|
||||
<p><b>2. GRANT OF RIGHTS</b></p>
|
||||
|
||||
<p class="list">a) Subject to the terms of this Agreement, each
|
||||
Contributor hereby grants Recipient a non-exclusive, worldwide,
|
||||
royalty-free copyright license to reproduce, prepare derivative works
|
||||
of, publicly display, publicly perform, distribute and sublicense the
|
||||
Contribution of such Contributor, if any, and such derivative works, in
|
||||
source code and object code form.</p>
|
||||
|
||||
<p class="list">b) Subject to the terms of this Agreement, each
|
||||
Contributor hereby grants Recipient a non-exclusive, worldwide,
|
||||
royalty-free patent license under Licensed Patents to make, use, sell,
|
||||
offer to sell, import and otherwise transfer the Contribution of such
|
||||
Contributor, if any, in source code and object code form. This patent
|
||||
license shall apply to the combination of the Contribution and the
|
||||
Program if, at the time the Contribution is added by the Contributor,
|
||||
such addition of the Contribution causes such combination to be covered
|
||||
by the Licensed Patents. The patent license shall not apply to any other
|
||||
combinations which include the Contribution. No hardware per se is
|
||||
licensed hereunder.</p>
|
||||
|
||||
<p class="list">c) Recipient understands that although each Contributor
|
||||
grants the licenses to its Contributions set forth herein, no assurances
|
||||
are provided by any Contributor that the Program does not infringe the
|
||||
patent or other intellectual property rights of any other entity. Each
|
||||
Contributor disclaims any liability to Recipient for claims brought by
|
||||
any other entity based on infringement of intellectual property rights
|
||||
or otherwise. As a condition to exercising the rights and licenses
|
||||
granted hereunder, each Recipient hereby assumes sole responsibility to
|
||||
secure any other intellectual property rights needed, if any. For
|
||||
example, if a third party patent license is required to allow Recipient
|
||||
to distribute the Program, it is Recipient's responsibility to acquire
|
||||
that license before distributing the Program.</p>
|
||||
|
||||
<p class="list">d) Each Contributor represents that to its knowledge it
|
||||
has sufficient copyright rights in its Contribution, if any, to grant
|
||||
the copyright license set forth in this Agreement.</p>
|
||||
|
||||
<p><b>3. REQUIREMENTS</b></p>
|
||||
|
||||
<p>A Contributor may choose to distribute the Program in object code
|
||||
form under its own license agreement, provided that:</p>
|
||||
|
||||
<p class="list">a) it complies with the terms and conditions of this
|
||||
Agreement; and</p>
|
||||
|
||||
<p class="list">b) its license agreement:</p>
|
||||
|
||||
<p class="list">i) effectively disclaims on behalf of all Contributors
|
||||
all warranties and conditions, express and implied, including warranties
|
||||
or conditions of title and non-infringement, and implied warranties or
|
||||
conditions of merchantability and fitness for a particular purpose;</p>
|
||||
|
||||
<p class="list">ii) effectively excludes on behalf of all Contributors
|
||||
all liability for damages, including direct, indirect, special,
|
||||
incidental and consequential damages, such as lost profits;</p>
|
||||
|
||||
<p class="list">iii) states that any provisions which differ from this
|
||||
Agreement are offered by that Contributor alone and not by any other
|
||||
party; and</p>
|
||||
|
||||
<p class="list">iv) states that source code for the Program is available
|
||||
from such Contributor, and informs licensees how to obtain it in a
|
||||
reasonable manner on or through a medium customarily used for software
|
||||
exchange.</p>
|
||||
|
||||
<p>When the Program is made available in source code form:</p>
|
||||
|
||||
<p class="list">a) it must be made available under this Agreement; and</p>
|
||||
|
||||
<p class="list">b) a copy of this Agreement must be included with each
|
||||
copy of the Program.</p>
|
||||
|
||||
<p>Contributors may not remove or alter any copyright notices contained
|
||||
within the Program.</p>
|
||||
|
||||
<p>Each Contributor must identify itself as the originator of its
|
||||
Contribution, if any, in a manner that reasonably allows subsequent
|
||||
Recipients to identify the originator of the Contribution.</p>
|
||||
|
||||
<p><b>4. COMMERCIAL DISTRIBUTION</b></p>
|
||||
|
||||
<p>Commercial distributors of software may accept certain
|
||||
responsibilities with respect to end users, business partners and the
|
||||
like. While this license is intended to facilitate the commercial use of
|
||||
the Program, the Contributor who includes the Program in a commercial
|
||||
product offering should do so in a manner which does not create
|
||||
potential liability for other Contributors. Therefore, if a Contributor
|
||||
includes the Program in a commercial product offering, such Contributor
|
||||
("Commercial Contributor") hereby agrees to defend and
|
||||
indemnify every other Contributor ("Indemnified Contributor")
|
||||
against any losses, damages and costs (collectively "Losses")
|
||||
arising from claims, lawsuits and other legal actions brought by a third
|
||||
party against the Indemnified Contributor to the extent caused by the
|
||||
acts or omissions of such Commercial Contributor in connection with its
|
||||
distribution of the Program in a commercial product offering. The
|
||||
obligations in this section do not apply to any claims or Losses
|
||||
relating to any actual or alleged intellectual property infringement. In
|
||||
order to qualify, an Indemnified Contributor must: a) promptly notify
|
||||
the Commercial Contributor in writing of such claim, and b) allow the
|
||||
Commercial Contributor to control, and cooperate with the Commercial
|
||||
Contributor in, the defense and any related settlement negotiations. The
|
||||
Indemnified Contributor may participate in any such claim at its own
|
||||
expense.</p>
|
||||
|
||||
<p>For example, a Contributor might include the Program in a commercial
|
||||
product offering, Product X. That Contributor is then a Commercial
|
||||
Contributor. If that Commercial Contributor then makes performance
|
||||
claims, or offers warranties related to Product X, those performance
|
||||
claims and warranties are such Commercial Contributor's responsibility
|
||||
alone. Under this section, the Commercial Contributor would have to
|
||||
defend claims against the other Contributors related to those
|
||||
performance claims and warranties, and if a court requires any other
|
||||
Contributor to pay any damages as a result, the Commercial Contributor
|
||||
must pay those damages.</p>
|
||||
|
||||
<p><b>5. NO WARRANTY</b></p>
|
||||
|
||||
<p>EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS
|
||||
PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
|
||||
OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION,
|
||||
ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY
|
||||
OR FITNESS FOR A PARTICULAR PURPOSE. Each Recipient is solely
|
||||
responsible for determining the appropriateness of using and
|
||||
distributing the Program and assumes all risks associated with its
|
||||
exercise of rights under this Agreement , including but not limited to
|
||||
the risks and costs of program errors, compliance with applicable laws,
|
||||
damage to or loss of data, programs or equipment, and unavailability or
|
||||
interruption of operations.</p>
|
||||
|
||||
<p><b>6. DISCLAIMER OF LIABILITY</b></p>
|
||||
|
||||
<p>EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT
|
||||
NOR ANY CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT,
|
||||
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING
|
||||
WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF
|
||||
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
|
||||
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OR
|
||||
DISTRIBUTION OF THE PROGRAM OR THE EXERCISE OF ANY RIGHTS GRANTED
|
||||
HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.</p>
|
||||
|
||||
<p><b>7. GENERAL</b></p>
|
||||
|
||||
<p>If any provision of this Agreement is invalid or unenforceable under
|
||||
applicable law, it shall not affect the validity or enforceability of
|
||||
the remainder of the terms of this Agreement, and without further action
|
||||
by the parties hereto, such provision shall be reformed to the minimum
|
||||
extent necessary to make such provision valid and enforceable.</p>
|
||||
|
||||
<p>If Recipient institutes patent litigation against any entity
|
||||
(including a cross-claim or counterclaim in a lawsuit) alleging that the
|
||||
Program itself (excluding combinations of the Program with other
|
||||
software or hardware) infringes such Recipient's patent(s), then such
|
||||
Recipient's rights granted under Section 2(b) shall terminate as of the
|
||||
date such litigation is filed.</p>
|
||||
|
||||
<p>All Recipient's rights under this Agreement shall terminate if it
|
||||
fails to comply with any of the material terms or conditions of this
|
||||
Agreement and does not cure such failure in a reasonable period of time
|
||||
after becoming aware of such noncompliance. If all Recipient's rights
|
||||
under this Agreement terminate, Recipient agrees to cease use and
|
||||
distribution of the Program as soon as reasonably practicable. However,
|
||||
Recipient's obligations under this Agreement and any licenses granted by
|
||||
Recipient relating to the Program shall continue and survive.</p>
|
||||
|
||||
<p>Everyone is permitted to copy and distribute copies of this
|
||||
Agreement, but in order to avoid inconsistency the Agreement is
|
||||
copyrighted and may only be modified in the following manner. The
|
||||
Agreement Steward reserves the right to publish new versions (including
|
||||
revisions) of this Agreement from time to time. No one other than the
|
||||
Agreement Steward has the right to modify this Agreement. The Eclipse
|
||||
Foundation is the initial Agreement Steward. The Eclipse Foundation may
|
||||
assign the responsibility to serve as the Agreement Steward to a
|
||||
suitable separate entity. Each new version of the Agreement will be
|
||||
given a distinguishing version number. The Program (including
|
||||
Contributions) may always be distributed subject to the version of the
|
||||
Agreement under which it was received. In addition, after a new version
|
||||
of the Agreement is published, Contributor may elect to distribute the
|
||||
Program (including its Contributions) under the new version. Except as
|
||||
expressly stated in Sections 2(a) and 2(b) above, Recipient receives no
|
||||
rights or licenses to the intellectual property of any Contributor under
|
||||
this Agreement, whether expressly, by implication, estoppel or
|
||||
otherwise. All rights in the Program not expressly granted under this
|
||||
Agreement are reserved.</p>
|
||||
|
||||
<p>This Agreement is governed by the laws of the State of New York and
|
||||
the intellectual property laws of the United States of America. No party
|
||||
to this Agreement will bring a legal action under this Agreement more
|
||||
than one year after the cause of action arose. Each party waives its
|
||||
rights to a jury trial in any resulting litigation.</p>
|
||||
|
||||
</body>
|
||||
|
||||
</html>
|
||||
69
pom.xml
Normal file
69
pom.xml
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<groupId>cemerick</groupId>
|
||||
<artifactId>bandalore</artifactId>
|
||||
<version>0.0.1-SNAPSHOT</version>
|
||||
<name>bandalore</name>
|
||||
<description>A Clojure library for Amazon's Simple Queue Service (SQS).</description>
|
||||
|
||||
<parent>
|
||||
<groupId>org.clojure</groupId>
|
||||
<artifactId>pom.contrib</artifactId>
|
||||
<version>0.0.20</version>
|
||||
</parent>
|
||||
|
||||
<developers>
|
||||
<developer>
|
||||
<name>Chas Emerick</name>
|
||||
<url>http://cemerick.com</url>
|
||||
<email>cemerick@snowtide.com</email>
|
||||
<timezone>-5</timezone>
|
||||
</developer>
|
||||
</developers>
|
||||
|
||||
<scm>
|
||||
<connection>scm:git:git@github.com:cemerick/bandalore.git</connection>
|
||||
<developerConnection>scm:git:git@github.com:cemerick/bandalore.git</developerConnection>
|
||||
<url>git@github.com:cemerick/bandalore.git</url>
|
||||
</scm>
|
||||
|
||||
<properties>
|
||||
<clojure.version>1.2.0</clojure.version>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.amazonaws</groupId>
|
||||
<artifactId>aws-java-sdk</artifactId>
|
||||
<version>1.1.5</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.clojure</groupId>
|
||||
<artifactId>clojure-contrib</artifactId>
|
||||
<version>1.2.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>com.theoryinpractise</groupId>
|
||||
<artifactId>clojure-maven-plugin</artifactId>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>clojure-test</id>
|
||||
<configuration>
|
||||
<clojureOptions>-Daws.id=${aws.id} -Daws.secret-key=${aws.secret-key}</clojureOptions>
|
||||
</configuration>
|
||||
</execution>
|
||||
</executions>
|
||||
<configuration>
|
||||
<warnOnReflection>true</warnOnReflection>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
</project>
|
||||
198
src/main/clojure/cemerick/bandalore.clj
Normal file
198
src/main/clojure/cemerick/bandalore.clj
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
; Copyright (c) Chas Emerick. All rights reserved.
|
||||
; The use and distribution terms for this software are covered by the
|
||||
; Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php)
|
||||
; which can be found in the file epl-v10.html at the root of this distribution.
|
||||
; By using this software in any fashion, you are agreeing to be bound by
|
||||
; the terms of this license.
|
||||
; You must not remove this notice, or any other, from this software.
|
||||
|
||||
(ns cemerick.bandalore
|
||||
(:import com.amazonaws.services.sqs.AmazonSQSClient
|
||||
(com.amazonaws.services.sqs.model
|
||||
AddPermissionRequest ChangeMessageVisibilityRequest CreateQueueRequest
|
||||
DeleteMessageRequest DeleteQueueRequest GetQueueAttributesRequest
|
||||
ListQueuesRequest Message ReceiveMessageRequest ReceiveMessageResult
|
||||
RemovePermissionRequest SendMessageRequest SendMessageResult
|
||||
SetQueueAttributesRequest))
|
||||
(:refer-clojure :exclude (send)))
|
||||
|
||||
(defn create-client
|
||||
"Creates a synchronous AmazonSQSClient using the provided account id, secret key,
|
||||
and optional com.amazonaws.ClientConfiguration."
|
||||
([id secret-key]
|
||||
(create-client id secret-key (com.amazonaws.ClientConfiguration.)))
|
||||
([id secret-key client-config]
|
||||
(AmazonSQSClient. (com.amazonaws.auth.BasicAWSCredentials. id secret-key)
|
||||
(.withUserAgent client-config "Bandalore - SQS for Clojure"))))
|
||||
|
||||
(defn create-queue
|
||||
"Creates a queue with the given name, returning the corresponding URL string.
|
||||
Returns successfully if the queue already exists.
|
||||
|
||||
Specify an optional :visibility keyword arg to set the new queue's default
|
||||
visibility timeout in seconds."
|
||||
[^AmazonSQSClient client queue-name & {:keys [visibility]}]
|
||||
(->> (if visibility
|
||||
(CreateQueueRequest. queue-name visibility)
|
||||
(CreateQueueRequest. queue-name))
|
||||
(.createQueue client)
|
||||
.getQueueUrl))
|
||||
|
||||
(defn delete-queue
|
||||
"Deletes the queue specified by the given URL string."
|
||||
[^AmazonSQSClient client queue-url]
|
||||
(.deleteQueue client (DeleteQueueRequest. queue-url)))
|
||||
|
||||
(defn list-queues
|
||||
"Returns a seq of all queues' URL strings."
|
||||
[^AmazonSQSClient client]
|
||||
(->> (ListQueuesRequest.)
|
||||
(.listQueues client)
|
||||
.getQueueUrls
|
||||
seq))
|
||||
|
||||
(defn queue-attrs
|
||||
"Gets or sets the attributes of a queue specified by its URL string.
|
||||
When setting attributes on a queue, the attribute map must have String keys
|
||||
and values."
|
||||
([^AmazonSQSClient client queue-url]
|
||||
(->>
|
||||
(.withAttributeNames (GetQueueAttributesRequest. queue-url) #{"All"})
|
||||
(.getQueueAttributes client)
|
||||
.getAttributes
|
||||
(into {})))
|
||||
([^AmazonSQSClient client queue-url attr-map]
|
||||
(.setQueueAttributes client
|
||||
(SetQueueAttributesRequest. queue-url attr-map))))
|
||||
|
||||
(defn send
|
||||
"Sends a new message with the given string body to the queue specified
|
||||
by the string URL. Returns a map with :id and :body-md5 slots."
|
||||
[^AmazonSQSClient client queue-url message]
|
||||
(let [resp (.sendMessage client (SendMessageRequest. queue-url message))]
|
||||
{:id (.getMessageId resp)
|
||||
:body-md5 (.getMD5OfMessageBody resp)}))
|
||||
|
||||
(defn- message-map
|
||||
[queue-url ^Message msg]
|
||||
{:attributes (.getAttributes msg)
|
||||
:body (.getBody msg)
|
||||
:body-md5 (.getMD5OfBody msg)
|
||||
:id (.getMessageId msg)
|
||||
:receipt-handle (.getReceiptHandle msg)
|
||||
:source-queue queue-url})
|
||||
|
||||
(defn receive
|
||||
"Receives one or more messages from the queue specified by the given URL.
|
||||
Optionally accepts keyword arguments:
|
||||
|
||||
:limit - between 1 (default) and 10, the maximum number of messages to receive
|
||||
:visibility - seconds the received messages should not be delivered to other
|
||||
receivers; defaults to the queue's visibility attribute
|
||||
:attributes - a collection of string names of :attributes to include in
|
||||
received messages; e.g. #{\"All\"} will include all attributes,
|
||||
#{\"SentTimestamp\"} will include only the SentTimestamp attribute, etc.
|
||||
Defaults to the empty set (i.e. no attributes will be included in
|
||||
received messages).
|
||||
See the SQS documentation for all support message attributes.
|
||||
|
||||
Returns a seq of maps with these slots:
|
||||
|
||||
:attributes - message attributes
|
||||
:body - the string body of the message
|
||||
:body-md5 - the MD5 checksum of :body
|
||||
:id - the message's ID
|
||||
:receipt-handle - the ID used to delete the message from the queue after
|
||||
it has been fully processed.
|
||||
:source-queue - the URL of the queue from which the message was received"
|
||||
[^AmazonSQSClient client queue-url & {:keys [limit visibility ^java.util.Collection attributes]
|
||||
:or {limit 1
|
||||
attributes #{"All"}}}]
|
||||
(let [req (-> (ReceiveMessageRequest. queue-url)
|
||||
(.withMaxNumberOfMessages (-> limit (min 10) (max 1)))
|
||||
(.withAttributeNames attributes))
|
||||
req (if visibility (.withVisibilityTimeout req visibility) req)]
|
||||
(->> (.receiveMessage client req)
|
||||
.getMessages
|
||||
(map (partial message-map queue-url)))))
|
||||
|
||||
(defn polling-receive
|
||||
"Receives one or more messages from the queue specified by the given URL.
|
||||
Returns a lazy seq of messages from multiple receive calls, performed
|
||||
as the seq is consumed.
|
||||
|
||||
Accepts the same keyword arguments as `receive`, in addition to:
|
||||
|
||||
:period - time in ms to wait after an unsuccessful `receive` request (default: 500)
|
||||
:max-wait - maximum time in ms to wait to successfully receive messages before
|
||||
terminating the lazy seq (default 5000)"
|
||||
[client queue-url & {:keys [period max-wait]
|
||||
:or {period 500
|
||||
max-wait 5000}
|
||||
:as receive-opts}]
|
||||
(let [waiting (atom 0)
|
||||
receive-opts (mapcat identity receive-opts)
|
||||
message-seq (fn message-seq []
|
||||
(lazy-seq
|
||||
(if-let [msgs (seq (apply receive client queue-url receive-opts))]
|
||||
(do
|
||||
(reset! waiting 0)
|
||||
(concat msgs (message-seq)))
|
||||
(do
|
||||
(when (<= (swap! waiting + period) max-wait)
|
||||
(Thread/sleep period)
|
||||
(message-seq))))))]
|
||||
(message-seq)))
|
||||
|
||||
(defn- receipt-handle
|
||||
[msg-ish]
|
||||
(cond
|
||||
(string? msg-ish) msg-ish
|
||||
(map? msg-ish) (:receipt-handle msg-ish)
|
||||
:else (.getReceiptHandle ^Message msg-ish)))
|
||||
|
||||
(defn delete
|
||||
"Deletes a message from the queue from which it was received.
|
||||
|
||||
The message must be a message map provided by `receive` or `polling-receive`
|
||||
if the `queue-url` is not provided explicitly. Otherwise,
|
||||
`message` may be a message map, Message object,
|
||||
or the :receipt-handle value from a received message map."
|
||||
([^AmazonSQSClient client message]
|
||||
(delete client (:source-queue message) message))
|
||||
([^AmazonSQSClient client queue-url message]
|
||||
(->> message
|
||||
receipt-handle
|
||||
(DeleteMessageRequest. queue-url)
|
||||
(.deleteMessage client))))
|
||||
|
||||
(defn deleting-consumer
|
||||
"Some minor middleware for simplifying the deletion of processed messages
|
||||
from their originating queue. Given a client and a function `f`, returns
|
||||
a function that accepts a single message (as provided by `receive` and `polling-receive`);
|
||||
it calls `f` with that message, and then deletes the message from its
|
||||
originating queue. `f`'s return value is passed along."
|
||||
[client f]
|
||||
(fn [message]
|
||||
(let [ret (f message)]
|
||||
(delete client message)
|
||||
ret)))
|
||||
|
||||
(defn change-message-visibility
|
||||
"Updates the visibility timeout for a single message in the queue identified by the
|
||||
given URL.
|
||||
|
||||
`message` may be a message map, Message object,
|
||||
or the :receipt-handle value from a received message map."
|
||||
[^AmazonSQSClient client queue-url message visibility-timeout]
|
||||
(.changeMessageVisibility client
|
||||
(ChangeMessageVisibilityRequest.
|
||||
queue-url (receipt-handle message) visibility-timeout)))
|
||||
|
||||
|
||||
;; ResponseMetadata getCachedResponseMetadata(AmazonWebServiceRequest request)
|
||||
;; Returns additional metadata for a previously executed successful, request, typically used for debugging issues where a service isn't acting as expected.
|
||||
;; void removePermission(RemovePermissionRequest removePermissionRequest)
|
||||
; The RemovePermission action revokes any permissions in the queue policy that matches the specified Label parameter.
|
||||
;; void addPermission(AddPermissionRequest addPermissionRequest)
|
||||
;;The AddPermission action adds a permission to a queue for a specific principal.
|
||||
41
src/test/clojure/cemerick/bandalore_test.clj
Normal file
41
src/test/clojure/cemerick/bandalore_test.clj
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
(ns cemerick.bandalore-test
|
||||
(:use cemerick.bandalore
|
||||
clojure.test
|
||||
clojure.contrib.core)
|
||||
(:refer-clojure :exclude (send)))
|
||||
|
||||
; kill the verbose aws logging
|
||||
(.setLevel (java.util.logging.Logger/getLogger "com.amazonaws")
|
||||
java.util.logging.Level/WARNING)
|
||||
|
||||
(def client
|
||||
(let [id (System/getProperty "aws.id")
|
||||
secret-key (System/getProperty "aws.secret-key")]
|
||||
(assert (and id secret-key))
|
||||
(create-client id secret-key)))
|
||||
|
||||
(def *test-queue-url* nil)
|
||||
|
||||
(defn- uuid
|
||||
[]
|
||||
(str (java.util.UUID/randomUUID)))
|
||||
|
||||
(defn- test-queue-name
|
||||
[]
|
||||
(str "bandalore-test-" (uuid)))
|
||||
|
||||
(defmacro defqueuetest
|
||||
[name & body]
|
||||
`(deftest ~name
|
||||
(binding [*test-queue-url* (create-queue client (test-queue-name))]
|
||||
(try
|
||||
(is *test-queue-url*)
|
||||
~@body
|
||||
(finally
|
||||
(delete-queue client *test-queue-url*))))))
|
||||
|
||||
(defqueuetest listing-queues
|
||||
(let [msg (uuid)]
|
||||
(send client *test-queue-url* msg)
|
||||
(is ((set (list-queues client)) *test-queue-url*))))
|
||||
|
||||
Loading…
Reference in a new issue