luciano/bandalore
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

readme asciidoc -> markdown

Browse source
This commit is contained in:
Chas Emerick 2012-02-24 10:36:21 -05:00
parent 6273806597
commit 112d6121d5
1 changed files with 81 additions and 79 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

160
README.asciidoc → README.md
View file

@ -1,44 +1,46 @@
= Bandalore # Bandalore
http://github.com/cemerick/bandalore[Bandalore] is a Clojure client [Bandalore](http://github.com/cemerick/bandalore) is a Clojure client
library for Amazon's http://aws.amazon.com/sqs/[Simple Queue Service (SQS)]. It depends upon library for Amazon's [Simple Queue Service](http://aws.amazon.com/sqs/). It depends upon
the standard http://aws.amazon.com/sdkforjava/[AWS SDK for Java], the standard [AWS SDK for Java](http://aws.amazon.com/sdkforjava/),
and provides a Clojure-idiomatic API for the SQS-related functionality and provides a Clojure-idiomatic API for the SQS-related functionality
therein. therein.
== "Installation" ## "Installation"
Bandalore is available in Maven central. Add it to your Maven project's `pom.xml`: Bandalore is available in Maven central. Add it to your Maven project's `pom.xml`:
---- ```xml
<dependency> <dependency>
<groupId>com.cemerick</groupId> <groupId>com.cemerick</groupId>
<artifactId>bandalore</artifactId> <artifactId>bandalore</artifactId>
<version>0.0.1</version> <version>0.0.2</version>
</dependency> </dependency>
---- ```
or your leiningen project.clj: or your leiningen project.clj:
---- ```clojure
[com.cemerick/bandalore "0.0.1"] [com.cemerick/bandalore "0.0.2"]
---- ```
== Logging Bandalore is compatible with Clojure 1.2.0 - 1.4.0.
## Logging
I strongly recommend squelching the AWS SDK's very verbose logging I strongly recommend squelching the AWS SDK's very verbose logging
before using Bandalore (the former spews a variety of stuff out on before using Bandalore (the former spews a variety of stuff out on
INFO that I personally think should be in DEBUG or TRACE). You can INFO that I personally think should be in DEBUG or TRACE). You can
do this with this snippet: do this with this snippet:
---- ```clojure
(.setLevel (java.util.logging.Logger/getLogger "com.amazonaws") (.setLevel (java.util.logging.Logger/getLogger "com.amazonaws")
java.util.logging.Level/WARNING) java.util.logging.Level/WARNING)
---- ```
Translate as necessary if you're using log4j, etc. Translate as necessary if you're using log4j, etc.
== Usage ## Usage
You should be familiar with http://aws.amazon.com/sqs/[SQS itself] You should be familiar with http://aws.amazon.com/sqs/[SQS itself]
before sensibly using this library. That said, Bandalore's API before sensibly using this library. That said, Bandalore's API
@ -47,23 +49,23 @@ is well-documented.
You'll first need to load the library and create a SQS client object You'll first need to load the library and create a SQS client object
to do anything: to do anything:
---- ```clojure
(require '[cemerick.bandalore :as sqs]) (require '[cemerick.bandalore :as sqs])
(def client (sqs/create-client "your aws id" "your aws secret-key")) (def client (sqs/create-client "your aws id" "your aws secret-key"))
---- ```
You can create, delete, and list queues: You can create, delete, and list queues:
---- ```clojure
=> (sqs/create-queue client "foo") #> (sqs/create-queue client "foo")
"https://queue.amazonaws.com/499312652346/foo" "https://queue.amazonaws.com/499312652346/foo"
=> (sqs/list-queues client) #> (sqs/list-queues client)
("https://queue.amazonaws.com/499312652346/foo") ("https://queue.amazonaws.com/499312652346/foo")
=> (sqs/delete-queue client (first *1)) #> (sqs/delete-queue client (first *1))
nil nil
=> (list-queues client) #> (list-queues client)
nil nil
---- ```
*Note that SQS is _eventually consistent_. This means that a created *Note that SQS is _eventually consistent_. This means that a created
queue won't necessarily show up in an immediate listing of queues, queue won't necessarily show up in an immediate listing of queues,
@ -71,95 +73,95 @@ messages aren't necessarily immediately available to be received, etc.*
You can send, receive, and delete messages: You can send, receive, and delete messages:
---- ```clojure
=> (def q (sqs/create-queue client "foo")) #> (def q (sqs/create-queue client "foo"))
#'cemerick.bandalore-test/q #'cemerick.bandalore-test/q
=> (sqs/send client q "my message body") #> (sqs/send client q "my message body")
{:id "75d5d7a1-2274-4163-97b2-aa4c75f209ee", :body-md5 "05d358de00fc63dd2fa2026b77e112f6"} {:id "75d5d7a1-2274-4163-97b2-aa4c75f209ee", :body-md5 "05d358de00fc63dd2fa2026b77e112f6"}
=> (sqs/receive client q) #> (sqs/receive client q)
({:attrs #<HashMap {}>, :body "my message body", :body-md5 "05d358de00fc63dd2fa2026b77e112f6", ({:attrs #<HashMap {}>, :body "my message body", :body-md5 "05d358de00fc63dd2fa2026b77e112f6",
:id "75d5d7a1-2274-4163-97b2-aa4c75f209ee", :id "75d5d7a1-2274-4163-97b2-aa4c75f209ee",
:receipt-handle "…very long string…"}) :receipt-handle "…very long string…"})
;; ;;
;; …presumably do something with the received message(s)… ;; …presumably do something with the received message(s)…
;; ;;
=> (sqs/delete client q (first *1)) #> (sqs/delete client q (first *1))
nil nil
=> (sqs/receive client q) #> (sqs/receive client q)
() ()
---- ```
That's cleaner than having to interop directly with the Java SDK, but it's all That's cleaner than having to interop directly with the Java SDK, but it's all
pretty pedestrian stuff. You can do more interesting things with some pretty pedestrian stuff. You can do more interesting things with some
simple higher-order functions and other nifty Clojure facilities. simple higher-order functions and other nifty Clojure facilities.
=== Sending and receiving Clojure values ### Sending and receiving Clojure values
SQS' message bodies are strings, so you can stuff anything in them that you can SQS' message bodies are strings, so you can stuff anything in them that you can
serialize to a string. That said, `pr-str` and `read-string` are too handy serialize to a string. That said, `pr-str` and `read-string` are too handy
to not use, assuming your consumers are using Clojure as well: to not use, assuming your consumers are using Clojure as well:
---- ```clojure
=> (sqs/send client q (pr-str {:a 5 :b "blah" :c 6.022e23})) #> (sqs/send client q (pr-str {:a 5 :b "blah" :c 6.022e23}))
{:id "3756c302-866a-4fcc-a7a3-746e6f531f47", :body-md5 "60052fc2ffb835257c26b9957c6e9ffd"} {:id "3756c302-866a-4fcc-a7a3-746e6f531f47", :body-md5 "60052fc2ffb835257c26b9957c6e9ffd"}
=> (-?> (sqs/receive client q) first :body read-string) #> (-?> (sqs/receive client q) first :body read-string)
{:a 5, :b "blah", :c 6.022E23} {:a 5, :b "blah", :c 6.022E23}
---- ```
=== Sending seqs of messages ### Sending seqs of messages
…with more gratuitous use of `pr-str` and `read-string` to send and receive …with more gratuitous use of `pr-str` and `read-string` to send and receive
Clojure values: Clojure values:
---- ```clojure
=> (->> [:foo 'bar ["some vector" 42] #{#"silly place for a regex"}] #> (->> [:foo 'bar ["some vector" 42] #{#"silly place for a regex"}]
(map (comp (partial sqs/send client q) pr-str)) (map (comp (partial sqs/send client q) pr-str))
dorun) dorun)
nil nil
=> (map (comp read-string :body) #> (map (comp read-string :body)
(sqs/receive client q :limit 10)) (sqs/receive client q :limit 10))
(bar ["some vector" 42]) (bar ["some vector" 42])
=> (map (comp read-string :body) #> (map (comp read-string :body)
(sqs/receive client q :limit 10)) (sqs/receive client q :limit 10))
(#{#"silly place for a regex"}) (#{#"silly place for a regex"})
=> (map (comp read-string :body) #> (map (comp read-string :body)
(sqs/receive client q :limit 10)) (sqs/receive client q :limit 10))
(:foo) (:foo)
---- ```
=== (Mostly) automatic deletion of consumed messages ### (Mostly) automatic deletion of consumed messages
When you're done processing a received message, you need to delete it from its When you're done processing a received message, you need to delete it from its
originaing queue: originaing queue:
---- ```clojure
; ensure our queue is empty to start ; ensure our queue is empty to start
=> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages") #> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages")
"0" "0"
=> (dorun (map (partial sqs/send client q) (map str (range 100)))) #> (dorun (map (partial sqs/send client q) (map str (range 100))))
nil nil
=> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages") #> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages")
"100" "100"
; received messages must be removed from the queue or they will ; received messages must be removed from the queue or they will
; be delivered again after their visibility timeout expires ; be delivered again after their visibility timeout expires
=> (sqs/receive client q) #> (sqs/receive client q)
(…message seq…) (…message seq…)
=> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages") #> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages")
"100" "100"
=> (->> (sqs/receive client q) first (sqs/delete client)) #> (->> (sqs/receive client q) first (sqs/delete client))
nil nil
=> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages") #> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages")
"99" "99"
---- ```
Rather than trying to remember to do this, just use the Rather than trying to remember to do this, just use the
`deleting-consumer` "middleware" to produce a function that calls `deleting-consumer` "middleware" to produce a function that calls
the message-processing function you provide to it, and then the message-processing function you provide to it, and then
automatically deletes the processed message from the origining queue: automatically deletes the processed message from the origining queue:
---- ```clojure
=> (doall (map #> (doall (map
(sqs/deleting-consumer client (comp println :body)) (sqs/deleting-consumer client (comp println :body))
(sqs/receive client q :limit 10))) (sqs/receive client q :limit 10)))
0 0
@ -173,11 +175,11 @@ automatically deletes the processed message from the origining queue:
52 52
55 55
(nil nil nil nil nil nil nil nil nil nil) (nil nil nil nil nil nil nil nil nil nil)
=> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages") #> (get (sqs/queue-attrs client q) "ApproximateNumberOfMessages")
"90" "90"
---- ```
=== Consuming queues as seqs ### Consuming queues as seqs
seqs being the _lingua franca_ of Clojure collections, it would be helpful if we seqs being the _lingua franca_ of Clojure collections, it would be helpful if we
could treat an SQS queue as a seq of messages. While `receive` does return could treat an SQS queue as a seq of messages. While `receive` does return
@ -187,11 +189,11 @@ a seq of messages, each `receive` call is limited to receiving a maximum of
The solution to this is `polling-receive`, which returns a lazy seq that The solution to this is `polling-receive`, which returns a lazy seq that
reaches out to SQS as necessary: reaches out to SQS as necessary:
---- ```clojure
=> (map (sqs/deleting-consumer client :body) #> (map (sqs/deleting-consumer client :body)
(sqs/polling-receive client q :limit 10)) (sqs/polling-receive client q :limit 10))
("3" "5" "7" "8" ... "81" "90" "91") ("3" "5" "7" "8" ... "81" "90" "91")
---- ```
`polling-receive` accepts all of the same optional kwargs as `receive` does, `polling-receive` accepts all of the same optional kwargs as `receive` does,
but adds two more to control its usage of `receive`: but adds two more to control its usage of `receive`:
@ -210,21 +212,21 @@ terminate because none have been available for a while.
Here's an example where one thread sends a message once a second for a minute, Here's an example where one thread sends a message once a second for a minute,
and another consumes those messages using a lazy seq provided by `polling-receive`: and another consumes those messages using a lazy seq provided by `polling-receive`:
---- ```clojure
=> (defn send-dummy-messages #> (defn send-dummy-messages
[client q count] [client q count]
(future (doseq [n (range count)] (future (doseq [n (range count)]
(Thread/sleep 100) (Thread/sleep 100)
(sqs/send client q (str n))))) (sqs/send client q (str n)))))
#'cemerick.bandalore-test/send-dummy-messages #'cemerick.bandalore-test/send-dummy-messages
=> (defn consume-dummy-messages #> (defn consume-dummy-messages
[client q] [client q]
(future (dorun (map (sqs/deleting-consumer client (comp println :body)) (future (dorun (map (sqs/deleting-consumer client (comp println :body))
(sqs/polling-receive client q :max-wait Integer/MAX_VALUE :limit 10))))) (sqs/polling-receive client q :max-wait Integer/MAX_VALUE :limit 10)))))
#'cemerick.bandalore-test/consume-dummy-messages #'cemerick.bandalore-test/consume-dummy-messages
=> (consume-dummy-messages client q) ;; start the consumer #> (consume-dummy-messages client q) ;; start the consumer
#<core$future_call$reify__5500@a6f00bc: :pending> #<core$future_call$reify__5500@a6f00bc: :pending>
=> (send-dummy-messages client q 1000) ;; start the sender #> (send-dummy-messages client q 1000) ;; start the sender
#<core$future_call$reify__5500@18986032: :pending> #<core$future_call$reify__5500@18986032: :pending>
3 3
4 4
@ -235,20 +237,20 @@ and another consumes those messages using a lazy seq provided by `polling-receiv
5 5
7 7
... ...
---- ```
You'd presumably want to set up some ways to control your consumer, but hopefully You'd presumably want to set up some ways to control your consumer, but hopefully
you see that it would be trivial to parallelize the processing function being you see that it would be trivial to parallelize the processing function being
wrapped by `deleting-consumer` using `pmap`, distribute processing among agents wrapped by `deleting-consumer` using `pmap`, distribute processing among agents
if that's more appropriate, etc. if that's more appropriate, etc.
== Building Bandalore ## Building Bandalore
Have maven. From the command line: Have maven. From the command line:
---- ```
$ mvn clean install $ mvn clean verify
---- ```
*The tests are all live*, so: *The tests are all live*, so:
@ -261,27 +263,27 @@ Since the tests are live, you either need to add your AWS credentials to your
`~/.m2/settings.xml` file as properties, or specify them on the command line `~/.m2/settings.xml` file as properties, or specify them on the command line
using `-D` switches: using `-D` switches:
---- ```
$ mvn -Daws.id=XXXXXXX -Daws.secret-key=YYYYYYY clean install $ mvn -Daws.id#XXXXXXX -Daws.secret-key#YYYYYYY clean install
---- ```
Or, you can skip the tests entirely: Or, you can skip the tests entirely:
---- ```
$ mvn -Dmaven.test.skip=true clean install $ mvn -Dmaven.test.skip#true clean install
---- ```
In any case, you'll find a built `.jar` file in the `target` directory, and in In any case, you'll find a built `.jar` file in the `target` directory, and in
its designated spot in `~/.m2/repository` (assuming you ran `install` rather than its designated spot in `~/.m2/repository` (assuming you ran `install` rather than
e.g. `package`). e.g. `package`).
== Need Help? ## Need Help?
Ping `cemerick` on freenode irc or twitter if you have questions Ping `cemerick` on freenode irc or twitter if you have questions
or would like to contribute patches. or would like to contribute patches.
== License ## License
Copyright © 2011 Chas Emerick Copyright © 2011-2012 Chas Emerick
Licensed under the EPL. (See the file epl-v10.html.) Licensed under the EPL. (See the file epl-v10.html.)