diff --git a/README.md b/README.md index 0727277a..d173f810 100644 --- a/README.md +++ b/README.md @@ -787,9 +787,10 @@ Differences with Clojure: A list of projects (scripts, libraries and tools) known to work with babashka. -## [Pods](doc/pods.md) +## Pods -Pods are a way of extending babashka using other programs. +Pods are a way of extending babashka using other programs. Documentation is +available in the [library repo](https://github.com/babashka/babashka.pods). ## Package babashka script as a AWS Lambda diff --git a/babashka.pods b/babashka.pods index fb6e739a..98d4f814 160000 --- a/babashka.pods +++ b/babashka.pods @@ -1 +1 @@ -Subproject commit fb6e739aeed8ada6720318b788549f72d37640e8 +Subproject commit 98d4f814323c50c3ea5427896c0eba6c40224bdc diff --git a/doc/pods.md b/doc/pods.md deleted file mode 100644 index 78eff391..00000000 --- a/doc/pods.md +++ /dev/null @@ -1,297 +0,0 @@ -# Pods - -Pods are standalone programs that can expose namespaces with vars to babashka or -a JVM. Pods can be created independently from babashka. Any program can be -invoked as a pod as long as it implements the _pod protocol_. This protocol is -influenced by and built upon battle-tested technologies: - -- the [nREPL](https://nrepl.org/) and [LSP](https://microsoft.github.io/language-server-protocol/) protocols -- [bencode](https://en.wikipedia.org/wiki/Bencode) -- [JSON](https://www.json.org/json-en.html) -- [EDN](https://github.com/edn-format/edn) -- composition of UNIX command line tools in via good old stdin and stdout - -Pods are a brand new way to extend babashka and you should consider the protocol -alpha. Breaking changes may occur at this phase. Pods were introduced in -babashka version `0.0.92`. - -Currently the following pods are available: - -- [clj-kondo](https://github.com/borkdude/clj-kondo/#babashka-pod): a Clojure - linter -- [pod-babashka-filewatcher](https://github.com/babashka/pod-babashka-filewatcher): a - filewatcher pod based on Rust notify. -- [pod-babashka-hsqldb](https://github.com/babashka/pod-babashka-hsqldb): a pod - that allows you to create and fire queries at a - [HSQLDB](http://www.hsqldb.org/) database. -- [pod-jaydeesimon-jsoup](https://github.com/jaydeesimon/pod-jaydeesimon-jsoup): - a pod for parsing HTML using CSS queries backed by Jsoup. -- [pod-lispyclouds-docker](https://github.com/lispyclouds/pod-lispyclouds-docker): - A pod for interacting with docker - -The name pod is inspired by [boot's pod -feature](https://github.com/boot-clj/boot/wiki/Pods). It means _underneath_ or -_below_ in Polish and Russian. In Romanian it means _bridge_ -([source](https://en.wiktionary.org/wiki/pod)). - -## Implementing your own pod - -We will refer to babashka or the JVM, invoking the pod, as the pod client. - -### Examples - -Beyond the already available pods mentioned above, eductional examples of pods -can be found [here](../examples/pods): - -- [pod-lispyclouds-sqlite](../examples/pods/pod-lispyclouds-sqlite): a pod that - allows you to create and fire queries at a [sqlite](https://www.sqlite.org/) - database. Implemented in Python. - -### Libraries - -If you are looking for libraries to deal with bencode, JSON or EDN, take a look -at the existing pods or [nREPL](https://nrepl.org/nrepl/beyond_clojure.html) -implementations for various languages. - -### Naming - -When choosing a name for your pod, we suggest the following naming scheme: - -``` -pod-- -``` - -where `` is your Github or Gitlab handle and `` describes -what your pod is about. - -Examples: - -- [pod-lispyclouds-sqlite](../examples/pods/pod-lispyclouds-sqlite): a pod to - communicate with [sqlite](https://www.sqlite.org/), provided by - [@lispyclouds](https://github.com/lispyclouds). - -Pods created by the babashka maintainers use the identifier `babashka`: - -- [pod-babashka-hsqldb](https://github.com/borkdude/pod-babashka-hsqldb): a pod - to communicate with [HSQLDB](http://www.hsqldb.org/) - -### The protocol - -#### Message and payload format - -Exchange of _messages_ between pod client and the pod happens in the -[bencode](https://en.wikipedia.org/wiki/Bencode) format. Bencode is a bare-bones -format that only has four types: - -- integers -- lists -- dictionaries (maps) -- byte strings - -Additionally, _payloads_ like `args` (arguments) or `value` (a function return -value) are encoded in either JSON or EDN. - -So remember: messages are in bencode, payloads (particular fields in the -message) are in either JSON or EDN. - -Bencode is chosen as the message format because it is a light-weight format -which can be implemented in 200-300 lines of code in most languages. If pods are -implemented in Clojure, they only need to depend on the -[bencode](https://github.com/nrepl/bencode) library and use `pr-str` and -`edn/read-string` for encoding and decoding payloads. - -Why isn't EDN or JSON chosen as the message format instead of bencode, you may -ask. Assuming EDN or JSON as the message and payload format for all pods is too -constraining: other languages might already have built-in JSON support and there -might not be a good EDN library available. So we use bencode as the first -encoding and choose one of multiple richer encodings on top of this. More -payload formats might be added in the future (e.g. transit). - -When calling the `babashka.pods/load-pod` function, the pod client will start -the pod and leave the pod running throughout the duration of a babashka script. - -#### describe - -The first message that the pod client will send to the pod on its stdin is: - -``` clojure -{"op" "describe"} -``` - -Encoded in bencode this looks like: - -``` clojure -(bencode/write-bencode System/out {"op" "describe"}) -;;=> d2:op8:describee -``` - -The pod should reply to this request with a message in the vein of: - -``` clojure -{"format" "json" - "namespaces" - [{"name" "pod.lispyclouds.sqlite" - "vars" [{"name" "execute!"}]}] - "ops" {"shutdown" {}}} -``` - -In this reply, the pod declares that payloads will be encoded and decoded using -JSON. It also declares that the pod exposes one namespace, -`pod.lispyclouds.sqlite` with one var `execute!`. - -The pod encodes the above map to bencode and writes it to stdoud. The pod client -reads this message from the pod's stdout. - -Upon receiving this message, the pod client creates these namespaces and vars. - -The optional `ops` value communicates which ops the pod supports, beyond -`describe` and `invoke`. It is a map of op names to option maps. In the above -example the pod declares that it supports the `shutdown` op. Since the -`shutdown` op does not need any additional options right now, the value is an -empty map. - -As a pod user, you can load the pod with: - -``` clojure -(require '[babashka.pods :as pods]) -(pods/load-pod "pod-lispyclouds-sqlite") -(some? (find-ns 'pod.lispyclouds.sqlite)) ;;=> true -;; yay, the namespace exists! - -;; let's give the namespace an alias -(require '[pod.lispyclouds.sqlite :as sql]) -``` - -#### invoke - -When invoking a var that is related to the pod, let's call it a _proxy var_, the -pod client reaches out to the pod with the arguments encoded in JSON or EDN. The -pod will then respond with a return value encoded in JSON or EDN. The pod client -will then decode the return value and present the user with that. - -Example: the user invokes `(sql/execute! "select * from foo")`. The pod client -sends this message to the pod: - -``` clojure -{"id" "1d17f8fe-4f70-48bf-b6a9-dc004e52d056" - "var" "pod.lispyclouds.sqlite/execute!" - "args" "[\"select * from foo\"]" -``` - -The `id` is unique identifier generated by the pod client which correlates this -request with a response from the pod. - -An example response from the pod could look like: - -``` clojure -{"id" "1d17f8fe-4f70-48bf-b6a9-dc004e52d056" - "value" "[[1] [2]]" - "status" "[\"done\"]"} -``` - -Here, the `value` payload is the return value of the function invocation. The -field `status` contains `"done"`. This tells the pod client that this is the last -message related to the request with `id` `1d17f8fe-4f70-48bf-b6a9-dc004e52d056`. - -Now you know most there is to know about the pod protocol! - -#### shutdown - -When the pod client is about to exit, it sends an `{"op" "shutdown"}` message, if the -pod has declared that it supports it in the `describe` response. Then it waits -for the pod process to end. This gives the pod a chance to clean up resources -before it exits. If the pod does not support the `shutdown` op, the pod process -is killed by the pod client. - -#### out and err - -Pods may send messages with an `out` and `err` string value. The Pod Client prints -these messages to `*out*` and `*err*`. Stderr from the pod is redirected to -`System/err`. - -``` clojure -{"id" "1d17f8fe-4f70-48bf-b6a9-dc004e52d056" - "out" "hello"} -``` - -``` clojure -{"id" "1d17f8fe-4f70-48bf-b6a9-dc004e52d056" - "err" "debug"} -``` - -#### Error handling - -Responses may contain an `ex-message` string and `ex-data` payload string (JSON -or EDN) along with an `"error"` value in `status`. This will cause the pod client to -throw an `ex-info` with the associated values. - -Example: - -``` clojure -{"id" "1d17f8fe-4f70-48bf-b6a9-dc004e52d056" - "ex-message" "Illegal input" - "ex-data" "{\"input\": 10} - "status" "[\"done\", \"error\"]"} -``` - -#### async - -Pods may implement async functions that return one or more values at a later -time in the future. Async functions must be declared as such as part of the -`describe` response message: - -``` clojure -{"format" "json" - "namespaces" - [{"name" "pod.babashka.filewatcher" - "vars" [{"name" "watch" "async" "true"}]}]} -``` - -When calling this function from the pod client, the return value is a `core.async` -channel on which the values will be received: - -``` clojure -(pods/load-pod "target/release/pod-babashka-filewatcher") -(def chan (pod.babashka.filewatcher/watch "/tmp")) -(require '[clojure.core.async :as async]) -(loop [] (prn (async/ ["changed" "/tmp"] -;;=> ["changed" "/tmp"] -``` - -#### Environment - -The pod client will set the `BABASHKA_POD` environment variable to `true` when -invoking the pod. This can be used by the invoked program to determine whether -it should behave as a pod or not. - -Added in v0.0.94. - -#### Client side code - -Pods may implement functions and macros by sending arbitrary code to the pod -client in a `"code"` field as part of a `"var"` section. The code is evaluated -by the pod client inside the declared namespace. - -For example, a pod can define a macro called `do-twice`: - -``` clojure -{"format" "json" - "namespaces" - [{"name" "pod.babashka.demo" - "vars" [{"name" "do-twice" "code" "(defmacro do-twice [x] `(do ~x ~x))"}]}]} -``` - -In the pod client: - -``` clojure -(pods/load-pod "pod-babashka-demo") -(require '[pod.babashka.demo :as demo]) -(demo/do-twice (prn :foo)) -;;=> -:foo -:foo -nil -``` - -Added in v0.0.96. diff --git a/examples/pods/pod-lispyclouds-sqlite/README.md b/examples/pods/pod-lispyclouds-sqlite/README.md deleted file mode 100644 index 0de0c5b8..00000000 --- a/examples/pods/pod-lispyclouds-sqlite/README.md +++ /dev/null @@ -1,20 +0,0 @@ -# pod-lispyclouds-sqlite - -To run this: - -- Install python3 and sqlite3 -- Create a virtualenv: `python3 -m venv ~/.virtualenvs/babashka` -- Switch to it: `source ~/.virtualenvs/babashka/bin/activate` -- Run: `pip install bcoding` to install the bencode lib -- Create a new db: `sqlite3 /tmp/babashka.db "create table foo (foo int);"` - -Then run as pod: - -``` clojure -(babashka.pods/load-pod ["./pod-lispyclouds-sqlite.py"]) -(require '[pod.lispyclouds.sqlite :as sqlite]) -(sqlite/execute! "create table if not exists foo ( int foo )") -(sqlite/execute! "delete from foo") -(sqlite/execute! "insert into foo values (1), (2)") -(sqlite/execute! "select * from foo") ;;=> ([1] [2]) -``` diff --git a/examples/pods/pod-lispyclouds-sqlite/pod-lispyclouds-sqlite.py b/examples/pods/pod-lispyclouds-sqlite/pod-lispyclouds-sqlite.py deleted file mode 100755 index 0b0c65cc..00000000 --- a/examples/pods/pod-lispyclouds-sqlite/pod-lispyclouds-sqlite.py +++ /dev/null @@ -1,61 +0,0 @@ -#!/usr/bin/env python3 - -import json -import sqlite3 -import sys - -from bcoding import bencode, bdecode - - -def read(): - return dict(bdecode(sys.stdin.buffer)) - - -def write(obj): - sys.stdout.buffer.write(bencode(obj)) - sys.stdout.flush() - -def debug(*msg): - with open("/tmp/debug.log", "a") as f: - f.write(str(msg) + "\n") - -def main(): - while True: - msg = read() - debug("msg", msg) - - op = msg["op"] - - if op == "describe": - write( - { - "format": "json", - "namespaces": [{"name": "pod.lispyclouds.sqlite", - "vars": [{"name": "execute!"}]}]} - ) - elif op == "invoke": - var = msg["var"] - id = msg["id"] - args = json.loads(msg["args"]) - debug(args) - conn = sqlite3.connect("/tmp/babashka.db") - c = conn.cursor() - - result = None - - if var == "pod.lispyclouds.sqlite/execute!": - try: - result = c.execute(*args) - except Exception as e: - debug(e) - - value = json.dumps(result.fetchall()) - debug("value", value) - - write({"value": value, "id": id, "status": ["done"]}) - - conn.commit() - conn.close() - -if __name__ == "__main__": - main()