luciano/pods
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'master' into feature/local-manifests

Browse source
This commit is contained in:
Wes Morgan 2025-04-08 13:37:47 -06:00 committed by GitHub
commit ef2e741412
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
13 changed files with 262 additions and 107 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.gitignore vendored
View file

@ -3,3 +3,7 @@
.lein-failures
/pom.xml
.lein-repl-history
.cache
.clj-kondo/babashka
.clj-kondo/rewrite-clj
src/scratch.clj

18
CHANGELOG.md
View file

@ -2,6 +2,20 @@
## Unreleased
- #49: don't log socket closed exception
- [#63](https://github.com/babashka/pods/issues/63): create directory before un-tarring
- [#59](https://github.com/babashka/pods/issues/59): delete port file on exit
- [#65](https://github.com/babashka/pods/issues/65): fix warnings when defining var with core name in JVM
- [#66](https://github.com/babashka/pods/issues/66): Allow metadata on fn arguments for transit+json
## ...
## v0.2.0
- [#61](https://github.com/babashka/pods/issues/61): add transit as explicit JVM dependency
- [#60](https://github.com/babashka/pods/issues/60): transform pod reader error into exception of caller
- Switch "out" and "err" messages to print and flush instead of `println` ([@justone](https://github.com/justone))
- Set TCP_NODELAY on transport socket ([@retrogradeorbit](https://github.com/retrogradeorbit))
- Allow env vars OS_NAME & OS_ARCH to override os props ([@cap10morgan](https://github.com/cap10morgan))
- [#49](https://github.com/babashka/pods/issues/49): don't log socket closed exception
## v0.1.0
Initial version

39
README.md
View file

@ -77,7 +77,7 @@ On the JVM:
When calling `load-pod` with a string or vector of strings (or declaring it in your `bb.edn`),
the pod is looked up on the local file system (either using the PATH, or using an absolute path).
When it is called with a qualified symbol and a version - like `(load-pod 'org.babashka/aws "0.0.5")`
then it will be looked up in and downloaded from the [pod-registry](https://github.com/babashka/pod-registry).
then it will be looked up in and downloaded from the [pod-registry](https://github.com/babashka/pod-registry). You can customize the file system location that `load-pod` will use by setting the `BABASHKA_PODS_DIR` environment variable.
By default babashka will search for a pod binary matching your system's OS and arch. If you want to download
pods for a different OS / arch (e.g. for deployment to servers), you can set one or both of the following
@ -130,7 +130,7 @@ light weight replacement for native interop (JNI, JNA, etc.).
### Examples
Beyond the already available pods mentioned above, eductional examples of pods
Beyond the already available pods mentioned above, educational examples of pods
can be found [here](examples):
- [pod-lispyclouds-sqlite](examples/pod-lispyclouds-sqlite): a pod that
@ -228,7 +228,7 @@ JSON. It also declares that the pod exposes one namespace,
To encode payloads in EDN use `"edn"` and for Transit JSON use `"transit+json"`.
The pod encodes the above map to bencode and writes it to stdoud. The pod client
The pod encodes the above map to bencode and writes it to stdout. The pod client
reads this message from the pod's stdout.
Upon receiving this message, the pod client creates these namespaces and vars.
@ -376,11 +376,15 @@ nil
#### Metadata
**From pod to pod client**
*Fixed Metadata on vars*
Pods may attach metadata to functions and macros by sending data to the pod client
in a `"meta"` field as part of a `"var"` section. The metadata must be an appropriate
map, encoded as an EDN string. This is only applicable to vars in the pod and will be
ignored if the var refers to Client-side code, since metadata can already be defined
in those code blocks.
in those code blocks (see 'Dynamic Metadata' below to enable the encoding of metadata).
For example, a pod can define a function called `add`:
@ -392,6 +396,33 @@ For example, a pod can define a function called `add`:
"meta" "{:doc \"arithmetic addition of 2 arguments\" :arglists ([a b])}"}]}]}
```
*Dynamic Metadata*
Pods may send metadata on values returned to the client if metadata encoding is enabled
for the particular transport format used by the pod.
For example, if your pod uses `:transit+json` as its format, you can enable metadata
encoding by adding `:transform transit/write-meta` (or whatever transit is aliased to)
to the optional map passed to `transit/writer`. e.g.:
````clojure
(transit/writer baos :json {:transform transit/write-meta})
````
##### From pod client to pod
Currently sending metadata on arguments passed to a pod function is available only for the
`transit+json` format and can be enabled on a per var basis.
A pod can enable metadata to be read on arguments by sending the "arg-meta" field to "true"
for the var representing that function. For example:
````clojure
{:format :transit+json
:namespaces [{:name "pod.babashka.demo"
:vars [{"name" "round-trip" "arg-meta" "true"}]}]}
````
#### Deferred namespace loading
When your pod exposes multiple namespaces that can be used independently from

5
project.clj
View file

@ -1,4 +1,4 @@
(defproject babashka/babashka.pods "0.1.0"
(defproject babashka/babashka.pods "0.2.0"
:description "babashka pods"
:url "https://github.com/babashka/babashka.pods"
:scm {:name "git"
@ -8,7 +8,8 @@
:dependencies [[org.clojure/clojure "1.10.3"]
[nrepl/bencode "1.1.0"]
[cheshire "5.10.0"]
[babashka/fs "0.1.6"]]
[babashka/fs "0.1.6"]
[com.cognitect/transit-clj "1.0.329"]]
:deploy-repositories [["clojars" {:url "https://clojars.org/repo"
:username :env/clojars_user
:password :env/clojars_pass

17
script/changelog.clj Executable file
View file

@ -0,0 +1,17 @@
#!/usr/bin/env bb
(ns changelog
(:require [clojure.string :as str]))
(let [changelog (slurp "CHANGELOG.md")
replaced (str/replace changelog
#" #(\d+)"
(fn [[_ issue after]]
(format " [#%s](https://github.com/babashka/pods/issues/%s)%s"
issue issue (str after))))
replaced (str/replace replaced
#"@([a-zA-Z0-9-_]+)([, \.)])"
(fn [[_ name after]]
(format "[@%s](https://github.com/%s)%s"
name name after)))]
(spit "CHANGELOG.md" replaced))

47
src/babashka/pods/impl.clj
View file

@ -28,6 +28,9 @@
(defn bytes->string [^"[B" bytes]
(String. bytes))
(defn bytes->boolean [^"[B" bytes]
(= "true" (String. bytes)))
(defn get-string [m k]
(-> (get m k)
bytes->string))
@ -36,6 +39,10 @@
(some-> (get m k)
bytes->string))
(defn get-maybe-boolean [m k]
(some-> (get m k)
bytes->boolean))
(defn next-id []
(str (java.util.UUID/randomUUID)))
@ -83,10 +90,12 @@
(let [wh (transit/write-handler tag-fn val-fn)]
(swap! transit-default-write-handlers assoc *pod-id* wh)))
(defn transit-json-write [pod-id ^String s]
(defn transit-json-write
[pod-id ^String s metadata?]
(with-open [baos (java.io.ByteArrayOutputStream. 4096)]
(let [w (transit/writer baos :json {:handlers (get @transit-write-handler-maps pod-id)
:default-handler (get @transit-default-write-handlers pod-id)})]
(let [w (transit/writer baos :json (cond-> {:handlers (get @transit-write-handler-maps pod-id)
:default-handler (get @transit-default-write-handlers pod-id)}
metadata? (assoc :transform transit/write-meta)))]
(transit/write w s)
(str baos))))
@ -98,7 +107,7 @@
write-fn (case format
:edn pr-str
:json cheshire/generate-string
:transit+json #(transit-json-write (:pod-id pod) %))
:transit+json #(transit-json-write (:pod-id pod) % (:arg-meta opts)))
id (next-id)
chan (if handlers handlers
(promise))
@ -128,11 +137,12 @@
edn/read-string)
name-sym (if vmeta
(with-meta name-sym vmeta)
name-sym)]
name-sym)
metadata? (get-maybe-boolean var "arg-meta")]
[name-sym
(or code
(fn [& args]
(let [res (invoke pod sym args {:async async?})]
(let [res (invoke pod sym args {:async async? :arg-meta metadata?})]
res)))]))
vars))
@ -177,14 +187,17 @@
(let [id (get reply "id")
id (bytes->string id)
value* (find reply "value")
value (some-> value*
second
bytes->string
read-fn)
[exception value] (try (some->> value*
second
bytes->string
read-fn
(vector nil))
(catch Exception e
[e nil]))
status (get reply "status")
status (set (map (comp keyword bytes->string) status))
error? (contains? status :error)
done? (or error? (contains? status :done))
error? (or exception (contains? status :error))
done? (or error? exception (contains? status :done))
[ex-message ex-data]
(when error?
[(or (some-> (get reply "ex-message")
@ -202,8 +215,9 @@
:vars (bencode->vars pod name-str v)}))
chan (get @chans id)
promise? (instance? clojure.lang.IPending chan)
exception (when (and promise? error?)
(ex-info ex-message ex-data))
exception (or exception
(when (and promise? error?)
(ex-info ex-message ex-data)))
;; NOTE: if we need more fine-grained handlers, we will add
;; a :raw handler that will just get the bencode message's raw
;; data
@ -306,7 +320,8 @@
(catch java.net.SocketException _ nil)))
(defn port-file [pid]
(io/file (str ".babashka-pod-" pid ".port")))
(doto (io/file (str ".babashka-pod-" pid ".port"))
(.deleteOnExit)))
(defn read-port [^java.io.File port-file]
(loop []
@ -350,7 +365,7 @@
(.redirectError pb java.lang.ProcessBuilder$Redirect/INHERIT))
_ (cond-> (doto (.environment pb)
(.put "BABASHKA_POD" "true"))
socket? (.put "BABASHKA_POD_TRANSPORT" "socket"))
socket? (.put "BABASHKA_POD_TRANSPORT" "socket"))
p (.start pb)
port-file (when socket? (port-file (.pid p)))
socket-port (when socket? (read-port port-file))

5
src/babashka/pods/impl/resolver.clj
View file

@ -91,7 +91,10 @@
^"[Ljava.nio.file.CopyOption;"
(into-array
[java.nio.file.StandardCopyOption/REPLACE_EXISTING])))
(sh "tar" "xf" (.getPath tmp-file) "--directory" (.getPath destination-dir))
(.mkdirs destination-dir)
(let [res (sh "tar" "xf" (.getPath tmp-file) "--directory" (.getPath destination-dir))]
(when-not (zero? (:exit res))
(throw (ex-info (:err res) res))))
(.delete tmp-file)))
(defn make-executable [dest-dir executables verbose?]

7
src/babashka/pods/jvm.clj
View file

@ -12,11 +12,12 @@
(defn- process-namespace [{:keys [:name :vars]}]
(binding [*ns* (load-string (format "(ns %s) *ns*" name))]
(doseq [[var-sym v] vars]
(when-let [maybe-core (some-> (ns-resolve *ns* var-sym) meta :ns str symbol)]
(when (= 'clojure.core maybe-core)
(ns-unmap *ns* var-sym)))
(cond
(ifn? v)
(do
(ns-unmap *ns* var-sym)
(intern name var-sym v))
(intern name var-sym v)
(string? v)
(load-string v)))))

130
test-pod/pod/test_pod.clj
View file

@ -61,6 +61,12 @@
(transit/write w s)
(str baos))))
(defn transit-json-write-meta [s]
(with-open [baos (java.io.ByteArrayOutputStream. 4096)]
(let [w (transit/writer baos :json {:transform transit/write-meta})]
(transit/write w s)
(str baos))))
(defn run-pod [cli-args]
(let [format (cond (contains? cli-args "--json") :json
(contains? cli-args "--transit+json") :transit+json
@ -130,6 +136,10 @@
{"name" "read-other-tag"
"code" "(defn read-other-tag [x] [x x])"
"meta" "{:doc \"unread\"}"}
{"name" "round-trip-meta"
"arg-meta" "true"}
{"name" "dont-round-trip-meta"
"arg-meta" "false"}
{"name" "-local-date-time"}
{"name" "transit-stuff"
"code" "
@ -151,7 +161,8 @@
(babashka.pods/add-transit-read-handler! \"java.array\"
into-array)
"}]
"}
{"name" "incorrect-edn"}]
dependents)}
{"name" "pod.test-pod.loaded"
"defer" "true"}
@ -175,70 +186,91 @@
pod.test-pod/add-sync
(try (let [ret (apply + args)]
(write out
{"value" (write-fn ret)
"id" id
"status" ["done"]}))
{"value" (write-fn ret)
"id" id
"status" ["done"]}))
(catch Exception e
(write out
{"ex-data" (write-fn {:args args})
"ex-message" (.getMessage e)
"status" ["done" "error"]
"id" id})))
{"ex-data" (write-fn {:args args})
"ex-message" (.getMessage e)
"status" ["done" "error"]
"id" id})))
pod.test-pod/range-stream
(let [rng (apply range args)]
(doseq [v rng]
(write out
{"value" (write-fn v)
"id" id})
{"value" (write-fn v)
"id" id})
(Thread/sleep 100))
(write out
{"status" ["done"]
"id" id}))
{"status" ["done"]
"id" id}))
pod.test-pod/assoc
(write out
{"value" (write-fn (apply assoc args))
"status" ["done"]
"id" id})
{"value" (write-fn (apply assoc args))
"status" ["done"]
"id" id})
pod.test-pod/error
(write out
{"ex-data" (write-fn {:args args})
"ex-message" (str "Illegal arguments")
"status" ["done" "error"]
"id" id})
{"ex-data" (write-fn {:args args})
"ex-message" (str "Illegal arguments")
"status" ["done" "error"]
"id" id})
pod.test-pod/print
(do (write out
{"out" (with-out-str (prn args))
"id" id})
{"out" (with-out-str (prn args))
"id" id})
(write out
{"status" ["done"]
"id" id}))
{"status" ["done"]
"id" id}))
pod.test-pod/print-err
(do (write out
{"err" (with-out-str (prn args))
"id" id})
{"err" (with-out-str (prn args))
"id" id})
(write out
{"status" ["done"]
"id" id}))
{"status" ["done"]
"id" id}))
pod.test-pod/return-nil
(write out
{"status" ["done"]
"id" id
"value" (write-fn nil)})
{"status" ["done"]
"id" id
"value" (write-fn nil)})
pod.test-pod/reader-tag
(write out
{"status" ["done"]
"id" id
"value" "#my/tag[1 2 3]"})
{"status" ["done"]
"id" id
"value" "#my/tag[1 2 3]"})
pod.test-pod/other-tag
(write out
{"status" ["done"]
"id" id
"value" "#my/other-tag[1]"})
{"status" ["done"]
"id" id
"value" "#my/other-tag[1]"})
pod.test-pod/round-trip-meta
(write out
{"status" ["done"]
"id" id
"value"
(case format
:transit+json (transit-json-write-meta (first args))
(write-fn (first args)))})
pod.test-pod/dont-round-trip-meta
(write out
{"status" ["done"]
"id" id
"value"
(case format
:transit+json (transit-json-write-meta (first args))
(write-fn (first args)))})
pod.test-pod/-local-date-time
(write out
{"status" ["done"]
"id" id
"value" (write-fn (first args))}))
"value" (write-fn (first args))})
pod.test-pod/incorrect-edn
(write out
{"status" ["done"]
"id" id
"value" (write-fn {(keyword "foo bar") 1})}))
(recur))
:shutdown (System/exit 0)
:load-ns (let [ns (-> (get message "ns")
@ -249,20 +281,20 @@
(case ns
pod.test-pod.loaded
(write out
{"status" ["done"]
"id" id
"name" "pod.test-pod.loaded"
"vars" [{"name" "loaded"
"code" "(defn loaded [x] (inc x))"}]})
{"status" ["done"]
"id" id
"name" "pod.test-pod.loaded"
"vars" [{"name" "loaded"
"code" "(defn loaded [x] (inc x))"}]})
pod.test-pod.loaded2
(write out
{"status" ["done"]
"id" id
"name" "pod.test-pod.loaded2"
"vars" [{"name" "x"
"code" "(require '[pod.test-pod.loaded :as loaded])"}
{"name" "loaded"
"code" "(defn loaded [x] (loaded/loaded x))"}]}))
{"status" ["done"]
"id" id
"name" "pod.test-pod.loaded2"
"vars" [{"name" "x"
"code" "(require '[pod.test-pod.loaded :as loaded])"}
{"name" "loaded"
"code" "(defn loaded [x] (loaded/loaded x))"}]}))
(recur)))))))
(catch Exception e
(binding [*out* *err*]

25
test-resources/test_program.clj
View file

@ -84,12 +84,31 @@
(.isArray (class v)))
true))
(def round-trip-meta
(if (= "transit+json" fmt)
(= {:my-meta 2} (meta (pod.test-pod/round-trip-meta (with-meta [2] {:my-meta 2}))))
true))
(def round-trip-meta-nested
(if (= "transit+json" fmt)
(= {:my-meta 3} (meta (first (pod.test-pod/round-trip-meta [(with-meta [3] {:my-meta 3})]))))
true))
(def dont-round-trip-meta
(if (= "transit+json" fmt)
(= nil (meta (pod.test-pod/dont-round-trip-meta (with-meta [2] {:my-meta 2}))))
true))
(require '[pod.test-pod.only-code :as only-code])
(def should-be-1 (only-code/foo))
(require '[pod.test-pod.loaded2 :as loaded2])
(def loaded (loaded2/loaded 1))
(def incorrect-edn-response
(try (pod.test-pod/incorrect-edn)
(catch Exception e (ex-message e))))
(pods/unload-pod pod-id)
(def successfully-removed (nil? (find-ns 'pod.test-pod)))
@ -112,7 +131,11 @@
fn-called
local-date-time
assoc-string-array
round-trip-meta
round-trip-meta-nested
dont-round-trip-meta
should-be-1
add-sync-meta
error-meta
read-other-tag-meta]
read-other-tag-meta
incorrect-edn-response]

2
test/babashka/pods/jvm_test.clj
View file

@ -19,7 +19,7 @@
(let [out (java.io.StringWriter.)
err (java.io.StringWriter.)
ex (binding [*out* out
*err* err]
*err* err]
(try (load-string
pod-registry)
(catch Exception e

4
test/babashka/pods/sci_test.clj
View file

@ -23,7 +23,9 @@
_ (vreset! ctx-ref ctx)
ret (sci/binding [sci/out out
sci/err err]
(sci/eval-string* ctx test-program))]
(binding [*out* out
*err* err]
(sci/eval-string* ctx test-program)))]
(assertions out err ret)))
(deftest pod-registry-test

66
test/babashka/pods/test_common.clj
View file

@ -1,5 +1,6 @@
(ns babashka.pods.test-common
(:require [clojure.java.io :as io]
[clojure.string :as str]
[clojure.test :refer [is]]))
(def test-program (slurp (io/file "test-resources" "test_program.clj")))
@ -9,34 +10,45 @@
;; (.println System/err out)
;; (.println System/err err)
(doseq [[expected actual]
(map vector '["pod.test-pod"
pod.test-pod
{:a 1, :b 2}
6
3
[1 2 3 4 5 6 7 8 9]
#"Illegal arguments / \{:args [\(\[]1 2 3[\)\]]\}"
nil
3
#"cast"
{:args ["1" 2]}
true
9
[1 2 3]
[[1] [1]]
2
3
true ;; local-date
true ;; roundtrip string array
1
"add the arguments"
nil
nil]
(map vector (replace
{::edn-error (if (= "edn"
(System/getenv "BABASHKA_POD_TEST_FORMAT"))
"Map literal must contain an even number of forms"
::dont-care)}
'["pod.test-pod"
pod.test-pod
{:a 1, :b 2}
6
3
[1 2 3 4 5 6 7 8 9]
#"Illegal arguments / \{:args [\(\[]1 2 3[\)\]]\}"
nil
3
#"cast"
{:args ["1" 2]}
true
9
[1 2 3]
[[1] [1]]
2
3
true ;; local-date
true ;; roundtrip string array
true ;; roundtrip metadata
true ;; roundtrip metadata nested
true ;; dont roundtrip metadata (when arg-meta "false"/ absent)
1
"add the arguments"
nil
nil
::edn-error])
(concat ret (repeat ::nil)))]
(if (instance? java.util.regex.Pattern expected)
(is (re-find expected actual))
(is (= expected actual))))
(cond (instance? java.util.regex.Pattern expected)
(is (re-find expected actual))
(= ::dont-care expected) nil
:else
(is (= expected actual))))
(is (= "(\"hello\" \"print\" \"this\" \"debugging\" \"message\")\n:foo\n:foo\n" (str out)))
(is (= "(\"hello\" \"print\" \"this\" \"error\")\n" (str err))))
(is (str/starts-with? (str err) "(\"hello\" \"print\" \"this\" \"error\")" )))
(def pod-registry (slurp (io/file "test-resources" "pod_registry.clj")))