Compare commits

..

No commits in common. "master" and "v0.1.0" have entirely different histories.

20 changed files with 223 additions and 592 deletions

View file

@ -1,28 +0,0 @@
version: 2.1
jobs:
test:
docker:
- image: clojure:openjdk-11-tools-deps-1.10.3.1087-slim-bullseye
working_directory: ~/repo
environment:
LEIN_ROOT: "true"
BABASHKA_PLATFORM: linux
resource_class: large
steps:
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ checksum "deps.edn" }}
# fallback to using latest cache if no exact match is found
- v1-dependencies-
- run: |
script/test
- save_cache:
paths:
- ~/.m2
key: v1-dependencies-{{ checksum "deps.edn" }}
workflows:
version: 2
ci:
jobs:
- test

4
.gitignore vendored
View file

@ -3,7 +3,3 @@
.lein-failures .lein-failures
/pom.xml /pom.xml
.lein-repl-history .lein-repl-history
.cache
.clj-kondo/babashka
.clj-kondo/rewrite-clj
src/scratch.clj

View file

@ -1,21 +0,0 @@
# Changelog
## Unreleased
- [#63](https://github.com/babashka/pods/issues/63): create directory before un-tarring
- [#59](https://github.com/babashka/pods/issues/59): delete port file on exit
- [#65](https://github.com/babashka/pods/issues/65): fix warnings when defining var with core name in JVM
- [#66](https://github.com/babashka/pods/issues/66): Allow metadata on fn arguments for transit+json
## v0.2.0
- [#61](https://github.com/babashka/pods/issues/61): add transit as explicit JVM dependency
- [#60](https://github.com/babashka/pods/issues/60): transform pod reader error into exception of caller
- Switch "out" and "err" messages to print and flush instead of `println` ([@justone](https://github.com/justone))
- Set TCP_NODELAY on transport socket ([@retrogradeorbit](https://github.com/retrogradeorbit))
- Allow env vars OS_NAME & OS_ARCH to override os props ([@cap10morgan](https://github.com/cap10morgan))
- [#49](https://github.com/babashka/pods/issues/49): don't log socket closed exception
## v0.1.0
Initial version

View file

@ -74,44 +74,7 @@ On the JVM:
### Where does the pod come from? ### Where does the pod come from?
When calling `load-pod` with a string or vector of strings (or declaring it in your `bb.edn`), When calling `load-pod` with a string or vector of strings, the pod is looked up on the local file system (either using the PATH, or using an absolute path). When it is called with a qualified symbol and a version - like `(load-pod 'org.babashka/aws "0.0.5")` then it will be looked up in and downloaded from the [pod-registry](https://github.com/babashka/pod-registry).
the pod is looked up on the local file system (either using the PATH, or using an absolute path).
When it is called with a qualified symbol and a version - like `(load-pod 'org.babashka/aws "0.0.5")`
then it will be looked up in and downloaded from the [pod-registry](https://github.com/babashka/pod-registry). You can customize the file system location that `load-pod` will use by setting the `BABASHKA_PODS_DIR` environment variable.
By default babashka will search for a pod binary matching your system's OS and arch. If you want to download
pods for a different OS / arch (e.g. for deployment to servers), you can set one or both of the following
environment variables:
- `BABASHKA_PODS_OS_NAME=Linux` (or `Mac OS X` or any other value returned by Java's `os.name` property)
- `BABASHKA_PODS_OS_ARCH=aarch64` (or `amd64` or any other value returned by Java's `os.arch` property)
### In a babashka project
As of babashka 0.8.0 you can declare the pods your babashka project uses in your `bb.edn` file like so:
```clojure
:pods {org.babashka/hsqldb {:version "0.1.0"} ; will be downloaded from the babashka pod registry
my.local/pod {:path "../pod-my-local/my-pod-binary"
:cache false}} ; optionally disable namespace caching if you're actively working on this pod
```
Then you can just require the pods in your code like any other clojure lib:
```clojure
(ns my.project
(:require [pod.babashka.hsqldb :as sql]
[my.local.pod :as my-pod]))
(def db "jdbc:hsqldb:mem:testdb;sql.syntax_mys=true")
(sql/execute! db ["create table foo ( foo int );"])
;;=> [#:next.jdbc{:update-count 0}]
(my-pod/do-a-thing "foo")
;;=> "something"
```
The pods will then be loaded on demand when you require them. No need to call `load-pod` explicitly.
## Sci ## Sci
@ -130,7 +93,7 @@ light weight replacement for native interop (JNI, JNA, etc.).
### Examples ### Examples
Beyond the already available pods mentioned above, educational examples of pods Beyond the already available pods mentioned above, eductional examples of pods
can be found [here](examples): can be found [here](examples):
- [pod-lispyclouds-sqlite](examples/pod-lispyclouds-sqlite): a pod that - [pod-lispyclouds-sqlite](examples/pod-lispyclouds-sqlite): a pod that
@ -228,7 +191,7 @@ JSON. It also declares that the pod exposes one namespace,
To encode payloads in EDN use `"edn"` and for Transit JSON use `"transit+json"`. To encode payloads in EDN use `"edn"` and for Transit JSON use `"transit+json"`.
The pod encodes the above map to bencode and writes it to stdout. The pod client The pod encodes the above map to bencode and writes it to stdoud. The pod client
reads this message from the pod's stdout. reads this message from the pod's stdout.
Upon receiving this message, the pod client creates these namespaces and vars. Upon receiving this message, the pod client creates these namespaces and vars.
@ -376,15 +339,11 @@ nil
#### Metadata #### Metadata
**From pod to pod client**
*Fixed Metadata on vars*
Pods may attach metadata to functions and macros by sending data to the pod client Pods may attach metadata to functions and macros by sending data to the pod client
in a `"meta"` field as part of a `"var"` section. The metadata must be an appropriate in a `"meta"` field as part of a `"var"` section. The metadata must be an appropriate
map, encoded as an EDN string. This is only applicable to vars in the pod and will be map, encoded as an EDN string. This is only applicable to vars in the pod and will be
ignored if the var refers to Client-side code, since metadata can already be defined ignored if the var refers to Client-side code, since metadata can already be defined
in those code blocks (see 'Dynamic Metadata' below to enable the encoding of metadata). in those code blocks.
For example, a pod can define a function called `add`: For example, a pod can define a function called `add`:
@ -396,33 +355,6 @@ For example, a pod can define a function called `add`:
"meta" "{:doc \"arithmetic addition of 2 arguments\" :arglists ([a b])}"}]}]} "meta" "{:doc \"arithmetic addition of 2 arguments\" :arglists ([a b])}"}]}]}
``` ```
*Dynamic Metadata*
Pods may send metadata on values returned to the client if metadata encoding is enabled
for the particular transport format used by the pod.
For example, if your pod uses `:transit+json` as its format, you can enable metadata
encoding by adding `:transform transit/write-meta` (or whatever transit is aliased to)
to the optional map passed to `transit/writer`. e.g.:
````clojure
(transit/writer baos :json {:transform transit/write-meta})
````
##### From pod client to pod
Currently sending metadata on arguments passed to a pod function is available only for the
`transit+json` format and can be enabled on a per var basis.
A pod can enable metadata to be read on arguments by sending the "arg-meta" field to "true"
for the var representing that function. For example:
````clojure
{:format :transit+json
:namespaces [{:name "pod.babashka.demo"
:vars [{"name" "round-trip" "arg-meta" "true"}]}]}
````
#### Deferred namespace loading #### Deferred namespace loading
When your pod exposes multiple namespaces that can be used independently from When your pod exposes multiple namespaces that can be used independently from
@ -497,7 +429,7 @@ The arguments to `babashka.pods/invoke` are:
- a pod identifier string derived from the first described namespace. - a pod identifier string derived from the first described namespace.
- the symbol of the var to invoke - the symbol of the var to invoke
- the arguments to the var - the arguments to the var
- an opts map containing `:handlers` containing callback functions: `:success`, `:error` and `:done` - an opts map containing `:handler` containing callback functions: `:success`, `:error` and `:done`
The return value of `babashka.pods/invoke` is a map containing `:result`. When The return value of `babashka.pods/invoke` is a map containing `:result`. When
not using callbacks, this is the return value from the pod var invocation. When not using callbacks, this is the return value from the pod var invocation. When
@ -523,9 +455,9 @@ callback is only called if no errors were sent by the pod.
In the above example the wrapper function calls the pod identified by In the above example the wrapper function calls the pod identified by
`"pod.babashka.filewatcher"`. It calls the var `"pod.babashka.filewatcher"`. It calls the var
`pod.babashka.filewatcher/watch*`. In `:success` it pulls out received `pod.babashka.filewatcher/watch*`. In `:on-success` it pulls out received
values, passing them to the user-provided callback. Additionally, it prints any values, passing them to the user-provided callback. Additionally, it prints any
errors received from the pod library in `:error` to `*err*`. errors received from the pod library in `:on-error` to `*err*`.
A user will then use `pod.babashka.filewatcher/watch` like this: A user will then use `pod.babashka.filewatcher/watch` like this:

2
bb.edn
View file

@ -1,2 +0,0 @@
{:tasks {test {:doc "Run tests"
:task (shell "script/test")}}}

View file

@ -1,7 +1,6 @@
{:deps {nrepl/bencode {:mvn/version "1.1.0"} {:deps {nrepl/bencode {:mvn/version "1.1.0"}
cheshire/cheshire {:mvn/version "5.10.0"} cheshire/cheshire {:mvn/version "5.10.0"}
com.cognitect/transit-clj {:mvn/version "1.0.324"} com.cognitect/transit-clj {:mvn/version "1.0.324"}}
babashka/fs {:mvn/version "0.1.6"}}
:aliases :aliases
{:sci {:sci
{:extra-deps {:extra-deps

View file

@ -1,4 +1,4 @@
(defproject babashka/babashka.pods "0.2.0" (defproject babashka/babashka.pods "0.1.0"
:description "babashka pods" :description "babashka pods"
:url "https://github.com/babashka/babashka.pods" :url "https://github.com/babashka/babashka.pods"
:scm {:name "git" :scm {:name "git"
@ -7,9 +7,7 @@
:url "https://www.eclipse.org/legal/epl-1.0/"} :url "https://www.eclipse.org/legal/epl-1.0/"}
:dependencies [[org.clojure/clojure "1.10.3"] :dependencies [[org.clojure/clojure "1.10.3"]
[nrepl/bencode "1.1.0"] [nrepl/bencode "1.1.0"]
[cheshire "5.10.0"] [cheshire "5.10.0"]]
[babashka/fs "0.1.6"]
[com.cognitect/transit-clj "1.0.329"]]
:deploy-repositories [["clojars" {:url "https://clojars.org/repo" :deploy-repositories [["clojars" {:url "https://clojars.org/repo"
:username :env/clojars_user :username :env/clojars_user
:password :env/clojars_pass :password :env/clojars_pass

View file

@ -1,17 +0,0 @@
#!/usr/bin/env bb
(ns changelog
(:require [clojure.string :as str]))
(let [changelog (slurp "CHANGELOG.md")
replaced (str/replace changelog
#" #(\d+)"
(fn [[_ issue after]]
(format " [#%s](https://github.com/babashka/pods/issues/%s)%s"
issue issue (str after))))
replaced (str/replace replaced
#"@([a-zA-Z0-9-_]+)([, \.)])"
(fn [[_ name after]]
(format "[@%s](https://github.com/%s)%s"
name name after)))]
(spit "CHANGELOG.md" replaced))

View file

@ -8,25 +8,24 @@ export BABASHKA_POD_TEST_SOCKET
# format = edn # format = edn
BABASHKA_POD_TEST_FORMAT=edn BABASHKA_POD_TEST_FORMAT=edn
echo "Testing edn" echo "Testing edn"
clojure -M:test -n babashka.pods.jvm-test clojure -A:test -n babashka.pods.jvm-test
clojure -M:sci:test -n babashka.pods.sci-test clojure -A:sci:test -n babashka.pods.sci-test
clojure -M:test -n babashka.pods.impl-test
# format = json # format = json
BABASHKA_POD_TEST_FORMAT=json BABASHKA_POD_TEST_FORMAT=json
echo "Testing json" echo "Testing json"
clojure -M:test -n babashka.pods.jvm-test clojure -A:test -n babashka.pods.jvm-test
clojure -M:sci:test -n babashka.pods.sci-test clojure -A:sci:test -n babashka.pods.sci-test
# format = json # format = json
BABASHKA_POD_TEST_FORMAT="transit+json" BABASHKA_POD_TEST_FORMAT="transit+json"
echo "Testing transit" echo "Testing transit"
clojure -M:test -n babashka.pods.jvm-test clojure -A:test -n babashka.pods.jvm-test
clojure -M:sci:test -n babashka.pods.sci-test clojure -A:sci:test -n babashka.pods.sci-test
# socket = true # socket = true
unset BABASHKA_POD_TEST_FORMAT unset BABASHKA_POD_TEST_FORMAT
BABASHKA_POD_TEST_SOCKET=true BABASHKA_POD_TEST_SOCKET=true
echo "Testing socket" echo "Testing socket"
clojure -M:test -n babashka.pods.jvm-test clojure -A:test -n babashka.pods.jvm-test
clojure -M:sci:test -n babashka.pods.sci-test clojure -A:sci:test -n babashka.pods.sci-test

View file

@ -28,9 +28,6 @@
(defn bytes->string [^"[B" bytes] (defn bytes->string [^"[B" bytes]
(String. bytes)) (String. bytes))
(defn bytes->boolean [^"[B" bytes]
(= "true" (String. bytes)))
(defn get-string [m k] (defn get-string [m k]
(-> (get m k) (-> (get m k)
bytes->string)) bytes->string))
@ -39,10 +36,6 @@
(some-> (get m k) (some-> (get m k)
bytes->string)) bytes->string))
(defn get-maybe-boolean [m k]
(some-> (get m k)
bytes->boolean))
(defn next-id [] (defn next-id []
(str (java.util.UUID/randomUUID))) (str (java.util.UUID/randomUUID)))
@ -90,12 +83,10 @@
(let [wh (transit/write-handler tag-fn val-fn)] (let [wh (transit/write-handler tag-fn val-fn)]
(swap! transit-default-write-handlers assoc *pod-id* wh))) (swap! transit-default-write-handlers assoc *pod-id* wh)))
(defn transit-json-write (defn transit-json-write [pod-id ^String s]
[pod-id ^String s metadata?]
(with-open [baos (java.io.ByteArrayOutputStream. 4096)] (with-open [baos (java.io.ByteArrayOutputStream. 4096)]
(let [w (transit/writer baos :json (cond-> {:handlers (get @transit-write-handler-maps pod-id) (let [w (transit/writer baos :json {:handlers (get @transit-write-handler-maps pod-id)
:default-handler (get @transit-default-write-handlers pod-id)} :default-handler (get @transit-default-write-handlers pod-id)})]
metadata? (assoc :transform transit/write-meta)))]
(transit/write w s) (transit/write w s)
(str baos)))) (str baos))))
@ -107,7 +98,7 @@
write-fn (case format write-fn (case format
:edn pr-str :edn pr-str
:json cheshire/generate-string :json cheshire/generate-string
:transit+json #(transit-json-write (:pod-id pod) % (:arg-meta opts))) :transit+json #(transit-json-write (:pod-id pod) %))
id (next-id) id (next-id)
chan (if handlers handlers chan (if handlers handlers
(promise)) (promise))
@ -137,12 +128,11 @@
edn/read-string) edn/read-string)
name-sym (if vmeta name-sym (if vmeta
(with-meta name-sym vmeta) (with-meta name-sym vmeta)
name-sym) name-sym)]
metadata? (get-maybe-boolean var "arg-meta")]
[name-sym [name-sym
(or code (or code
(fn [& args] (fn [& args]
(let [res (invoke pod sym args {:async async? :arg-meta metadata?})] (let [res (invoke pod sym args {:async async?})]
res)))])) res)))]))
vars)) vars))
@ -178,26 +168,19 @@
(loop [] (loop []
(let [reply (try (read stdout) (let [reply (try (read stdout)
(catch java.io.EOFException _ (catch java.io.EOFException _
::EOF) ::EOF))]
(catch java.net.SocketException e
(if (= "Socket closed" (ex-message e))
::EOF
(throw e))))]
(when-not (identical? ::EOF reply) (when-not (identical? ::EOF reply)
(let [id (get reply "id") (let [id (get reply "id")
id (bytes->string id) id (bytes->string id)
value* (find reply "value") value* (find reply "value")
[exception value] (try (some->> value* value (some-> value*
second second
bytes->string bytes->string
read-fn read-fn)
(vector nil))
(catch Exception e
[e nil]))
status (get reply "status") status (get reply "status")
status (set (map (comp keyword bytes->string) status)) status (set (map (comp keyword bytes->string) status))
error? (or exception (contains? status :error)) error? (contains? status :error)
done? (or error? exception (contains? status :done)) done? (or error? (contains? status :done))
[ex-message ex-data] [ex-message ex-data]
(when error? (when error?
[(or (some-> (get reply "ex-message") [(or (some-> (get reply "ex-message")
@ -215,9 +198,8 @@
:vars (bencode->vars pod name-str v)})) :vars (bencode->vars pod name-str v)}))
chan (get @chans id) chan (get @chans id)
promise? (instance? clojure.lang.IPending chan) promise? (instance? clojure.lang.IPending chan)
exception (or exception exception (when (and promise? error?)
(when (and promise? error?) (ex-info ex-message ex-data))
(ex-info ex-message ex-data)))
;; NOTE: if we need more fine-grained handlers, we will add ;; NOTE: if we need more fine-grained handlers, we will add
;; a :raw handler that will just get the bencode message's raw ;; a :raw handler that will just get the bencode message's raw
;; data ;; data
@ -233,12 +215,9 @@
;; listening to output synchronous. ;; listening to output synchronous.
(when out (when out
(binding [*out* out-stream] (binding [*out* out-stream]
(print out) (println out)))
(.flush ^java.io.Writer out-stream))) (when err (binding [*out* err-stream]
(when err (println err)))
(binding [*out* err-stream]
(print err)
(.flush ^java.io.Writer err-stream)))
(when (or value* error? namespace) (when (or value* error? namespace)
(cond promise? (cond promise?
(deliver chan (cond error? exception (deliver chan (cond error? exception
@ -269,24 +248,27 @@
(defn lookup-pod [pod-id] (defn lookup-pod [pod-id]
(get @pods pod-id)) (get @pods pod-id))
(defn destroy* [{:keys [:stdin :process :ops]}]
(if (contains? ops :shutdown)
(do (write stdin
{"op" "shutdown"
"id" (next-id)})
(.waitFor ^Process process))
(.destroy ^Process process)))
(defn destroy [pod-id-or-pod] (defn destroy [pod-id-or-pod]
(let [pod-id (get-pod-id pod-id-or-pod)] (let [pod-id (get-pod-id pod-id-or-pod)]
(when-let [pod (lookup-pod pod-id)] (when-let [pod (lookup-pod pod-id)]
(destroy* pod) (if (contains? (:ops pod) :shutdown)
(do (write (:stdin pod)
{"op" "shutdown"
"id" (next-id)})
(.waitFor ^Process (:process pod)))
(.destroy ^Process (:process pod)))
(when-let [rns (:remove-ns pod)] (when-let [rns (:remove-ns pod)]
(doseq [[ns-name _] (:namespaces pod)] (doseq [[ns-name _] (:namespaces pod)]
(rns ns-name)))) (rns ns-name))))
(swap! pods dissoc pod-id) (swap! pods dissoc pod-id)
nil)) nil))
(def next-pod-id
(let [counter (atom 0)]
(fn []
(let [[o _] (swap-vals! counter inc)]
o))))
(def bytes->symbol (def bytes->symbol
(comp symbol bytes->string)) (comp symbol bytes->string))
@ -309,8 +291,7 @@
the socket is connected." the socket is connected."
^Socket ^Socket
[^String hostname ^Integer port] [^String hostname ^Integer port]
(doto (Socket. hostname port) (Socket. hostname port))
(.setTcpNoDelay true)))
(defn close-socket (defn close-socket
"Close the socket, and also closes its input and output streams." "Close the socket, and also closes its input and output streams."
@ -320,8 +301,7 @@
(catch java.net.SocketException _ nil))) (catch java.net.SocketException _ nil)))
(defn port-file [pid] (defn port-file [pid]
(doto (io/file (str ".babashka-pod-" pid ".port")) (io/file (str ".babashka-pod-" pid ".port")))
(.deleteOnExit)))
(defn read-port [^java.io.File port-file] (defn read-port [^java.io.File port-file]
(loop [] (loop []
@ -330,35 +310,29 @@
(let [s (slurp f)] (let [s (slurp f)]
(when (str/ends-with? s "\n") (when (str/ends-with? s "\n")
(str/trim s))))] (str/trim s))))]
(Integer/parseInt s) (Integer. s)
(recur))))) (recur)))))
(defn debug [& strs] (defn debug [& strs]
(binding [*out* *err*] (binding [*out* *err*]
(println (str/join " " (map pr-str strs))))) (println (str/join " " (map pr-str strs)))))
(defn resolve-pod [pod-spec {:keys [:version :path :force] :as opts}] (defn load-pod
(when (qualified-symbol? pod-spec) ([pod-spec] (load-pod pod-spec nil))
(when (and (not version) (not path)) ([pod-spec opts]
(throw (IllegalArgumentException. "Version or path must be provided"))) (let [{:keys [:version :force]} opts
(when (and version path) resolved (when (qualified-symbol? pod-spec)
(throw (IllegalArgumentException. "You must provide either version or path, not both"))))
(let [resolved (when (and (qualified-symbol? pod-spec) version)
(resolver/resolve pod-spec version force)) (resolver/resolve pod-spec version force))
opts (if resolved opts (if resolved
(if-let [extra-opts (:options resolved)] (if-let [extra-opts (:options resolved)]
(merge opts extra-opts) (merge opts extra-opts)
opts) opts)
opts) opts)
pod-spec (cond {:keys [:remove-ns :resolve :transport]} opts
resolved [(:executable resolved)] pod-spec (cond resolved [(:executable resolved)]
path [path]
(string? pod-spec) [pod-spec] (string? pod-spec) [pod-spec]
:else pod-spec)] :else pod-spec)
{:pod-spec pod-spec, :opts opts})) pb (ProcessBuilder. ^java.util.List pod-spec)
(defn run-pod [pod-spec {:keys [:transport] :as _opts}]
(let [pb (ProcessBuilder. ^java.util.List pod-spec)
socket? (identical? :socket transport) socket? (identical? :socket transport)
_ (if socket? _ (if socket?
(.inheritIO pb) (.inheritIO pb)
@ -381,53 +355,14 @@
[socket [socket
(.getOutputStream socket) (.getOutputStream socket)
(PushbackInputStream. (.getInputStream socket))]) (PushbackInputStream. (.getInputStream socket))])
[nil (.getOutputStream p) (java.io.PushbackInputStream. (.getInputStream p))])] [nil (.getOutputStream p) (java.io.PushbackInputStream. (.getInputStream p))])
{:process p _ (write stdin {"op" "describe"
:socket socket
:stdin stdin
:stdout stdout}))
(defn describe-pod [{:keys [:stdin :stdout]}]
(write stdin {"op" "describe"
"id" (next-id)}) "id" (next-id)})
(read stdout)) reply (read stdout)
format (-> (get reply "format") bytes->string keyword)
(defn describe->ops [describe-reply] ops (some->> (get reply "ops") keys (map keyword) set)
(some->> (get describe-reply "ops") keys (map keyword) set))
(defn describe->metadata [describe-reply resolve-fn]
(let [format (-> (get describe-reply "format") bytes->string keyword)
ops (describe->ops describe-reply)
readers (when (identical? :edn format) readers (when (identical? :edn format)
(read-readers describe-reply resolve-fn))] (read-readers reply resolve))
{:format format, :ops ops, :readers readers}))
(defn run-pod-for-metadata [pod-spec opts]
(let [running-pod (run-pod pod-spec opts)
describe-reply (describe-pod running-pod)
ops (describe->ops describe-reply)]
(destroy* (assoc running-pod :ops ops))
describe-reply))
(defn load-pod-metadata [unresolved-pod-spec {:keys [:download-only] :as opts}]
(let [{:keys [:pod-spec :opts]} (resolve-pod unresolved-pod-spec opts)]
(if download-only
(resolver/warn "Not running pod" unresolved-pod-spec "to pre-cache metadata because OS and/or arch are different than system")
(run-pod-for-metadata pod-spec opts))))
(defn load-pod
([pod-spec] (load-pod pod-spec nil))
([pod-spec opts]
(let [{:keys [:pod-spec :opts]} (resolve-pod pod-spec opts)
{:keys [:remove-ns :resolve]} opts
{p :process, stdin :stdin, stdout :stdout, socket :socket
:as running-pod}
(run-pod pod-spec opts)
reply (or (:metadata opts)
(describe-pod running-pod))
{:keys [:format :ops :readers]} (describe->metadata reply resolve)
pod {:process p pod {:process p
:pod-spec pod-spec :pod-spec pod-spec
:stdin stdin :stdin stdin

View file

@ -15,43 +15,29 @@
"x86_64" "x86_64"
arch)) arch))
(defn normalize-os [os] (def os {:os/name (System/getProperty "os.name")
(-> os str/lower-case (str/replace #"\s+" "_"))) :os/arch (let [arch (System/getProperty "os.arch")]
(normalize-arch arch))})
(def os
(delay
{:os/name (or (System/getenv "BABASHKA_PODS_OS_NAME")
(System/getProperty "os.name"))
:os/arch (let [arch (or (System/getenv "BABASHKA_PODS_OS_ARCH")
(System/getProperty "os.arch"))]
(normalize-arch arch))}))
(defn warn [& strs] (defn warn [& strs]
(binding [*out* *err*] (binding [*out* *err*]
(apply println strs))) (apply println strs)))
(defn match-artifacts (defn match-artifacts [package]
([package] (match-artifacts package (:os/arch @os)))
([package arch]
(let [artifacts (:pod/artifacts package) (let [artifacts (:pod/artifacts package)
res (filter (fn [{os-name :os/name res (filter (fn [{os-name :os/name
os-arch :os/arch}] os-arch :os/arch}]
(let [os-arch (normalize-arch os-arch)] (let [os-arch (normalize-arch os-arch)]
(and (re-matches (re-pattern os-name) (:os/name @os)) (and (re-matches (re-pattern os-name) (:os/name os))
(re-matches (re-pattern os-arch) (re-matches (re-pattern os-arch)
arch)))) (:os/arch os)))))
artifacts)] artifacts)]
(if (empty? res) (when (empty? res)
(if (and (= "Mac OS X" (:os/name @os))
(= "aarch64" (:os/arch @os)))
;; Rosetta2 fallback on Apple M1 machines
(match-artifacts package "x86_64")
(throw (IllegalArgumentException. (format "No executable found for pod %s (%s) and OS %s/%s" (throw (IllegalArgumentException. (format "No executable found for pod %s (%s) and OS %s/%s"
(:pod/name package) (:pod/name package)
(:pod/version package) (:pod/version package)
(:os/name @os) (:os/name os)
(:os/arch @os))))) (:os/arch os)))))
res)))) res))
(defn unzip [{:keys [^java.io.File zip-file (defn unzip [{:keys [^java.io.File zip-file
^java.io.File destination-dir ^java.io.File destination-dir
@ -91,10 +77,7 @@
^"[Ljava.nio.file.CopyOption;" ^"[Ljava.nio.file.CopyOption;"
(into-array (into-array
[java.nio.file.StandardCopyOption/REPLACE_EXISTING]))) [java.nio.file.StandardCopyOption/REPLACE_EXISTING])))
(.mkdirs destination-dir) (sh "tar" "xf" (.getPath tmp-file) "--directory" (.getPath destination-dir))
(let [res (sh "tar" "xf" (.getPath tmp-file) "--directory" (.getPath destination-dir))]
(when-not (zero? (:exit res))
(throw (ex-info (:err res) res))))
(.delete tmp-file))) (.delete tmp-file)))
(defn make-executable [dest-dir executables verbose?] (defn make-executable [dest-dir executables verbose?]
@ -114,20 +97,11 @@
(with-open [is (.getInputStream conn)] (with-open [is (.getInputStream conn)]
(io/copy is dest)))) (io/copy is dest))))
(defn repo-dir [] (def pod-manifests-dir
(io/file (if-let [pods-dir (System/getenv "BABASHKA_PODS_DIR")]
(io/file pods-dir)
(io/file (or
(System/getenv "XDG_DATA_HOME")
(System/getProperty "user.home"))
".babashka"
"pods"))
"repository"))
(def pods-repo-dir
;; wrapped in delay for GraalVM native-image ;; wrapped in delay for GraalVM native-image
(delay (delay (io/file (or (System/getenv "XDG_DATA_HOME")
(repo-dir))) (System/getProperty "user.home"))
".babashka" "pods" "repository")))
(defn github-url [qsym version] (defn github-url [qsym version]
(format (format
@ -136,7 +110,7 @@
(defn pod-manifest (defn pod-manifest
[qsym version force?] [qsym version force?]
(let [f (io/file @pods-repo-dir (str qsym) (str version) "manifest.edn")] (let [f (io/file @pod-manifests-dir (str qsym) (str version) "manifest.edn")]
(if (and (not force?) (if (and (not force?)
(.exists f)) (.exists f))
(edn/read-string (slurp f)) (edn/read-string (slurp f))
@ -147,30 +121,27 @@
^java.io.File ^java.io.File
[{pod-name :pod/name [{pod-name :pod/name
pod-version :pod/version}] pod-version :pod/version}]
(let [base-file
(if-let [pods-dir (System/getenv "BABASHKA_PODS_DIR")]
(io/file pods-dir)
(io/file (or (io/file (or
(System/getenv "XDG_CACHE_HOME") (System/getenv "XDG_CACHE_HOME")
(System/getProperty "user.home")) (System/getProperty "user.home"))
".babashka" ".babashka"
"pods"))] "pods"
(io/file base-file
"repository" "repository"
(str pod-name) (str pod-name)
pod-version pod-version))
(normalize-os (:os/name @os))
(:os/arch @os))))
(defn data-dir (defn data-dir
^java.io.File ^java.io.File
[{pod-name :pod/name [{pod-name :pod/name
pod-version :pod/version}] pod-version :pod/version}]
(io/file @pods-repo-dir (io/file (or
(System/getenv "XDG_DATA_HOME")
(System/getProperty "user.home"))
".babashka"
"pods"
"repository"
(str pod-name) (str pod-name)
pod-version pod-version))
(normalize-os (:os/name @os))
(:os/arch @os)))
(defn sha256 [file] (defn sha256 [file]
(let [buf (byte-array 8192) (let [buf (byte-array 8192)

View file

@ -12,12 +12,11 @@
(defn- process-namespace [{:keys [:name :vars]}] (defn- process-namespace [{:keys [:name :vars]}]
(binding [*ns* (load-string (format "(ns %s) *ns*" name))] (binding [*ns* (load-string (format "(ns %s) *ns*" name))]
(doseq [[var-sym v] vars] (doseq [[var-sym v] vars]
(when-let [maybe-core (some-> (ns-resolve *ns* var-sym) meta :ns str symbol)]
(when (= 'clojure.core maybe-core)
(ns-unmap *ns* var-sym)))
(cond (cond
(ifn? v) (ifn? v)
(intern name var-sym v) (do
(ns-unmap *ns* var-sym)
(intern name var-sym v))
(string? v) (string? v)
(load-string v))))) (load-string v)))))

View file

@ -1,12 +1,6 @@
(ns babashka.pods.sci (ns babashka.pods.sci
(:require [babashka.pods.impl :as impl] (:require [babashka.pods.impl :as impl]
[sci.core :as sci] [sci.core :as sci]))
[clojure.java.io :as io]
[babashka.pods.impl.resolver :as resolver]
[babashka.fs :as fs])
(:import (java.io PushbackInputStream File)))
(set! *warn-on-reflection* true)
(defn- process-namespace [ctx {:keys [:name :vars]}] (defn- process-namespace [ctx {:keys [:name :vars]}]
(let [env (:env ctx) (let [env (:env ctx)
@ -25,50 +19,6 @@
(string? var-value) (string? var-value)
(sci/eval-string* ctx var-value)))))) (sci/eval-string* ctx var-value))))))
(defn metadata-cache-file ^File [^File bb-edn-file pod-spec {:keys [:version :path]}]
(if version
(io/file (resolver/cache-dir {:pod/name pod-spec :pod/version version})
"metadata.cache")
(let [config-dir (.getParentFile bb-edn-file)
cache-dir (io/file config-dir ".babashka")
pod-file (-> path io/file .getName)
cache-file (io/file cache-dir (str pod-file ".metadata.cache"))]
cache-file)))
(defn load-metadata-from-cache [bb-edn-file pod-spec opts]
(let [cache-file (metadata-cache-file bb-edn-file pod-spec opts)]
(when (.exists cache-file)
(with-open [r (PushbackInputStream. (io/input-stream cache-file))]
(impl/read r)))))
(defn load-pod-metadata* [bb-edn-file pod-spec {:keys [:version :cache] :as opts}]
(let [metadata (impl/load-pod-metadata pod-spec opts)
cache-file (when (and metadata cache)
(metadata-cache-file bb-edn-file pod-spec opts))]
(when cache-file
(io/make-parents cache-file)
(when (fs/writable? (fs/parent cache-file))
(with-open [w (io/output-stream cache-file)]
(impl/write w metadata))))
metadata))
(defn load-pod-metadata
([pod-spec opts] (load-pod-metadata nil pod-spec opts))
([bb-edn-file pod-spec {:keys [:cache] :as opts}]
(let [metadata
(if-let [cached-metadata (when cache
(load-metadata-from-cache bb-edn-file
pod-spec
opts))]
cached-metadata
(load-pod-metadata* bb-edn-file pod-spec opts))]
(reduce
(fn [pod-namespaces ns]
(let [ns-sym (-> ns (get "name") impl/bytes->string symbol)]
(assoc pod-namespaces ns-sym {:pod-spec pod-spec
:opts (assoc opts :metadata metadata)})))
{} (get metadata "namespaces")))))
(defn load-pod (defn load-pod
([ctx pod-spec] (load-pod ctx pod-spec nil)) ([ctx pod-spec] (load-pod ctx pod-spec nil))
([ctx pod-spec version opts] (load-pod ctx pod-spec (assoc opts :version version))) ([ctx pod-spec version opts] (load-pod ctx pod-spec (assoc opts :version version)))

View file

@ -61,12 +61,6 @@
(transit/write w s) (transit/write w s)
(str baos)))) (str baos))))
(defn transit-json-write-meta [s]
(with-open [baos (java.io.ByteArrayOutputStream. 4096)]
(let [w (transit/writer baos :json {:transform transit/write-meta})]
(transit/write w s)
(str baos))))
(defn run-pod [cli-args] (defn run-pod [cli-args]
(let [format (cond (contains? cli-args "--json") :json (let [format (cond (contains? cli-args "--json") :json
(contains? cli-args "--transit+json") :transit+json (contains? cli-args "--transit+json") :transit+json
@ -136,10 +130,6 @@
{"name" "read-other-tag" {"name" "read-other-tag"
"code" "(defn read-other-tag [x] [x x])" "code" "(defn read-other-tag [x] [x x])"
"meta" "{:doc \"unread\"}"} "meta" "{:doc \"unread\"}"}
{"name" "round-trip-meta"
"arg-meta" "true"}
{"name" "dont-round-trip-meta"
"arg-meta" "false"}
{"name" "-local-date-time"} {"name" "-local-date-time"}
{"name" "transit-stuff" {"name" "transit-stuff"
"code" " "code" "
@ -161,8 +151,7 @@
(babashka.pods/add-transit-read-handler! \"java.array\" (babashka.pods/add-transit-read-handler! \"java.array\"
into-array) into-array)
"} "}]
{"name" "incorrect-edn"}]
dependents)} dependents)}
{"name" "pod.test-pod.loaded" {"name" "pod.test-pod.loaded"
"defer" "true"} "defer" "true"}
@ -218,14 +207,14 @@
"id" id}) "id" id})
pod.test-pod/print pod.test-pod/print
(do (write out (do (write out
{"out" (with-out-str (prn args)) {"out" (pr-str args)
"id" id}) "id" id})
(write out (write out
{"status" ["done"] {"status" ["done"]
"id" id})) "id" id}))
pod.test-pod/print-err pod.test-pod/print-err
(do (write out (do (write out
{"err" (with-out-str (prn args)) {"err" (pr-str args)
"id" id}) "id" id})
(write out (write out
{"status" ["done"] {"status" ["done"]
@ -245,32 +234,11 @@
{"status" ["done"] {"status" ["done"]
"id" id "id" id
"value" "#my/other-tag[1]"}) "value" "#my/other-tag[1]"})
pod.test-pod/round-trip-meta
(write out
{"status" ["done"]
"id" id
"value"
(case format
:transit+json (transit-json-write-meta (first args))
(write-fn (first args)))})
pod.test-pod/dont-round-trip-meta
(write out
{"status" ["done"]
"id" id
"value"
(case format
:transit+json (transit-json-write-meta (first args))
(write-fn (first args)))})
pod.test-pod/-local-date-time pod.test-pod/-local-date-time
(write out (write out
{"status" ["done"] {"status" ["done"]
"id" id "id" id
"value" (write-fn (first args))}) "value" (write-fn (first args))}))
pod.test-pod/incorrect-edn
(write out
{"status" ["done"]
"id" id
"value" (write-fn {(keyword "foo bar") 1})}))
(recur)) (recur))
:shutdown (System/exit 0) :shutdown (System/exit 0)
:load-ns (let [ns (-> (get message "ns") :load-ns (let [ns (-> (get message "ns")

View file

@ -1,6 +1,6 @@
(require '[babashka.pods :as pods]) (require '[babashka.pods :as pods])
(pods/load-pod 'org.babashka/buddy "0.1.0") (pods/load-pod 'org.babashka/buddy "0.0.1")
(require '[pod.babashka.buddy.codecs :as codecs] (require '[pod.babashka.buddy.codecs :as codecs]
'[pod.babashka.buddy.hash :as hash]) '[pod.babashka.buddy.hash :as hash])
@ -8,4 +8,4 @@
(println (-> (hash/sha256 "foobar") (println (-> (hash/sha256 "foobar")
(codecs/bytes->hex))) (codecs/bytes->hex)))
(pods/load-pod 'org.babashka/etaoin) ;; should cause error when version & path are missing (pods/load-pod 'org.babashka/etaoin) ;; should cause error when version is missing

View file

@ -84,31 +84,12 @@
(.isArray (class v))) (.isArray (class v)))
true)) true))
(def round-trip-meta
(if (= "transit+json" fmt)
(= {:my-meta 2} (meta (pod.test-pod/round-trip-meta (with-meta [2] {:my-meta 2}))))
true))
(def round-trip-meta-nested
(if (= "transit+json" fmt)
(= {:my-meta 3} (meta (first (pod.test-pod/round-trip-meta [(with-meta [3] {:my-meta 3})]))))
true))
(def dont-round-trip-meta
(if (= "transit+json" fmt)
(= nil (meta (pod.test-pod/dont-round-trip-meta (with-meta [2] {:my-meta 2}))))
true))
(require '[pod.test-pod.only-code :as only-code]) (require '[pod.test-pod.only-code :as only-code])
(def should-be-1 (only-code/foo)) (def should-be-1 (only-code/foo))
(require '[pod.test-pod.loaded2 :as loaded2]) (require '[pod.test-pod.loaded2 :as loaded2])
(def loaded (loaded2/loaded 1)) (def loaded (loaded2/loaded 1))
(def incorrect-edn-response
(try (pod.test-pod/incorrect-edn)
(catch Exception e (ex-message e))))
(pods/unload-pod pod-id) (pods/unload-pod pod-id)
(def successfully-removed (nil? (find-ns 'pod.test-pod))) (def successfully-removed (nil? (find-ns 'pod.test-pod)))
@ -131,11 +112,7 @@
fn-called fn-called
local-date-time local-date-time
assoc-string-array assoc-string-array
round-trip-meta
round-trip-meta-nested
dont-round-trip-meta
should-be-1 should-be-1
add-sync-meta add-sync-meta
error-meta error-meta
read-other-tag-meta read-other-tag-meta]
incorrect-edn-response]

View file

@ -1,11 +0,0 @@
(ns babashka.pods.impl-test
(:require [clojure.test :refer :all]
[babashka.pods.impl :refer :all]))
(deftest load-pod-test
(testing "resolve fn gets called when pod has EDN data readers"
(let [resolved? (atom false)
test-resolve (fn [_sym]
(reset! resolved? true))]
(load-pod ["clojure" "-M:test-pod"] {:resolve test-resolve})
(is @resolved?))))

View file

@ -25,4 +25,4 @@
(catch Exception e (catch Exception e
e)))] e)))]
(is (str/includes? (str out) "c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2")) (is (str/includes? (str out) "c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2"))
(is (str/includes? (pr-str ex) "Version or path must be provided")))) (is (str/includes? (pr-str ex) "Version must be provided" ))))

View file

@ -23,9 +23,7 @@
_ (vreset! ctx-ref ctx) _ (vreset! ctx-ref ctx)
ret (sci/binding [sci/out out ret (sci/binding [sci/out out
sci/err err] sci/err err]
(binding [*out* out (sci/eval-string* ctx test-program))]
*err* err]
(sci/eval-string* ctx test-program)))]
(assertions out err ret))) (assertions out err ret)))
(deftest pod-registry-test (deftest pod-registry-test
@ -38,4 +36,4 @@
(catch Exception e (catch Exception e
e)))] e)))]
(is (str/includes? (str out) "c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2")) (is (str/includes? (str out) "c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2"))
(is (str/includes? (pr-str ex) "Version or path must be provided")))) (is (str/includes? (pr-str ex) "Version must be provided" ))))

View file

@ -1,6 +1,5 @@
(ns babashka.pods.test-common (ns babashka.pods.test-common
(:require [clojure.java.io :as io] (:require [clojure.java.io :as io]
[clojure.string :as str]
[clojure.test :refer [is]])) [clojure.test :refer [is]]))
(def test-program (slurp (io/file "test-resources" "test_program.clj"))) (def test-program (slurp (io/file "test-resources" "test_program.clj")))
@ -10,12 +9,7 @@
;; (.println System/err out) ;; (.println System/err out)
;; (.println System/err err) ;; (.println System/err err)
(doseq [[expected actual] (doseq [[expected actual]
(map vector (replace (map vector '["pod.test-pod"
{::edn-error (if (= "edn"
(System/getenv "BABASHKA_POD_TEST_FORMAT"))
"Map literal must contain an even number of forms"
::dont-care)}
'["pod.test-pod"
pod.test-pod pod.test-pod
{:a 1, :b 2} {:a 1, :b 2}
6 6
@ -34,21 +28,15 @@
3 3
true ;; local-date true ;; local-date
true ;; roundtrip string array true ;; roundtrip string array
true ;; roundtrip metadata
true ;; roundtrip metadata nested
true ;; dont roundtrip metadata (when arg-meta "false"/ absent)
1 1
"add the arguments" "add the arguments"
nil nil
nil nil]
::edn-error])
(concat ret (repeat ::nil)))] (concat ret (repeat ::nil)))]
(cond (instance? java.util.regex.Pattern expected) (if (instance? java.util.regex.Pattern expected)
(is (re-find expected actual)) (is (re-find expected actual))
(= ::dont-care expected) nil
:else
(is (= expected actual)))) (is (= expected actual))))
(is (= "(\"hello\" \"print\" \"this\" \"debugging\" \"message\")\n:foo\n:foo\n" (str out))) (is (= "(\"hello\" \"print\" \"this\" \"debugging\" \"message\")\n:foo\n:foo\n" (str out)))
(is (str/starts-with? (str err) "(\"hello\" \"print\" \"this\" \"error\")" ))) (is (= "(\"hello\" \"print\" \"this\" \"error\")\n" (str err))))
(def pod-registry (slurp (io/file "test-resources" "pod_registry.clj"))) (def pod-registry (slurp (io/file "test-resources" "pod_registry.clj")))