luciano/coffi
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

WIP: Update documentation for arenas instead of sessions

Browse source 
This is a part of the push for completing coffi 1.0 for a release tested
against JDK 22 and newer.
This commit is contained in:
Joshua Suskalo 2024-09-30 12:17:04 -04:00
parent 7da20f74cf
commit 2073f15767
No known key found for this signature in database
GPG key ID: 9B6BA586EFF1B9F0
3 changed files with 172 additions and 285 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

135
README.md
View file

@ -1,15 +1,15 @@
# coffi
[![Clojars Project](https://img.shields.io/clojars/v/org.suskalo/coffi.svg)](https://clojars.org/org.suskalo/coffi)
Coffi is a foreign function interface library for Clojure, using the new
[Project Panama](https://openjdk.java.net/projects/panama/) that's a part of the
preview in Java 19. This allows calling native code directly from Clojure
without the need for either Java or native code specific to the library, as e.g.
the JNI does. Coffi focuses on ease of use, including functions and macros for
creating wrappers to allow the resulting native functions to act just like
Clojure ones, however this doesn't remove the ability to write systems which
minimize the cost of marshaling data and optimize for performance, to make use
of the low-level access Panama gives us.
Coffi is a foreign function interface library for Clojure, using the [Foreign
Function & Memory API](https://openjdk.org/jeps/454) in JDK 22 and later. This
allows calling native code directly from Clojure without the need for either
Java or native code specific to the library, as e.g. the JNI does. Coffi focuses
on ease of use, including functions and macros for creating wrappers to allow
the resulting native functions to act just like Clojure ones, however this
doesn't remove the ability to write systems which minimize the cost of
marshaling data and optimize for performance, to make use of the low-level
access Panama gives us.
## Installation
This library is available on Clojars. Add one of the following entries to the
@ -27,19 +27,20 @@ library.
$ clj -X:deps prep
```
Coffi requires usage of the package `java.lang.foreign`, and everything in this
package is considered to be a preview release, which are disabled by default. In
order to use coffi, add the following JVM arguments to your application.
Coffi requires usage of the package `java.lang.foreign`, and most of the
operations are considered unsafe by the JDK, and are therefore unavailable to
your code without passing some command line flags. In order to use coffi, add
the following JVM arguments to your application.
```sh
--enable-preview --enable-native-access=ALL-UNNAMED
--enable-native-access=ALL-UNNAMED
```
You can specify JVM arguments in a particular invocation of the Clojure CLI with
the -J flag like so:
``` sh
clj -J--enable-preview -J--enable-native-access=ALL-UNNAMED
clj -J--enable-native-access=ALL-UNNAMED
```
You can also specify them in an alias in your `deps.edn` file under the
@ -47,15 +48,20 @@ You can also specify them in an alias in your `deps.edn` file under the
using `-M`, `-A`, or `-X`.
``` clojure
{:aliases {:dev {:jvm-opts ["--enable-preview" "--enable-native-access=ALL-UNNAMED"]}}}
{:aliases {:dev {:jvm-opts ["--enable-native-access=ALL-UNNAMED"]}}}
```
Other build tools should provide similar functionality if you check their
documentation.
When creating an executable jar file, you can avoid the need to pass this
argument by adding the manifest attribute `Enable-Native-Access: ALL-UNNAMED` to
your jar.
Coffi also includes support for the linter clj-kondo. If you use clj-kondo and
this library's macros are not linting correctly, you may need to install the
config bundled with the library. You can do so with the following shell command:
config bundled with the library. You can do so with the following shell command,
run from your project directory:
```sh
$ clj-kondo --copy-configs --dependencies --lint "$(clojure -Spath)"
@ -193,7 +199,7 @@ be found in `coffi.layout`.
[[:a ::mem/char]
[:x ::mem/float]]]))
(mem/size-of ::needs-padding))
(mem/size-of ::needs-padding)
;; => 8
(mem/align-of ::needs-padding)
@ -331,7 +337,7 @@ Clojure code to make this easier.
native-fn
[ints]
(let [arr-len (count ints)
int-array (serialize ints [::mem/array ::mem/int arr-len])]
int-array (mem/serialize ints [::mem/array ::mem/int arr-len])]
(native-fn (mem/address-of int-array) arr-len)))
```
@ -349,38 +355,34 @@ This can be used to implement out variables often seen in native code.
"out_int" [::mem/pointer] ::mem/void
native-fn
[i]
(let [int-ptr (serialize i [::mem/pointer ::mem/int])]
(let [int-ptr (mem/serialize i [::mem/pointer ::mem/int])]
(native-fn int-ptr)
(deserialize int-ptr [::mem/pointer ::mem/int])))
(mem/deserialize int-ptr [::mem/pointer ::mem/int])))
```
### Sessions
**Before JDK 19 Sessions were called Scopes. Coffi retains functions that are
named for creating scopes for backwards compatibility, but they will be removed
in version 1.0.**
### Arenas
In order to serialize any non-primitive type (such as the previous
`[::mem/pointer ::mem/int]`), off-heap memory needs to be allocated. When memory
is allocated inside the JVM, the memory is associated with a session. Because
none was provided here, the session is an implicit session, and the memory will
be freed when the serialized object is garbage collected.
is allocated inside the JVM, the memory is associated with an arena. Because
none was provided here, the arena is an implicit arena, and the memory will be
freed when the serialized object is garbage collected.
In many cases this is not desirable, because the memory is not freed in a
deterministic manner, causing garbage collection pauses to become longer, as
well as changing allocation performance. Instead of an implicit session, there
are other kinds of sessions as well. A `stack-session` is a thread-local
session. Stack sessions are `Closeable`, which means they should usually be used
in a `with-open` form. When a `stack-session` is closed, it immediately frees
all the memory associated with it. The previous example, `out-int`, can be
implemented with a stack session.
well as changing allocation performance. Instead of an implicit arena, there
are other kinds of arenas as well. A `confined-arena` is a thread-local arena.
Confined arenas are `Closeable`, which means they should usually be used in a
`with-open` form. When a `confined-arena` is closed, it immediately frees all
the memory associated with it. The previous example, `out-int`, can be
implemented with a confined arena.
```clojure
(defcfn out-int
"out_int" [::mem/pointer] ::mem/void
native-fn
[i]
(with-open [session (mem/stack-session)]
(let [int-ptr (mem/serialize i [::mem/pointer ::mem/int] session)]
(with-open [arena (mem/confined-arena)]
(let [int-ptr (mem/serialize i [::mem/pointer ::mem/int] arena)]
(native-fn int-ptr)
(mem/deserialize int-ptr [::mem/pointer ::mem/int]))))
```
@ -388,15 +390,15 @@ implemented with a stack session.
This will free the pointer immediately upon leaving the function.
When memory needs to be accessible from multiple threads, there's
`shared-session`. When using a `shared-session`, it should be accessed inside a
`with-acquired` block. When a `shared-session` is `.close`d, it will release all
its associated memory when every `with-acquired` block associated with it is
exited.
`shared-arena`. When a `shared-arena` is `.close`d, it will release all its
associated memory immediately, and so this should only be done once all other
threads are done accessing memory associated with it.
In addition, two non-`Closeable` sessions are `global-session`, which never
frees the resources associated with it, and `connected-session`, which is a
session that frees its resources on garbage collection, like an implicit
session.
In addition, two non-`Closeable` arenas are `global-arena`, which never frees
the resources associated with it, and `auto-arena`, which is an arena that frees
its resources once all of them are unreachable during a garbage collection
cycle, like an implicit arena, but potentially for multiple allocations rather
than just one.
### Serialization and Deserialization
Custom serializers and deserializers may also be created. This is done using two
@ -426,34 +428,35 @@ serialize to primitives.
```clojure
(defmethod mem/serialize* ::vector
[obj _type session]
(mem/address-of (mem/serialize obj [::mem/array ::mem/float 3] session)))
[obj _type arena]
(mem/serialize obj [::mem/array ::mem/float 3] arena))
(defmethod mem/deserialize* ::vector
[addr _type]
(mem/deserialize (mem/slice-global addr (mem/size-of [::mem/array ::mem/float 3]))
[segment _type]
(mem/deserialize (mem/reinterpret segment (mem/size-of [::mem/array ::mem/float 3]))
[::mem/array ::mem/float 3]))
```
The `slice-global` function allows you to take an address without an associated
session and get a memory segment which can be deserialized.
The `reinterpret` function allows you to take a segment and decorate it with a
new size, and possibly associate it with an arena or add cleanup functions on
it.
In cases like this where we don't know the session of the pointer, we could use
`add-close-action!` to ensure it's freed. For example if a `free-vector!`
function that takes a pointer exists, we could use this:
In cases like this where we don't know the arena of the pointer, we could use
`reinterpret` to ensure it's freed. For example if a `free-vector!` function
that takes a pointer exists, we could use this:
```clojure
(defcfn returns-vector
"returns_vector" [] ::mem/pointer
native-fn
[session]
[arena]
(let [ret-ptr (native-fn)]
(add-close-action! session #(free-vector! ret-ptr))
(deserialize ret-ptr ::vector)))
(-> (reinterpret ret-ptr (mem/size-of ::vector) arena free-vector!)
(deserialize ::vector))))
```
This function takes a session and returns the deserialized vector, and it will
free the pointer when the session closes.
This function takes an arena and returns the deserialized vector, and it will
free the pointer when the arena closes.
#### Tagged Union
For the tagged union type, we will represent the value as a vector of a keyword
@ -505,7 +508,7 @@ deserialize the value into and out of memory segments. This is accomplished with
(map first))))
(defmethod mem/serialize-into ::tagged-union
[obj [_tagged-union tags type-map] segment session]
[obj [_tagged-union tags type-map] segment arena]
(mem/serialize-into
{:tag (item-index tags (first obj))
:value (second obj)}
@ -513,7 +516,7 @@ deserialize the value into and out of memory segments. This is accomplished with
[[:tag ::mem/long]
[:value (get type-map (first obj))]]]
segment
session))
arena))
```
This serialization method is rather simple, it just turns the vector value into
@ -570,7 +573,7 @@ it could be represented for serialization purposes like so:
This union however would not include the tag when serialized.
If a union is deserialized, then all that coffi does is to allocate a new
segment of the appropriate size with an implicit session so that it may later be
segment of the appropriate size with an implicit arena so that it may later be
garbage collected, and copies the data from the source segment into it. It's up
to the user to call `deserialize-from` on that segment with the appropriate
type.
@ -593,14 +596,14 @@ create raw function handles.
With raw handles, the argument types are expected to exactly match the types
expected by the native function. For primitive types, those are primitives. For
addresses, that is `MemoryAddress`, and for composite types like structs and
unions, that is `MemorySegment`. Both `MemoryAddress` and `MemorySegment` come
from the `java.lang.foreign` package.
pointers, that is `MemorySegment`, and for composite types like structs and
unions, that is also `MemorySegment`. `MemorySegment` comes from the
`java.lang.foreign` package.
In addition, when a raw handle returns a composite type represented with a
`MemorySegment`, it requires an additional first argument, a `SegmentAllocator`,
which can be acquired with `session-allocator` to get one associated with a
specific session. The returned value will live until that session is released.
which can be acquired with `arena-allocator` to get one associated with a
specific arena. The returned value will live until that arena is released.
In addition, function types can be specified as being raw, in the following
manner:

42
src/clj/coffi/ffi.clj
View file

@ -236,20 +236,20 @@
The return type and any arguments that are primitives will not
be (de)serialized except to be cast. If all arguments and return are
primitive, the `downcall` is returned directly. In cases where arguments must
be serialized, a new [[mem/stack-session]] is generated."
be serialized, a new [[mem/confined-arena]] is generated."
[downcall arg-types ret-type]
(let [;; Complexity of types
const-args? (or (vector? arg-types) (nil? arg-types))
simple-args? (when const-args?
(and (every? mem/primitive? arg-types)
;; NOTE(Joshua): Pointer types with serdes (e.g. [::mem/pointer ::mem/int])
;; still require a session, making them not qualify as "simple".
;; still require an arena, making them not qualify as "simple".
(every? keyword? (filter (comp #{::mem/pointer} mem/primitive-type) arg-types))))
const-ret? (s/valid? ::mem/type ret-type)
primitive-ret? (and const-ret?
(or (and (mem/primitive? ret-type)
;; NOTE(Joshua): Pointer types with serdes require deserializing the
;; return value, but don't require passing a session to the downcall,
;; return value, but don't require passing an arena to the downcall,
;; making them cause the return to not be primitive, but it may still
;; be "simple".
(or (keyword? ret-type) (not (#{::mem/pointer} (mem/primitive-type ret-type)))))
@ -265,7 +265,7 @@
~ret-type
downcall#)
(let [;; All our symbols
session (gensym "session")
arena (gensym "arena")
downcall-sym (gensym "downcall")
args-sym (when-not const-args?
(gensym "args"))
@ -284,7 +284,7 @@
(some->>
(cond
(not (s/valid? ::mem/type type))
`(mem/serialize ~sym ~type-sym ~session)
`(mem/serialize ~sym ~type-sym ~arena)
(and (mem/primitive? type)
(not (#{::mem/pointer} (mem/primitive-type type))))
@ -295,11 +295,11 @@
`(or ~sym (MemorySegment/NULL))
(mem/primitive-type type)
`(mem/serialize* ~sym ~type-sym ~session)
`(mem/serialize* ~sym ~type-sym ~arena)
:else
`(let [alloc# (mem/alloc-instance ~type-sym)]
(mem/serialize-into ~sym ~type-sym alloc# ~session)
(mem/serialize-into ~sym ~type-sym alloc# ~arena)
alloc#))
(list sym)))
@ -326,7 +326,7 @@
:else
`(let [~args-sym (map (fn [obj# type#]
(mem/serialize obj# type# ~session))
(mem/serialize obj# type# ~arena))
~args-sym ~args-types-sym)]
~expr)))
@ -335,7 +335,7 @@
;; taking restargs, and so the downcall must be applied
(-> `(~@(when (symbol? args) [`apply])
~downcall-sym
~@(when allocator? [`(mem/arena-allocator ~session)])
~@(when allocator? [`(mem/arena-allocator ~arena)])
~@(if (symbol? args)
[args]
args))
@ -358,12 +358,12 @@
:else
(deserialize-segment expr)))
wrap-session (fn [expr]
`(with-open [~session (mem/stack-session)]
wrap-arena (fn [expr]
`(with-open [~arena (mem/confined-arena)]
~expr))
wrap-fn (fn [call needs-session?]
wrap-fn (fn [call needs-arena?]
`(fn [~@(if const-args? arg-syms ['& args-sym])]
~(cond-> call needs-session? wrap-session)))]
~(cond-> call needs-arena? wrap-arena)))]
`(let [;; NOTE(Joshua): To ensure all arguments are evaluated once and
;; in-order, they must be bound here
~downcall-sym ~downcall
@ -395,15 +395,15 @@
[downcall arg-types ret-type]
(if (mem/primitive-type ret-type)
(fn native-fn [& args]
(with-open [session (mem/stack-session)]
(with-open [arena (mem/confined-arena)]
(mem/deserialize*
(apply downcall (map #(mem/serialize %1 %2 session) args arg-types))
(apply downcall (map #(mem/serialize %1 %2 arena) args arg-types))
ret-type)))
(fn native-fn [& args]
(with-open [session (mem/stack-session)]
(with-open [arena (mem/confined-arena)]
(mem/deserialize-from
(apply downcall (mem/arena-allocator session)
(map #(mem/serialize %1 %2 session) args arg-types))
(apply downcall (mem/arena-allocator arena)
(map #(mem/serialize %1 %2 arena) args arg-types))
ret-type)))))
(defn make-serde-varargs-wrapper
@ -530,13 +530,13 @@
(defn- upcall-serde-wrapper
"Creates a function that wraps `f` which deserializes the arguments and
serializes the return type in the [[global-session]]."
serializes the return type in the [[global-arena]]."
[f arg-types ret-type]
(fn [& args]
(mem/serialize
(apply f (map mem/deserialize args arg-types))
ret-type
(mem/global-session))))
(mem/global-arena))))
(defmethod mem/serialize* ::fn
[f [_fn arg-types ret-type & {:keys [raw-fn?]}] arena]
@ -608,7 +608,7 @@
(mem/serialize-into
newval (.-type static-var)
(.-seg static-var)
(mem/global-session))
(mem/global-arena))
newval)
(defn fswap!

280
src/clj/coffi/mem.clj
View file

@ -1,5 +1,5 @@
(ns coffi.mem
"Functions for managing native allocations, memory sessions, and (de)serialization.
"Functions for managing native allocations, memory arenas, and (de)serialization.
For any new type to be implemented, three multimethods must be overriden, but
which three depends on the native representation of the type.
@ -34,6 +34,7 @@
ValueLayout$OfFloat
ValueLayout$OfDouble)
(java.lang.ref Cleaner)
(java.util.function Consumer)
(java.nio ByteOrder)))
(set! *warn-on-reflection* true)
@ -49,24 +50,6 @@
(^Arena []
(Arena/ofConfined)))
(defn ^:deprecated stack-session
"Constructs a new session for use only in this thread.
The memory allocated within this session is cheap to allocate, like a native
stack."
(^Arena []
(confined-arena))
(^Arena [^Cleaner cleaner]
(assert false "Stack sessions with associated cleaners have been removed.")))
(defn ^:deprecated stack-scope
"Constructs a new scope for use only in this thread.
The memory allocated within this scope is cheap to allocate, like a native
stack."
^Arena []
(confined-arena))
(defn shared-arena
"Constructs a new shared memory arena.
@ -75,24 +58,6 @@
(^Arena []
(Arena/ofShared)))
(defn ^:deprecated shared-session
"Constructs a new shared memory session.
This session can be shared across threads and memory allocated in it will only
be cleaned up once every thread accessing the session closes it."
(^Arena []
(shared-arena))
(^Arena [^Cleaner cleaner]
(assert false "Shared sessions with associated cleaners have been removed.")))
(defn ^:deprecated shared-scope
"Constructs a new shared scope.
This scope can be shared across threads and memory allocated in it will only
be cleaned up once every thread accessing the scope closes it."
^Arena []
(shared-arena))
(defn auto-arena
"Constructs a new memory arena that is managed by the garbage collector.
@ -104,28 +69,6 @@
^Arena []
(Arena/ofAuto))
(defn ^:deprecated connected-session
"Constructs a new memory session to reclaim all connected resources at once.
The session may be shared across threads, and all resources created with it
will be cleaned up at the same time, when all references have been collected.
This type of session cannot be closed, and therefore should not be created in
a [[with-open]] clause."
^Arena []
(auto-arena))
(defn ^:deprecated connected-scope
"Constructs a new scope to reclaim all connected resources at once.
The scope may be shared across threads, and all resources created with it will
be cleaned up at the same time, when all references have been collected.
This type of scope cannot be closed, and therefore should not be created in
a [[with-open]] clause."
^Arena []
(auto-arena))
(defn global-arena
"Constructs the global arena, which will never reclaim its resources.
@ -136,64 +79,16 @@
^Arena []
(Arena/global))
(defn ^:deprecated global-session
"Constructs the global session, which will never reclaim its resources.
This session may be shared across threads, but is intended mainly in cases
where memory is allocated with [[alloc]] but is either never freed or whose
management is relinquished to a native library, such as when returned from a
callback."
^Arena []
(global-arena))
(defn ^:deprecated global-scope
"Constructs the global scope, which will never reclaim its resources.
This scope may be shared across threads, but is intended mainly in cases where
memory is allocated with [[alloc]] but is either never freed or whose
management is relinquished to a native library, such as when returned from a
callback."
^Arena []
(global-arena))
(defn arena-allocator
"Constructs a [[SegmentAllocator]] from the given [[Arena]].
This is primarily used when working with unwrapped downcall functions. When a
downcall function returns a non-primitive type, it must be provided with an
allocator."
^SegmentAllocator [^Arena scope]
^SegmentAllocator [^Arena arena]
(reify SegmentAllocator
(^MemorySegment allocate [_this ^long byte-size ^long byte-alignment]
(.allocate scope ^long byte-size ^long byte-alignment))))
(defn ^:deprecated session-allocator
"Constructs a segment allocator from the given `session`.
This is primarily used when working with unwrapped downcall functions. When a
downcall function returns a non-primitive type, it must be provided with an
allocator."
^SegmentAllocator [^Arena session]
(arena-allocator session))
(defn ^:deprecated scope-allocator
"Constructs a segment allocator from the given `scope`.
This is primarily used when working with unwrapped downcall functions. When a
downcall function returns a non-primitive type, it must be provided with an
allocator."
^SegmentAllocator [^Arena scope]
(arena-allocator scope))
(defn ^:deprecated segment-session
"Gets the memory session used to construct the `segment`."
^Arena [^MemorySegment segment]
(assert false "Segment sessions no longer provide linkes to the arenas that allocated them."))
(defn segment-scope
"Gets the scope associated with the `segment`."
^MemorySegment$Scope [segment]
(.scope ^MemorySegment segment))
(.allocate arena ^long byte-size ^long byte-alignment))))
(defn alloc
"Allocates `size` bytes.
@ -210,19 +105,6 @@
(^MemorySegment [allocator size alignment]
(.allocate ^SegmentAllocator allocator (long size) (long alignment))))
(defmacro ^:deprecated with-acquired
"Acquires one or more `sessions` until the `body` completes.
This is only necessary to do on shared sessions, however if you are operating
on an arbitrary passed session, it is best practice to wrap code that
interacts with it wrapped in this."
{:style/indent 1}
[sessions & body]
(assert false "Support was removed for keeping a shared arena open."))
(s/fdef with-acquired
:args (s/cat :sessions any?
:body (s/* any?)))
(defn address-of
"Gets the address of a given segment.
@ -249,32 +131,34 @@
(^MemorySegment [segment offset size]
(.asSlice ^MemorySegment segment (long offset) (long size))))
(defn ^:deprecated slice-into
"Get a slice into the `segment` starting at the `address`."
(^MemorySegment [address segment]
(.asSlice ^MemorySegment segment (address-of address)))
(^MemorySegment [address segment size]
(.asSlice ^MemorySegment segment (address-of address) (long size))))
(defn reinterpret
"Reinterprets the `segment` as having the passed `size`.
(defn ^:deprecated with-offset
"Get a new address `offset` from the old `address`."
^MemorySegment [address offset]
(slice address offset))
;; TODO(Joshua): Figure out if this can be replicated with [[Cleaner]]
#_
(defn add-close-action!
"Adds a 0-arity function to be run when the `session` closes."
[^MemorySession session ^Runnable action]
(.addCloseAction session action)
nil)
If `arena` is passed, the scope of the `segment` is associated with the arena,
as well as its access constraints. If `cleanup` is passed, it will be a
1-argument function of a fresh memory segment backed by the same memory as the
returned segment which should perform any required cleanup operations. It will
be called when the `arena` is closed."
(^MemorySegment [^MemorySegment segment size]
(.reinterpret segment (long size) (auto-arena) nil))
(^MemorySegment [^MemorySegment segment size ^Arena arena]
(.reinterpret segment (long size) arena nil))
(^MemorySegment [^MemorySegment segment size ^Arena arena cleanup]
(.reinterpret segment (long size) arena
(reify Consumer
(accept [_this segment]
(cleanup segment))))))
(defn as-segment
"Dereferences an `address` into a memory segment associated with the `session`."
(^MemorySegment [^MemorySegment address size]
(.reinterpret (MemorySegment/ofAddress address) (long size) (connected-session) nil))
(^MemorySegment [^MemorySegment address size session]
(.reinterpret (MemorySegment/ofAddress address) (long size) session nil)))
"Dereferences an `address` into a memory segment associated with the `arena` (default global)."
(^MemorySegment [^long address]
(MemorySegment/ofAddress address))
(^MemorySegment [^long address size]
(reinterpret (MemorySegment/ofAddress address) size))
(^MemorySegment [^long address size ^Arena arena]
(reinterpret (MemorySegment/ofAddress address) (long size) arena nil))
(^MemorySegment [^long address size ^Arena arena cleanup]
(reinterpret (MemorySegment/ofAddress address) (long size) arena cleanup)))
(defn copy-segment
"Copies the content to `dest` from `src`.
@ -285,9 +169,9 @@
(defn clone-segment
"Clones the content of `segment` into a new segment of the same size."
(^MemorySegment [segment] (clone-segment segment (connected-session)))
(^MemorySegment [^MemorySegment segment session]
(copy-segment ^MemorySegment (alloc (.byteSize segment) session) segment)))
(^MemorySegment [segment] (clone-segment segment (auto-arena)))
(^MemorySegment [^MemorySegment segment ^Arena arena]
(copy-segment ^MemorySegment (alloc (.byteSize segment) arena) segment)))
(defn slice-segments
"Constructs a lazy seq of `size`-length memory segments, sliced from `segment`."
@ -938,67 +822,67 @@
"Constructs a serialized version of the `obj` and returns it.
Any new allocations made during the serialization should be tied to the given
`session`, except in extenuating circumstances.
`arena`, except in extenuating circumstances.
This method should only be implemented for types that serialize to primitives."
(fn
#_{:clj-kondo/ignore [:unused-binding]}
[obj type session]
[obj type arena]
(type-dispatch type)))
(defmethod serialize* :default
[obj type _session]
[obj type _arena]
(throw (ex-info "Attempted to serialize a non-primitive type with primitive methods"
{:type type
:object obj})))
(defmethod serialize* ::byte
[obj _type _session]
[obj _type _arena]
(byte obj))
(defmethod serialize* ::short
[obj _type _session]
[obj _type _arena]
(short obj))
(defmethod serialize* ::int
[obj _type _session]
[obj _type _arena]
(int obj))
(defmethod serialize* ::long
[obj _type _session]
[obj _type _arena]
(long obj))
(defmethod serialize* ::char
[obj _type _session]
[obj _type _arena]
(char obj))
(defmethod serialize* ::float
[obj _type _session]
[obj _type _arena]
(float obj))
(defmethod serialize* ::double
[obj _type _session]
[obj _type _arena]
(double obj))
(defmethod serialize* ::pointer
[obj type session]
[obj type arena]
(if-not (null? obj)
(if (sequential? type)
(let [segment (alloc-instance (second type) session)]
(serialize-into obj (second type) segment session)
(let [segment (alloc-instance (second type) arena)]
(serialize-into obj (second type) segment arena)
(address-of segment))
obj)
(MemorySegment/NULL)))
(defmethod serialize* ::void
[_obj _type _session]
[_obj _type _arena]
nil)
(defmulti serialize-into
"Writes a serialized version of the `obj` to the given `segment`.
Any new allocations made during the serialization should be tied to the given
`session`, except in extenuating circumstances.
`arena`, except in extenuating circumstances.
This method should be implemented for any type which does not
override [[c-layout]].
@ -1007,61 +891,61 @@
the result value into the `segment`."
(fn
#_{:clj-kondo/ignore [:unused-binding]}
[obj type segment session]
[obj type segment arena]
(type-dispatch type)))
(defmethod serialize-into :default
[obj type segment session]
[obj type segment arena]
(if-some [prim-layout (primitive-type type)]
(serialize-into (serialize* obj type session) prim-layout segment session)
(serialize-into (serialize* obj type arena) prim-layout segment arena)
(throw (ex-info "Attempted to serialize an object to a type that has not been overridden"
{:type type
:object obj}))))
(defmethod serialize-into ::byte
[obj _type segment _session]
[obj _type segment _arena]
(write-byte segment (byte obj)))
(defmethod serialize-into ::short
[obj type segment _session]
[obj type segment _arena]
(if (sequential? type)
(write-short segment 0 (second type) (short obj))
(write-short segment (short obj))))
(defmethod serialize-into ::int
[obj type segment _session]
[obj type segment _arena]
(if (sequential? type)
(write-int segment 0 (second type) (int obj))
(write-int segment (int obj))))
(defmethod serialize-into ::long
[obj type segment _session]
[obj type segment _arena]
(if (sequential? type)
(write-long segment 0 (second type) (long obj))
(write-long segment (long obj))))
(defmethod serialize-into ::char
[obj _type segment _session]
[obj _type segment _arena]
(write-char segment (char obj)))
(defmethod serialize-into ::float
[obj type segment _session]
[obj type segment _arena]
(if (sequential? type)
(write-float segment 0 (second type) (float obj))
(write-float segment (float obj))))
(defmethod serialize-into ::double
[obj type segment _session]
[obj type segment _arena]
(if (sequential? type)
(write-double segment 0 (second type) (double obj))
(write-double segment (double obj))))
(defmethod serialize-into ::pointer
[obj type segment session]
[obj type segment arena]
(write-address
segment
(cond-> obj
(sequential? type) (serialize* type session))))
(sequential? type) (serialize* type arena))))
(defn serialize
"Serializes an arbitrary type.
@ -1069,12 +953,12 @@
For types which have a primitive representation, this serializes into that
representation. For types which do not, it allocates a new segment and
serializes into that."
([obj type] (serialize obj type (connected-session)))
([obj type session]
([obj type] (serialize obj type (auto-arena)))
([obj type arena]
(if (primitive-type type)
(serialize* obj type session)
(let [segment (alloc-instance type session)]
(serialize-into obj type segment session)
(serialize* obj type arena)
(let [segment (alloc-instance type arena)]
(serialize-into obj type segment arena)
segment))))
(declare deserialize deserialize*)
@ -1227,7 +1111,7 @@
(c-layout type))
(defmethod serialize-into ::raw
[obj _type segment _session]
[obj _type segment _arena]
(if (instance? MemorySegment obj)
(copy-segment segment obj)
obj))
@ -1245,9 +1129,9 @@
::pointer)
(defmethod serialize* ::c-string
[obj _type ^Arena session]
[obj _type ^Arena arena]
(if obj
(.allocateFrom session ^String obj)
(.allocateFrom arena ^String obj)
(MemorySegment/NULL)))
(defmethod deserialize* ::c-string
@ -1264,7 +1148,7 @@
(into-array MemoryLayout items))))
(defmethod serialize-into ::union
[obj [_union _types & {:keys [dispatch extract]} :as type] segment session]
[obj [_union _types & {:keys [dispatch extract]} :as type] segment arena]
(when-not dispatch
(throw (ex-info "Attempted to serialize a union with no dispatch function"
{:type type
@ -1276,7 +1160,7 @@
obj)
type
segment
session)))
arena)))
(defmethod deserialize-from ::union
[segment type]
@ -1293,7 +1177,7 @@
(into-array MemoryLayout fields))))
(defmethod serialize-into ::struct
[obj [_struct fields] segment session]
[obj [_struct fields] segment arena]
(loop [offset 0
fields fields]
(when (seq fields)
@ -1301,7 +1185,7 @@
size (size-of type)]
(serialize-into
(get obj field) type
(slice segment offset size) session)
(slice segment offset size) arena)
(recur (long (+ offset size)) (rest fields))))))
(defmethod deserialize-from ::struct
@ -1327,7 +1211,7 @@
(MemoryLayout/paddingLayout size))
(defmethod serialize-into ::padding
[_obj [_padding _size] _segment _session]
[_obj [_padding _size] _segment _arena]
nil)
(defmethod deserialize-from ::padding
@ -1343,9 +1227,9 @@
(c-layout type)))
(defmethod serialize-into ::array
[obj [_array type count] segment session]
[obj [_array type count] segment arena]
(dorun
(map #(serialize-into %1 type %2 session)
(map #(serialize-into %1 type %2 arena)
obj
(slice-segments (slice segment 0 (* count (size-of type)))
(size-of type)))))
@ -1390,10 +1274,10 @@
variants))))
(defmethod serialize* ::enum
[obj [_enum variants & {:keys [repr]}] session]
[obj [_enum variants & {:keys [repr]}] arena]
(serialize* ((enum-variants-map variants) obj)
(or repr ::int)
session))
arena))
(defmethod deserialize* ::enum
[obj [_enum variants & {:keys [_repr]}]]
@ -1408,9 +1292,9 @@
::int))
(defmethod serialize* ::flagset
[obj [_flagset bits & {:keys [repr]}] session]
[obj [_flagset bits & {:keys [repr]}] arena]
(let [bits-map (enum-variants-map bits)]
(reduce #(bit-set %1 (get bits-map %2)) (serialize* 0 (or repr ::int) session) obj)))
(reduce #(bit-set %1 (get bits-map %2)) (serialize* 0 (or repr ::int) arena) obj)))
(defmethod deserialize* ::flagset
[obj [_flagset bits & {:keys [repr]}]]
@ -1442,8 +1326,8 @@
[_type#]
(primitive-type aliased#))
(defmethod serialize* ~new-type
[obj# _type# session#]
(serialize* obj# aliased# session#))
[obj# _type# arena#]
(serialize* obj# aliased# arena#))
(defmethod deserialize* ~new-type
[obj# _type#]
(deserialize* obj# aliased#)))
@ -1452,8 +1336,8 @@
[_type#]
(c-layout aliased#))
(defmethod serialize-into ~new-type
[obj# _type# segment# session#]
(serialize-into obj# aliased# segment# session#))
[obj# _type# segment# arena#]
(serialize-into obj# aliased# segment# arena#))
(defmethod deserialize-from ~new-type
[segment# _type#]
(deserialize-from segment# aliased#)))))