Update changelog

This reverts commit c1fa08586a.

.clj-kondo/config.edn

@@ -1 +1,2 @@
-{:config-paths ["org.suskalo/coffi"]}
+{:config-paths ["org.suskalo/coffi"]
+ :linters {:clojure-lsp/unused-public-var {:level :off}}}

.envrc

@@ -0,0 +1 @@
+use flake

.gitignore

@@ -14,3 +14,4 @@
 /.socket-repl-port
 .hgignore
 .hg/
+/.direnv

CHANGELOG.md

@@ -1,6 +1,145 @@
 # Change Log
 All notable changes to this project will be documented in this file. This change log follows the conventions of [keepachangelog.com](http://keepachangelog.com/).
 
+## [Unreleased]
+### Added
+- New `coffi.mem/defstruct` macro to allow the definition of struct types with more performant serdes
+- Support for named union members in c-layout (thanks to @jjttjj)
+
+## [1.0.486] - 2024-10-04
+### Fixed
+- Bug where one too many indirections is used when serializing/deserializing pointer types
+
+## [1.0.472] - 2024-10-03
+### Added
+- New `coffi.mem/null` var for implementing custom types
+
+### Performance
+- Upcall functions serialized from functions returned by deserializing function pointers now use the backing function pointer directly
+- Upcall and downcall classes have been changed to be memoized, meaning ASM is no longer invoked every time a function is serialized, which should drastically improve performance where functions are serialized in a hot loop
+
+### Fixed
+- Incorrect docstring on `coffi.mem/address-of` that implied some type of pointer type was returned rather than a long
+- Usage of deprecated `(Class/STATIC_FIELD)` access pattern
+
+## [1.0.450] - 2024-10-02
+### Added
+- Support for JDK 22
+- `reinterpret` function which changes the size associated with a segment, optionally associating it with an arena and cleanup action
+
+### Changed
+- Arglists and docstrings of functions to refer to arenas rather than the outdated terms scope or session
+- Change the arguments to `as-segment` to take longs to account for the removal of an Address type
+
+### Removed
+- Deprecated functions referring to sessions and scopes
+- Deprecated functions `slice-into` and `with-offset`, replaced by the function `slice`
+
+### Fixed
+- Prep step when using coffi as a dependency wouldn't re-run if it failed during execution, e.g. when using the incorrect java version
+
+## [0.6.409] - 2023-03-31
+### Added
+- Support for JDK 19
+- New macros for defining vars with values from native code
+- New function to allow getting the backing memory segment of a `coffi.ffi.StaticVariable`, to replace the `Addressable` implementation lost in the migration to JDK 18
+
+### Fixed
+- Bug where `static-variable`s with primitive types would not deserialize properly on `deref`
+- Uses of `defvar` not compiling
+- Bug where nil values would not be correctly coerced to null pointers when passed to inlined functions
+- Bug where inline serde functions would fail on complex pointer types
+- Bug where padding in structs may be increased when fields have alignments less than their size
+- Bug where pointer alignment was incorrectly defined
+
+### Changed
+- References to `scope` as a term have been changed to `session` to match Panama messaging. Where this conflicts with function names, old versions have been deprecated and new names have been introduced.
+
+## [0.5.357] - 2022-07-07
+### Removed
+- `:coffi.mem/long-long` primitive type
+- `coffi.mem/slice-into`; the function no longer has an equivalent in panama, but see 2-arity of `coffi.mem/as-segment` for an alternative
+
+### Changed
+- `coffi.mem/as-segment` no longer has a close action arity
+- JDK version from 17 to 18
+
+## [0.4.341] - 2022-01-23
+### Added
+- Constants for size and alignment of primitive types
+- Support for non-native byte orders of primitive types
+- Functions for reading and writing primitive types (e.g. `coffi.mem/read-float`, `coffi.mem/write-long`, etc.)
+- Layout objects may now be passed to `coffi.mem/size-of` and `coffi.mem/align-of`
+- Constants for native-order primitive layouts
+- Constants for byte orders
+
+### Changed
+- The `coffi.mem/primitive?` predicate is now actually a function instead of a set
+
+## [0.3.298] - 2022-01-10
+### Added
+- New `coffi.layout` namespace with support for forcing C layout rules on structs
+
+### Fixed
+- C-characters were being read as UTF-16 rather than ASCII code points
+
+## [0.2.277] - 2021-10-25
+### Fixed
+- Non-primitive arguments on upcalls would generate invalid bytecode with `nil` instructions
+
+## [0.2.259] - 2021-10-16
+### Fixed
+- Long and double arguments to upcalls failed to compile in some cases
+- Void return types on upcalls would crash on serialization
+
+## [0.1.251] - 2021-10-14
+### Fixed
+- Bug with the inline expansion of `make-serde-wrapper`, make it more maintainable
+
+## [0.1.246] - 2021-10-14
+### Fixed
+- Incorrect inline expansion of `make-serde-wrapper` in cases where a function has no arguments
+
+## [0.1.241] - 2021-10-14
+### Performance
+- Added an `:inline` function to `make-serde-wrapper` to remove serialization overhead on primitives
+- Added multimethod implementations for primitives in (de)serialization functions, rather than using the default
+
+### Fixed
+- `cfn` didn't add serializers with non-primitive types in some cases
+
+## [0.1.220] - 2021-10-09
+### Fixed
+- All-primitive method types still used serialization when called from `cfn`
+- Arrays deserialized to non-vector sequences
+- Non-primitive argument types fail to link
+
+## [0.1.205] - 2021-10-06
+### Added
+- An `address?` predicate
+
+### Fixed
+- Compound types caused problems in arglists meta on expansion of `defcfn`
+- Compound types were not allowed as return types in `defcfn`
+- `nil` was not considered a null pointer
+- Primitive-serializing types fail to compile as arguments to downcall handles
+- Primitive-serializing types fail to load as arguments to upcall functions
+- Void return types on upcalls crash the JVM
+- Invalid implementation of serialize-into for primitive types
+
+## [0.1.192] - 2021-09-30
+### Added
+- An `::ffi/address` key to wrapper functions' metadata
+
+### Fixed
+- Usage of a method no longer in Panama that breaks `with-acquired`
+
+## [0.1.184] - 2021-09-30
+### Fixed
+- Deserializing nullpointers as functions threw an exception
+- Upcall stubs with non-primitive arguments failed to compile
+- Upcall stubs had incorrect types
+
 ## [0.1.176] - 2021-09-29
 ### Fixed
 - Usage of `defcfn` without a docstring produced an invalid `def` form
@@ -15,5 +154,22 @@ All notable changes to this project will be documented in this file. This change
 - Support for serializing and deserializing arbitrary Clojure functions
 - Support for serializing and deserializing arbitrary Clojure data structures
 
-[Unreleased]: https://github.com/IGJoshua/coffi/compare/v0.1.169...HEAD
+[Unreleased]: https://github.com/IGJoshua/coffi/compare/v1.0.486...develop
+[1.0.486]: https://github.com/IGJoshua/coffi/compare/v1.0.472...v1.0.486
+[1.0.472]: https://github.com/IGJoshua/coffi/compare/v1.0.450...v1.0.472
+[1.0.450]: https://github.com/IGJoshua/coffi/compare/v0.6.409...v1.0.450
+[0.6.409]: https://github.com/IGJoshua/coffi/compare/v0.5.357...v0.6.409
+[0.5.357]: https://github.com/IGJoshua/coffi/compare/v0.4.341...v0.5.357
+[0.4.341]: https://github.com/IGJoshua/coffi/compare/v0.3.298...v0.4.341
+[0.3.298]: https://github.com/IGJoshua/coffi/compare/v0.2.277...v0.3.298
+[0.2.277]: https://github.com/IGJoshua/coffi/compare/v0.2.259...v0.2.277
+[0.2.259]: https://github.com/IGJoshua/coffi/compare/v0.1.251...v0.2.259
+[0.1.251]: https://github.com/IGJoshua/coffi/compare/v0.1.246...v0.1.251
+[0.1.246]: https://github.com/IGJoshua/coffi/compare/v0.1.241...v0.1.246
+[0.1.241]: https://github.com/IGJoshua/coffi/compare/v0.1.220...v0.1.241
+[0.1.220]: https://github.com/IGJoshua/coffi/compare/v0.1.205...v0.1.220
+[0.1.205]: https://github.com/IGJoshua/coffi/compare/v0.1.192...v0.1.205
+[0.1.192]: https://github.com/IGJoshua/coffi/compare/v0.1.184...v0.1.192
+[0.1.184]: https://github.com/IGJoshua/coffi/compare/v0.1.176...v0.1.184
+[0.1.176]: https://github.com/IGJoshua/coffi/compare/v0.1.169...v0.1.176
 [0.1.169]: https://github.com/IGJoshua/coffi/compare/16f56bc31d69142ec4d2fb61b15b069d78b127ca...v0.1.169

README.md

@@ -1,24 +1,28 @@
 # coffi
-[![cljdoc badge](https://cljdoc.org/badge/org.suskalo/coffi)](https://cljdoc.org/d/org.suskalo/coffi/CURRENT)
 [![Clojars Project](https://img.shields.io/clojars/v/org.suskalo/coffi.svg)](https://clojars.org/org.suskalo/coffi)
+[![cljdoc badge](https://cljdoc.org/badge/org.suskalo/coffi)](https://cljdoc.org/d/org.suskalo/coffi)
 
-Coffi is a foreign function interface library for Clojure, using the new
+Coffi is a foreign function interface library for Clojure, using the [Foreign
-[Project Panama](https://openjdk.java.net/projects/panama/) that's a part of the
+Function & Memory API](https://openjdk.org/jeps/454) in JDK 22 and later. This
-incubator in Java 17. This allows calling native code directly from Clojure
+allows calling native code directly from Clojure without the need for either
-without the need for either Java or native code specific to the library, as e.g.
+Java or native code specific to the library, as e.g. the JNI does. Coffi focuses
-the JNI does. Coffi focuses on ease of use, including functions and macros for
+on ease of use, including functions and macros for creating wrappers to allow
-creating wrappers to allow the resulting native functions to act just like
+the resulting native functions to act just like Clojure ones, however this
-Clojure ones, however this doesn't remove the ability to write systems which
+doesn't remove the ability to write systems which minimize the cost of
-minimize the cost of marshaling data and optimize for performance, to make use
+marshaling data and optimize for performance, to make use of the low-level
-of the low-level access Panama gives us.
+access the FF&M API gives us.
+
+- [Getting Started](https://cljdoc.org/d/org.suskalo/coffi/CURRENT/doc/getting-started)
+- [API Documentation](https://cljdoc.org/d/org.suskalo/coffi/CURRENT/api/coffi)
+- [Recent Changes](CHANGELOG.md)
 
 ## Installation
-This library is available on Clojars. Add one of the following entries to the
+This library is available on Clojars, or as a git dependency. Add one of the
-`:deps` key of your `deps.edn`:
+following entries to the `:deps` key of your `deps.edn`:
 
 ```clojure
-org.suskalo/coffi {:mvn/version "0.1.176"}
+org.suskalo/coffi {:mvn/version "1.0.486"}
-io.github.IGJoshua/coffi {:git/tag "v0.1.169" :git/sha "7ec2748"}
+io.github.IGJoshua/coffi {:git/tag "v1.0.486" :git/sha "c61090c"}
 ```
 
 If you use this library as a git dependency, you will need to prepare the
@@ -28,26 +32,54 @@ library.
 $ clj -X:deps prep
 ```
 
+Coffi requires usage of the package `java.lang.foreign`, and most of the
+operations are considered unsafe by the JDK, and are therefore unavailable to
+your code without passing some command line flags. In order to use coffi, add
+the following JVM arguments to your application.
+
+```sh
+--enable-native-access=ALL-UNNAMED
+```
+
+You can specify JVM arguments in a particular invocation of the Clojure CLI with
+the `-J` flag like so:
+
+``` sh
+clj -J--enable-native-access=ALL-UNNAMED
+```
+
+You can also specify them in an alias in your `deps.edn` file under the
+`:jvm-opts` key (see the next example) and then invoking the CLI with that alias
+using `-M`, `-A`, or `-X`.
+
+``` clojure
+{:aliases {:dev {:jvm-opts ["--enable-native-access=ALL-UNNAMED"]}}}
+```
+
+Other build tools should provide similar functionality if you check their
+documentation.
+
+When creating an executable jar file, you can avoid the need to pass this
+argument by adding the manifest attribute `Enable-Native-Access: ALL-UNNAMED` to
+your jar. See your build tool's documentation for how to add this.
+
 Coffi also includes support for the linter clj-kondo. If you use clj-kondo and
 this library's macros are not linting correctly, you may need to install the
-config bundled with the library. You can do so with the following shell command:
+config bundled with the library. You can do so with the following shell command,
+run from your project directory:
 
 ```sh
 $ clj-kondo --copy-configs --dependencies --lint "$(clojure -Spath)"
 ```
 
-And then adding `"org.suskalo/coffi"` to the `:config-paths` key in your
-`.clj-kondo/config.edn` file.
-
 ## Usage
-There are two major components to coffi and interacting with native code:
+The two main namespaces are `coffi.mem` which provides functions for allocating
-manipulating off-heap memory, and loading native code for use with Clojure.
+and manipulating off-heap memory and (de)serializing values, and `coffi.ffi`
-
+which can load native libraries, declare native function wrappers, and
-In the simplest cases, the native functions you call will work exclusively with
+(de)serialize functions as callbacks.
-built-in types, for example the function `strlen` from libc.
 
 ```clojure
-(require '[coffi.mem :as mem :refer [defalias]])
+(require '[coffi.mem :as mem])
 (require '[coffi.ffi :as ffi :refer [defcfn]])
 
 (defcfn strlen
@@ -56,545 +88,74 @@ built-in types, for example the function `strlen` from libc.
 
 (strlen "hello")
 ;; => 5
-```
 
-The first argument to `defcfn` is the name of the Clojure var that will hold the
-native function reference, followed by an optional docstring and attribute map,
-then the C function identifier, including the name of the native symbol, a
-vector of argument types, and the return type.
-
-If you wish to use a native function as an anonymous function, it can be done
-with the `cfn` function.
-
-```clojure
-((ffi/cfn "strlen" [::mem/c-string] ::mem/long) "hello")
-;; => 5
-```
-
-If you want to use functions from libraries other than libc, then you'll need to
-load them. Two functions are provided for this, `load-system-library`, and
-`load-library`. `load-system-library` takes a string which represents the name
-of a library that should be loaded via system lookup.
-
-```clojure
 (ffi/load-system-library "z")
 ```
 
-This will load libz from the appropriate place on the user's load path.
+In the `coffi.mem` namespace there are types for all the signed primitive
-
+numeric types in C, plus `::mem/pointer` and `::mem/c-string`, and ways to use
-Alternatively, `load-library` takes a file path to a dynamically loaded library.
+malli-like type declarations to define structs, unions, arrays, enums, and
-
+flagsets.
-```clojure
+
-(ffi/load-library "lib/libz.so")
+## Alternatives
-```
+This library is not the only Clojure library providing access to native code. In
-
+addition the following libraries (among others) exist:
-This will load libz from the lib subdirectory of the current working directory.
+
-As you can see this requires the entire filename, including platform-specific
+- [dtype-next](https://github.com/cnuernber/dtype-next)
-file extensions.
+- [tech.jna](https://github.com/techascent/tech.jna)
-
+- [clojure-jna](https://github.com/Chouser/clojure-jna)
-If a library is attempted to be loaded but doesn't exist or otherwise can't be
+
-loaded, an exception is thrown. This can be convenient as any namespace with a
+Dtype-next has support for Java versions 8-15, 17+, and GraalVM, but is focused
-`load-library` call at the top level cannot be required without the library
+strongly on array-based programming, as well as being focused on keeping memory
-being able to be loaded.
+in the native side rather than marshaling data to and from Clojure-native
-
+structures. In Java 17+, this uses the Foreign Function & Memory API (a part of
-### Primitive Types
+Project Panama until stabilization in JDK 22), while in other Java versions it
-Coffi defines a basic set of primitive types:
+uses JNA.
-- byte
+
-- short
+Tech.jna and clojure-jna both use the JNA library in all cases, and neither
-- int
+provide explicit support for callbacks. JNA allows the use of
-- long
+`java.nio.ByteBuffer`s to pass structs by value, and both libraries provide ways
-- long-long
+to use this by-value construction to call by-reference apis.
-- char
+
-- float
+An additional alternative to coffi is to directly use the JNI, which is the
-- double
+longest-standing method of wrapping native code in the JVM, but comes with the
-- pointer
+downside that it requires you to write both native and Java code to use, even if
-
+you only intend to use it from Clojure.
-Each of these types maps to their C counterpart. Values of any of these
+
-primitive types except for `pointer` will be cast with their corresponding
+If your application needs to be able to run in earlier versions of the JVM than
-Clojure function (with `long-long` mapping to the `long` function) when they are
+22, you should consider these other options. Dtype-next provides the most robust
-passed as arguments to native functions. Additionally, the `c-string` type is
+support for native code, but if you are wrapping a simple library then the other
-defined, although it is not primitive.
+libraries may be more appealing, as they have a smaller API surface area and
-
+it's easier to wrap functions.
-### Composite Types
+
-In addition, some composite types are also defined in coffi, including struct
+There is also a [third party round up](https://docs.google.com/spreadsheets/d/1ViLHNUgrO2osh2AH0h7MaCaXz8g0UpLbyWojY5f10kk/edit?gid=332155605#gid=332155605)
-and union types (unions will be discussed with serialization and
+of FFI options for Clojure.
-deserialization). For an example c struct and function:
-
-```c
-typedef struct point {
-    float x;
-    float y;
-} Point;
-
-Point zero(void) {
-    Point res = {};
-
-    res.x = 0.0;
-    res.y = 0.0;
-
-    return res;
-}
-```
-
-The corresponding coffi definition is like so:
-
-```clojure
-(defcfn zero-point
-  "zero" [] [::mem/struct [[:x ::mem/float] [:y ::mem/float]]])
-
-(zero-point)
-;; => {:x 0.0,
-;;     :y 0.0}
-```
-
-Writing out struct definitions like this every time would get tedious, so the
-macro `defalias` is used to define a struct alias.
-
-```clojure
-(defalias ::point
-  [::mem/struct
-   [[:x ::mem/float]
-    [:y ::mem/float]]])
-
-(defcfn zero-point
-  "zero" [] ::point)
-```
-
-In cases where a pointer to some data is required to pass as an argument to a
-native function, but dosn't need to be read back in, the `pointer` primitive
-type can take a type argument.
-
-```clojure
-[::mem/pointer ::mem/int]
-```
-
-Arrays are also supported via a type argument. Keep in mind that they are the
-array itself, and not a pointer to the array like you might see in certain cases
-in C.
-
-```clojure
-[::mem/array ::mem/int 3]
-```
-
-### Callbacks
-In addition to these composite types, there is also support for Clojure
-functions.
-
-```clojure
-[::ffi/fn [::mem/c-string] ::mem/int]
-```
-
-Be aware though that if an exception is thrown out of a callback that is called
-from C, the JVM will crash. The resulting crash log should include the exception
-type and message in the registers section, but it's important to be aware of all
-the same. Ideally you should test your callbacks before actually passing them to
-native code.
-
-### Variadic Functions
-Some native functions can take any number of arguments, and in these cases coffi
-provides `vacfn-factory` (for "varargs C function factory").
-
-```clojure
-(def printf-factory (ffi/vacfn-factory "printf" [::mem/c-string] ::mem/int))
-```
-
-This returns a function of the types of the rest of the arguments which itself
-returns a native function wrapper.
-
-```clojure
-(def print-int (printf-factory ::mem/int))
-
-(print-int "Some integer: %d\n" 5)
-;; Some integer: 5
-```
-
-At the moment there is no equivalent to `defcfn` for varargs functions.
-
-Some native functions that are variadic use the type `va_list` to make it easier
-for other languages to call them in their FFI. At the time of writing, coffi
-does not support va-list, however it is a planned feature.
-
-### Global Variables
-Some libraries include global variables or constants accessible through symbols.
-To start with, constant values stored in symbols can be fetched with `const`
-
-```clojure
-(def some-const (ffi/const "some_const" ::mem/int))
-```
-
-This value is fetched once when you call `const` and is turned into a Clojure
-value. If you need to refer to a global variable, then `static-variable` can be
-used to create a reference to the native value.
-
-```clojure
-(def some-var (ffi/static-variable "some_var" ::mem/int))
-```
-
-This variable is an `IDeref`. Each time you dereference it, the value will be
-deserialized from the native memory and returned. Additional functions are
-provided for mutating the variable.
-
-```clojure
-(ffi/freset! some-var 5)
-;; => 5
-@some-var
-;; => 5
-```
-
-Be aware however that there is no synchronization on these types. The value
-being read is not read atomically, so you may see an inconsistent state if the
-value is being mutated on another thread.
-
-A parallel function `fswap!` is also provided, but it does not provide any
-atomic semantics either.
-
-### Complex Wrappers
-Some functions require more complex code to map nicely to a Clojure function.
-The `defcfn` macro provides facilities to wrap the native function with some
-Clojure code to make this easier.
-
-```clojure
-(defcfn takes-array
-  "takes_array_with_count" [::mem/pointer ::mem/long] ::mem/void
-  native-fn
-  [ints]
-  (let [arr-len (count ints)
-        int-array (serialize ints [::mem/array ::mem/int arr-len]
-    (native-fn (mem/address-of int-array) arr-len))]))
-```
-
-The symbol `native-fn` can be any unqualified symbol, and names the native
-function being wrapped. It must be called in the function body below if you want
-to call the native code.
-
-This `serialize` function has a paired `deserialize`, and allows marshaling
-Clojure data back and forth to native data structures.
-
-This can be used to implement out variables often seen in native code.
-
-```clojure
-(defcfn out-int
-  "out_int" [::mem/pointer] ::mem/void
-  native-fn
-  [i]
-  (let [int-ptr (serialize i [::mem/pointer ::mem/int])]
-    (native-fn int-ptr)
-    (deserialize int-ptr [::mem/pointer ::mem/int])))
-```
-
-### Scopes
-In order to serialize any non-primitive type (such as the previous
-`[::mem/pointer ::mem/int]`), off-heap memory needs to be allocated. When memory
-is allocated inside the JVM, the memory is associated with a scope. Because none
-was provided here, the scope is an implicit scope, and the memory will be freed
-when the serialized object is garbage collected.
-
-In many cases this is not desirable, because the memory is not freed in a
-deterministic manner, causing garbage collection pauses to become longer, as
-well as changing allocation performance. Instead of an implicit scope, there are
-other kinds of scopes as well. A `stack-scope` is a thread-local scope. Stack
-scopes are `Closeable`, which means they should usually be used in a `with-open`
-form. When a `stack-scope` is closed, it immediately frees all the memory
-associated with it. The previous example, `out-int`, can be implemented with a
-stack scope.
-
-```clojure
-(defcfn out-int
-  "out_int" [::mem/pointer] ::mem/void
-  native-fn
-  [i]
-  (with-open [scope (mem/stack-scope)]
-    (let [int-ptr (mem/serialize i [::mem/pointer ::mem/int] scope)]
-      (native-fn int-ptr)
-      (mem/deserialize int-ptr [::mem/pointer ::mem/int]))))
-```
-
-This will free the pointer immediately upon leaving the function.
-
-When memory needs to be accessible from multiple threads, there's
-`shared-scope`. When using a `shared-scope`, it should be accessed inside a
-`with-acquired` block. When a `shared-scope` is `.close`d, it will release all
-its associated memory when every `with-acquired` block associated with it is
-exited.
-
-In addition, two non-`Closeable` scopes are `global-scope`, which never frees
-the resources associated with it, and `connected-scope`, which is a scope that
-frees its resources on garbage collection, like an implicit scope.
-
-### Serialization and Deserialization
-Custom serializers and deserializers may also be created. This is done using two
-sets of three multimethods which can be extended by the user. For any given
-type, only one set need be implemented.
-
-Two examples of custom types are given here, one is a 3d vector, and the other
-an example of a tagged union.
-
-#### Vector3
-For the vector type, it will serialize to a pointer to an array of three floats.
-
-The multimethod `primitive-type` returns the primitive type that a given type
-serializes to. For this example, it should be a pointer.
-
-```clojure
-(defmethod mem/primitive-type ::vector
-  [_type]
-  ::mem/pointer)
-```
-
-For any type which doesn't serialize to a primitive, it returns nil, and
-therefore need not be overriden.
-
-Next is `serialize*` and `deserialize*`, multimethods that work with types that
-serialize to primitives.
-
-```clojure
-(defmethod mem/serialize* ::vector
-  [obj _type scope]
-  (mem/address-of (mem/serialize obj [::mem/array ::mem/float 3] scope)))
-
-(defmethod mem/deserialize* ::vector
-  [addr _type]
-  (mem/deserialize (mem/slice-global addr (mem/size-of [::mem/array ::mem/float 3]))
-                   [::mem/array ::mem/float 3]))
-```
-
-The `slice-global` function allows you to take an address without an associated
-scope and get a memory segment which can be deserialized.
-
-In cases like this where we don't know the scope of the pointer, we could use
-`add-close-action!` to ensure it's freed. For example if a `free-vector!`
-function that takes a pointer exists, we could use this:
-
-```clojure
-(defcfn returns-vector
-  "returns_vector" [] ::mem/pointer
-  native-fn
-  [scope]
-  (let [ret-ptr (native-fn)]
-    (add-close-action! scope #(free-vector! ret-ptr))
-    (deserialize ret-ptr ::vector)))
-```
-
-This function takes a scope and returns the deserialized vector, and it will
-free the pointer when the scope closes.
-
-#### Tagged Union
-For the tagged union type, we will represent the value as a vector of a keyword
-naming the tag and the value. The type itself will need to take arguments,
-similar to `struct`. For example, if we were to represent a result type like in
-Rust, we might have the following values:
-
-```clojure
-[:ok 5]
-[:err "Invalid number format"]
-```
-
-To represent this, we can have a `tagged-union` type. For this instance of the
-result type, it may look like this:
-
-```clojure
-[::tagged-union [:ok :err] {:ok ::mem/int :err ::mem/c-string}]
-```
-
-The native representation of these objects is a struct of the tag and a union of
-the value. In order to correctly serialize the data and pass it to native code,
-we need a representation of the native layout of the data. The `c-layout`
-multimethod provides that.
-
-```clojure
-(defmethod mem/c-layout ::tagged-union
-  [[_tagged-union tags type-map]]
-  (mem/c-layout [::mem/struct
-                 [[:tag ::mem/long]
-                  [:value [::mem/union (vals type-map)]]]]))
-```
-
-Types with type arguments are represented as vectors of the type name and any
-additional arguments. The type name is what is dispatched on for the
-multimethods.
-
-Now that we have a native layout, we need to be able to serialize and
-deserialize the value into and out of memory segments. This is accomplished with
-`serialize-into` and `deserialize-from`.
-
-```clojure
-(defn item-index
-  "Gets the index of the first occurance of `item` in `coll`."
-  [coll item]
-  (first
-   (->> coll
-        (map-indexed vector)
-        (filter (comp #{item} second))
-        (map first))))
-
-(defmethod mem/serialize-into ::tagged-union
-  [obj [_tagged-union tags type-map] segment scope]
-  (mem/serialize-into
-   {:tag (item-index tags (first obj))
-    :value (second obj)}
-   [::mem/struct
-    [[:tag ::mem/long]
-     [:value (get type-map (first obj))]]]
-   segment
-   scope))
-```
-
-This serialization method is rather simple, it just turns the vector value into
-a map, and serializes it as a struct, choosing the type of the value based on
-the tag.
-
-```clojure
-(defmethod mem/deserialize-from ::tagged-union
-  [segment [_tagged-union tags type-map]]
-  (let [tag (mem/deserialize-from segment ::mem/long)]
-    [(nth tags tag)
-     (mem/deserialize-from
-      (mem/slice segment (mem/size-of ::mem/long))
-      (get type-map tag))]))
-```
-
-Deserialization is a little more complex. First the tag is retrieved from the
-beginning of the segment, and then the type of the value is decided based on
-that before it is deserialized.
-
-### Unions
-In the last section the custom serialization and deserialization of a tagged
-union used a union from coffi in order to define the native layout, but not for
-actual serialization or deserialization. This is intentional. A union in coffi
-is rather limited. It can be serialized, but not deserialized without external
-information.
-
-```clojure
-[::mem/union
- #{::mem/float ::mem/double}
- :dispatch #(cond
-             (float? %) ::mem/float
-             (double? %) ::mem/double)]
-```
-
-This is a minimal union in coffi. If the `:dispatch` keyword argument is not
-passed, then the union cannot be serialized, as coffi would not know which type
-to serialize the values as. In the example with a tagged union, a dispatch
-function was not provided because the type was only used for the native layout.
-
-In addition to a dispatch function, when serializing a union an extract function
-may also be provided. In the case of the value in the tagged union from before,
-it could be represented for serialization purposes like so:
-
-```clojure
-[::mem/union
- #{::mem/int ::mem/c-string}
- :dispatch #(case (first %)
-              :ok ::mem/int
-              :err ::mem/c-string)
- :extract second]
-```
-
-This union however would not include the tag when serialized.
-
-If a union is deserialized, then all that coffi does is to allocate a new
-segment of the appropriate size with an implicit scope so that it may later be
-garbage collected, and copies the data from the source segment into it. It's up
-to the user to call `deserialize-from` on that segment with the appropriate
-type.
-
-### Unwrapped Native Handles
-Sometimes the overhead brought by the automatic serialization and
-deserialization from the methods explained so far is too much. In cases like
-these, unwrapped native handles are desirable.
-
-The functions `make-downcall` and `make-varargs-factory` are provided to create
-these raw handles.
-
-```clojure
-(def raw-strlen (ffi/make-downcall "strlen" [::mem/c-string] ::mem/long))
-(raw-strlen (mem/serialize "hello" ::mem/c-string))
-;; => 5
-```
-
-In these cases, the argument types are expected to exactly match the types
-expected by the native function. For primitive types, those are primitives. For
-addresses, that is `MemoryAddress`, and for composite types like structs and
-unions, that is `MemorySegment`. Both `MemoryAddress` and `MemorySegment` come
-from the `jdk.incubator.foreign` package.
-
-In addition, when a raw handle returns a composite type represented with a
-`MemorySegment`, it requires an additional first argument, a `SegmentAllocator`,
-which can be acquired with `scope-allocator` to get one associated with a
-specific scope. The returned value will live until that scope is released.
-
-In addition, function types can be specified as being raw, in the following
-manner:
-
-```clojure
-[::ffi/fn [::mem/int] ::mem/int :raw-fn? true]
-```
-
-Clojure functions serialized to this type will have their arguments and return
-value exactly match the types specified and will not perform any serialization
-or deserialization at their boundaries.
-
-### Data Model
-In addition to the macros and functions provided to build a Clojure API for
-native libraries, facilities are provided for taking data and loading all the
-symbols specified by it. This can be useful if a library provides (or an
-external provider maintains) a data representation of their API, as Clojure data
-to represent it may be programmatically generated from these sources.
-
-The data to represent an API is a map with the following form:
-
-```clojure
-(def strlen-libspec
-  {:strlen {:type :function
-            :symbol "strlen"
-            :function/args [::mem/c-string]
-            :function/ret ::mem/long}})
-```
-
-Each key in this map represents a single symbol to be loaded. The value is a map
-with at least the keys `:type` and `:symbol`. These are the currently recognized
-types:
-
-- function
-- varargs-factory
-- const
-- static-var
-
-Each one has its own set of additional keys which can be added to the map. Both
-`function` and `varargs-factory` have the three keys `:function/args`,
-`:function/ret`, and `:function/raw-fn?`. The `const` type has `:const/type` and
-`static-var` has `:static-var/type`.
-
-This data can be passed to the function `reify-libspec`, which will take the
-data and return a map from the same keys as the input map to whatever value is
-appropriate for a given symbol type (e.g. a Clojure function for `function`, a
-value for `const`, etc.).
-
-```clojure
-(ffi/reify-libspec strlen-libspec)
-;; => {:strlen #function[...]}
-```
-
-This functionality can be extended by specifying new types as implementations of
-the multimethod `reify-symbolspec`, although it's recommended that for any
-library authors who do so, namespaced keywords be used to name types.
 
 ## Known Issues
 The project author is aware of these issues and plans to fix them in a future
 release:
 
-There are currently no known issues! Hooray!
+- When generating docs with codox in a library that depends on coffi, the below error will be produced. A temporary workaround is to add an explicit dependency in your codox build on insn at version 0.2.1
+  ```
+  Unable to find static field: ACC_OPEN in interface org.objectweb.asm.Opcodes
+  ```
 
 ## Future Plans
 These features are planned for future releases.
 
 - Support for va_args type
-- Functions for wrapping structs in padding following various standards
+- Header parsing tool for generating a data model? (maybe just work with [clong](https://github.com/phronmophobic/clong)?)
-- Header parsing tool for generating a data model?
 - Generic type aliases
+- Unsigned integer types
+- Record-based struct types
+- Helper macro for out arguments
+- Improve error messages from defcfn macro
+- Mapped memory
+- Helper macros for custom serde implementations for composite data types (this is in progress [for structs](https://github.com/IGJoshua/coffi/issues/12)!)
+- Support for GraalVM Native Image (once their support for FFM becomes mature)
 
 ## License
 
-Copyright © 2021 Joshua Suskalo
+Copyright © 2023 Joshua Suskalo
 
 Distributed under the Eclipse Public License version 1.0.

build.clj

@@ -17,7 +17,7 @@
    [clojure.tools.build.api :as b]))
 
 (def lib-coord 'org.suskalo/coffi)
-(def version (format "0.1.%s" (b/git-count-revs nil)))
+(def version (format "1.0.%s" (b/git-count-revs nil)))
 
 (def resource-dirs ["resources/"])
 
@@ -49,11 +49,13 @@
   "Compiles java classes required for interop."
   [opts]
   (.mkdirs (io/file class-dir))
-  (b/process {:command-args ["javac" "--add-modules=jdk.incubator.foreign"
+  (let [compilation-result
+        (b/process {:command-args ["javac"
                                    "src/java/coffi/ffi/Loader.java"
                                    "-d" class-dir
-                             "-target" "17"
+                                   "--release" "22"]})]
-                             "-source" "17"]})
+    (when-not (zero? (:exit compilation-result))
+      (b/delete {:path class-dir})))
   opts)
 
 (defn- write-pom

deps.edn

@@ -1,6 +1,6 @@
 {:paths ["src/clj" "target/classes" "resources"]
- :deps {org.clojure/clojure {:mvn/version "1.10.3"}
+ :deps {org.clojure/clojure {:mvn/version "1.11.1"}
-        insn/insn {:mvn/version "0.2.1"}}
+        insn/insn {:mvn/version "0.5.4"}}
 
  :deps/prep-lib {:alias :build
                  :fn build/compile-java
@@ -9,18 +9,31 @@
  :aliases
  {:dev {:extra-paths ["."]
         :extra-deps {io.github.clojure/tools.build {:git/tag "v0.3.0" :git/sha "e418fc9"}
-                     nodisassemble/nodisassemble {:mvn/version "0.1.3"}}
+                     nodisassemble/nodisassemble {:mvn/version "0.1.3"}
+                     criterium/criterium {:mvn/version "0.4.6"}}
         ;; NOTE(Joshua): If you want to use nodisassemble you should also add a
         ;; -javaagent for the resolved location
-        :jvm-opts ["--add-modules=jdk.incubator.foreign" "--enable-native-access=ALL-UNNAMED"]}
+        :jvm-opts ["--enable-native-access=ALL-UNNAMED"]}
   :test {:extra-paths ["test/clj"]
          :extra-deps {org.clojure/test.check {:mvn/version "1.1.0"}
                       io.github.cognitect-labs/test-runner
                       {:git/url "https://github.com/cognitect-labs/test-runner"
                        :sha "62ef1de18e076903374306060ac0e8a752e57c86"}}
-         :jvm-opts ["--add-modules=jdk.incubator.foreign" "--enable-native-access=ALL-UNNAMED"]
+         :jvm-opts ["--enable-native-access=ALL-UNNAMED"]
          :exec-fn cognitect.test-runner.api/test}
 
+  :codox {:extra-deps {codox/codox {:mvn/version "0.10.8"}
+                       insn/insn {:mvn/version "0.2.1"}}
+          :exec-fn codox.main/generate-docs
+          :exec-args {:name "coffi"
+                      :version "v1.0.486"
+                      :description "A Foreign Function Interface in Clojure for JDK 22+."
+                      :source-paths  ["src/clj"]
+                      :doc-paths ["docs/articles"]
+                      :output-path "docs"
+                      :source-uri "https://github.com/IGJoshua/coffi/blob/{git-commit}/{filepath}#L{line}"
+                      :metadata {:doc/format :markdown}}}
+
   :build {:replace-deps {org.clojure/clojure {:mvn/version "1.10.3"}
                          io.github.clojure/tools.build {:git/tag "v0.3.0" :git/sha "e418fc9"}}
           :ns-default build

docs/01-Getting-Started.html

@@ -0,0 +1,178 @@
+<!DOCTYPE html PUBLIC ""
+    "">
+<html><head><meta charset="UTF-8" /><title>Getting Started</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">coffi</span> <span class="project-version">v1.0.486</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1  current"><a href="01-Getting-Started.html"><div class="inner"><span>Getting Started</span></div></a></li><li class="depth-1 "><a href="02-Memory-Management.html"><div class="inner"><span>Memory Management</span></div></a></li><li class="depth-1 "><a href="03-Builtin-Types.html"><div class="inner"><span>Built-in Types **WIP**</span></div></a></li><li class="depth-1 "><a href="04-Custom-Types.html"><div class="inner"><span>Custom Types</span></div></a></li><li class="depth-1 "><a href="05-Low-Level-Wrappers.html"><div class="inner"><span>Low-Level Wrappers</span></div></a></li><li class="depth-1 "><a href="50-Data-Model.html"><div class="inner"><span>Data Model</span></div></a></li><li class="depth-1 "><a href="99-Benchmarks.html"><div class="inner"><span>Benchmarks **OUTDATED**</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>coffi</span></div></div></li><li class="depth-2 branch"><a href="coffi.ffi.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ffi</span></div></a></li><li class="depth-2 branch"><a href="coffi.layout.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>layout</span></div></a></li><li class="depth-2"><a href="coffi.mem.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mem</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1><a href="#getting-started" id="getting-started"></a>Getting Started</h1>
+<h2><a href="#installation" id="installation"></a>Installation</h2>
+<p>This library is available on Clojars. Add one of the following entries to the <code>:deps</code> key of your <code>deps.edn</code>:</p>
+<pre><code class="language-clojure">org.suskalo/coffi {:mvn/version "x.y.z"}
+io.github.IGJoshua/coffi {:git/tag "x.y.z" :git/sha "abcdef0"}
+</code></pre>
+<p>See GitHub for the <a href="https://github.com/IGJoshua/coffi/releases">latest releases</a>.</p>
+<p>If you use this library as a git dependency, you will need to prepare the library.</p>
+<pre><code class="language-sh">$ clj -X:deps prep
+</code></pre>
+<p>Coffi requires usage of the package <code>java.lang.foreign</code>, and most of the operations are considered unsafe by the JDK, and are therefore unavailable to your code without passing some command line flags. In order to use coffi, add the following JVM arguments to your application.</p>
+<pre><code class="language-sh">--enable-native-access=ALL-UNNAMED
+</code></pre>
+<p>You can specify JVM arguments in a particular invocation of the Clojure CLI with the -J flag like so:</p>
+<pre><code class="language-sh">clj -J--enable-native-access=ALL-UNNAMED
+</code></pre>
+<p>You can also specify them in an alias in your <code>deps.edn</code> file under the <code>:jvm-opts</code> key (see the next example) and then invoking the CLI with that alias using <code>-M</code>, <code>-A</code>, or <code>-X</code>.</p>
+<pre><code class="language-clojure">{:aliases {:dev {:jvm-opts ["--enable-native-access=ALL-UNNAMED"]}}}
+</code></pre>
+<p>Other build tools should provide similar functionality if you check their documentation.</p>
+<p>When creating an executable jar file, you can avoid the need to pass this argument by adding the manifest attribute <code>Enable-Native-Access: ALL-UNNAMED</code> to your jar.</p>
+<h2><a href="#basic-usage" id="basic-usage"></a>Basic Usage</h2>
+<p>There are two major components to coffi and interacting with native code: manipulating off-heap memory, and loading native code for use with Clojure.</p>
+<p>In the simplest cases, the native functions you call will work exclusively with built-in types, for example the function <code>strlen</code> from libc.</p>
+<pre><code class="language-clojure">(require '[coffi.mem :as mem :refer [defalias]])
+(require '[coffi.ffi :as ffi :refer [defcfn]])
+
+(defcfn strlen
+  "Given a string, measures its length in bytes."
+  strlen [::mem/c-string] ::mem/long)
+
+(strlen "hello")
+;; =&gt; 5
+</code></pre>
+<p>The first argument to <code>defcfn</code> is the name of the Clojure var that will hold the native function reference, followed by an optional docstring and attribute map, then the C function identifier, including the name of the native symbol, a vector of argument types, and the return type.</p>
+<p>If you wish to use a native function as an anonymous function, it can be done with the <code>cfn</code> function.</p>
+<pre><code class="language-clojure">((ffi/cfn "strlen" [::mem/c-string] ::mem/long) "hello")
+;; =&gt; 5
+</code></pre>
+<p>If you want to use functions from libraries other than libc, then you’ll need to load them. Two functions are provided for this, <code>load-system-library</code>, and <code>load-library</code>. <code>load-system-library</code> takes a string which represents the name of a library that should be loaded via system lookup.</p>
+<pre><code class="language-clojure">(ffi/load-system-library "z")
+</code></pre>
+<p>This will load libz from the appropriate place on the user’s load path.</p>
+<p>Alternatively, <code>load-library</code> takes a file path to a dynamically loaded library.</p>
+<pre><code class="language-clojure">(ffi/load-library "lib/libz.so")
+</code></pre>
+<p>This will load libz from the lib subdirectory of the current working directory. As you can see this requires the entire filename, including platform-specific file extensions.</p>
+<p>If a library is attempted to be loaded but doesn’t exist or otherwise can’t be loaded, an exception is thrown. This can be convenient as any namespace with a <code>load-library</code> call at the top level cannot be required without the library being able to be loaded.</p>
+<h3><a href="#primitive-types" id="primitive-types"></a>Primitive Types</h3>
+<p>Coffi defines a basic set of primitive types:</p>
+<ul>
+<li>byte</li>
+<li>short</li>
+<li>int</li>
+<li>long</li>
+<li>char</li>
+<li>float</li>
+<li>double</li>
+<li>pointer</li>
+</ul>
+<p>Each of these types maps to their C counterpart. Values of any of these primitive types except for <code>pointer</code> will be cast with their corresponding Clojure function when they are passed as arguments to native functions. Additionally, the <code>c-string</code> type is defined, although it is not primitive.</p>
+<h3><a href="#composite-types" id="composite-types"></a>Composite Types</h3>
+<p>In addition, some composite types are also defined in coffi, including struct and union types (unions will be discussed with serialization and deserialization). For an example C struct and function:</p>
+<pre><code class="language-c">typedef struct point {
+    float x;
+    float y;
+} Point;
+
+Point zero(void) {
+    Point res = {};
+
+    res.x = 0.0;
+    res.y = 0.0;
+
+    return res;
+}
+</code></pre>
+<p>The corresponding coffi definition is like so:</p>
+<pre><code class="language-clojure">(defcfn zero-point
+  "zero" [] [::mem/struct [[:x ::mem/float] [:y ::mem/float]]])
+
+(zero-point)
+;; =&gt; {:x 0.0,
+;;     :y 0.0}
+</code></pre>
+<p>Writing out struct definitions like this every time would get tedious, so the macro <code>defalias</code> is used to define a struct alias.</p>
+<pre><code class="language-clojure">(defalias ::point
+  [::mem/struct
+   [[:x ::mem/float]
+    [:y ::mem/float]]])
+
+(defcfn zero-point
+  "zero" [] ::point)
+</code></pre>
+<p>Struct definitions do not include any padding by default. Functions for transforming struct types to include padding conforming to various standards can be found in <code>coffi.layout</code>.</p>
+<pre><code class="language-clojure">(require '[coffi.layout :as layout])
+
+(defalias ::needs-padding
+  (layout/with-c-layout
+   [::mem/struct
+    [[:a ::mem/char]
+     [:x ::mem/float]]]))
+
+(mem/size-of ::needs-padding)
+;; =&gt; 8
+
+(mem/align-of ::needs-padding)
+;; =&gt; 4
+</code></pre>
+<p>Values deserialized with types produced from layout functions may include an extra <code>:coffi.layout/padding</code> key with a nil value.</p>
+<p>A limitation of the <code>defcfn</code> macro in its current form is that types provided to it must be provided in a literal form, not as an expression that evaluates to a type. This means that if you wish to use a layout function on a struct you must define an alias for it before the type can be used as a type in <code>defcfn</code>.</p>
+<p>In cases where a pointer to some data is required to pass as an argument to a native function, but doesn’t need to be read back in, the <code>pointer</code> primitive type can take a type argument.</p>
+<pre><code class="language-clojure">[::mem/pointer ::mem/int]
+</code></pre>
+<p>Arrays are also supported via a type argument. Keep in mind that they are the array itself, and not a pointer to the array like you might see in certain cases in C.</p>
+<pre><code class="language-clojure">[::mem/array ::mem/int 3]
+</code></pre>
+<h3><a href="#callbacks" id="callbacks"></a>Callbacks</h3>
+<p>In addition to these composite types, there is also support for Clojure functions.</p>
+<pre><code class="language-clojure">[::ffi/fn [::mem/c-string] ::mem/int]
+</code></pre>
+<p>Be aware though that if an exception is thrown out of a callback that is called from C, the JVM will crash. The resulting crash log should include the exception type and message in the registers section, but it’s important to be aware of all the same. Ideally you should test your callbacks before actually passing them to native code.</p>
+<p>When writing a wrapper library for a C library, it may be a good choice to wrap all passed Clojure functions in an additional function which catches all throwables, potentially notifies the user in some manner (e.g. logging), and returns a default value. This is on the wrapper library’s developer to decide when and where this is appropriate, as in some cases no reasonable default return value can be determined and it is most sensible to simply crash the JVM. This is the reason that coffi defaults to this behavior, as in the author’s opinion it is better to fail hard and fast rather than to attempt to produce a default and cause unexpected behavior later.</p>
+<p>Another important thing to keep in mind is the expected lifetime of the function that you pass to native code. For example it is perfectly fine to pass an anonymous function to a native function if the callback will never be called again once the native function returns. If however it saves the callback for later use the JVM may collect it prematurely, causing a crash when the callback is later called by native code.</p>
+<h3><a href="#variadic-functions" id="variadic-functions"></a>Variadic Functions</h3>
+<p>Some native functions can take any number of arguments, and in these cases coffi provides <code>vacfn-factory</code> (for “varargs C function factory”).</p>
+<pre><code class="language-clojure">(def printf-factory (ffi/vacfn-factory "printf" [::mem/c-string] ::mem/int))
+</code></pre>
+<p>This returns a function of the types of the rest of the arguments which itself returns a native function wrapper.</p>
+<pre><code class="language-clojure">(def print-int (printf-factory ::mem/int))
+
+(print-int "Some integer: %d\n" 5)
+;; Some integer: 5
+</code></pre>
+<p>At the moment there is no equivalent to <code>defcfn</code> for varargs functions.</p>
+<p>Some native functions that are variadic use the type <code>va_list</code> to make it easier for other languages to call them in their FFI. At the time of writing, coffi does not support va-list, however it is a planned feature.</p>
+<h3><a href="#global-variables" id="global-variables"></a>Global Variables</h3>
+<p>Some libraries include global variables or constants accessible through symbols. To start with, constant values stored in symbols can be fetched with <code>const</code>, or the parallel macro <code>defconst</code></p>
+<pre><code class="language-clojure">(def some-const (ffi/const "some_const" ::mem/int))
+(ffi/defconst some-const "some_const" ::mem/int)
+</code></pre>
+<p>This value is fetched once when you call <code>const</code> and is turned into a Clojure value. If you need to refer to a global variable, then <code>static-variable</code> (or parallel <code>defvar</code>) can be used to create a reference to the native value.</p>
+<pre><code class="language-clojure">(def some-var (ffi/static-variable "some_var" ::mem/int))
+(ffi/defvar some-var "some_var" ::mem/int)
+</code></pre>
+<p>This variable is an <code>IDeref</code>. Each time you dereference it, the value will be deserialized from the native memory and returned. Additional functions are provided for mutating the variable.</p>
+<pre><code class="language-clojure">(ffi/freset! some-var 5)
+;; =&gt; 5
+@some-var
+;; =&gt; 5
+</code></pre>
+<p>Be aware however that there is no synchronization on these types. The value being read is not read atomically, so you may see an inconsistent state if the value is being mutated on another thread.</p>
+<p>A parallel function <code>fswap!</code> is also provided, but it does not provide any atomic semantics either.</p>
+<p>The memory that backs the static variable can be fetched with the function <code>static-variable-segment</code>, which can be used to pass a pointer to the static variable to native functions that require it.</p>
+<h3><a href="#complex-wrappers" id="complex-wrappers"></a>Complex Wrappers</h3>
+<p>Some functions require more complex code to map nicely to a Clojure function. The <code>defcfn</code> macro provides facilities to wrap the native function with some Clojure code to make this easier.</p>
+<pre><code class="language-clojure">(defcfn takes-array
+  "takes_array_with_count" [::mem/pointer ::mem/long] ::mem/void
+  native-fn
+  [ints]
+  (let [arr-len (count ints)
+        int-array (mem/serialize ints [::mem/array ::mem/int arr-len])]
+    (native-fn int-array arr-len)))
+</code></pre>
+<p>The symbol <code>native-fn</code> can be any unqualified symbol, and names the native function being wrapped. It must be called in the function body below if you want to call the native code.</p>
+<p>This <code>serialize</code> function has a paired <code>deserialize</code>, and allows marshaling Clojure data back and forth to native data structures.</p>
+<p>This can be used to implement out variables often seen in native code.</p>
+<pre><code class="language-clojure">(defcfn out-int
+  "out_int" [::mem/pointer] ::mem/void
+  native-fn
+  [i]
+  (let [int-ptr (mem/serialize i [::mem/pointer ::mem/int])]
+    (native-fn int-ptr)
+    (mem/deserialize int-ptr [::mem/pointer ::mem/int])))
+</code></pre>
+</div></div></div></body></html>

docs/02-Memory-Management.html

@@ -0,0 +1,18 @@
+<!DOCTYPE html PUBLIC ""
+    "">
+<html><head><meta charset="UTF-8" /><title>Memory Management</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">coffi</span> <span class="project-version">v1.0.486</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="01-Getting-Started.html"><div class="inner"><span>Getting Started</span></div></a></li><li class="depth-1  current"><a href="02-Memory-Management.html"><div class="inner"><span>Memory Management</span></div></a></li><li class="depth-1 "><a href="03-Builtin-Types.html"><div class="inner"><span>Built-in Types **WIP**</span></div></a></li><li class="depth-1 "><a href="04-Custom-Types.html"><div class="inner"><span>Custom Types</span></div></a></li><li class="depth-1 "><a href="05-Low-Level-Wrappers.html"><div class="inner"><span>Low-Level Wrappers</span></div></a></li><li class="depth-1 "><a href="50-Data-Model.html"><div class="inner"><span>Data Model</span></div></a></li><li class="depth-1 "><a href="99-Benchmarks.html"><div class="inner"><span>Benchmarks **OUTDATED**</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>coffi</span></div></div></li><li class="depth-2 branch"><a href="coffi.ffi.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ffi</span></div></a></li><li class="depth-2 branch"><a href="coffi.layout.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>layout</span></div></a></li><li class="depth-2"><a href="coffi.mem.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mem</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1><a href="#memory-management" id="memory-management"></a>Memory Management</h1>
+<p>In order to serialize any non-primitive type, off-heap memory needs to be allocated. When memory is allocated inside the JVM, the memory is associated with an arena. If none is provided, the arena is an implicit arena, and the memory will be freed when the serialized object is garbage collected.</p>
+<p>In many cases this is not desirable, because the memory is not freed in a deterministic manner, causing garbage collection pauses to become longer, as well as changing allocation performance. Instead of an implicit arena, there are other kinds of arenas as well. A <code>confined-arena</code> is a thread-local arena. Confined arenas are <code>Closeable</code>, which means they should usually be used in a <code>with-open</code> form. When a <code>confined-arena</code> is closed, it immediately frees all the memory associated with it. The previous example, <code>out-int</code>, can be implemented with a confined arena.</p>
+<pre><code class="language-clojure">(defcfn out-int
+  "out_int" [::mem/pointer] ::mem/void
+  native-fn
+  [i]
+  (with-open [arena (mem/confined-arena)]
+    (let [int-ptr (mem/serialize i [::mem/pointer ::mem/int] arena)]
+      (native-fn int-ptr)
+      (mem/deserialize int-ptr [::mem/pointer ::mem/int]))))
+</code></pre>
+<p>This will free the pointer immediately upon leaving the function.</p>
+<p>When memory needs to be accessible from multiple threads, there’s <code>shared-arena</code>. When a <code>shared-arena</code> is <code>.close</code>d, it will release all its associated memory immediately, and so this should only be done once all other threads are done accessing memory associated with it.</p>
+<p>In addition, two non-<code>Closeable</code> arenas are <code>global-arena</code>, which never frees the resources associated with it, and <code>auto-arena</code>, which is an arena that frees its resources once all of them are unreachable during a garbage collection cycle, like an implicit arena, but potentially for multiple allocations rather than just one.</p>
+</div></div></div></body></html>

docs/03-Builtin-Types.html

@@ -0,0 +1,31 @@
+<!DOCTYPE html PUBLIC ""
+    "">
+<html><head><meta charset="UTF-8" /><title>Built-in Types **WIP**</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">coffi</span> <span class="project-version">v1.0.486</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="01-Getting-Started.html"><div class="inner"><span>Getting Started</span></div></a></li><li class="depth-1 "><a href="02-Memory-Management.html"><div class="inner"><span>Memory Management</span></div></a></li><li class="depth-1  current"><a href="03-Builtin-Types.html"><div class="inner"><span>Built-in Types **WIP**</span></div></a></li><li class="depth-1 "><a href="04-Custom-Types.html"><div class="inner"><span>Custom Types</span></div></a></li><li class="depth-1 "><a href="05-Low-Level-Wrappers.html"><div class="inner"><span>Low-Level Wrappers</span></div></a></li><li class="depth-1 "><a href="50-Data-Model.html"><div class="inner"><span>Data Model</span></div></a></li><li class="depth-1 "><a href="99-Benchmarks.html"><div class="inner"><span>Benchmarks **OUTDATED**</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>coffi</span></div></div></li><li class="depth-2 branch"><a href="coffi.ffi.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ffi</span></div></a></li><li class="depth-2 branch"><a href="coffi.layout.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>layout</span></div></a></li><li class="depth-2"><a href="coffi.mem.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mem</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1><a href="#built-in-types-wip" id="built-in-types-wip"></a>Built-in Types <strong>WIP</strong></h1>
+<h3><a href="#primitives" id="primitives"></a>Primitives</h3>
+<h3><a href="#arrays" id="arrays"></a>Arrays</h3>
+<h3><a href="#pointers" id="pointers"></a>Pointers</h3>
+<h3><a href="#structs" id="structs"></a>Structs</h3>
+<h3><a href="#enums" id="enums"></a>Enums</h3>
+<h3><a href="#flagsets" id="flagsets"></a>Flagsets</h3>
+<h3><a href="#functions" id="functions"></a>Functions</h3>
+<h3><a href="#unions" id="unions"></a>Unions</h3>
+<p>Unions in coffi are rather limited. They can be serialized, but not deserialized without external information.</p>
+<pre><code class="language-clojure">[::mem/union
+ #{::mem/float ::mem/double}
+ :dispatch #(cond
+             (float? %) ::mem/float
+             (double? %) ::mem/double)]
+</code></pre>
+<p>This is a minimal union in coffi. If the <code>:dispatch</code> keyword argument is not passed, then the union cannot be serialized, as coffi would not know which type to serialize the values as. In <a href="04-Custom-Types.md#tagged-union">the example with a tagged union</a>, a dispatch function was not provided because the type was only used for the native layout.</p>
+<p>In addition to a dispatch function, when serializing a union an extract function may also be provided. In the case of the value in the tagged union from before, it could be represented for serialization purposes like so:</p>
+<pre><code class="language-clojure">[::mem/union
+ #{::mem/int ::mem/c-string}
+ :dispatch #(case (first %)
+              :ok ::mem/int
+              :err ::mem/c-string)
+ :extract second]
+</code></pre>
+<p>This union however would not include the tag when serialized.</p>
+<p>If a union is deserialized, then all that coffi does is to allocate a new segment of the appropriate size with an implicit arena so that it may later be garbage collected, and copies the data from the source segment into it. It’s up to the user to call <code>deserialize-from</code> on that segment with the appropriate type.</p>
+<h3><a href="#raw-types" id="raw-types"></a>Raw Types</h3>
+</div></div></div></body></html>

docs/04-Custom-Types.html

@@ -0,0 +1,82 @@
+<!DOCTYPE html PUBLIC ""
+    "">
+<html><head><meta charset="UTF-8" /><title>Custom Types</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">coffi</span> <span class="project-version">v1.0.486</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="01-Getting-Started.html"><div class="inner"><span>Getting Started</span></div></a></li><li class="depth-1 "><a href="02-Memory-Management.html"><div class="inner"><span>Memory Management</span></div></a></li><li class="depth-1 "><a href="03-Builtin-Types.html"><div class="inner"><span>Built-in Types **WIP**</span></div></a></li><li class="depth-1  current"><a href="04-Custom-Types.html"><div class="inner"><span>Custom Types</span></div></a></li><li class="depth-1 "><a href="05-Low-Level-Wrappers.html"><div class="inner"><span>Low-Level Wrappers</span></div></a></li><li class="depth-1 "><a href="50-Data-Model.html"><div class="inner"><span>Data Model</span></div></a></li><li class="depth-1 "><a href="99-Benchmarks.html"><div class="inner"><span>Benchmarks **OUTDATED**</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>coffi</span></div></div></li><li class="depth-2 branch"><a href="coffi.ffi.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ffi</span></div></a></li><li class="depth-2 branch"><a href="coffi.layout.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>layout</span></div></a></li><li class="depth-2"><a href="coffi.mem.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mem</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1><a href="#custom-types" id="custom-types"></a>Custom Types</h1>
+<p>Custom types with serializers and deserializers may be created. This is done using two sets of three multimethods which can be extended by the user. For any given type, only one set need be implemented.</p>
+<p>Two examples of custom types are given here, one is a 3d vector, and the other an example of a tagged union.</p>
+<h3><a href="#vector3" id="vector3"></a>Vector3</h3>
+<p>For the vector type, it will serialize to a pointer to an array of three floats.</p>
+<p>The multimethod <code>primitive-type</code> returns the primitive type that a given type serializes to. For this example, it should be a pointer.</p>
+<pre><code class="language-clojure">(defmethod mem/primitive-type ::vector
+  [_type]
+  ::mem/pointer)
+</code></pre>
+<p>For any type which doesn’t serialize to a primitive, it returns nil, and therefore need not be overriden.</p>
+<p>Next is <code>serialize*</code> and <code>deserialize*</code>, multimethods that work with types that serialize to primitives.</p>
+<pre><code class="language-clojure">(defmethod mem/serialize* ::vector
+  [obj _type arena]
+  (mem/serialize obj [::mem/array ::mem/float 3] arena))
+
+(defmethod mem/deserialize* ::vector
+  [segment _type]
+  (mem/deserialize (mem/reinterpret segment (mem/size-of [::mem/array ::mem/float 3]))
+                   [::mem/array ::mem/float 3]))
+</code></pre>
+<p>The <code>reinterpret</code> function allows you to take a segment and decorate it with a new size, and possibly associate it with an arena or add cleanup functions on it.</p>
+<p>In cases like this where we don’t know the arena of the pointer, we could use <code>reinterpret</code> to ensure it’s freed. For example if a <code>free-vector!</code> function that takes a pointer exists, we could use this:</p>
+<pre><code class="language-clojure">(defcfn returns-vector
+  "returns_vector" [] ::mem/pointer
+  native-fn
+  [arena]
+  (let [ret-ptr (native-fn)]
+    (-&gt; (reinterpret ret-ptr (mem/size-of ::vector) arena free-vector!)
+        (deserialize ::vector))))
+</code></pre>
+<p>This function takes an arena and returns the deserialized vector, and it will free the pointer when the arena closes.</p>
+<h3><a href="#tagged-union" id="tagged-union"></a>Tagged Union</h3>
+<p>For the tagged union type, we will represent the value as a vector of a keyword naming the tag and the value. The type itself will need to take arguments, similar to <code>struct</code>. For example, if we were to represent a result type like in Rust, we might have the following values:</p>
+<pre><code class="language-clojure">[:ok 5]
+[:err "Invalid number format"]
+</code></pre>
+<p>To represent this, we can have a <code>tagged-union</code> type. For this instance of the result type, it may look like this:</p>
+<pre><code class="language-clojure">[::tagged-union [:ok :err] {:ok ::mem/int :err ::mem/c-string}]
+</code></pre>
+<p>The native representation of these objects is a struct of the tag and a union of the value. In order to correctly serialize the data and pass it to native code, we need a representation of the native layout of the data. The <code>c-layout</code> multimethod provides that.</p>
+<pre><code class="language-clojure">(defmethod mem/c-layout ::tagged-union
+  [[_tagged-union tags type-map]]
+  (mem/c-layout [::mem/struct
+                 [[:tag ::mem/long]
+                  [:value [::mem/union (vals type-map)]]]]))
+</code></pre>
+<p>Types with type arguments are represented as vectors of the type name and any additional arguments. The type name is what is dispatched on for the multimethods.</p>
+<p>Now that we have a native layout, we need to be able to serialize and deserialize the value into and out of memory segments. This is accomplished with <code>serialize-into</code> and <code>deserialize-from</code>.</p>
+<pre><code class="language-clojure">(defn item-index
+  "Gets the index of the first occurance of `item` in `coll`."
+  [coll item]
+  (first
+   (-&gt;&gt; coll
+        (map-indexed vector)
+        (filter (comp #{item} second))
+        (map first))))
+
+(defmethod mem/serialize-into ::tagged-union
+  [obj [_tagged-union tags type-map] segment arena]
+  (mem/serialize-into
+   {:tag (item-index tags (first obj))
+    :value (second obj)}
+   [::mem/struct
+    [[:tag ::mem/long]
+     [:value (get type-map (first obj))]]]
+   segment
+   arena))
+</code></pre>
+<p>This serialization method is rather simple, it just turns the vector value into a map, and serializes it as a struct, choosing the type of the value based on the tag.</p>
+<pre><code class="language-clojure">(defmethod mem/deserialize-from ::tagged-union
+  [segment [_tagged-union tags type-map]]
+  (let [tag (mem/deserialize-from segment ::mem/long)]
+    [(nth tags tag)
+     (mem/deserialize-from
+      (mem/slice segment (mem/size-of ::mem/long))
+      (get type-map tag))]))
+</code></pre>
+<p>Deserialization is a little more complex. First the tag is retrieved from the beginning of the segment, and then the type of the value is decided based on that before it is deserialized.</p>
+</div></div></div></body></html>

docs/05-Low-Level-Wrappers.html

@@ -0,0 +1,51 @@
+<!DOCTYPE html PUBLIC ""
+    "">
+<html><head><meta charset="UTF-8" /><title>Low-Level Wrappers</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">coffi</span> <span class="project-version">v1.0.486</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="01-Getting-Started.html"><div class="inner"><span>Getting Started</span></div></a></li><li class="depth-1 "><a href="02-Memory-Management.html"><div class="inner"><span>Memory Management</span></div></a></li><li class="depth-1 "><a href="03-Builtin-Types.html"><div class="inner"><span>Built-in Types **WIP**</span></div></a></li><li class="depth-1 "><a href="04-Custom-Types.html"><div class="inner"><span>Custom Types</span></div></a></li><li class="depth-1  current"><a href="05-Low-Level-Wrappers.html"><div class="inner"><span>Low-Level Wrappers</span></div></a></li><li class="depth-1 "><a href="50-Data-Model.html"><div class="inner"><span>Data Model</span></div></a></li><li class="depth-1 "><a href="99-Benchmarks.html"><div class="inner"><span>Benchmarks **OUTDATED**</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>coffi</span></div></div></li><li class="depth-2 branch"><a href="coffi.ffi.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ffi</span></div></a></li><li class="depth-2 branch"><a href="coffi.layout.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>layout</span></div></a></li><li class="depth-2"><a href="coffi.mem.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mem</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1><a href="#low-level-wrappers" id="low-level-wrappers"></a>Low-Level Wrappers</h1>
+<h3><a href="#unwrapped-native-handles" id="unwrapped-native-handles"></a>Unwrapped Native Handles</h3>
+<p>Some native libraries work with handles to large amounts of data at once, making it undesirable to marshal data back and forth from Clojure, both because it’s not necessary to work with the data in Clojure directly, or also because of the high (de)serialization costs associated with marshaling. In cases like these, unwrapped native handles are desirable.</p>
+<p>The functions <code>make-downcall</code> and <code>make-varargs-factory</code> are also provided to create raw function handles.</p>
+<pre><code class="language-clojure">(def raw-strlen (ffi/make-downcall "strlen" [::mem/c-string] ::mem/long))
+(raw-strlen (mem/serialize "hello" ::mem/c-string))
+;; =&gt; 5
+</code></pre>
+<p>With raw handles, the argument types are expected to exactly match the types expected by the native function. For primitive types, those are primitives. For pointers, that is <code>MemorySegment</code>, and for composite types like structs and unions, that is also <code>MemorySegment</code>. <code>MemorySegment</code> comes from the <code>java.lang.foreign</code> package.</p>
+<p>In addition, when a raw handle returns a composite type represented with a <code>MemorySegment</code>, it requires an additional first argument, a <code>SegmentAllocator</code>, which can be acquired with <code>arena-allocator</code> to get one associated with a specific arena. The returned value will live until that arena is released.</p>
+<p>In addition, function types can be specified as being raw, in the following manner:</p>
+<pre><code class="language-clojure">[::ffi/fn [::mem/int] ::mem/int :raw-fn? true]
+</code></pre>
+<p>Clojure functions serialized to this type will have their arguments and return value exactly match the types specified and will not perform any serialization or deserialization at their boundaries.</p>
+<p>One important caveat to consider when writing wrappers for performance-sensitive functions is that the convenience macro <code>defcfn</code> that coffi provides will already perform no serialization or deserialization on primitive arguments and return types, so for functions with only primitive argument and return types there is no performance reason to choose unwrapped native handles over the convenience macro.</p>
+<h3><a href="#manual-deserialization" id="manual-deserialization"></a>Manual (De)Serialization</h3>
+<p>Coffi uses multimethods to dispatch to (de)serialization functions to enable code that’s generic over the types it operates on. However, in cases where you know the exact types that you will be (de)serializing and the multimethod dispatch overhead is too high a cost, it may be appropriate to manually handle (de)serializing data. This will often be done paired with <a href="#unwrapped-native-handles">Unwrapped Native Handles</a>.</p>
+<p>Convenience functions are provided to both read and write all primitive types and addresses, including byte order.</p>
+<p>As an example, when wrapping a function that returns an array of big-endian floats, the following code might be used.</p>
+<pre><code class="language-clojure">;; int returns_float_array(float **arr)
+(def ^:private returns-float-array* (ffi/make-downcall "returns_float_array" [::mem/pointer] ::mem/int))
+;; void releases_float_array(float *arr)
+(def ^:private release-floats* (ffi/make-downcall "releases_float_array" [::mem/pointer] ::mem/void))
+
+(defn returns-float-array
+  []
+  (with-open [arena (mem/confined-arena)]
+    ;; float *out_floats;
+    ;; int num_floats = returns_float_array(&amp;out_floats);
+    (let [out-floats (mem/alloc mem/pointer-size arena)
+          num-floats (returns-float-array* out-floats)
+          floats-addr (mem/read-address out-floats)
+          floats-slice (mem/reinterpret floats-addr (unchecked-multiply-int mem/float-size num-floats))]
+      ;; Using a try/finally to perform an operation when the stack frame exits,
+      ;; but not to try to catch anything.
+      (try
+        (loop [floats (transient [])
+               index 0]
+          (if (&gt;= index num-floats)
+            (persistent! floats)
+            (recur (conj! floats (mem/read-float floats-slice
+                                                 (unchecked-multiply-int index mem/float-size)
+                                                 mem/big-endian))
+                   (unchecked-inc-int index))))
+        (finally
+          (release-floats* floats-addr))))))
+</code></pre>
+<p>The above code manually performs all memory operations rather than relying on coffi’s dispatch. This will be more performant, but because multimethod overhead is usually relatively low, it’s recommended to use the multimethod variants for convenience in colder functions.</p>
+</div></div></div></body></html>

docs/50-Data-Model.html

@@ -0,0 +1,25 @@
+<!DOCTYPE html PUBLIC ""
+    "">
+<html><head><meta charset="UTF-8" /><title>Data Model</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">coffi</span> <span class="project-version">v1.0.486</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="01-Getting-Started.html"><div class="inner"><span>Getting Started</span></div></a></li><li class="depth-1 "><a href="02-Memory-Management.html"><div class="inner"><span>Memory Management</span></div></a></li><li class="depth-1 "><a href="03-Builtin-Types.html"><div class="inner"><span>Built-in Types **WIP**</span></div></a></li><li class="depth-1 "><a href="04-Custom-Types.html"><div class="inner"><span>Custom Types</span></div></a></li><li class="depth-1 "><a href="05-Low-Level-Wrappers.html"><div class="inner"><span>Low-Level Wrappers</span></div></a></li><li class="depth-1  current"><a href="50-Data-Model.html"><div class="inner"><span>Data Model</span></div></a></li><li class="depth-1 "><a href="99-Benchmarks.html"><div class="inner"><span>Benchmarks **OUTDATED**</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>coffi</span></div></div></li><li class="depth-2 branch"><a href="coffi.ffi.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ffi</span></div></a></li><li class="depth-2 branch"><a href="coffi.layout.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>layout</span></div></a></li><li class="depth-2"><a href="coffi.mem.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mem</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1><a href="#data-model" id="data-model"></a>Data Model</h1>
+<p>In addition to the macros and functions provided to build a Clojure API for native libraries, facilities are provided for taking data and loading all the symbols specified by it. This can be useful if a library provides (or an external provider maintains) a data representation of their API, as Clojure data to represent it may be programmatically generated from these sources.</p>
+<p>The data to represent an API is a map with the following form:</p>
+<pre><code class="language-clojure">(def strlen-libspec
+  {:strlen {:type :function
+            :symbol "strlen"
+            :function/args [::mem/c-string]
+            :function/ret ::mem/long}})
+</code></pre>
+<p>Each key in this map represents a single symbol to be loaded. The value is a map with at least the keys <code>:type</code> and <code>:symbol</code>. These are the currently recognized types:</p>
+<ul>
+<li>function</li>
+<li>varargs-factory</li>
+<li>const</li>
+<li>static-var</li>
+</ul>
+<p>Each one has its own set of additional keys which can be added to the map. Both <code>function</code> and <code>varargs-factory</code> have the three keys <code>:function/args</code>, <code>:function/ret</code>, and <code>:function/raw-fn?</code>. The <code>const</code> type has <code>:const/type</code> and <code>static-var</code> has <code>:static-var/type</code>.</p>
+<p>This data can be passed to the function <code>reify-libspec</code>, which will take the data and return a map from the same keys as the input map to whatever value is appropriate for a given symbol type (e.g. a Clojure function for <code>function</code>, a value for <code>const</code>, etc.).</p>
+<pre><code class="language-clojure">(ffi/reify-libspec strlen-libspec)
+;; =&gt; {:strlen #function[...]}
+</code></pre>
+<p>This functionality can be extended by specifying new types as implementations of the multimethod <code>reify-symbolspec</code>, although it’s recommended that for any library authors who do so, namespaced keywords be used to name types.</p>
+</div></div></div></body></html>

docs/99-Benchmarks.html

@@ -0,0 +1,258 @@
+<!DOCTYPE html PUBLIC ""
+    "">
+<html><head><meta charset="UTF-8" /><title>Benchmarks **OUTDATED**</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">coffi</span> <span class="project-version">v1.0.486</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="01-Getting-Started.html"><div class="inner"><span>Getting Started</span></div></a></li><li class="depth-1 "><a href="02-Memory-Management.html"><div class="inner"><span>Memory Management</span></div></a></li><li class="depth-1 "><a href="03-Builtin-Types.html"><div class="inner"><span>Built-in Types **WIP**</span></div></a></li><li class="depth-1 "><a href="04-Custom-Types.html"><div class="inner"><span>Custom Types</span></div></a></li><li class="depth-1 "><a href="05-Low-Level-Wrappers.html"><div class="inner"><span>Low-Level Wrappers</span></div></a></li><li class="depth-1 "><a href="50-Data-Model.html"><div class="inner"><span>Data Model</span></div></a></li><li class="depth-1  current"><a href="99-Benchmarks.html"><div class="inner"><span>Benchmarks **OUTDATED**</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>coffi</span></div></div></li><li class="depth-2 branch"><a href="coffi.ffi.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ffi</span></div></a></li><li class="depth-2 branch"><a href="coffi.layout.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>layout</span></div></a></li><li class="depth-2"><a href="coffi.mem.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mem</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1><a href="#benchmarks-outdated" id="benchmarks-outdated"></a>Benchmarks <strong>OUTDATED</strong></h1>
+<p><strong>BENCHMARKS FOR COFFI AND DTYPE-NEXT ARE BASED ON AN OLD VERSION. NEW BENCHMARKS WILL BE CREATED SOON.</strong></p>
+<p>An additional consideration when thinking about alternatives is the performance of each available option. It’s an established fact that JNA (used by all three alternative libraries on JDK &lt;16) introduces more overhead when calling native code than JNI does.</p>
+<p>In order to provide a benchmark to see how much of a difference the different native interfaces make, we can use <a href="https://github.com/hugoduncan/criterium">criterium</a> to benchmark each. <a href="https://www.glfw.org">GLFW</a>’s <a href="https://www.glfw.org/docs/latest/group__input.html#gaa6cf4e7a77158a3b8fd00328b1720a4a"><code>glfwGetTime</code></a> function will be used for the test as it performs a simple operation, and is conveniently already wrapped in JNI by the excellent <a href="https://www.lwjgl.org/">LWJGL</a> library.</p>
+<p>The following benchmarks were run on a Lenovo Thinkpad with an Intel i7-10610U running Manjaro Linux, using Clojure 1.10.3 on Java 17.</p>
+<h3><a href="#jni" id="jni"></a>JNI</h3>
+<p>The baseline for performance is the JNI. Using LWJGL it’s relatively simple to benchmark. The following Clojure CLI command will start a repl with LWJGL and criterium loaded.</p>
+<pre><code class="language-sh">$ clj -Sdeps '{:deps {org.lwjgl/lwjgl {:mvn/version "3.2.3"}
+                      org.lwjgl/lwjgl-glfw {:mvn/version "3.2.3"}
+                      org.lwjgl/lwjgl$natives-linux {:mvn/version "3.2.3"}
+                      org.lwjgl/lwjgl-glfw$natives-linux {:mvn/version "3.2.3"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}'
+</code></pre>
+<p>Then from the repl</p>
+<pre><code class="language-clojure">user=&gt; (import 'org.lwjgl.glfw.GLFW)
+org.lwjgl.glfw.GLFW
+user=&gt; (require '[criterium.core :as bench])
+nil
+user=&gt; (GLFW/glfwInit)
+true
+user=&gt; (bench/bench (GLFW/glfwGetTime) :verbose)
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: -Dclojure.basis=/home/jsusk/.clojure/.cpcache/2667074721.basis
+Evaluation count : 1613349900 in 60 samples of 26889165 calls.
+      Execution time sample mean : 32.698446 ns
+             Execution time mean : 32.697811 ns
+Execution time sample std-deviation : 1.274600 ns
+    Execution time std-deviation : 1.276437 ns
+   Execution time lower quantile : 30.750813 ns ( 2.5%)
+   Execution time upper quantile : 33.757662 ns (97.5%)
+                   Overhead used : 6.400704 ns
+nil
+</code></pre>
+<p>GLFW requires that we initialize it before calling the <code>glfwGetTime</code> function. Besides that this is a simple interop call which directly maps to the native function.</p>
+<p>This gives us a basis of 32.7 ns +/-1.3 ns. All other libraries will be evaluated relative to this result.</p>
+<p>To ensure fairness, we’ll also get that overhead value to be used in further tests.</p>
+<pre><code class="language-clojure">user=&gt; bench/estimated-overhead-cache
+6.400703613065185E-9
+</code></pre>
+<h3><a href="#coffi" id="coffi"></a>Coffi</h3>
+<p>The dependencies when using coffi are simpler, but it also requires some JVM options to support the foreign access api.</p>
+<pre><code class="language-sh">$ clj -Sdeps '{:deps {org.suskalo/coffi {:mvn/version "0.1.205"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}' \
+      -J--add-modules=jdk.incubator.foreign \
+      -J--enable-native-access=ALL-UNNAMED
+</code></pre>
+<p>In order to ensure fair comparisons, we’re going to use the same overhead value on each run, so before we do the benchmark we’ll set it to the observed value from last time.</p>
+<pre><code class="language-clojure">user=&gt; (require '[criterium.core :as bench])
+nil
+user=&gt; (alter-var-root #'bench/estimated-overhead-cache (constantly 6.400703613065185E-9))
+6.400703613065185E-9
+user=&gt; (require '[coffi.ffi :as ffi])
+nil
+user=&gt; (require '[coffi.mem :as mem])
+nil
+user=&gt; (ffi/load-system-library "glfw")
+nil
+user=&gt; ((ffi/cfn "glfwInit" [] ::mem/int))
+1
+user=&gt; (let [f (ffi/cfn "glfwGetTime" [] ::mem/double)]
+         (bench/bench (f) :verbose))
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: --add-modules=jdk.incubator.foreign --enable-native-access=ALL-UNNAMED -Dclojure.basis=/home/jsusk/.clojure/.cpcache/72793624.basis
+Evaluation count : 1657995600 in 60 samples of 27633260 calls.
+      Execution time sample mean : 31.382665 ns
+             Execution time mean : 31.386493 ns
+Execution time sample std-deviation : 1.598571 ns
+    Execution time std-deviation : 1.608818 ns
+   Execution time lower quantile : 29.761194 ns ( 2.5%)
+   Execution time upper quantile : 33.228276 ns (97.5%)
+                   Overhead used : 6.400704 ns
+nil
+</code></pre>
+<p>This result is about 1.3 ns faster, and while that is less than the standard deviation of 1.6, it’s quite close to it.</p>
+<h3><a href="#clojure-jna" id="clojure-jna"></a>Clojure-JNA</h3>
+<p>Clojure-JNA uses the JNA library, which was designed to provide Java with an easy way to access native libraries, but which is known for not having the greatest performance. Since this is an older project, I’m also including the clojure dependency to ensure the correct version is used.</p>
+<pre><code class="language-sh">$ clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.3"}
+                      net.n01se/clojure-jna {:mvn/version "1.0.0"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}'
+</code></pre>
+<p>The naive way to call the function using Clojure-JNA is to use <code>jna/invoke</code>.</p>
+<pre><code class="language-clojure">user=&gt; (require '[criterium.core :as bench])
+nil
+user=&gt; (alter-var-root #'bench/estimated-overhead-cache (constantly 6.400703613065185E-9))
+6.400703613065185E-9
+user=&gt; (require '[net.n01se.clojure-jna :as jna])
+nil
+user=&gt; (jna/invoke Integer glfw/glfwInit)
+1
+user=&gt; (bench/bench (jna/invoke Double glfw/glfwGetTime) :verbose)
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: -Dclojure.basis=/home/jsusk/.clojure/.cpcache/3229486237.basis
+Evaluation count : 195948720 in 60 samples of 3265812 calls.
+      Execution time sample mean : 350.335614 ns
+             Execution time mean : 350.373520 ns
+Execution time sample std-deviation : 24.833070 ns
+    Execution time std-deviation : 24.755929 ns
+   Execution time lower quantile : 300.000019 ns ( 2.5%)
+   Execution time upper quantile : 365.759273 ns (97.5%)
+                   Overhead used : 6.400704 ns
+
+Found 13 outliers in 60 samples (21.6667 %)
+	low-severe	 12 (20.0000 %)
+	low-mild	 1 (1.6667 %)
+ Variance from outliers : 53.4220 % Variance is severely inflated by outliers
+nil
+</code></pre>
+<p>As you can see, this method of calling functions is very bad for performance, with call overhead dominating function runtime by an order of magnitude. That said, this isn’t a completely fair comparison, nor the most realistic, because this way of calling functions looks the function up on each invocation.</p>
+<p>To adjust for this, we’ll use the <code>jna/to-fn</code> function to give a persistent handle to the function that we can call.</p>
+<pre><code class="language-clojure">user=&gt; (let [f (jna/to-fn Double glfw/glfwGetTime)]
+         (bench/bench (f) :verbose))
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: -Dclojure.basis=/home/jsusk/.clojure/.cpcache/3229486237.basis
+Evaluation count : 611095020 in 60 samples of 10184917 calls.
+      Execution time sample mean : 104.623634 ns
+             Execution time mean : 104.638406 ns
+Execution time sample std-deviation : 7.649296 ns
+    Execution time std-deviation : 7.638963 ns
+   Execution time lower quantile : 92.446016 ns ( 2.5%)
+   Execution time upper quantile : 110.258832 ns (97.5%)
+                   Overhead used : 6.400704 ns
+nil
+</code></pre>
+<p>This is much better, but is still about 3x slower than JNI, meaning the overhead from using JNA is still bigger than the function runtime.</p>
+<p>This performance penalty is still small in the scope of longer-running functions, and so may not be a concern for your application, but it is something to be aware of.</p>
+<h3><a href="#techjna" id="techjna"></a>tech.jna</h3>
+<p>The tech.jna library is similar in scope to Clojure-JNA, however was written to fit into an ecosystem of libraries meant for array-based programming for machine learning and data science.</p>
+<pre><code class="language-sh">$ clj -Sdeps '{:deps {techascent/tech.jna {:mvn/version "4.05"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}'
+</code></pre>
+<p>This library is also quite simple to use, the only slightly odd thing I’m doing here is to dereference the var outside the benchmark in order to ensure it’s an apples-to-apples comparison. We don’t want var dereference time mucking up our benchmark.</p>
+<pre><code class="language-clojure">user=&gt; (require '[criterium.core :as bench])
+nil
+user=&gt; (alter-var-root #'bench/estimated-overhead-cache (constantly 6.400703613065185E-9))
+6.400703613065185E-9
+user=&gt; (require '[tech.v3.jna :as jna])
+nil
+user=&gt; (jna/def-jna-fn "glfw" glfwInit "initialize glfw" Integer)
+#'user/glfwInit
+user=&gt; (glfwInit)
+Oct 09, 2021 10:30:50 AM clojure.tools.logging$eval1122$fn__1125 invoke
+INFO: Library glfw found at [:system "glfw"]
+1
+user=&gt; (jna/def-jna-fn "glfw" glfwGetTime "gets the time as a double since init" Double)
+#'user/glfwGetTime
+user=&gt; (let [f @#'glfwGetTime]
+         (bench/bench (f) :verbose))
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: -Dclojure.basis=/home/jsusk/.clojure/.cpcache/2910209237.basis
+Evaluation count : 323281680 in 60 samples of 5388028 calls.
+      Execution time sample mean : 203.976803 ns
+             Execution time mean : 203.818712 ns
+Execution time sample std-deviation : 14.557312 ns
+    Execution time std-deviation : 14.614080 ns
+   Execution time lower quantile : 179.732593 ns ( 2.5%)
+   Execution time upper quantile : 213.929374 ns (97.5%)
+                   Overhead used : 6.400704 ns
+nil
+</code></pre>
+<p>This version is even slower than Clojure-JNA. I’m unsure where this overhead is coming from, but I’ll admit that I haven’t looked at their implementations very closely.</p>
+<h3><a href="#dtype-next" id="dtype-next"></a>dtype-next</h3>
+<p>The library dtype-next replaced tech.jna in the toolkit of the group working on machine learning and array-based programming, and it includes support for composite data types including structs, as well as primitive functions and callbacks.</p>
+<p>In addition, dtype-next has two different ffi backends. First is JNA, which is usable on any JDK version, and is what we’ll use for the first benchmark. Second is the Java 16 version of Project Panama, which will be shown next.</p>
+<p>In order to use the dtype-next ffi with the JNA backend, the JNA library has to be included in the dependencies.</p>
+<pre><code class="language-sh">$ clj -Sdeps '{:deps {cnuernber/dtype-next {:mvn/version "8.032"}
+                      net.java.dev.jna/jna {:mvn/version "5.8.0"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}'
+</code></pre>
+<p>The dtype-next library also requires some more ceremony around declaring native functions. One advantage this has is that multiple symbols with the same name can be loaded from different shared libraries, but it also does increase friction when defining native wrappers.</p>
+<p>Some easier ways to define native wrappers are provided than what is seen here, but they share some disadvantages in documentation over the core methods provided in coffi, although they are comparable to the data model provided in coffi.</p>
+<pre><code class="language-clojure">user=&gt; (require '[criterium.core :as bench])
+nil
+user=&gt; (alter-var-root #'bench/estimated-overhead-cache (constantly 6.400703613065185E-9))
+6.400703613065185E-9
+user=&gt; (require '[tech.v3.datatype.ffi :as dt-ffi])
+nil
+user=&gt; (def fn-defs {:glfwInit {:rettype :int32} :glfwGetTime {:rettype :float64}})
+#'user/fn-defs
+user=&gt; (def library-def (dt-ffi/define-library fn-defs))
+#'user/library-def
+user=&gt; (def library-instance (dt-ffi/instantiate-library library-def "/usr/lib/libglfw.so"))
+#'user/library-instance
+user=&gt; (def init (:glfwInit @library-instance))
+#'user/init
+user=&gt; (init)
+1
+user=&gt; (let [f (:glfwGetTime @library-instance)]
+         (bench/bench (f) :verbose))
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: -Dclojure.basis=/home/jsusk/.clojure/.cpcache/643862289.basis
+Evaluation count : 710822100 in 60 samples of 11847035 calls.
+      Execution time sample mean : 90.900112 ns
+             Execution time mean : 90.919917 ns
+Execution time sample std-deviation : 6.463312 ns
+    Execution time std-deviation : 6.470108 ns
+   Execution time lower quantile : 79.817126 ns ( 2.5%)
+   Execution time upper quantile : 95.454652 ns (97.5%)
+                   Overhead used : 6.400704 ns
+nil
+</code></pre>
+<p>This version of JNA usage is significantly faster than either of the other JNA libraries, but is still substantially slower than using JNI or coffi.</p>
+<p>In addition to the JNA backend, dtype-next has a Java 16-specific backend that uses an older version of Panama. This version requires similar setup to coffi in order to run.</p>
+<pre><code class="language-sh">$ clj -Sdeps '{:deps {cnuernber/dtype-next {:mvn/version "8.032"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}' \
+      -J--add-modules=jdk.incubator.foreign \
+      -J-Dforeign.restricted=permit \
+      -J--add-opens=java.base/java.lang=ALL-UNNAMED \
+      -J-Djava.library.path=/usr/lib/x86_64-linux-gnu
+</code></pre>
+<p>The actual code to run the benchmark is identical to the last example, but is reproduced here for completeness.</p>
+<pre><code class="language-clojure">user=&gt; (require '[criterium.core :as bench])
+nil
+user=&gt; (alter-var-root #'bench/estimated-overhead-cache (constantly 6.400703613065185E-9))
+6.400703613065185E-9
+user=&gt; (require '[tech.v3.datatype.ffi :as dt-ffi])
+nil
+user=&gt; (def fn-defs {:glfwInit {:rettype :int32} :glfwGetTime {:rettype :float64}})
+#'user/fn-defs
+user=&gt; (def library-def (dt-ffi/define-library fn-defs))
+#'user/library-def
+user=&gt; (def library-instance (dt-ffi/instantiate-library library-def "/usr/lib/libglfw.so"))
+#'user/library-instance
+user=&gt; (def init (:glfwInit @library-instance))
+#'user/init
+user=&gt; (init)
+1
+user=&gt; (let [f (:glfwGetTime @library-instance)]
+         (bench/bench (f) :verbose))
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 16.0.2+7
+Runtime arguments: --add-modules=jdk.incubator.foreign -Dforeign.restricted=permit --add-opens=java.base/java.lang=ALL-UNNAMED -Djava.library.path=/usr/lib/x86_64-linux-gnu -Dclojure.basis=/home/jsusk/.clojure/.cpcache/2337051659.basis
+Evaluation count : 1588513080 in 60 samples of 26475218 calls.
+      Execution time sample mean : 58.732468 ns
+             Execution time mean : 58.647361 ns
+Execution time sample std-deviation : 9.732389 ns
+    Execution time std-deviation : 9.791738 ns
+   Execution time lower quantile : 31.318115 ns ( 2.5%)
+   Execution time upper quantile : 65.449222 ns (97.5%)
+                   Overhead used : 6.400704 ns
+
+Found 14 outliers in 60 samples (23.3333 %)
+	low-severe	 8 (13.3333 %)
+	low-mild	 4 (6.6667 %)
+	high-mild	 2 (3.3333 %)
+ Variance from outliers : 87.6044 % Variance is severely inflated by outliers
+nil
+</code></pre>
+<p>Not reproduced here, but notable for comparison, in my testing Java 16’s version of the JNI version performed about the same.</p>
+<p>This is significantly faster than the JNA version of dtype-next, but it is still slower than modern Panama. This is likely to simply be a result of optimizations and changes to the Panama API, and when dtype-next is updated to use the Java 17 version of Panama I expect it will perform in line with coffi, but this benchmark will be reproduced when this happens. Still, this shows that as it stands, coffi is the fastest FFI available to Clojure developers.</p>
+</div></div></div></body></html>

docs/articles/01-Getting-Started.md

@@ -0,0 +1,344 @@
+# Getting Started
+
+## Installation
+This library is available on Clojars. Add one of the following entries to the
+`:deps` key of your `deps.edn`:
+
+```clojure
+org.suskalo/coffi {:mvn/version "x.y.z"}
+io.github.IGJoshua/coffi {:git/tag "x.y.z" :git/sha "abcdef0"}
+```
+
+See GitHub for the [latest releases](https://github.com/IGJoshua/coffi/releases).
+
+If you use this library as a git dependency, you will need to prepare the
+library.
+
+```sh
+$ clj -X:deps prep
+```
+
+Coffi requires usage of the package `java.lang.foreign`, and most of the
+operations are considered unsafe by the JDK, and are therefore unavailable to
+your code without passing some command line flags. In order to use coffi, add
+the following JVM arguments to your application.
+
+```sh
+--enable-native-access=ALL-UNNAMED
+```
+
+You can specify JVM arguments in a particular invocation of the Clojure CLI with
+the -J flag like so:
+
+``` sh
+clj -J--enable-native-access=ALL-UNNAMED
+```
+
+You can also specify them in an alias in your `deps.edn` file under the
+`:jvm-opts` key (see the next example) and then invoking the CLI with that alias
+using `-M`, `-A`, or `-X`.
+
+``` clojure
+{:aliases {:dev {:jvm-opts ["--enable-native-access=ALL-UNNAMED"]}}}
+```
+
+Other build tools should provide similar functionality if you check their
+documentation.
+
+When creating an executable jar file, you can avoid the need to pass this
+argument by adding the manifest attribute `Enable-Native-Access: ALL-UNNAMED` to
+your jar.
+
+## Basic Usage
+There are two major components to coffi and interacting with native code:
+manipulating off-heap memory, and loading native code for use with Clojure.
+
+In the simplest cases, the native functions you call will work exclusively with
+built-in types, for example the function `strlen` from libc.
+
+```clojure
+(require '[coffi.mem :as mem :refer [defalias]])
+(require '[coffi.ffi :as ffi :refer [defcfn]])
+
+(defcfn strlen
+  "Given a string, measures its length in bytes."
+  strlen [::mem/c-string] ::mem/long)
+
+(strlen "hello")
+;; => 5
+```
+
+The first argument to `defcfn` is the name of the Clojure var that will hold the
+native function reference, followed by an optional docstring and attribute map,
+then the C function identifier, including the name of the native symbol, a
+vector of argument types, and the return type.
+
+If you wish to use a native function as an anonymous function, it can be done
+with the `cfn` function.
+
+```clojure
+((ffi/cfn "strlen" [::mem/c-string] ::mem/long) "hello")
+;; => 5
+```
+
+If you want to use functions from libraries other than libc, then you'll need to
+load them. Two functions are provided for this, `load-system-library`, and
+`load-library`. `load-system-library` takes a string which represents the name
+of a library that should be loaded via system lookup.
+
+```clojure
+(ffi/load-system-library "z")
+```
+
+This will load libz from the appropriate place on the user's load path.
+
+Alternatively, `load-library` takes a file path to a dynamically loaded library.
+
+```clojure
+(ffi/load-library "lib/libz.so")
+```
+
+This will load libz from the lib subdirectory of the current working directory.
+As you can see this requires the entire filename, including platform-specific
+file extensions.
+
+If a library is attempted to be loaded but doesn't exist or otherwise can't be
+loaded, an exception is thrown. This can be convenient as any namespace with a
+`load-library` call at the top level cannot be required without the library
+being able to be loaded.
+
+### Primitive Types
+Coffi defines a basic set of primitive types:
+
+- byte
+- short
+- int
+- long
+- char
+- float
+- double
+- pointer
+
+Each of these types maps to their C counterpart. Values of any of these
+primitive types except for `pointer` will be cast with their corresponding
+Clojure function when they are passed as arguments to native functions.
+Additionally, the `c-string` type is defined, although it is not primitive.
+
+### Composite Types
+In addition, some composite types are also defined in coffi, including struct
+and union types (unions will be discussed with serialization and
+deserialization). For an example C struct and function:
+
+```c
+typedef struct point {
+    float x;
+    float y;
+} Point;
+
+Point zero(void) {
+    Point res = {};
+
+    res.x = 0.0;
+    res.y = 0.0;
+
+    return res;
+}
+```
+
+The corresponding coffi definition is like so:
+
+```clojure
+(defcfn zero-point
+  "zero" [] [::mem/struct [[:x ::mem/float] [:y ::mem/float]]])
+
+(zero-point)
+;; => {:x 0.0,
+;;     :y 0.0}
+```
+
+Writing out struct definitions like this every time would get tedious, so the
+macro `defalias` is used to define a struct alias.
+
+```clojure
+(defalias ::point
+  [::mem/struct
+   [[:x ::mem/float]
+    [:y ::mem/float]]])
+
+(defcfn zero-point
+  "zero" [] ::point)
+```
+
+Struct definitions do not include any padding by default. Functions for
+transforming struct types to include padding conforming to various standards can
+be found in `coffi.layout`.
+
+``` clojure
+(require '[coffi.layout :as layout])
+
+(defalias ::needs-padding
+  (layout/with-c-layout
+   [::mem/struct
+    [[:a ::mem/char]
+     [:x ::mem/float]]]))
+
+(mem/size-of ::needs-padding)
+;; => 8
+
+(mem/align-of ::needs-padding)
+;; => 4
+```
+
+Values deserialized with types produced from layout functions may include an
+extra `:coffi.layout/padding` key with a nil value.
+
+A limitation of the `defcfn` macro in its current form is that types provided to
+it must be provided in a literal form, not as an expression that evaluates to a
+type. This means that if you wish to use a layout function on a struct you must
+define an alias for it before the type can be used as a type in `defcfn`.
+
+In cases where a pointer to some data is required to pass as an argument to a
+native function, but doesn't need to be read back in, the `pointer` primitive
+type can take a type argument.
+
+```clojure
+[::mem/pointer ::mem/int]
+```
+
+Arrays are also supported via a type argument. Keep in mind that they are the
+array itself, and not a pointer to the array like you might see in certain cases
+in C.
+
+```clojure
+[::mem/array ::mem/int 3]
+```
+
+### Callbacks
+In addition to these composite types, there is also support for Clojure
+functions.
+
+```clojure
+[::ffi/fn [::mem/c-string] ::mem/int]
+```
+
+Be aware though that if an exception is thrown out of a callback that is called
+from C, the JVM will crash. The resulting crash log should include the exception
+type and message in the registers section, but it's important to be aware of all
+the same. Ideally you should test your callbacks before actually passing them to
+native code.
+
+When writing a wrapper library for a C library, it may be a good choice to wrap
+all passed Clojure functions in an additional function which catches all
+throwables, potentially notifies the user in some manner (e.g. logging), and
+returns a default value. This is on the wrapper library's developer to decide
+when and where this is appropriate, as in some cases no reasonable default
+return value can be determined and it is most sensible to simply crash the JVM.
+This is the reason that coffi defaults to this behavior, as in the author's
+opinion it is better to fail hard and fast rather than to attempt to produce a
+default and cause unexpected behavior later.
+
+Another important thing to keep in mind is the expected lifetime of the function
+that you pass to native code. For example it is perfectly fine to pass an
+anonymous function to a native function if the callback will never be called
+again once the native function returns. If however it saves the callback for
+later use the JVM may collect it prematurely, causing a crash when the callback
+is later called by native code.
+
+### Variadic Functions
+Some native functions can take any number of arguments, and in these cases coffi
+provides `vacfn-factory` (for "varargs C function factory").
+
+```clojure
+(def printf-factory (ffi/vacfn-factory "printf" [::mem/c-string] ::mem/int))
+```
+
+This returns a function of the types of the rest of the arguments which itself
+returns a native function wrapper.
+
+```clojure
+(def print-int (printf-factory ::mem/int))
+
+(print-int "Some integer: %d\n" 5)
+;; Some integer: 5
+```
+
+At the moment there is no equivalent to `defcfn` for varargs functions.
+
+Some native functions that are variadic use the type `va_list` to make it easier
+for other languages to call them in their FFI. At the time of writing, coffi
+does not support va-list, however it is a planned feature.
+
+### Global Variables
+Some libraries include global variables or constants accessible through symbols.
+To start with, constant values stored in symbols can be fetched with `const`, or
+the parallel macro `defconst`
+
+```clojure
+(def some-const (ffi/const "some_const" ::mem/int))
+(ffi/defconst some-const "some_const" ::mem/int)
+```
+
+This value is fetched once when you call `const` and is turned into a Clojure
+value. If you need to refer to a global variable, then `static-variable` (or
+parallel `defvar`) can be used to create a reference to the native value.
+
+```clojure
+(def some-var (ffi/static-variable "some_var" ::mem/int))
+(ffi/defvar some-var "some_var" ::mem/int)
+```
+
+This variable is an `IDeref`. Each time you dereference it, the value will be
+deserialized from the native memory and returned. Additional functions are
+provided for mutating the variable.
+
+```clojure
+(ffi/freset! some-var 5)
+;; => 5
+@some-var
+;; => 5
+```
+
+Be aware however that there is no synchronization on these types. The value
+being read is not read atomically, so you may see an inconsistent state if the
+value is being mutated on another thread.
+
+A parallel function `fswap!` is also provided, but it does not provide any
+atomic semantics either.
+
+The memory that backs the static variable can be fetched with the function
+`static-variable-segment`, which can be used to pass a pointer to the static
+variable to native functions that require it.
+
+### Complex Wrappers
+Some functions require more complex code to map nicely to a Clojure function.
+The `defcfn` macro provides facilities to wrap the native function with some
+Clojure code to make this easier.
+
+```clojure
+(defcfn takes-array
+  "takes_array_with_count" [::mem/pointer ::mem/long] ::mem/void
+  native-fn
+  [ints]
+  (let [arr-len (count ints)
+        int-array (mem/serialize ints [::mem/array ::mem/int arr-len])]
+    (native-fn int-array arr-len)))
+```
+
+The symbol `native-fn` can be any unqualified symbol, and names the native
+function being wrapped. It must be called in the function body below if you want
+to call the native code.
+
+This `serialize` function has a paired `deserialize`, and allows marshaling
+Clojure data back and forth to native data structures.
+
+This can be used to implement out variables often seen in native code.
+
+```clojure
+(defcfn out-int
+  "out_int" [::mem/pointer] ::mem/void
+  native-fn
+  [i]
+  (let [int-ptr (mem/serialize i [::mem/pointer ::mem/int])]
+    (native-fn int-ptr)
+    (mem/deserialize int-ptr [::mem/pointer ::mem/int])))
+```

docs/articles/02-Memory-Management.md

@@ -0,0 +1,38 @@
+# Memory Management
+In order to serialize any non-primitive type, off-heap memory needs to be
+allocated. When memory is allocated inside the JVM, the memory is associated
+with an arena. If none is provided, the arena is an implicit arena, and the
+memory will be freed when the serialized object is garbage collected.
+
+In many cases this is not desirable, because the memory is not freed in a
+deterministic manner, causing garbage collection pauses to become longer, as
+well as changing allocation performance. Instead of an implicit arena, there
+are other kinds of arenas as well. A `confined-arena` is a thread-local arena.
+Confined arenas are `Closeable`, which means they should usually be used in a
+`with-open` form. When a `confined-arena` is closed, it immediately frees all
+the memory associated with it. The previous example, `out-int`, can be
+implemented with a confined arena.
+
+```clojure
+(defcfn out-int
+  "out_int" [::mem/pointer] ::mem/void
+  native-fn
+  [i]
+  (with-open [arena (mem/confined-arena)]
+    (let [int-ptr (mem/serialize i [::mem/pointer ::mem/int] arena)]
+      (native-fn int-ptr)
+      (mem/deserialize int-ptr [::mem/pointer ::mem/int]))))
+```
+
+This will free the pointer immediately upon leaving the function.
+
+When memory needs to be accessible from multiple threads, there's
+`shared-arena`. When a `shared-arena` is `.close`d, it will release all its
+associated memory immediately, and so this should only be done once all other
+threads are done accessing memory associated with it.
+
+In addition, two non-`Closeable` arenas are `global-arena`, which never frees
+the resources associated with it, and `auto-arena`, which is an arena that frees
+its resources once all of them are unreachable during a garbage collection
+cycle, like an implicit arena, but potentially for multiple allocations rather
+than just one.

docs/articles/03-Builtin-Types.md

@@ -0,0 +1,56 @@
+# Built-in Types **WIP**
+
+### Primitives
+
+### Arrays
+
+### Pointers
+
+### Structs
+
+### Enums
+
+### Flagsets
+
+### Functions
+
+### Unions
+Unions in coffi are rather limited. They can be serialized, but not deserialized
+without external information.
+
+```clojure
+[::mem/union
+ #{::mem/float ::mem/double}
+ :dispatch #(cond
+             (float? %) ::mem/float
+             (double? %) ::mem/double)]
+```
+
+This is a minimal union in coffi. If the `:dispatch` keyword argument is not
+passed, then the union cannot be serialized, as coffi would not know which type
+to serialize the values as. In [the example with a tagged
+union](04-Custom-Types.md#tagged-union), a dispatch function was not provided
+because the type was only used for the native layout.
+
+In addition to a dispatch function, when serializing a union an extract function
+may also be provided. In the case of the value in the tagged union from before,
+it could be represented for serialization purposes like so:
+
+```clojure
+[::mem/union
+ #{::mem/int ::mem/c-string}
+ :dispatch #(case (first %)
+              :ok ::mem/int
+              :err ::mem/c-string)
+ :extract second]
+```
+
+This union however would not include the tag when serialized.
+
+If a union is deserialized, then all that coffi does is to allocate a new
+segment of the appropriate size with an implicit arena so that it may later be
+garbage collected, and copies the data from the source segment into it. It's up
+to the user to call `deserialize-from` on that segment with the appropriate
+type.
+
+### Raw Types

docs/articles/04-Custom-Types.md

@@ -0,0 +1,136 @@
+# Custom Types
+Custom types with serializers and deserializers may be created. This is done
+using two sets of three multimethods which can be extended by the user. For any
+given type, only one set need be implemented.
+
+Two examples of custom types are given here, one is a 3d vector, and the other
+an example of a tagged union.
+
+### Vector3
+For the vector type, it will serialize to a pointer to an array of three floats.
+
+The multimethod `primitive-type` returns the primitive type that a given type
+serializes to. For this example, it should be a pointer.
+
+```clojure
+(defmethod mem/primitive-type ::vector
+  [_type]
+  ::mem/pointer)
+```
+
+For any type which doesn't serialize to a primitive, it returns nil, and
+therefore need not be overriden.
+
+Next is `serialize*` and `deserialize*`, multimethods that work with types that
+serialize to primitives.
+
+```clojure
+(defmethod mem/serialize* ::vector
+  [obj _type arena]
+  (mem/serialize obj [::mem/array ::mem/float 3] arena))
+
+(defmethod mem/deserialize* ::vector
+  [segment _type]
+  (mem/deserialize (mem/reinterpret segment (mem/size-of [::mem/array ::mem/float 3]))
+                   [::mem/array ::mem/float 3]))
+```
+
+The `reinterpret` function allows you to take a segment and decorate it with a
+new size, and possibly associate it with an arena or add cleanup functions on
+it.
+
+In cases like this where we don't know the arena of the pointer, we could use
+`reinterpret` to ensure it's freed. For example if a `free-vector!` function
+that takes a pointer exists, we could use this:
+
+```clojure
+(defcfn returns-vector
+  "returns_vector" [] ::mem/pointer
+  native-fn
+  [arena]
+  (let [ret-ptr (native-fn)]
+    (-> (reinterpret ret-ptr (mem/size-of ::vector) arena free-vector!)
+        (deserialize ::vector))))
+```
+
+This function takes an arena and returns the deserialized vector, and it will
+free the pointer when the arena closes.
+
+### Tagged Union
+For the tagged union type, we will represent the value as a vector of a keyword
+naming the tag and the value. The type itself will need to take arguments,
+similar to `struct`. For example, if we were to represent a result type like in
+Rust, we might have the following values:
+
+```clojure
+[:ok 5]
+[:err "Invalid number format"]
+```
+
+To represent this, we can have a `tagged-union` type. For this instance of the
+result type, it may look like this:
+
+```clojure
+[::tagged-union [:ok :err] {:ok ::mem/int :err ::mem/c-string}]
+```
+
+The native representation of these objects is a struct of the tag and a union of
+the value. In order to correctly serialize the data and pass it to native code,
+we need a representation of the native layout of the data. The `c-layout`
+multimethod provides that.
+
+```clojure
+(defmethod mem/c-layout ::tagged-union
+  [[_tagged-union tags type-map]]
+  (mem/c-layout [::mem/struct
+                 [[:tag ::mem/long]
+                  [:value [::mem/union (vals type-map)]]]]))
+```
+
+Types with type arguments are represented as vectors of the type name and any
+additional arguments. The type name is what is dispatched on for the
+multimethods.
+
+Now that we have a native layout, we need to be able to serialize and
+deserialize the value into and out of memory segments. This is accomplished with
+`serialize-into` and `deserialize-from`.
+
+```clojure
+(defn item-index
+  "Gets the index of the first occurance of `item` in `coll`."
+  [coll item]
+  (first
+   (->> coll
+        (map-indexed vector)
+        (filter (comp #{item} second))
+        (map first))))
+
+(defmethod mem/serialize-into ::tagged-union
+  [obj [_tagged-union tags type-map] segment arena]
+  (mem/serialize-into
+   {:tag (item-index tags (first obj))
+    :value (second obj)}
+   [::mem/struct
+    [[:tag ::mem/long]
+     [:value (get type-map (first obj))]]]
+   segment
+   arena))
+```
+
+This serialization method is rather simple, it just turns the vector value into
+a map, and serializes it as a struct, choosing the type of the value based on
+the tag.
+
+```clojure
+(defmethod mem/deserialize-from ::tagged-union
+  [segment [_tagged-union tags type-map]]
+  (let [tag (mem/deserialize-from segment ::mem/long)]
+    [(nth tags tag)
+     (mem/deserialize-from
+      (mem/slice segment (mem/size-of ::mem/long))
+      (get type-map tag))]))
+```
+
+Deserialization is a little more complex. First the tag is retrieved from the
+beginning of the segment, and then the type of the value is decided based on
+that before it is deserialized.

docs/articles/05-Low-Level-Wrappers.md

@@ -0,0 +1,95 @@
+# Low-Level Wrappers
+
+### Unwrapped Native Handles
+Some native libraries work with handles to large amounts of data at once, making
+it undesirable to marshal data back and forth from Clojure, both because it's
+not necessary to work with the data in Clojure directly, or also because of the
+high (de)serialization costs associated with marshaling. In cases like these,
+unwrapped native handles are desirable.
+
+The functions `make-downcall` and `make-varargs-factory` are also provided to
+create raw function handles.
+
+```clojure
+(def raw-strlen (ffi/make-downcall "strlen" [::mem/c-string] ::mem/long))
+(raw-strlen (mem/serialize "hello" ::mem/c-string))
+;; => 5
+```
+
+With raw handles, the argument types are expected to exactly match the types
+expected by the native function. For primitive types, those are primitives. For
+pointers, that is `MemorySegment`, and for composite types like structs and
+unions, that is also `MemorySegment`. `MemorySegment` comes from the
+`java.lang.foreign` package.
+
+In addition, when a raw handle returns a composite type represented with a
+`MemorySegment`, it requires an additional first argument, a `SegmentAllocator`,
+which can be acquired with `arena-allocator` to get one associated with a
+specific arena. The returned value will live until that arena is released.
+
+In addition, function types can be specified as being raw, in the following
+manner:
+
+```clojure
+[::ffi/fn [::mem/int] ::mem/int :raw-fn? true]
+```
+
+Clojure functions serialized to this type will have their arguments and return
+value exactly match the types specified and will not perform any serialization
+or deserialization at their boundaries.
+
+One important caveat to consider when writing wrappers for performance-sensitive
+functions is that the convenience macro `defcfn` that coffi provides will
+already perform no serialization or deserialization on primitive arguments and
+return types, so for functions with only primitive argument and return types
+there is no performance reason to choose unwrapped native handles over the
+convenience macro.
+
+### Manual (De)Serialization
+Coffi uses multimethods to dispatch to (de)serialization functions to enable
+code that's generic over the types it operates on. However, in cases where you
+know the exact types that you will be (de)serializing and the multimethod
+dispatch overhead is too high a cost, it may be appropriate to manually handle
+(de)serializing data. This will often be done paired with [Unwrapped Native
+Handles](#unwrapped-native-handles).
+
+Convenience functions are provided to both read and write all primitive types
+and addresses, including byte order.
+
+As an example, when wrapping a function that returns an array of big-endian
+floats, the following code might be used.
+
+``` clojure
+;; int returns_float_array(float **arr)
+(def ^:private returns-float-array* (ffi/make-downcall "returns_float_array" [::mem/pointer] ::mem/int))
+;; void releases_float_array(float *arr)
+(def ^:private release-floats* (ffi/make-downcall "releases_float_array" [::mem/pointer] ::mem/void))
+
+(defn returns-float-array
+  []
+  (with-open [arena (mem/confined-arena)]
+    ;; float *out_floats;
+    ;; int num_floats = returns_float_array(&out_floats);
+    (let [out-floats (mem/alloc mem/pointer-size arena)
+          num-floats (returns-float-array* out-floats)
+          floats-addr (mem/read-address out-floats)
+          floats-slice (mem/reinterpret floats-addr (unchecked-multiply-int mem/float-size num-floats))]
+      ;; Using a try/finally to perform an operation when the stack frame exits,
+      ;; but not to try to catch anything.
+      (try
+        (loop [floats (transient [])
+               index 0]
+          (if (>= index num-floats)
+            (persistent! floats)
+            (recur (conj! floats (mem/read-float floats-slice
+                                                 (unchecked-multiply-int index mem/float-size)
+                                                 mem/big-endian))
+                   (unchecked-inc-int index))))
+        (finally
+          (release-floats* floats-addr))))))
+```
+
+The above code manually performs all memory operations rather than relying on
+coffi's dispatch. This will be more performant, but because multimethod overhead
+is usually relatively low, it's recommended to use the multimethod variants for
+convenience in colder functions.

docs/articles/50-Data-Model.md

@@ -0,0 +1,44 @@
+# Data Model
+In addition to the macros and functions provided to build a Clojure API for
+native libraries, facilities are provided for taking data and loading all the
+symbols specified by it. This can be useful if a library provides (or an
+external provider maintains) a data representation of their API, as Clojure data
+to represent it may be programmatically generated from these sources.
+
+The data to represent an API is a map with the following form:
+
+```clojure
+(def strlen-libspec
+  {:strlen {:type :function
+            :symbol "strlen"
+            :function/args [::mem/c-string]
+            :function/ret ::mem/long}})
+```
+
+Each key in this map represents a single symbol to be loaded. The value is a map
+with at least the keys `:type` and `:symbol`. These are the currently recognized
+types:
+
+- function
+- varargs-factory
+- const
+- static-var
+
+Each one has its own set of additional keys which can be added to the map. Both
+`function` and `varargs-factory` have the three keys `:function/args`,
+`:function/ret`, and `:function/raw-fn?`. The `const` type has `:const/type` and
+`static-var` has `:static-var/type`.
+
+This data can be passed to the function `reify-libspec`, which will take the
+data and return a map from the same keys as the input map to whatever value is
+appropriate for a given symbol type (e.g. a Clojure function for `function`, a
+value for `const`, etc.).
+
+```clojure
+(ffi/reify-libspec strlen-libspec)
+;; => {:strlen #function[...]}
+```
+
+This functionality can be extended by specifying new types as implementations of
+the multimethod `reify-symbolspec`, although it's recommended that for any
+library authors who do so, namespaced keywords be used to name types.

docs/articles/99-Benchmarks.md

@@ -0,0 +1,373 @@
+# Benchmarks **OUTDATED**
+**BENCHMARKS FOR COFFI AND DTYPE-NEXT ARE BASED ON AN OLD VERSION. NEW BENCHMARKS WILL BE CREATED SOON.**
+
+An additional consideration when thinking about alternatives is the performance
+of each available option. It's an established fact that JNA (used by all three
+alternative libraries on JDK <16) introduces more overhead when calling native
+code than JNI does.
+
+In order to provide a benchmark to see how much of a difference the different
+native interfaces make, we can use
+[criterium](https://github.com/hugoduncan/criterium) to benchmark each.
+[GLFW](https://www.glfw.org)'s
+[`glfwGetTime`](https://www.glfw.org/docs/latest/group__input.html#gaa6cf4e7a77158a3b8fd00328b1720a4a)
+function will be used for the test as it performs a simple operation, and is
+conveniently already wrapped in JNI by the excellent
+[LWJGL](https://www.lwjgl.org/) library.
+
+The following benchmarks were run on a Lenovo Thinkpad with an Intel i7-10610U
+running Manjaro Linux, using Clojure 1.10.3 on Java 17.
+
+### JNI
+The baseline for performance is the JNI. Using LWJGL it's relatively simple to
+benchmark. The following Clojure CLI command will start a repl with LWJGL and
+criterium loaded.
+
+```sh
+$ clj -Sdeps '{:deps {org.lwjgl/lwjgl {:mvn/version "3.2.3"}
+                      org.lwjgl/lwjgl-glfw {:mvn/version "3.2.3"}
+                      org.lwjgl/lwjgl$natives-linux {:mvn/version "3.2.3"}
+                      org.lwjgl/lwjgl-glfw$natives-linux {:mvn/version "3.2.3"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}'
+```
+
+Then from the repl
+
+```clojure
+user=> (import 'org.lwjgl.glfw.GLFW)
+org.lwjgl.glfw.GLFW
+user=> (require '[criterium.core :as bench])
+nil
+user=> (GLFW/glfwInit)
+true
+user=> (bench/bench (GLFW/glfwGetTime) :verbose)
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: -Dclojure.basis=/home/jsusk/.clojure/.cpcache/2667074721.basis
+Evaluation count : 1613349900 in 60 samples of 26889165 calls.
+      Execution time sample mean : 32.698446 ns
+             Execution time mean : 32.697811 ns
+Execution time sample std-deviation : 1.274600 ns
+    Execution time std-deviation : 1.276437 ns
+   Execution time lower quantile : 30.750813 ns ( 2.5%)
+   Execution time upper quantile : 33.757662 ns (97.5%)
+                   Overhead used : 6.400704 ns
+nil
+```
+
+GLFW requires that we initialize it before calling the `glfwGetTime` function.
+Besides that this is a simple interop call which directly maps to the native
+function.
+
+This gives us a basis of 32.7 ns +/-1.3 ns. All other libraries will be
+evaluated relative to this result.
+
+To ensure fairness, we'll also get that overhead value to be used in further
+tests.
+
+```clojure
+user=> bench/estimated-overhead-cache
+6.400703613065185E-9
+```
+
+### Coffi
+The dependencies when using coffi are simpler, but it also requires some JVM
+options to support the foreign access api.
+
+```sh
+$ clj -Sdeps '{:deps {org.suskalo/coffi {:mvn/version "0.1.205"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}' \
+      -J--add-modules=jdk.incubator.foreign \
+      -J--enable-native-access=ALL-UNNAMED
+```
+
+In order to ensure fair comparisons, we're going to use the same overhead value
+on each run, so before we do the benchmark we'll set it to the observed value
+from last time.
+
+```clojure
+user=> (require '[criterium.core :as bench])
+nil
+user=> (alter-var-root #'bench/estimated-overhead-cache (constantly 6.400703613065185E-9))
+6.400703613065185E-9
+user=> (require '[coffi.ffi :as ffi])
+nil
+user=> (require '[coffi.mem :as mem])
+nil
+user=> (ffi/load-system-library "glfw")
+nil
+user=> ((ffi/cfn "glfwInit" [] ::mem/int))
+1
+user=> (let [f (ffi/cfn "glfwGetTime" [] ::mem/double)]
+         (bench/bench (f) :verbose))
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: --add-modules=jdk.incubator.foreign --enable-native-access=ALL-UNNAMED -Dclojure.basis=/home/jsusk/.clojure/.cpcache/72793624.basis
+Evaluation count : 1657995600 in 60 samples of 27633260 calls.
+      Execution time sample mean : 31.382665 ns
+             Execution time mean : 31.386493 ns
+Execution time sample std-deviation : 1.598571 ns
+    Execution time std-deviation : 1.608818 ns
+   Execution time lower quantile : 29.761194 ns ( 2.5%)
+   Execution time upper quantile : 33.228276 ns (97.5%)
+                   Overhead used : 6.400704 ns
+nil
+```
+
+This result is about 1.3 ns faster, and while that is less than the standard
+deviation of 1.6, it's quite close to it.
+
+### Clojure-JNA
+Clojure-JNA uses the JNA library, which was designed to provide Java with an
+easy way to access native libraries, but which is known for not having the
+greatest performance. Since this is an older project, I'm also including the
+clojure dependency to ensure the correct version is used.
+
+```sh
+$ clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.3"}
+                      net.n01se/clojure-jna {:mvn/version "1.0.0"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}'
+```
+
+The naive way to call the function using Clojure-JNA is to use `jna/invoke`.
+
+```clojure
+user=> (require '[criterium.core :as bench])
+nil
+user=> (alter-var-root #'bench/estimated-overhead-cache (constantly 6.400703613065185E-9))
+6.400703613065185E-9
+user=> (require '[net.n01se.clojure-jna :as jna])
+nil
+user=> (jna/invoke Integer glfw/glfwInit)
+1
+user=> (bench/bench (jna/invoke Double glfw/glfwGetTime) :verbose)
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: -Dclojure.basis=/home/jsusk/.clojure/.cpcache/3229486237.basis
+Evaluation count : 195948720 in 60 samples of 3265812 calls.
+      Execution time sample mean : 350.335614 ns
+             Execution time mean : 350.373520 ns
+Execution time sample std-deviation : 24.833070 ns
+    Execution time std-deviation : 24.755929 ns
+   Execution time lower quantile : 300.000019 ns ( 2.5%)
+   Execution time upper quantile : 365.759273 ns (97.5%)
+                   Overhead used : 6.400704 ns
+
+Found 13 outliers in 60 samples (21.6667 %)
+	low-severe	 12 (20.0000 %)
+	low-mild	 1 (1.6667 %)
+ Variance from outliers : 53.4220 % Variance is severely inflated by outliers
+nil
+```
+
+As you can see, this method of calling functions is very bad for performance,
+with call overhead dominating function runtime by an order of magnitude. That
+said, this isn't a completely fair comparison, nor the most realistic, because
+this way of calling functions looks the function up on each invocation.
+
+To adjust for this, we'll use the `jna/to-fn` function to give a persistent
+handle to the function that we can call.
+
+```clojure
+user=> (let [f (jna/to-fn Double glfw/glfwGetTime)]
+         (bench/bench (f) :verbose))
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: -Dclojure.basis=/home/jsusk/.clojure/.cpcache/3229486237.basis
+Evaluation count : 611095020 in 60 samples of 10184917 calls.
+      Execution time sample mean : 104.623634 ns
+             Execution time mean : 104.638406 ns
+Execution time sample std-deviation : 7.649296 ns
+    Execution time std-deviation : 7.638963 ns
+   Execution time lower quantile : 92.446016 ns ( 2.5%)
+   Execution time upper quantile : 110.258832 ns (97.5%)
+                   Overhead used : 6.400704 ns
+nil
+```
+
+This is much better, but is still about 3x slower than JNI, meaning the overhead
+from using JNA is still bigger than the function runtime.
+
+This performance penalty is still small in the scope of longer-running
+functions, and so may not be a concern for your application, but it is something
+to be aware of.
+
+### tech.jna
+The tech.jna library is similar in scope to Clojure-JNA, however was written to
+fit into an ecosystem of libraries meant for array-based programming for machine
+learning and data science.
+
+```sh
+$ clj -Sdeps '{:deps {techascent/tech.jna {:mvn/version "4.05"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}'
+```
+
+This library is also quite simple to use, the only slightly odd thing I'm doing
+here is to dereference the var outside the benchmark in order to ensure it's an
+apples-to-apples comparison. We don't want var dereference time mucking up our
+benchmark.
+
+```clojure
+user=> (require '[criterium.core :as bench])
+nil
+user=> (alter-var-root #'bench/estimated-overhead-cache (constantly 6.400703613065185E-9))
+6.400703613065185E-9
+user=> (require '[tech.v3.jna :as jna])
+nil
+user=> (jna/def-jna-fn "glfw" glfwInit "initialize glfw" Integer)
+#'user/glfwInit
+user=> (glfwInit)
+Oct 09, 2021 10:30:50 AM clojure.tools.logging$eval1122$fn__1125 invoke
+INFO: Library glfw found at [:system "glfw"]
+1
+user=> (jna/def-jna-fn "glfw" glfwGetTime "gets the time as a double since init" Double)
+#'user/glfwGetTime
+user=> (let [f @#'glfwGetTime]
+         (bench/bench (f) :verbose))
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: -Dclojure.basis=/home/jsusk/.clojure/.cpcache/2910209237.basis
+Evaluation count : 323281680 in 60 samples of 5388028 calls.
+      Execution time sample mean : 203.976803 ns
+             Execution time mean : 203.818712 ns
+Execution time sample std-deviation : 14.557312 ns
+    Execution time std-deviation : 14.614080 ns
+   Execution time lower quantile : 179.732593 ns ( 2.5%)
+   Execution time upper quantile : 213.929374 ns (97.5%)
+                   Overhead used : 6.400704 ns
+nil
+```
+
+This version is even slower than Clojure-JNA. I'm unsure where this overhead is
+coming from, but I'll admit that I haven't looked at their implementations very
+closely.
+
+### dtype-next
+The library dtype-next replaced tech.jna in the toolkit of the group working on
+machine learning and array-based programming, and it includes support for
+composite data types including structs, as well as primitive functions and
+callbacks.
+
+In addition, dtype-next has two different ffi backends. First is JNA, which is
+usable on any JDK version, and is what we'll use for the first benchmark. Second
+is the Java 16 version of Project Panama, which will be shown next.
+
+In order to use the dtype-next ffi with the JNA backend, the JNA library has to
+be included in the dependencies.
+
+```sh
+$ clj -Sdeps '{:deps {cnuernber/dtype-next {:mvn/version "8.032"}
+                      net.java.dev.jna/jna {:mvn/version "5.8.0"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}'
+```
+
+The dtype-next library also requires some more ceremony around declaring native
+functions. One advantage this has is that multiple symbols with the same name
+can be loaded from different shared libraries, but it also does increase
+friction when defining native wrappers.
+
+Some easier ways to define native wrappers are provided than what is seen here,
+but they share some disadvantages in documentation over the core methods
+provided in coffi, although they are comparable to the data model provided in
+coffi.
+
+```clojure
+user=> (require '[criterium.core :as bench])
+nil
+user=> (alter-var-root #'bench/estimated-overhead-cache (constantly 6.400703613065185E-9))
+6.400703613065185E-9
+user=> (require '[tech.v3.datatype.ffi :as dt-ffi])
+nil
+user=> (def fn-defs {:glfwInit {:rettype :int32} :glfwGetTime {:rettype :float64}})
+#'user/fn-defs
+user=> (def library-def (dt-ffi/define-library fn-defs))
+#'user/library-def
+user=> (def library-instance (dt-ffi/instantiate-library library-def "/usr/lib/libglfw.so"))
+#'user/library-instance
+user=> (def init (:glfwInit @library-instance))
+#'user/init
+user=> (init)
+1
+user=> (let [f (:glfwGetTime @library-instance)]
+         (bench/bench (f) :verbose))
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 17+35-2724
+Runtime arguments: -Dclojure.basis=/home/jsusk/.clojure/.cpcache/643862289.basis
+Evaluation count : 710822100 in 60 samples of 11847035 calls.
+      Execution time sample mean : 90.900112 ns
+             Execution time mean : 90.919917 ns
+Execution time sample std-deviation : 6.463312 ns
+    Execution time std-deviation : 6.470108 ns
+   Execution time lower quantile : 79.817126 ns ( 2.5%)
+   Execution time upper quantile : 95.454652 ns (97.5%)
+                   Overhead used : 6.400704 ns
+nil
+```
+
+This version of JNA usage is significantly faster than either of the other JNA
+libraries, but is still substantially slower than using JNI or coffi.
+
+In addition to the JNA backend, dtype-next has a Java 16-specific backend that
+uses an older version of Panama. This version requires similar setup to coffi in
+order to run.
+
+```sh
+$ clj -Sdeps '{:deps {cnuernber/dtype-next {:mvn/version "8.032"}
+                      criterium/criterium {:mvn/version "0.4.6"}}}' \
+      -J--add-modules=jdk.incubator.foreign \
+      -J-Dforeign.restricted=permit \
+      -J--add-opens=java.base/java.lang=ALL-UNNAMED \
+      -J-Djava.library.path=/usr/lib/x86_64-linux-gnu
+```
+
+The actual code to run the benchmark is identical to the last example, but is
+reproduced here for completeness.
+
+```clojure
+user=> (require '[criterium.core :as bench])
+nil
+user=> (alter-var-root #'bench/estimated-overhead-cache (constantly 6.400703613065185E-9))
+6.400703613065185E-9
+user=> (require '[tech.v3.datatype.ffi :as dt-ffi])
+nil
+user=> (def fn-defs {:glfwInit {:rettype :int32} :glfwGetTime {:rettype :float64}})
+#'user/fn-defs
+user=> (def library-def (dt-ffi/define-library fn-defs))
+#'user/library-def
+user=> (def library-instance (dt-ffi/instantiate-library library-def "/usr/lib/libglfw.so"))
+#'user/library-instance
+user=> (def init (:glfwInit @library-instance))
+#'user/init
+user=> (init)
+1
+user=> (let [f (:glfwGetTime @library-instance)]
+         (bench/bench (f) :verbose))
+amd64 Linux 5.10.68-1-MANJARO 8 cpu(s)
+OpenJDK 64-Bit Server VM 16.0.2+7
+Runtime arguments: --add-modules=jdk.incubator.foreign -Dforeign.restricted=permit --add-opens=java.base/java.lang=ALL-UNNAMED -Djava.library.path=/usr/lib/x86_64-linux-gnu -Dclojure.basis=/home/jsusk/.clojure/.cpcache/2337051659.basis
+Evaluation count : 1588513080 in 60 samples of 26475218 calls.
+      Execution time sample mean : 58.732468 ns
+             Execution time mean : 58.647361 ns
+Execution time sample std-deviation : 9.732389 ns
+    Execution time std-deviation : 9.791738 ns
+   Execution time lower quantile : 31.318115 ns ( 2.5%)
+   Execution time upper quantile : 65.449222 ns (97.5%)
+                   Overhead used : 6.400704 ns
+
+Found 14 outliers in 60 samples (23.3333 %)
+	low-severe	 8 (13.3333 %)
+	low-mild	 4 (6.6667 %)
+	high-mild	 2 (3.3333 %)
+ Variance from outliers : 87.6044 % Variance is severely inflated by outliers
+nil
+```
+
+Not reproduced here, but notable for comparison, in my testing Java 16's version
+of the JNI version performed about the same.
+
+This is significantly faster than the JNA version of dtype-next, but it is still
+slower than modern Panama. This is likely to simply be a result of optimizations
+and changes to the Panama API, and when dtype-next is updated to use the Java 17
+version of Panama I expect it will perform in line with coffi, but this
+benchmark will be reproduced when this happens. Still, this shows that as it
+stands, coffi is the fastest FFI available to Clojure developers.

docs/coffi.layout.html

@@ -0,0 +1,6 @@
+<!DOCTYPE html PUBLIC ""
+    "">
+<html><head><meta charset="UTF-8" /><title>coffi.layout documentation</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">coffi</span> <span class="project-version">v1.0.486</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="01-Getting-Started.html"><div class="inner"><span>Getting Started</span></div></a></li><li class="depth-1 "><a href="02-Memory-Management.html"><div class="inner"><span>Memory Management</span></div></a></li><li class="depth-1 "><a href="03-Builtin-Types.html"><div class="inner"><span>Built-in Types **WIP**</span></div></a></li><li class="depth-1 "><a href="04-Custom-Types.html"><div class="inner"><span>Custom Types</span></div></a></li><li class="depth-1 "><a href="05-Low-Level-Wrappers.html"><div class="inner"><span>Low-Level Wrappers</span></div></a></li><li class="depth-1 "><a href="50-Data-Model.html"><div class="inner"><span>Data Model</span></div></a></li><li class="depth-1 "><a href="99-Benchmarks.html"><div class="inner"><span>Benchmarks **OUTDATED**</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>coffi</span></div></div></li><li class="depth-2 branch"><a href="coffi.ffi.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ffi</span></div></a></li><li class="depth-2 branch current"><a href="coffi.layout.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>layout</span></div></a></li><li class="depth-2"><a href="coffi.mem.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mem</span></div></a></li></ul></div><div class="sidebar secondary"><h3><a href="#top"><span class="inner">Public Vars</span></a></h3><ul><li class="depth-1"><a href="coffi.layout.html#var-with-c-layout"><div class="inner"><span>with-c-layout</span></div></a></li></ul></div><div class="namespace-docs" id="content"><h1 class="anchor" id="top">coffi.layout</h1><div class="doc"><div class="markdown"><p>Functions for adjusting the layout of structs.</p>
+</div></div><div class="public anchor" id="var-with-c-layout"><h3>with-c-layout</h3><div class="usage"><code>(with-c-layout struct-spec)</code></div><div class="doc"><div class="markdown"><p>Forces a struct specification to C layout rules.</p>
+<p>This will add padding fields between fields to match C alignment requirements.</p>
+</div></div><div class="src-link"><a href="https://github.com/IGJoshua/coffi/blob/2d708fa7724cd2055357f37cefb93a6177ddf281/src/clj/coffi/layout.clj#L6">view source</a></div></div></div></body></html>

docs/css/default.css

@@ -0,0 +1,551 @@
+body {
+    font-family: Helvetica, Arial, sans-serif;
+    font-size: 15px;
+}
+
+pre, code {
+    font-family: Monaco, DejaVu Sans Mono, Consolas, monospace;
+    font-size: 9pt;
+    margin: 15px 0;
+}
+
+h1 {
+    font-weight: normal;
+    font-size: 29px;
+    margin: 10px 0 2px 0;
+    padding: 0;
+}
+
+h2 {
+    font-weight: normal;
+    font-size: 25px;
+}
+
+h5.license {
+    margin: 9px 0 22px 0;
+    color: #555;
+    font-weight: normal;
+    font-size: 12px;
+    font-style: italic;
+}
+
+.document h1, .namespace-index h1 {
+    font-size: 32px;
+    margin-top: 12px;
+}
+
+#header, #content, .sidebar {
+    position: fixed;
+}
+
+#header {
+    top: 0;
+    left: 0;
+    right: 0;
+    height: 22px;
+    color: #f5f5f5;
+    padding: 5px 7px;
+}
+
+#content {
+    top: 32px;
+    right: 0;
+    bottom: 0;
+    overflow: auto;
+    background: #fff;
+    color: #333;
+    padding: 0 18px;
+}
+
+.sidebar {
+    position: fixed;
+    top: 32px;
+    bottom: 0;
+    overflow: auto;
+}
+
+.sidebar.primary {
+    background: #e2e2e2;
+    border-right: solid 1px #cccccc;
+    left: 0;
+    width: 250px;
+}
+
+.sidebar.secondary {
+    background: #f2f2f2;
+    border-right: solid 1px #d7d7d7;
+    left: 251px;
+    width: 200px;
+}
+
+#content.namespace-index, #content.document {
+    left: 251px;
+}
+
+#content.namespace-docs {
+    left: 452px;
+}
+
+#content.document {
+    padding-bottom: 10%;
+}
+
+#header {
+    background: #3f3f3f;
+    box-shadow: 0 0 8px rgba(0, 0, 0, 0.4);
+    z-index: 100;
+}
+
+#header h1 {
+    margin: 0;
+    padding: 0;
+    font-size: 18px;
+    font-weight: lighter;
+    text-shadow: -1px -1px 0px #333;
+}
+
+#header h1 .project-version {
+    font-weight: normal;
+}
+
+.project-version {
+    padding-left: 0.15em;
+}
+
+#header a, .sidebar a {
+    display: block;
+    text-decoration: none;
+}
+
+#header a {
+    color: #f5f5f5;
+}
+
+.sidebar a {
+    color: #333;
+}
+
+#header h2 {
+    float: right;
+    font-size: 9pt;
+    font-weight: normal;
+    margin: 4px 3px;
+    padding: 0;
+    color: #bbb;
+}
+
+#header h2 a {
+    display: inline;
+}
+
+.sidebar h3 {
+    margin: 0;
+    padding: 10px 13px 0 13px;
+    font-size: 19px;
+    font-weight: lighter;
+}
+
+.sidebar h3 a {
+    color: #444;
+}
+
+.sidebar h3.no-link {
+    color: #636363;
+}
+
+.sidebar ul {
+    padding: 7px 0 6px 0;
+    margin: 0;
+}
+
+.sidebar ul.index-link {
+    padding-bottom: 4px;
+}
+
+.sidebar li {
+    display: block;
+    vertical-align: middle;
+}
+
+.sidebar li a, .sidebar li .no-link {
+    border-left: 3px solid transparent;
+    padding: 0 10px;
+    white-space: nowrap;
+}
+
+.sidebar li .no-link {
+    display: block;
+    color: #777;
+    font-style: italic;
+}
+
+.sidebar li .inner {
+    display: inline-block;
+    padding-top: 7px;
+    height: 24px;
+}
+
+.sidebar li a, .sidebar li .tree {
+    height: 31px;
+}
+
+.depth-1 .inner { padding-left: 2px; }
+.depth-2 .inner { padding-left: 6px; }
+.depth-3 .inner { padding-left: 20px; }
+.depth-4 .inner { padding-left: 34px; }
+.depth-5 .inner { padding-left: 48px; }
+.depth-6 .inner { padding-left: 62px; }
+
+.sidebar li .tree {
+    display: block;
+    float: left;
+    position: relative;
+    top: -10px;
+    margin: 0 4px 0 0;
+    padding: 0;
+}
+
+.sidebar li.depth-1 .tree {
+    display: none;
+}
+
+.sidebar li .tree .top, .sidebar li .tree .bottom {
+    display: block;
+    margin: 0;
+    padding: 0;
+    width: 7px;
+}
+
+.sidebar li .tree .top {
+    border-left: 1px solid #aaa;
+    border-bottom: 1px solid #aaa;
+    height: 19px;
+}
+
+.sidebar li .tree .bottom {
+    height: 22px;
+}
+
+.sidebar li.branch .tree .bottom {
+    border-left: 1px solid #aaa;
+}
+
+.sidebar.primary li.current a {
+    border-left: 3px solid #a33;
+    color: #a33;
+}
+
+.sidebar.secondary li.current a {
+    border-left: 3px solid #33a;
+    color: #33a;
+}
+
+.namespace-index h2 {
+    margin: 30px 0 0 0;
+}
+
+.namespace-index h3 {
+    font-size: 16px;
+    font-weight: bold;
+    margin-bottom: 0;
+}
+
+.namespace-index .topics {
+    padding-left: 30px;
+    margin: 11px 0 0 0;
+}
+
+.namespace-index .topics li {
+    padding: 5px 0;
+}
+
+.namespace-docs h3 {
+    font-size: 18px;
+    font-weight: bold;
+}
+
+.public h3 {
+    margin: 0;
+    float: left;
+}
+
+.usage {
+    clear: both;
+}
+
+.public {
+    margin: 0;
+    border-top: 1px solid #e0e0e0;
+    padding-top: 14px;
+    padding-bottom: 6px;
+}
+
+.public:last-child {
+    margin-bottom: 20%;
+}
+
+.members .public:last-child {
+    margin-bottom: 0;
+}
+
+.members {
+    margin: 15px 0;
+}
+
+.members h4 {
+    color: #555;
+    font-weight: normal;
+    font-variant: small-caps;
+    margin: 0 0 5px 0;
+}
+
+.members .inner {
+    padding-top: 5px;
+    padding-left: 12px;
+    margin-top: 2px;
+    margin-left: 7px;
+    border-left: 1px solid #bbb;
+}
+
+#content .members .inner h3 {
+    font-size: 12pt;
+}
+
+.members .public {
+    border-top: none;
+    margin-top: 0;
+    padding-top: 6px;
+    padding-bottom: 0;
+}
+
+.members .public:first-child {
+    padding-top: 0;
+}
+
+h4.type,
+h4.dynamic,
+h4.added,
+h4.deprecated {
+    float: left;
+    margin: 3px 10px 15px 0;
+    font-size: 15px;
+    font-weight: bold;
+    font-variant: small-caps;
+}
+
+.public h4.type,
+.public h4.dynamic,
+.public h4.added,
+.public h4.deprecated {
+    font-size: 13px;
+    font-weight: bold;
+    margin: 3px 0 0 10px;
+}
+
+.members h4.type,
+.members h4.added,
+.members h4.deprecated {
+    margin-top: 1px;
+}
+
+h4.type {
+    color: #717171;
+}
+
+h4.dynamic {
+    color: #9933aa;
+}
+
+h4.added {
+    color: #508820;
+}
+
+h4.deprecated {
+    color: #880000;
+}
+
+.namespace {
+    margin-bottom: 30px;
+}
+
+.namespace:last-child {
+    margin-bottom: 10%;
+}
+
+.index {
+    padding: 0;
+    font-size: 80%;
+    margin: 15px 0;
+    line-height: 16px;
+}
+
+.index * {
+    display: inline;
+}
+
+.index p {
+    padding-right: 3px;
+}
+
+.index li {
+    padding-right: 5px;
+}
+
+.index ul {
+    padding-left: 0;
+}
+
+.type-sig {
+    clear: both;
+    color: #088;
+}
+
+.type-sig pre {
+    padding-top: 10px;
+    margin: 0;
+}
+
+.usage code {
+    display: block;
+    color: #008;
+    margin: 2px 0;
+}
+
+.usage code:first-child {
+    padding-top: 10px;
+}
+
+p {
+    margin: 15px 0;
+}
+
+.public p:first-child, .public pre.plaintext {
+    margin-top: 12px;
+}
+
+.doc {
+    margin: 0 0 26px 0;
+    clear: both;
+}
+
+.public .doc {
+    margin: 0;
+}
+
+.namespace-index .doc {
+    margin-bottom: 20px;
+}
+
+.namespace-index .namespace .doc {
+    margin-bottom: 10px;
+}
+
+.markdown p, .markdown li, .markdown dt, .markdown dd, .markdown td {
+    line-height: 22px;
+}
+
+.markdown li {
+    padding: 2px 0;
+}
+
+.markdown h2 {
+    font-weight: normal;
+    font-size: 25px;
+    margin: 30px 0 10px 0;
+}
+
+.markdown h3 {
+    font-weight: normal;
+    font-size: 20px;
+    margin: 30px 0 0 0;
+}
+
+.markdown h4 {
+    font-size: 15px;
+    margin: 22px 0 -4px 0;
+}
+
+.doc, .public, .namespace .index {
+    max-width: 680px;
+    overflow-x: visible;
+}
+
+.markdown pre > code {
+    display: block;
+    padding: 10px;
+}
+
+.markdown pre > code, .src-link a {
+    border: 1px solid #e4e4e4;
+    border-radius: 2px;
+}
+
+.markdown code:not(.hljs), .src-link a {
+    background: #f6f6f6;
+}
+
+pre.deps {
+    display: inline-block;
+    margin: 0 10px;
+    border: 1px solid #e4e4e4;
+    border-radius: 2px;
+    padding: 10px;
+    background-color: #f6f6f6;
+}
+
+.markdown hr {
+    border-style: solid;
+    border-top: none;
+    color: #ccc;
+}
+
+.doc ul, .doc ol {
+    padding-left: 30px;
+}
+
+.doc table {
+    border-collapse: collapse;
+    margin: 0 10px;
+}
+
+.doc table td, .doc table th {
+    border: 1px solid #dddddd;
+    padding: 4px 6px;
+}
+
+.doc table th {
+    background: #f2f2f2;
+}
+
+.doc dl {
+    margin: 0 10px 20px 10px;
+}
+
+.doc dl dt {
+    font-weight: bold;
+    margin: 0;
+    padding: 3px 0;
+    border-bottom: 1px solid #ddd;
+}
+
+.doc dl dd {
+    padding: 5px 0;
+    margin: 0 0 5px 10px;
+}
+
+.doc abbr {
+    border-bottom: 1px dotted #333;
+    font-variant: none;
+    cursor: help;
+}
+
+.src-link {
+    margin-bottom: 15px;
+}
+
+.src-link a {
+    font-size: 70%;
+    padding: 1px 4px;
+    text-decoration: none;
+    color: #5555bb;
+}

docs/css/highlight.css

@@ -0,0 +1,97 @@
+/*
+github.com style (c) Vasily Polovnyov <vast@whiteants.net>
+*/
+
+.hljs {
+  display: block;
+  overflow-x: auto;
+  padding: 0.5em;
+  color: #333;
+  background: #f8f8f8;
+}
+
+.hljs-comment,
+.hljs-quote {
+  color: #998;
+  font-style: italic;
+}
+
+.hljs-keyword,
+.hljs-selector-tag,
+.hljs-subst {
+  color: #333;
+  font-weight: bold;
+}
+
+.hljs-number,
+.hljs-literal,
+.hljs-variable,
+.hljs-template-variable,
+.hljs-tag .hljs-attr {
+  color: #008080;
+}
+
+.hljs-string,
+.hljs-doctag {
+  color: #d14;
+}
+
+.hljs-title,
+.hljs-section,
+.hljs-selector-id {
+  color: #900;
+  font-weight: bold;
+}
+
+.hljs-subst {
+  font-weight: normal;
+}
+
+.hljs-type,
+.hljs-class .hljs-title {
+  color: #458;
+  font-weight: bold;
+}
+
+.hljs-tag,
+.hljs-name,
+.hljs-attribute {
+  color: #000080;
+  font-weight: normal;
+}
+
+.hljs-regexp,
+.hljs-link {
+  color: #009926;
+}
+
+.hljs-symbol,
+.hljs-bullet {
+  color: #990073;
+}
+
+.hljs-built_in,
+.hljs-builtin-name {
+  color: #0086b3;
+}
+
+.hljs-meta {
+  color: #999;
+  font-weight: bold;
+}
+
+.hljs-deletion {
+  background: #fdd;
+}
+
+.hljs-addition {
+  background: #dfd;
+}
+
+.hljs-emphasis {
+  font-style: italic;
+}
+
+.hljs-strong {
+  font-weight: bold;
+}

docs/js/page_effects.js

@@ -0,0 +1,112 @@
+function visibleInParent(element) {
+    var position = $(element).position().top
+    return position > -50 && position < ($(element).offsetParent().height() - 50)
+}
+
+function hasFragment(link, fragment) {
+    return $(link).attr("href").indexOf("#" + fragment) != -1
+}
+
+function findLinkByFragment(elements, fragment) {
+    return $(elements).filter(function(i, e) { return hasFragment(e, fragment)}).first()
+}
+
+function scrollToCurrentVarLink(elements) {
+    var elements = $(elements);
+    var parent   = elements.offsetParent();
+
+    if (elements.length == 0) return;
+
+    var top    = elements.first().position().top;
+    var bottom = elements.last().position().top + elements.last().height();
+
+    if (top >= 0 && bottom <= parent.height()) return;
+
+    if (top < 0) {
+        parent.scrollTop(parent.scrollTop() + top);
+    }
+    else if (bottom > parent.height()) {
+        parent.scrollTop(parent.scrollTop() + bottom - parent.height());
+    }
+}
+
+function setCurrentVarLink() {
+    $('.secondary a').parent().removeClass('current')
+    $('.anchor').
+        filter(function(index) { return visibleInParent(this) }).
+        each(function(index, element) {
+            findLinkByFragment(".secondary a", element.id).
+                parent().
+                addClass('current')
+        });
+    scrollToCurrentVarLink('.secondary .current');
+}
+
+var hasStorage = (function() { try { return localStorage.getItem } catch(e) {} }())
+
+function scrollPositionId(element) {
+    var directory = window.location.href.replace(/[^\/]+\.html$/, '')
+    return 'scroll::' + $(element).attr('id') + '::' + directory
+}
+
+function storeScrollPosition(element) {
+    if (!hasStorage) return;
+    localStorage.setItem(scrollPositionId(element) + "::x", $(element).scrollLeft())
+    localStorage.setItem(scrollPositionId(element) + "::y", $(element).scrollTop())
+}
+
+function recallScrollPosition(element) {
+    if (!hasStorage) return;
+    $(element).scrollLeft(localStorage.getItem(scrollPositionId(element) + "::x"))
+    $(element).scrollTop(localStorage.getItem(scrollPositionId(element) + "::y"))
+}
+
+function persistScrollPosition(element) {
+    recallScrollPosition(element)
+    $(element).scroll(function() { storeScrollPosition(element) })
+}
+
+function sidebarContentWidth(element) {
+    var widths = $(element).find('.inner').map(function() { return $(this).innerWidth() })
+    return Math.max.apply(Math, widths)
+}
+
+function calculateSize(width, snap, margin, minimum) {
+    if (width == 0) {
+        return 0
+    }
+    else {
+        return Math.max(minimum, (Math.ceil(width / snap) * snap) + (margin * 2))
+    }
+}
+
+function resizeSidebars() {
+    var primaryWidth   = sidebarContentWidth('.primary')
+    var secondaryWidth = 0
+
+    if ($('.secondary').length != 0) {
+        secondaryWidth = sidebarContentWidth('.secondary')
+    }
+
+    // snap to grid
+    primaryWidth   = calculateSize(primaryWidth, 32, 13, 160)
+    secondaryWidth = calculateSize(secondaryWidth, 32, 13, 160)
+
+    $('.primary').css('width', primaryWidth)
+    $('.secondary').css('width', secondaryWidth).css('left', primaryWidth + 1)
+
+    if (secondaryWidth > 0) {
+        $('#content').css('left', primaryWidth + secondaryWidth + 2)
+    }
+    else {
+        $('#content').css('left', primaryWidth + 1)
+    }
+}
+
+$(window).ready(resizeSidebars)
+$(window).ready(setCurrentVarLink)
+$(window).ready(function() { persistScrollPosition('.primary')})
+$(window).ready(function() {
+    $('#content').scroll(setCurrentVarLink)
+    $(window).resize(setCurrentVarLink)
+})

flake.lock

@@ -0,0 +1,26 @@
+{
+  "nodes": {
+    "nixpkgs": {
+      "locked": {
+        "lastModified": 1727634051,
+        "narHash": "sha256-S5kVU7U82LfpEukbn/ihcyNt2+EvG7Z5unsKW9H/yFA=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "06cf0e1da4208d3766d898b7fdab6513366d45b9",
+        "type": "github"
+      },
+      "original": {
+        "id": "nixpkgs",
+        "ref": "nixos-unstable",
+        "type": "indirect"
+      }
+    },
+    "root": {
+      "inputs": {
+        "nixpkgs": "nixpkgs"
+      }
+    }
+  },
+  "root": "root",
+  "version": 7
+}

flake.nix

@@ -0,0 +1,33 @@
+{
+  inputs = {
+    nixpkgs.url = "nixpkgs/nixos-unstable";
+  };
+  outputs = { self, nixpkgs }:
+    let
+      system = "x86_64-linux";
+      pkgs = import nixpkgs {
+        inherit system;
+        overlays = [
+          (final: prev: {
+            clojure = prev.clojure.override { jdk = final.jdk22; };
+          })
+        ];
+      };
+    in
+      {
+        devShells.${system}.default = pkgs.mkShell rec {
+          packages = [
+          ];
+
+          nativeBuildInputs = with pkgs; [
+            clojure
+          ];
+
+          buildInputs = with pkgs; [
+          ];
+
+          inputsFrom = with pkgs; [
+          ];
+        };
+      };
+}

pom.xml

@@ -1,8 +1,8 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
-  <name>coffi</name>
+  <name>org.suskalo/coffi</name>
-  <description>A Foreign Function Interface in Clojure for JDK 17.</description>
+  <description>A Foreign Function Interface in Clojure for JDK 22+.</description>
   <url>https://github.com/IGJoshua/coffi</url>
   <licenses>
     <license>

src/clj/coffi/ffi.clj

@@ -1,7 +1,5 @@
 (ns coffi.ffi
-  "Functions for managing native allocations and resource scopes, creating handles
+  "Functions for creating handles to native functions and loading native libraries."
-  to native functions, serializing and deserializing Clojure data to native
-  structures, and loading native libraries."
   (:require
    [clojure.java.io :as io]
    [clojure.spec.alpha :as s]
@@ -15,13 +13,16 @@
     MethodHandle
     MethodHandles
     MethodType)
-   (jdk.incubator.foreign
+   (java.lang.foreign
-    Addressable
+    Linker
-    CLinker
+    Linker$Option
     FunctionDescriptor
     MemoryLayout
+    MemorySegment
     SegmentAllocator)))
 
+(set! *warn-on-reflection* true)
+
 ;;; FFI Code loading and function access
 
 (defn load-system-library
@@ -35,18 +36,9 @@
   (Loader/loadLibrary (.getAbsolutePath (io/file path))))
 
 (defn find-symbol
-  "Gets the [[MemoryAddress]] of a symbol from the loaded libraries."
+  "Gets the [[MemorySegment]] of a symbol from the loaded libraries."
   [sym]
-  (let [sym (name sym)]
+  (Loader/findSymbol (name sym)))
-    (Loader/findSymbol sym)))
-
-(defn- method-type
-  "Gets the [[MethodType]] for a set of `args` and `ret` types."
-  ([args] (method-type args ::mem/void))
-  ([args ret]
-   (MethodType/methodType
-    ^Class (mem/java-layout ret)
-    ^"[Ljava.lang.Class;" (into-array Class (map mem/java-layout args)))))
 
 (defn- function-descriptor
   "Gets the [[FunctionDescriptor]] for a set of `args` and `ret` types."
@@ -61,9 +53,10 @@
         args-arr)))))
 
 (defn- downcall-handle
-  "Gets the [[MethodHandle]] for the function at the `address`."
+  "Gets the [[MethodHandle]] for the function at the `sym`."
-  [address method-type function-descriptor]
+  [sym function-descriptor]
-  (.downcallHandle (CLinker/getInstance) address method-type function-descriptor))
+  (.downcallHandle (Linker/nativeLinker) sym function-descriptor
+                   (make-array Linker$Option 0)))
 
 (def ^:private load-instructions
   "Mapping from primitive types to the instruction used to load them onto the stack."
@@ -71,7 +64,6 @@
    ::mem/short :sload
    ::mem/int :iload
    ::mem/long :lload
-   ::mem/long-long :lload
    ::mem/char :cload
    ::mem/float :fload
    ::mem/double :dload
@@ -83,7 +75,6 @@
    ::mem/short Short
    ::mem/int Integer
    ::mem/long Long
-   ::mem/long-long Long
    ::mem/char Character
    ::mem/float Float
    ::mem/double Double})
@@ -108,8 +99,9 @@
 (defn- insn-layout
   "Gets the type keyword or class for referring to the type in bytecode."
   [type]
-  (if (some-> (mem/primitive-type type) (not= ::mem/pointer))
+  (or (when-some [prim (mem/primitive-type type)]
-    (keyword (name type))
+        (when (not= prim ::mem/pointer)
+          (keyword (name prim))))
       (mem/java-layout type)))
 
 (def ^:private unbox-fn-for-type
@@ -118,7 +110,6 @@
    ::mem/short "shortValue"
    ::mem/int "intValue"
    ::mem/long "longValue"
-   ::mem/long-long "longValue"
    ::mem/char "charValue"
    ::mem/float "floatValue"
    ::mem/double "doubleValue"})
@@ -139,11 +130,15 @@
          [:invokevirtual (prim-classes prim-type) (unbox-fn-for-type prim-type) [prim]]]
         []))))
 
-(defn- downcall-class
+(defn- downcall-class-ctor*
-  "Class definition for an implementation of [[IFn]] which calls a closed over
+  "Returns a function to construct a downcall class for the given `args` and `ret` types.
+
+  A downcall class is an implementation of [[IFn]] which calls a closed over
   method handle without reflection, unboxing primitives when needed."
   [args ret]
+  (let [klass (insn/define
                 {:flags #{:public :final}
+                 :version 8
                  :super clojure.lang.AFunction
                  :fields [{:name "downcall_handle"
                            :type MethodHandle
@@ -180,18 +175,32 @@
                                       (not (mem/primitive-type ret)) (cons SegmentAllocator))]
                                    (to-object-asm ret)
                                    [:areturn]]}]})
+        ctor (.getConstructor klass
+                              (doto ^"[Ljava.lang.Class;" (make-array Class 1)
+                                (aset 0 MethodHandle)))]
+    (fn [^MethodHandle h]
+      (.newInstance ctor
+                    (doto (object-array 1)
+                      (aset 0 h))))))
+
+(def ^:private downcall-class-ctor
+  "Returns a function to construct a downcall class for the given memoized `args` and `ret` types.
+
+  A downcall class is an implementation of [[IFn]] which calls a closed over
+  method handle without reflection, unboxing primitives when needed."
+  (memoize downcall-class-ctor*))
 
 (defn- downcall-fn
   "Creates a function to call `handle` without reflection."
   [handle args ret]
-  (insn/new-instance (downcall-class args ret) ^MethodHandle handle))
+  ((downcall-class-ctor args ret) ^MethodHandle handle))
 
-(defn- ensure-address
+(defn ensure-symbol
-  "Gets the address if the argument is [[Addressable]], otherwise
+  "Returns the argument if it is a [[MemorySegment]], otherwise
   calls [[find-symbol]] on it."
-  [symbol-or-addr]
+  ^MemorySegment [symbol-or-addr]
-  (if (instance? Addressable symbol-or-addr)
+  (if (instance? MemorySegment symbol-or-addr)
-    (mem/address-of symbol-or-addr)
+    symbol-or-addr
     (find-symbol symbol-or-addr)))
 
 (defn make-downcall
@@ -206,10 +215,8 @@
   first argument of a [[SegmentAllocator]]."
   [symbol-or-addr args ret]
   (-> symbol-or-addr
-      ensure-address
+      ensure-symbol
-      (downcall-handle
+      (downcall-handle (function-descriptor args ret))
-       (method-type args ret)
-       (function-descriptor args ret))
       (downcall-fn args ret)))
 
 (defn make-varargs-factory
@@ -229,21 +236,191 @@
      (let [args (concat required-args types)]
        (make-downcall symbol args ret)))))
 
+(def ^:private primitive-cast-sym
+  "Map from non-pointer primitive types to functions that cast to the appropriate
+  java primitive."
+  {::mem/byte `byte
+   ::mem/short `short
+   ::mem/int `int
+   ::mem/long `long
+   ::mem/char `char
+   ::mem/float `float
+   ::mem/double `double})
+
+(defn- inline-serde-wrapper
+  "Builds a form that returns a function that calls `downcall` with serdes.
+
+  The return type and any arguments that are primitives will not
+  be (de)serialized except to be cast. If all arguments and return are
+  primitive, the `downcall` is returned directly. In cases where arguments must
+  be serialized, a new [[mem/confined-arena]] is generated."
+  [downcall arg-types ret-type]
+  (let [;; Complexity of types
+        const-args? (or (vector? arg-types) (nil? arg-types))
+        simple-args? (when const-args?
+                       (and (every? mem/primitive? arg-types)
+                            ;; NOTE(Joshua): Pointer types with serdes (e.g. [::mem/pointer ::mem/int])
+                            ;; still require an arena, making them not qualify as "simple".
+                            (every? keyword? (filter (comp #{::mem/pointer} mem/primitive-type) arg-types))))
+        const-ret? (s/valid? ::mem/type ret-type)
+        primitive-ret? (and const-ret?
+                            (or (and (mem/primitive? ret-type)
+                                     ;; NOTE(Joshua): Pointer types with serdes require deserializing the
+                                     ;; return value, but don't require passing an arena to the downcall,
+                                     ;; making them cause the return to not be primitive, but it may still
+                                     ;; be "simple".
+                                     (or (keyword? ret-type) (not (#{::mem/pointer} (mem/primitive-type ret-type)))))
+                                (#{::mem/void} ret-type)))
+        simple-ret? (and const-ret? (mem/primitive-type ret-type))
+        no-serde? (and const-args? (empty? arg-types)
+                       primitive-ret?)]
+    (if no-serde?
+      `(let [downcall# ~downcall]
+         ;; NOTE(Joshua): These are here to ensure that evaluation order is
+         ;; preserved as equivalent to a function call.
+         ~arg-types
+         ~ret-type
+         downcall#)
+      (let [;; All our symbols
+            arena (gensym "arena")
+            downcall-sym (gensym "downcall")
+            args-sym (when-not const-args?
+                       (gensym "args"))
+            args-types-sym (when-not const-args?
+                             (gensym "args-types"))
+            arg-syms (when const-args?
+                       (repeatedly (count arg-types) #(gensym "arg")))
+            arg-type-syms (when const-args?
+                            (repeatedly (count arg-types) #(gensym "arg-type")))
+            ret-type-sym (gensym "ret-type")
+
+            ;; Helper Functions
+            make-serialized-binding
+            ;; Given a symbol and its type, make a partial binding to serialize and shadow it
+            (fn [sym type type-sym]
+              (some->>
+               (cond
+                 (not (s/valid? ::mem/type type))
+                 `(mem/serialize ~sym ~type-sym ~arena)
+
+                 (and (mem/primitive? type)
+                      (not (#{::mem/pointer} (mem/primitive-type type))))
+                 (list (primitive-cast-sym (mem/primitive-type type)) sym)
+
+                 ;; cast null pointers to something understood by panama
+                 (#{::mem/pointer} type)
+                 `(or ~sym mem/null)
+
+                 (mem/primitive-type type)
+                 `(mem/serialize* ~sym ~type-sym ~arena)
+
+                 :else
+                 `(let [alloc# (mem/alloc-instance ~type-sym)]
+                    (mem/serialize-into ~sym ~type-sym alloc# ~arena)
+                    alloc#))
+               (list sym)))
+
+            arg-serializers
+            ;; Binding forms that rebind the arg symbols to their serialized counterparts
+            (when const-args?
+              (->> (map make-serialized-binding
+                        arg-syms arg-types arg-type-syms)
+                   (filter some?)))
+
+            wrap-serialize
+            ;; Wrap an expression to shadow args to their serialized counterparts
+            (fn [expr]
+              (cond
+                (and const-args?
+                     (zero? (count arg-types)))
+                expr
+
+                const-args?
+                (if (seq arg-serializers)
+                  `(let [~@(mapcat identity arg-serializers)]
+                     ~expr)
+                  expr)
+
+                :else
+                `(let [~args-sym (map (fn [obj# type#]
+                                        (mem/serialize obj# type# ~arena))
+                                      ~args-sym ~args-types-sym)]
+                   ~expr)))
+
+            make-call (fn [args & {:keys [allocator?]}]
+                        ;; NOTE(Joshua): If `args` is a symbol, that means we're
+                        ;; taking restargs, and so the downcall must be applied
+                        (-> `(~@(when (symbol? args) [`apply])
+                              ~downcall-sym
+                              ~@(when allocator? [`(mem/arena-allocator ~arena)])
+                              ~@(if (symbol? args)
+                                  [args]
+                                  args))
+                            wrap-serialize))
+
+            deserialize-prim (fn [expr]
+                               `(mem/deserialize* ~expr ~ret-type-sym))
+            deserialize-segment (fn [expr]
+                                  `(mem/deserialize-from ~expr ~ret-type-sym))
+            deserialize-ret (fn [expr]
+                              (cond
+                                (and (or (mem/primitive? ret-type)
+                                         (#{::mem/void} ret-type))
+                                     (not (#{::mem/pointer} (mem/primitive-type ret-type))))
+                                expr
+
+                                (mem/primitive-type ret-type)
+                                (deserialize-prim expr)
+
+                                :else
+                                (deserialize-segment expr)))
+
+            wrap-arena (fn [expr]
+                           `(with-open [~arena (mem/confined-arena)]
+                              ~expr))
+            wrap-fn (fn [call needs-arena?]
+                      `(fn [~@(if const-args? arg-syms ['& args-sym])]
+                         ~(cond-> call needs-arena? wrap-arena)))]
+        `(let [;; NOTE(Joshua): To ensure all arguments are evaluated once and
+               ;; in-order, they must be bound here
+               ~downcall-sym ~downcall
+               ~@(if const-args?
+                   (mapcat vector arg-type-syms arg-types)
+                   [args-types-sym arg-types])
+               ~ret-type-sym ~ret-type]
+           ~(if const-ret?
+              (-> (make-call (if const-args? arg-syms args-sym)
+                             :allocator? (not (mem/primitive-type ret-type)))
+                  deserialize-ret
+                  (wrap-fn (or (not simple-args?)
+                               (not simple-ret?))))
+              (let [prim-call (-> (make-call (if const-args? arg-syms args-sym)
+                                             :allocator? false)
+                                  deserialize-prim)
+                    non-prim-call (-> (make-call (if const-args? arg-syms args-sym)
+                                                 :allocator? true)
+                                      deserialize-segment)]
+                `(if (mem/primitive-type ~ret-type-sym)
+                   ~(wrap-fn prim-call (not simple-args?))
+                   ~(wrap-fn non-prim-call true)))))))))
+
 (defn make-serde-wrapper
   "Constructs a wrapper function for the `downcall` which serializes the arguments
   and deserializes the return value."
+  {:inline (fn [downcall arg-types ret-type]
+             (inline-serde-wrapper downcall arg-types ret-type))}
   [downcall arg-types ret-type]
   (if (mem/primitive-type ret-type)
     (fn native-fn [& args]
-      (with-open [scope (mem/stack-scope)]
+      (with-open [arena (mem/confined-arena)]
-        (mem/deserialize
+        (mem/deserialize*
-         (apply downcall (map #(mem/serialize %1 %2 scope) args arg-types))
+         (apply downcall (map #(mem/serialize %1 %2 arena) args arg-types))
          ret-type)))
     (fn native-fn [& args]
-      (with-open [scope (mem/stack-scope)]
+      (with-open [arena (mem/confined-arena)]
-        (mem/deserialize
+        (mem/deserialize-from
-         (apply downcall (mem/scope-allocator scope)
+         (apply downcall (mem/arena-allocator arena)
-                (map #(mem/serialize %1 %2 scope) args arg-types))
+                (map #(mem/serialize %1 %2 arena) args arg-types))
          ret-type)))))
 
 (defn make-serde-varargs-wrapper
@@ -262,10 +439,14 @@
   "Constructs a Clojure function to call the native function referenced by `symbol`.
 
   The function returned will serialize any passed arguments into the `args`
-  types, and deserialize the return to the `ret` type."
+  types, and deserialize the return to the `ret` type.
+
+  If your `args` and `ret` are constants, then it is more efficient to
+  call [[make-downcall]] followed by [[make-serde-wrapper]] because the latter
+  has an inline definition which will result in less overhead from serdes."
+  ;; TODO(Joshua): Add an inline arity for when the args and ret types are constant
   [symbol args ret]
   (-> symbol
-      ensure-address
       (make-downcall args ret)
       (make-serde-wrapper args ret)))
 
@@ -277,7 +458,7 @@
   arguments."
   [symbol required-args ret]
   (-> symbol
-      ensure-address
+      ensure-symbol
       (make-varargs-factory required-args ret)
       (make-serde-varargs-wrapper required-args ret)))
 
@@ -289,17 +470,24 @@
    ::mem/short :sreturn
    ::mem/int :ireturn
    ::mem/long :lreturn
-   ::mem/long-long :lreturn
    ::mem/char :creturn
    ::mem/float :freturn
    ::mem/double :dreturn
    ::mem/void :return})
 
-(defn- upcall-class
+(def ^:private double-sized?
-  "Constructs a class definition for a class with a single method, `upcall`, which
+  "Set of primitive types which require 2 indices in the constant pool."
-  boxes any primitives passed to it and calls a closed over [[IFn]]."
+  #{::mem/double ::mem/long})
+
+(defn- upcall-class-ctor*
+  "Returns a function to construct an upcall class for the given `arg-types` and `ret-types`.
+
+  An upcall class is a class with a single method, `upcall`, which boxes any
+  primitives passed to it and calls a closed over [[IFn]]."
   [arg-types ret-type]
+  (let [klass (insn/define
                 {:flags #{:public :final}
+                 :version 8
                  :fields [{:name "upcall_ifn"
                            :type IFn
                            :flags #{:final}}]
@@ -314,23 +502,54 @@
                                    [:return]]}
                            {:name :upcall
                             :flags #{:public}
-              :desc (conj (mapv mem/java-layout arg-types)
+                            :desc (conj (mapv insn-layout arg-types)
-                          (mem/java-layout ret-type))
+                                        (insn-layout ret-type))
                             :emit [[:aload 0]
                                    [:getfield :this "upcall_ifn" IFn]
-                     (map-indexed
+                                   (loop [types arg-types
-                      (fn [idx arg]
+                                          acc []
-                        [[(load-instructions arg) (inc idx)]
+                                          idx 1]
-                         (to-object-asm arg)])
+                                     (if (seq types)
-                      arg-types)
+                                       (let [prim (mem/primitive-type (first types))]
+                                         (recur (rest types)
+                                                (conj acc [[(load-instructions prim :aload) idx]
+                                                           (to-object-asm (first types))])
+                                                (cond-> (inc idx)
+                                                  (double-sized? prim)
+                                                  inc)))
+                                       acc))
                                    [:invokeinterface IFn "invoke" (repeat (inc (count arg-types)) Object)]
                                    (to-prim-asm ret-type)
                                    [(return-for-type ret-type :areturn)]]}]})
+        ctor (.getConstructor klass
+                              (doto ^"[Ljava.lang.Class;" (make-array Class 1)
+                                (aset 0 IFn)))]
+    (fn [^IFn f]
+      (.newInstance ctor
+                    (doto (object-array 1)
+                      (aset 0 f))))))
+
+(def ^:private upcall-class-ctor
+  "Returns a function to construct an upcall class for the given memoized `arg-types` and `ret-types`.
+
+  An upcall class is a class with a single method, `upcall`, which boxes any
+  primitives passed to it and calls a closed over [[IFn]]."
+  (memoize upcall-class-ctor*))
 
 (defn- upcall
-  "Constructs an instance of [[upcall-class]], closing over `f`."
+  "Constructs an instance of an upcall class, closing over `f`.
+
+  See [[upcall-class-ctor]]."
   [f arg-types ret-type]
-  (insn/new-instance (upcall-class arg-types ret-type) ^IFn f))
+  ((upcall-class-ctor arg-types ret-type) ^IFn f))
+
+(defn- method-type
+  "Gets the [[MethodType]] for a set of `args` and `ret` types."
+  ([args] (method-type args ::mem/void))
+  ([args ret]
+   (MethodType/methodType
+    ^Class (mem/java-layout ret)
+    ^"[Ljava.lang.Class;" (into-array Class (map mem/java-layout args)))))
 
 (defn- upcall-handle
   "Constructs a method handle for invoking `f`, a function of `arg-count` args."
@@ -347,52 +566,75 @@
 
 (defn- upcall-serde-wrapper
   "Creates a function that wraps `f` which deserializes the arguments and
-  serializes the return type in the [[global-scope]]."
+  serializes the return type in the [[global-arena]]."
   [f arg-types ret-type]
   (fn [& args]
     (mem/serialize
      (apply f (map mem/deserialize args arg-types))
      ret-type
-     (mem/global-scope))))
+     (mem/global-arena))))
 
 (defmethod mem/serialize* ::fn
-  [f [_fn arg-types ret-type & {:keys [raw-fn?]}] scope]
+  [f [_fn arg-types ret-type & {:keys [raw-fn?]} :as typ] arena]
+  (if-let [address (::address (meta f))]
+    (do (assert (= typ (::type (meta f)))
+                "The type of a deserialized function must match the type it is re-serialized to.")
+        address)
     (.upcallStub
-   (CLinker/getInstance)
+     (Linker/nativeLinker)
-   (cond-> f
+     ^MethodHandle (cond-> f
                      (not raw-fn?) (upcall-serde-wrapper arg-types ret-type)
                      :always (upcall-handle arg-types ret-type))
-   (function-descriptor arg-types ret-type)
+     ^FunctionDescriptor (function-descriptor arg-types ret-type)
-   scope))
+     ^Arena arena
+     (make-array Linker$Option 0))))
 
 (defmethod mem/deserialize* ::fn
-  [addr [_fn arg-types ret-type & {:keys [raw-fn?]}]]
+  [addr [_fn arg-types ret-type & {:keys [raw-fn?] :as typ}]]
-  (-> addr
+  (when-not (mem/null? addr)
-      (downcall-handle
+    (vary-meta
-       (method-type arg-types ret-type)
+      (-> ^MemorySegment addr
-       (function-descriptor arg-types ret-type))
+          (downcall-handle (function-descriptor arg-types ret-type))
           (downcall-fn arg-types ret-type)
-      (cond->
+          (cond-> (not raw-fn?) (make-serde-wrapper arg-types ret-type)))
-        (not raw-fn?) (make-serde-wrapper arg-types ret-type))))
+      assoc
+      ::address addr
+      ::type typ)))
 
 ;;; Static memory access
 
 (defn const
   "Gets the value of a constant stored in `symbol-or-addr`."
   [symbol-or-addr type]
-  (mem/deserialize (ensure-address symbol-or-addr) [::mem/pointer type]))
+  (mem/deserialize (ensure-symbol symbol-or-addr) [::mem/pointer type]))
 
-(deftype StaticVariable [addr type meta]
+(s/def ::defconst-args
-  Addressable
+  (s/cat :var-name simple-symbol?
-  (address [_]
+         :docstring (s/? string?)
-    addr)
+         :symbol-or-addr any?
+         :type ::mem/type))
+
+(defmacro defconst
+  "Defines a var named by `symbol` to be the value of the given `type` from `symbol-or-addr`."
+  {:arglists '([symbol docstring? symbol-or-addr type])}
+  [& args]
+  (let [args (s/conform ::defconst-args args)]
+    `(let [symbol# (ensure-symbol ~(:symbol-or-addr args))]
+       (def ~(:var-name args)
+         ~@(when-let [doc (:docstring args)]
+             (list doc))
+         (const symbol# ~(:type args))))))
+(s/fdef defconst
+  :args ::defconst-args)
+
+(deftype StaticVariable [seg type meta]
   IDeref
   (deref [_]
-    (mem/deserialize addr [::mem/pointer type]))
+    (mem/deserialize-from seg type))
 
   IObj
   (withMeta [_ meta-map]
-    (StaticVariable. addr type (atom meta-map)))
+    (StaticVariable. seg type (atom meta-map)))
   IMeta
   (meta [_]
     @meta)
@@ -407,8 +649,8 @@
   [^StaticVariable static-var newval]
   (mem/serialize-into
    newval (.-type static-var)
-   (mem/slice-global (.-addr static-var) (mem/size-of (.-type static-var)))
+   (.-seg static-var)
-   (mem/global-scope))
+   (mem/global-arena))
   newval)
 
 (defn fswap!
@@ -419,15 +661,37 @@
   [static-var f & args]
   (freset! static-var (apply f @static-var args)))
 
+(defn static-variable-segment
+  "Gets the backing [[MemorySegment]] from `static-var`.
+
+  This is primarily useful when you need to pass the static variable's address
+  to a native function which takes an [[Addressable]]."
+  ^MemorySegment [static-var]
+  (.-seg ^StaticVariable static-var))
+
 (defn static-variable
   "Constructs a reference to a mutable value stored in `symbol-or-addr`.
 
-  The returned value can be dereferenced, and has metadata, and the address of
+  The returned value can be dereferenced, and has metadata.
-  the value can be queried with [[address-of]].
 
   See [[freset!]], [[fswap!]]."
   [symbol-or-addr type]
-  (StaticVariable. (ensure-address symbol-or-addr) type (atom nil)))
+  (StaticVariable. (.reinterpret ^MemorySegment (ensure-symbol symbol-or-addr)
+                                 ^long (mem/size-of type))
+                   type (atom nil)))
+
+(defmacro defvar
+  "Defines a var named by `symbol` to be a reference to the native memory from `symbol-or-addr`."
+  {:arglists '([symbol docstring? symbol-or-addr type])}
+  [& args]
+  (let [args (s/conform ::defconst-args args)]
+    `(let [symbol# (ensure-symbol ~(:symbol-or-addr args))]
+       (def ~(:var-name args)
+         ~@(when-let [doc (:docstring args)]
+             (list doc))
+         (static-variable symbol# ~(:type args))))))
+(s/fdef defvar
+  :args ::defconst-args)
 
 (s/def :coffi.ffi.symbolspec/symbol string?)
 (s/def :coffi.ffi.symbolspec/type keyword?)
@@ -500,7 +764,7 @@
                    (s/or :string string?
                          :symbol simple-symbol?))
           :native-arglist (s/coll-of ::mem/type :kind vector?)
-          :return-type qualified-keyword?
+          :return-type ::mem/type
           :wrapper (s/?
                     (s/cat
                      :native-fn simple-symbol?
@@ -540,9 +804,7 @@
    :style/indent [:defn]}
   [& args]
   (let [args (s/conform ::defcfn-args args)
-        args-types (gensym "args-types")
+        address (gensym "symbol")
-        ret-type (gensym "ret-type")
-        invoke (gensym "invoke")
         native-sym (gensym "native")
         [arity fn-tail] (-> args :wrapper :fn-tail)
         fn-tail (case arity
@@ -553,16 +815,11 @@
                               :single-arity [fn-tail]
                               :multi-arity fn-tail
                               nil))]
-    `(let [~args-types ~(:native-arglist args)
+    `(let [~address (find-symbol ~(name (:symbol args)))
-           ~ret-type ~(:return-type args)
+           ~(or (-> args :wrapper :native-fn)
-           ~invoke (make-downcall ~(name (:symbol args)) ~args-types ~ret-type)
+                native-sym)
-           ~(or (-> args :wrapper :native-fn) native-sym)
+           (-> (make-downcall ~address ~(:native-arglist args) ~(:return-type args))
-           ~(if (and (every? #(= % (mem/primitive-type %))
+               (make-serde-wrapper ~(:native-arglist args) ~(:return-type args)))
-                             (:native-arglist args))
-                     (= (:return-type args)
-                        (mem/primitive-type (:return-type args))))
-              invoke
-              `(make-serde-wrapper ~invoke ~args-types ~ret-type))
            fun# ~(if (:wrapper args)
                    `(fn ~(:name args)
                       ~@fn-tail)
@@ -576,9 +833,14 @@
                               (or old-list
                                   (seq arglists)
                                   (list
-                                   (mapv (comp symbol name)
+                                   (mapv (fn [type]
+                                           (-> (cond-> type
+                                                 (vector? type) first)
+                                               name
+                                               symbol))
                                          (:native-arglist args)))))))
-                   (:attr-map args)))
+                   (assoc (:attr-map args)
+                          ::address address)))
          ~@(when-let [doc (:doc args)]
              (list doc))
          fun#))))

src/clj/coffi/layout.clj

@@ -0,0 +1,31 @@
+(ns coffi.layout
+  "Functions for adjusting the layout of structs."
+  (:require
+   [coffi.mem :as mem]))
+
+(defn with-c-layout
+  "Forces a struct specification to C layout rules.
+
+  This will add padding fields between fields to match C alignment
+  requirements."
+  [struct-spec]
+  (let [aligned-fields
+        (loop [offset 0
+               aligned-fields []
+               fields (nth struct-spec 1)]
+          (if (seq fields)
+            (let [[[_ type :as field] & fields] fields
+                  size (mem/size-of type)
+                  align (mem/align-of type)
+                  r (rem offset align)]
+              (recur (cond-> (+ offset size)
+                       (pos? r) (+ (- align r)))
+                     (cond-> aligned-fields
+                       (pos? r) (conj [:coffi.layout/padding [:coffi.mem/padding (- align r)]])
+                       :always (conj field))
+                     fields))
+            (let [strongest-alignment (reduce max (map (comp mem/align-of second) (nth struct-spec 1)))
+                  r (rem offset strongest-alignment)]
+              (cond-> aligned-fields
+                (pos? r) (conj [:coffi.layout/padding [:coffi.mem/padding (- strongest-alignment r)]])))))]
+    (assoc struct-spec 1 aligned-fields)))

src/java/coffi/ffi/Loader.java

@@ -1,6 +1,6 @@
 package coffi.ffi;
 
-import jdk.incubator.foreign.*;
+import java.lang.foreign.*;
 
 /**
  * Loading libraries with the {@link System#load} and {@link System#loadLibrary}
@@ -10,6 +10,8 @@ import jdk.incubator.foreign.*;
  */
 public class Loader {
 
+    static SymbolLookup lookup = Linker.nativeLinker().defaultLookup().or(SymbolLookup.loaderLookup());
+
     /**
      * Loads a library from a given absolute file path.
      *
@@ -36,8 +38,7 @@ public class Loader {
      *
      * @param symbol The name of the symbol to load from a library.
      */
-    public static MemoryAddress findSymbol(String symbol) {
+    public static MemorySegment findSymbol(String symbol) {
-        return CLinker.systemLookup().lookup(symbol)
+        return lookup.find(symbol).orElse(null);
-            .orElseGet(() -> SymbolLookup.loaderLookup().lookup(symbol).orElse(null));
     }
 }

test/c/ffi_test.c

@@ -1,4 +1,8 @@
 #include <stdio.h>
+#include <stdlib.h>
+
+const int c = 42;
+const char *s = "Test string";
 
 int add_numbers(int a, int b) {
     return a + b;
@@ -26,10 +30,19 @@ CString upcall_test(StringFactory fun) {
     return fun();
 }
 
-static int counter = 0;
+int upcall_test2(int (*f)(void)) {
+    return f();
+}
+
+char *mut_str = NULL;
+int counter = 0;
 
 static char* responses[] = { "Hello, world!", "Goodbye friend.", "co'oi prenu" };
 
+char* upcall_test_int_fn_string_ret(int (*f)(void)) {
+    return responses[f()];
+}
+
 CString get_string1(void) {
     return responses[counter++ % 3];
 }
@@ -48,3 +61,63 @@ StringFactory get_downcall(int whichString) {
         return 0;
     }
 }
+
+typedef struct alignment_test {
+    char a;
+    double x;
+    float y;
+} AlignmentTest;
+
+AlignmentTest get_struct() {
+    AlignmentTest ret = {};
+    ret.a = 'x';
+    ret.x = 3.14;
+    ret.y = 42.0;
+
+    return ret;
+}
+
+void test_call_with_trailing_string_arg(int a, int b, char* text) {
+    printf("call of `test_call_with_trailing_string_arg` with a=%i b=%i text='%s'",1,2,text);
+    printf("\r                                                                          ");
+    return;
+}
+
+int freed = 0;
+
+int get_variable_length_array(float **arr) {
+    freed = 0;
+    *arr = malloc(sizeof(float) * 7);
+
+    for (int i = 0; i < 7; ++i) {
+        (*arr)[i] = 1.5f * i;
+    }
+
+    return 7;
+}
+
+void free_variable_length_array(float *arr) {
+    freed = 1;
+    free(arr);
+}
+
+typedef struct complextype {
+    Point x;
+    char  y;
+    int   z[4];
+    char *w;
+} ComplexType;
+
+ComplexType complexTypeTest(ComplexType a) {
+  ComplexType ret = {};
+  ret.x = a.x;
+  ret.x.x++;
+  ret.x.y++;
+  ret.y = a.y-1;
+  ret.z[0] = a.z[0];
+  ret.z[1] = a.z[1];
+  ret.z[2] = a.z[2];
+  ret.z[3] = a.z[3];
+  ret.w = "hello from c";
+  return ret;
+}

test/clj/coffi/ffi_test.clj

@@ -1,14 +1,20 @@
 (ns coffi.ffi-test
   (:require
    [clojure.test :as t]
+   [coffi.ffi :as ffi]
+   [coffi.layout :as layout]
    [coffi.mem :as mem]
-   [coffi.ffi :as ffi]))
+   [clojure.pprint]))
 
 (ffi/load-library "target/ffi_test.so")
 
 (t/deftest can-load-symbols
   (t/is (not (nil? (ffi/find-symbol "add_numbers")))))
 
+(t/deftest can-fetch-constant
+  (t/is (= 42 (ffi/const "c" ::mem/int)))
+  (t/is (= "Test string" (ffi/const "s" ::mem/c-string))))
+
 (t/deftest can-call-primitive-fns
   (t/is (= 5 ((ffi/cfn "add_numbers" [::mem/int ::mem/int] ::mem/int) 2 3))))
 
@@ -28,5 +34,103 @@
 
 (t/deftest can-make-upcall
   (t/is (= ((ffi/cfn "upcall_test" [[::ffi/fn [] ::mem/c-string]] ::mem/c-string)
-            (fn [] "hello"))
+            (fn [] "hello from clojure from c from clojure"))
-           "hello")))
+           "hello from clojure from c from clojure")))
+
+(t/deftest can-make-upcall2
+  (t/is (= ((ffi/cfn "upcall_test2" [[::ffi/fn [] ::mem/int]] ::mem/int)
+            (fn [] 5))
+           5)))
+
+(t/deftest can-make-upcall-int-fn-string-ret
+  (t/is (= ((ffi/cfn "upcall_test_int_fn_string_ret" [[::ffi/fn [] ::mem/int]] ::mem/c-string)
+            (fn [] 2))
+           "co'oi prenu")))
+
+(mem/defalias ::alignment-test
+  (layout/with-c-layout
+    [::mem/struct
+     [[:a ::mem/char]
+      [:x ::mem/double]
+      [:y ::mem/float]]]))
+
+(t/deftest padding-matches
+  (t/is (= (dissoc ((ffi/cfn "get_struct" [] ::alignment-test)) ::layout/padding)
+           {:a \x
+            :x 3.14
+            :y 42.0})))
+
+(t/deftest static-variables-are-mutable
+  (let [mut-str (ffi/static-variable "mut_str" ::mem/c-string)]
+    (ffi/freset! mut-str nil)
+    (t/is (nil? @mut-str))
+    (ffi/freset! mut-str "Hello world!")
+    (t/is (= "Hello world!" @mut-str)))
+  (ffi/freset! (ffi/static-variable "counter" ::mem/int) 1)
+  (t/is (= ((ffi/cfn "get_string1" [] ::mem/c-string))
+           "Goodbye friend.")))
+
+(t/deftest can-call-with-trailing-string-arg
+  (t/is
+   (= (try ((ffi/cfn "test_call_with_trailing_string_arg"
+                     [::mem/int ::mem/int ::mem/c-string]
+                     ::mem/void)
+            1 2 "third arg")
+           :ok
+           (catch Throwable _t
+             :err))
+      :ok)))
+
+(ffi/defvar freed? "freed" ::mem/int)
+
+(def get-variable-length-array* (ffi/make-downcall "get_variable_length_array" [::mem/pointer] ::mem/int))
+(def free-variable-length-array* (ffi/make-downcall "free_variable_length_array" [::mem/pointer] ::mem/void))
+
+(t/deftest get-variable-length-array
+  (let [floats
+        (with-open [stack (mem/confined-arena)]
+          (let [out-floats (mem/alloc mem/pointer-size stack)
+                num-floats (get-variable-length-array* out-floats)
+                floats-addr (mem/read-address out-floats)
+                floats-slice (mem/reinterpret floats-addr (unchecked-multiply-int mem/float-size num-floats))]
+            (try
+              (loop [floats (transient [])
+                     index 0]
+                (if (>= index num-floats)
+                  (persistent! floats)
+                  (recur (conj! floats (mem/read-float floats-slice (unchecked-multiply-int index mem/float-size)))
+                         (unchecked-inc-int index))))
+              (finally
+                (free-variable-length-array* floats-addr)))))]
+    (t/is (not (zero? @freed?)))
+    (t/is (= floats (mapv #(* (float 1.5) %) (range (count floats)))))))
+
+(mem/defstruct Point [x ::mem/float y ::mem/float])
+
+(t/deftest can-call-with-defstruct
+  (t/is (= {:x 2.0 :y 2.0}
+           ((ffi/cfn "add_points" [::Point ::Point] ::Point) (Point. 1 2) (Point. 1 0)))))
+
+(mem/defstruct AlignmentTest [a ::mem/char x ::mem/double y ::mem/float])
+
+(t/deftest padding-matches-defstruct
+  (t/is (= ((ffi/cfn "get_struct" [] ::AlignmentTest))
+           {:a \x
+            :x 3.14
+            :y 42.0})))
+
+(mem/defstruct ComplexType [x ::Point y ::mem/byte z [::mem/array ::mem/int 4 :raw? true] w ::mem/c-string])
+
+(t/deftest can-call-with-complex-defstruct
+  (t/are [x y] (= x (y ((ffi/cfn "complexTypeTest" [::ComplexType] ::ComplexType)
+                        (ComplexType. (Point. 2 3) 4 (int-array [5 6 7 8]) "hello from clojure"))))
+    {:x {:x 3.0 :y 4.0} :y 3 :w "hello from c"} #(dissoc % :z)
+    [5 6 7 8] (comp vec :z)))
+
+(mem/defstruct ComplexTypeWrapped [x ::Point y ::mem/byte z [::mem/array ::mem/int 4] w ::mem/c-string])
+
+(t/deftest can-call-with-wrapped-complex-defstruct
+  (t/are [x y] (= x (y ((ffi/cfn "complexTypeTest" [::ComplexTypeWrapped] ::ComplexTypeWrapped)
+                        (ComplexTypeWrapped. (Point. 2 3) 4 (int-array [5 6 7 8]) "hello from clojure"))))
+    {:x {:x 3.0 :y 4.0} :y 3 :w "hello from c"} #(dissoc % :z)
+    [5 6 7 8] (comp vec :z)))

test/clj/coffi/mem_test.clj

@@ -0,0 +1,135 @@
+(ns coffi.mem-test
+  (:require
+   [clojure.test :as t]
+   [coffi.ffi :as ffi]
+   [coffi.mem :as mem])
+  (:import
+   (java.lang.foreign
+    AddressLayout
+    Arena
+    MemoryLayout
+    MemorySegment
+    MemorySegment$Scope
+    SegmentAllocator
+    ValueLayout
+    ValueLayout$OfByte
+    ValueLayout$OfShort
+    ValueLayout$OfInt
+    ValueLayout$OfLong
+    ValueLayout$OfChar
+    ValueLayout$OfFloat
+    ValueLayout$OfDouble)
+   (java.lang.ref Cleaner)
+   (java.nio ByteOrder)))
+
+(ffi/load-library "target/ffi_test.so")
+
+(t/deftest can-serialize-string
+  (t/is
+   (instance? MemorySegment (mem/serialize "this is a string" ::mem/c-string))))
+
+(t/deftest can-define-struct
+  (t/is
+   (eval
+    `(mem/defstruct ~'TestType [~'a ::mem/int ~'b ::mem/byte]))))
+
+(mem/defstruct TestType [a ::mem/int b ::mem/byte c ::mem/short])
+
+(t/deftest can-initialize-struct
+  (t/is (TestType. 5 10 15)))
+
+(t/deftest can-use-common-map-functions
+  (let [v1 (TestType. 5 10 15)
+        v2 (TestType. 6 11 16)]
+    (t/are [x y] (= x (y v1))
+     5  :a
+     10 :b
+     15 :c
+     5  (fn [v] (v :a))
+     10 (fn [v] (v :b))
+     15 (fn [v] (v :c))
+     5  #(get % :a)
+     10 #(get % :b)
+     15 #(get % :c)
+     20 #(get % :d 20)
+     nil #(get % :d)
+     [:a :b :c] keys
+     [5 10 15]  vals
+     {:a 5 :c 15} #(dissoc % :b)
+     {:a 5 :b 10 :c 0} #(assoc % :c 0)
+     {:a 5 :b 10 :c 15 :d 20} #(assoc % :d 20)
+     [[:a 5] [:b 10] [:c 15]] seq
+     {:a 5 :b 10 :c 15 :d 20} #(merge % {:d 20})
+     {:a [5 6] :b [10 11] :c [15 16]} #(merge-with vector % {:a 6 :b 11 :c 16})
+     {:a [5 6] :b [10 11] :c [15 16]} #(merge-with vector % v2)
+     [:a 5] #(find % :a)
+     nil #(find % :d)
+     {:a 5 :b 10 :c 15} identity
+     v1 identity
+     v1 (fn [s] {:a 5 :b 10 :c 15}))))
+
+(t/deftest can-serialize-struct-type
+  (t/is
+   (instance? MemorySegment (mem/serialize (TestType. 5 10 15) ::TestType))))
+
+(t/deftest can-deserialize-struct-type
+  (t/is
+   (= {:a 5 :b 10 :c 15}
+      (mem/deserialize (mem/serialize (TestType. 5 10 15) ::TestType) ::TestType))))
+
+(mem/defstruct NestedTestType [x ::mem/int y ::mem/byte z ::TestType])
+
+(t/deftest can-instantiated-nested-structs
+  (t/is
+   (= {:x 5 :y 6 :z {:a 5 :b 10 :c 15}}
+      (NestedTestType. 5 6 (TestType. 5 10 15)))))
+
+(t/deftest can-define-structs-with-array-members
+  (t/is
+   (eval
+    `(mem/defstruct ~'ArrayTestType [~'x ::mem/int ~'y ::mem/byte ~'z [::mem/array ::mem/int 4 :raw? true]]))))
+
+(mem/defstruct ArrayTestType [x ::mem/int y ::mem/byte z [::mem/array ::mem/int 4 :raw? true]])
+
+(t/deftest can-instantiated-array-member-structs
+  (t/are [x y z] (z x (y (ArrayTestType. 5 6 (int-array [1 2 3 4]))))
+    {:x 5 :y 6} #(dissoc % :z) =
+    (int-array [1 2 3 4]) :z java.util.Arrays/equals))
+
+(t/deftest can-serialize-array-struct
+  (t/is
+   (= [5 6 1 2 3 4]
+      (vec (filter #(not= 0 %) (vec (.toArray (mem/serialize (ArrayTestType. 5 6 (int-array [1 2 3 4])) ::ArrayTestType) mem/byte-layout)))))))
+
+(t/deftest can-serialize-deserialize-array-struct
+  (t/is
+   (java.util.Arrays/equals
+    (int-array [1 2 3 4])
+    (.z (mem/deserialize (mem/serialize (ArrayTestType. 5 6 (int-array [1 2 3 4])) ::ArrayTestType) ::ArrayTestType)))))
+
+(mem/defstruct ComplexTestType [x [::mem/array ::ArrayTestType 4 :raw? true] y ::mem/byte z [::mem/array ::mem/int 4 :raw? true] w ::NestedTestType])
+
+(t/deftest can-serialize-deserialize-complex-struct-type
+  (t/is
+   (let [x (object-array (map #(ArrayTestType. % % (int-array (range 4))) (range 4)))
+         y 12
+         z (int-array (range 4))
+         w (NestedTestType. 5 6 (TestType. 5 10 15))]
+     (->
+      (ComplexTestType. x y z w)
+      (mem/serialize ::ComplexTestType)
+      (mem/deserialize ::ComplexTestType)))))
+
+(mem/defstruct ComplexTestTypeWrapped [x [::mem/array ::ArrayTestType 4] y ::mem/byte z [::mem/array ::mem/int 4] w ::NestedTestType])
+
+(t/deftest can-serialize-deserialize-complex-wrapped-struct-type
+  (t/is
+   (let [x (vec (map #(ArrayTestType. % % (int-array (range 4))) (range 4)))
+         y 12
+         z (vec (range 4))
+         w (NestedTestType. 5 6 (TestType. 5 10 15))]
+     (->
+      (ComplexTestTypeWrapped. x y z w)
+      (mem/serialize ::ComplexTestTypeWrapped)
+      (mem/deserialize ::ComplexTestTypeWrapped)))))
+