telemere/README.md

353 lines
22 KiB
Markdown
Raw Permalink Normal View History

2023-10-16 08:50:36 +00:00
<a href="https://www.taoensso.com/clojure" title="More stuff by @ptaoussanis at www.taoensso.com"><img src="https://www.taoensso.com/open-source.png" alt="Taoensso open source" width="340"/></a>
2025-12-16 09:16:51 +00:00
[**API**][cljdoc] | [**Wiki**][GitHub wiki] | [Slack][] | Latest release: [v1.2.1](../../releases/tag/v1.2.1) (2025-12-16)
[![Clj tests][Clj tests SVG]][Clj tests URL]
[![Cljs tests][Cljs tests SVG]][Cljs tests URL]
[![Graal tests][Graal tests SVG]][Graal tests URL]
2023-10-16 08:50:36 +00:00
2024-04-15 07:57:40 +00:00
# <img src="https://raw.githubusercontent.com/taoensso/telemere/master/imgs/telemere-logo.svg" alt="Telemere logo" width="360"/>
2023-10-16 08:50:36 +00:00
2025-02-28 09:53:11 +00:00
### Structured logs and telemetry for Clojure/Script
2023-10-16 08:50:36 +00:00
**Telemere** is the next-gen version of [Timbre](https://www.taoensso.com/timbre). It offers **one API** to cover:
2024-03-06 10:54:51 +00:00
2024-08-28 20:22:20 +00:00
- **Traditional logging** (string messages)
- **Structured logging** (rich Clojure data types and structures)
- **Tracing** (nested flow tracking, with optional data)
- Basic **performance monitoring** (nested form runtimes)
2024-08-24 08:57:56 +00:00
It's pure Clj/s, small, **easy to use**, super fast, and **seriously flexible**:
```clojure
(tel/log! {:level :info, :id ::login, :data {:user-id 1234}, :msg "User logged in!"})
```
Works great with:
2025-04-30 12:53:54 +00:00
- [Trove](https://www.taoensso.com/trove) for logging by **library authors**
- [Tufte](https://www.taoensso.com/tufte) for rich **performance monitoring**
- [Truss](https://www.taoensso.com/truss) for **assertions** and error handling
2025-04-30 12:53:54 +00:00
## Why structured logging?
2024-03-06 10:54:51 +00:00
- Traditional logging outputs **strings** (messages).
- Structured logging in contrast outputs **data**. It retains **rich data types and (nested) structures** throughout the logging pipeline from logging callsite → filters → middleware → handlers.
2023-10-16 08:50:36 +00:00
A data-oriented pipeline can make a huge difference - supporting **easier filtering**, **transformation**, and **analysis**. Its also usually **faster**, since you only pay for serialization if/when you need it. In a lot of cases you can avoid serialization altogether if your final target (DB, etc.) supports the relevant types.
2024-08-28 20:22:20 +00:00
The structured (data-oriented) approach is inherently more flexible, faster, and well suited to the tools and idioms offered by Clojure and ClojureScript.
2024-08-28 20:22:20 +00:00
## Examples
2024-08-28 20:22:20 +00:00
See [examples.cljc](https://github.com/taoensso/telemere/blob/master/examples.cljc) for REPL-ready snippets, or expand below:
2024-08-28 20:22:20 +00:00
<details><summary>Create signals</summary><br/>
2024-10-02 08:42:22 +00:00
2024-08-22 07:02:21 +00:00
```clojure
(require '[taoensso.telemere :as tel])
2024-08-22 07:02:21 +00:00
2025-06-13 09:10:24 +00:00
;; No config needed for typical use cases!!
;; Signals print to console by default for both Clj and Cljs
2024-08-24 08:57:56 +00:00
;; Traditional style logging (data formatted into message string):
(tel/log! {:level :info, :msg (str "User " 1234 " logged in!")})
;; Modern/structured style logging (explicit id and data)
(tel/log! {:level :info, :id :auth/login, :data {:user-id 1234}})
2024-08-22 07:02:21 +00:00
;; Mixed style (explicit id and data, with message string)
(tel/log! {:level :info, :id :auth/login, :data {:user-id 1234}, :msg "User logged in!"})
2024-08-22 07:02:21 +00:00
2024-08-28 20:22:20 +00:00
;; Trace (can interop with OpenTelemetry)
2024-08-24 08:57:56 +00:00
;; Tracks form runtime, return value, and (nested) parent tree
(tel/trace! {:id ::my-id :data {...}}
2024-08-22 07:02:21 +00:00
(do-some-work))
2024-08-24 08:57:56 +00:00
;; Check resulting signal content for debug/tests
(tel/with-signal (tel/log! {...})) ; => {:keys [ns level id data msg_ ...]}
2024-08-22 07:02:21 +00:00
;; Getting fancy (all costs are conditional!)
(tel/log!
{:level :debug
:sample 0.75 ; 75% sampling (noop 25% of the time)
:when (my-conditional)
:limit {"1 per sec" [1 1000]
"5 per min" [5 60000]} ; Rate limit
:limit-by my-user-ip-address ; Rate limit scope
2024-08-22 07:02:21 +00:00
:do (inc-my-metric!)
:let
[diagnostics (my-expensive-diagnostics)
formatted (my-expensive-format diagnostics)]
:data
{:diagnostics diagnostics
:formatted formatted
:local-state *my-dynamic-context*}}
;; Message string or vector to join as string
["Something interesting happened!" formatted])
2024-10-02 08:42:22 +00:00
```
2024-09-25 12:07:56 +00:00
2024-10-02 08:42:22 +00:00
</details>
<details><summary>Filter signals</summary><br/>
```clojure
2024-09-25 12:07:56 +00:00
;; Set minimum level
(tel/set-min-level! :warn) ; For all signals
(tel/set-min-level! :log :debug) ; For `log!` signals specifically
2024-09-25 12:07:56 +00:00
;; Set id and namespace filters
(tel/set-id-filter! {:allow #{::my-particular-id "my-app/*"}})
(tel/set-ns-filter! {:disallow "taoensso.*" :allow "taoensso.sente.*"})
;; SLF4J signals will have their `:ns` key set to the logger's name
;; (typically a source class)
(tel/set-ns-filter! {:disallow "com.noisy.java.package.*"})
2024-09-25 12:07:56 +00:00
;; Set minimum level for `log!` signals for particular ns pattern
(tel/set-min-level! :log "taoensso.sente.*" :warn)
2024-09-25 12:07:56 +00:00
;; Use transforms (xfns) to filter and/or arbitrarily modify signals
;; by signal data/content/etc.
2024-09-25 12:07:56 +00:00
(tel/set-xfn!
2024-09-25 12:07:56 +00:00
(fn [signal]
(if (-> signal :data :skip-me?)
nil ; Filter signal (don't handle)
(assoc signal :transformed? true))))
2024-09-25 12:07:56 +00:00
(tel/with-signal (tel/log! {... :data {:skip-me? true}})) ; => nil
(tel/with-signal (tel/log! {... :data {:skip-me? false}})) ; => {...}
2024-09-25 12:07:56 +00:00
;; See `tel/help:filters` docstring for more filtering options
2024-08-22 07:02:21 +00:00
```
2024-10-02 08:42:22 +00:00
</details>
2024-03-06 10:54:51 +00:00
2024-10-02 08:42:22 +00:00
<details><summary>Add handlers</summary><br/>
2024-04-11 09:05:24 +00:00
2024-08-22 07:02:21 +00:00
```clojure
2024-09-25 12:07:56 +00:00
;; Add your own signal handler
(tel/add-handler! :my-handler
2024-09-25 12:07:56 +00:00
(fn
([signal] (println signal))
2024-10-02 08:42:22 +00:00
([] (println "Handler has shut down"))))
2024-09-25 12:07:56 +00:00
;; Use `add-handler!` to set handler-level filtering and back-pressure
(tel/add-handler! :my-handler
2024-09-25 12:07:56 +00:00
(fn
([signal] (println signal))
2024-10-02 08:42:22 +00:00
([] (println "Handler has shut down")))
2024-08-05 10:59:18 +00:00
2024-08-22 07:02:21 +00:00
{:async {:mode :dropping, :buffer-size 1024, :n-threads 1}
:priority 100
:sample 0.5
:min-level :info
:ns-filter {:disallow "taoensso.*"}
:limit {"1 per sec" [1 1000]}
;; See `tel/help:handler-dispatch-options` for more
2024-08-22 07:02:21 +00:00
})
2024-08-24 08:57:56 +00:00
2024-09-25 12:07:56 +00:00
;; See current handlers
(tel/get-handlers) ; => {<handler-id> {:keys [handler-fn handler-stats_ dispatch-opts]}}
2024-09-25 12:07:56 +00:00
2024-10-20 18:49:37 +00:00
;; Add console handler to print signals as human-readable text
(tel/add-handler! :my-handler
(tel/handler:console
{:output-fn (tel/format-signal-fn {})}))
2024-08-24 08:57:56 +00:00
2024-10-20 18:49:37 +00:00
;; Add console handler to print signals as edn
(tel/add-handler! :my-handler
(tel/handler:console
{:output-fn (tel/pr-signal-fn {:pr-fn :edn})}))
2024-08-24 08:57:56 +00:00
2024-10-20 18:49:37 +00:00
;; Add console handler to print signals as JSON
2024-08-24 08:57:56 +00:00
;; Ref. <https://github.com/metosin/jsonista> (or any alt JSON lib)
#?(:clj (require '[jsonista.core :as jsonista]))
(tel/add-handler! :my-handler
(tel/handler:console
2024-08-24 08:57:56 +00:00
{:output-fn
#?(:cljs :json ; Use js/JSON.stringify
2024-09-25 12:07:56 +00:00
:clj jsonista/write-value-as-string)}))
2024-04-11 09:05:24 +00:00
```
2024-10-02 08:42:22 +00:00
</details>
## Why Telemere?
### Ergonomics
- Elegant unified API that's **easy to use** and **deeply flexible**.
- Pure **Clojure vals and fns** for easy config, composition, and REPL debugging.
- **Sensible defaults** to get started fast.
- **Beginner-oriented** [documentation][GitHub wiki], [docstrings](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere), and error messages.
2024-10-02 08:42:22 +00:00
### Interop
- **Interop ready** with [tools.logging](../../wiki/3-Config#toolslogging), [Java logging via SLF4J v2](../../wiki/3-Config#java-logging), [OpenTelemetry](../../wiki/3-Config#opentelemetry), and [Tufte](../../wiki/3-Config#tufte).
- [Timbre shim](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.timbre) for easy/gradual [migration from Timbre](../../wiki/5-Migrating).
2024-10-02 08:42:22 +00:00
- Extensive set of [handlers](../../wiki/4-Handlers#included-handlers) included out-the-box.
### Scaling
- Rich [filtering](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:filters) by namespace, id pattern, level, level by namespace pattern, etc.
- Fully [configurable](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:handler-dispatch-options) **a/sync dispatch support** with per-handler [performance monitoring](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#get-handlers-stats).
- Turn-key **sampling**, **rate limiting**, and **back-pressure monitoring**.
- Highly optimized and [blazing fast](#performance)!
2024-10-02 08:42:22 +00:00
## Comparisons
- Telemere [compared](../../wiki/5-Migrating#from-timbre) to [Timbre](https://www.taoensso.com/timbre) (Telemere's predecessor)
2025-12-10 07:42:36 +00:00
- Telemere [compared](../../wiki/6-FAQ#how-does-telemere-compare-to-%CE%BClog) to [μ/log](https://github.com/BrunoBonacci/mulog) (structured micro-logging library)
2024-10-02 08:42:22 +00:00
2024-12-20 13:44:34 +00:00
## Videos
2024-10-02 08:42:22 +00:00
2024-12-20 13:44:34 +00:00
### Lightning intro (7 mins):
<a href="https://www.youtube.com/watch?v=lOaZ0SgPVu4" target="_blank">
<img src="https://img.youtube.com/vi/lOaZ0SgPVu4/maxresdefault.jpg" alt="Telemere lightning intro" width="480" border="0" />
</a>
### REPL demo (24 mins):
2024-10-02 08:42:22 +00:00
<a href="https://www.youtube.com/watch?v=-L9irDG8ysM" target="_blank">
<img src="https://img.youtube.com/vi/-L9irDG8ysM/maxresdefault.jpg" alt="Telemere demo video" width="480" border="0" />
</a>
2024-04-15 07:57:40 +00:00
2024-04-11 09:05:24 +00:00
## API overview
### Creating signals
2024-04-11 09:05:24 +00:00
80% of Telemere's functionality is available through one macro: [`signal!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#signal!) and a rich set of [opts](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-options).
Use that directly, or any of the wrapper macros that you find most convenient. They're **semantically equivalent** but have ergonomics slightly tweaked for different common use cases:
| Name | Args | Returns |
| :---------------------------------------------------------------------------------------------------------- | :------------------------- | :--------------------------- |
| [`log!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#log!) | `[opts]` or `[?level msg]` | nil |
| [`event!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#event!) | `[opts]` or `[id ?level]` | nil |
| [`trace!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#trace!) | `[opts]` or `[?id run]` | Form result |
| [`spy!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#spy!) | `[opts]` or `[?level run]` | Form result |
| [`error!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#error!) | `[opts]` or `[?id error]` | Given error |
| [`catch->error!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#catch-%3Eerror!) | `[opts]` or `[?id error]` | Form value or given fallback |
| [`signal!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#signal!) | `[opts]` | Depends on opts |
2024-03-06 10:54:51 +00:00
2024-04-11 09:05:24 +00:00
### Internal help
2024-08-05 10:59:18 +00:00
Detailed help is available without leaving your IDE:
2024-08-22 07:02:21 +00:00
| Var | Help with |
| :---------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------- |
| [`help:signal-creators`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-creators) | Creating signals |
| [`help:signal-options`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-options) | Options when creating signals |
| [`help:signal-content`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-content) | Signal content (map given to transforms/handlers) |
2024-08-22 07:02:21 +00:00
| [`help:filters`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:filters) | Signal filtering and transformation |
| [`help:handlers`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:handlers) | Signal handler management |
| [`help:handler-dispatch-options`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:handler-dispatch-options) | Signal handler dispatch options |
| [`help:environmental-config`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:environmental-config) | Config via JVM properties, environment variables, or classpath resources |
2024-04-11 09:05:24 +00:00
2024-08-28 20:22:20 +00:00
## Performance
Telemere is **highly optimized** and offers great performance at any scale, handling up to **4.2 million filtered signals/sec** on a 2020 Macbook Pro M1.
Signal call benchmarks (per thread):
| Compile-time filtering? | Runtime filtering? | Profile? | Trace? | nsecs / call |
| :---------------------: | :----------------: | :------: | :----: | -----------: |
| ✓ (elide) | - | - | - | 0 |
| - | ✓ | - | - | 350 |
| - | ✓ | ✓ | - | 450 |
| - | ✓ | ✓ | ✓ | 1000 |
- Nanoseconds per signal call ~ **milliseconds per 1e6 calls**
- Times exclude [handler runtime](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#get-handlers-stats) (which depends on handler/s, is usually async)
- Benched on a 2020 Macbook Pro M1, running Clojure v1.12 and OpenJDK v22
### Performance philosophy
Telemere is optimized for *real-world* performance. This means **prioritizing flexibility** and realistic usage over synthetic micro-benchmarks.
Large applications can produce absolute *heaps* of data, not all equally valuable. Quickly processing infinite streams of unmanageable junk is an anti-pattern. As scale and complexity increase, it becomes more important to **strategically plan** what data to collect, when, in what quantities, and how to manage it.
Telemere is designed to help with all that. It offers [rich data](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-content) and unmatched [filtering](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:filters) support - including per-signal and per-handler **sampling** and **rate limiting**, and zero cost compile-time filtering.
2024-08-28 20:22:20 +00:00
Use these to ensure that you're not capturing useless/low-value/high-noise information in production! With appropriate planning, Telemere is designed to scale to systems of any size and complexity.
See [here](../../wiki/7-Tips) for detailed tips on real-world usage.
## Included handlers
2024-04-11 09:05:24 +00:00
2024-08-22 07:02:21 +00:00
See ✅ links below for **features and usage**,
2024-08-24 08:57:56 +00:00
See ❤️ links below to **vote on future handlers**:
| Target (↓) | Clj | Cljs |
| :--------------------------------------------- | :-----------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------: |
| [Apache Kafka](https://kafka.apache.org/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| [AWS Kinesis](https://aws.amazon.com/kinesis/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| Console | [](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console) | [](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console) |
| Console (raw) | - | [](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console-raw) |
| [Datadog](https://www.datadoghq.com/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | [❤️](https://github.com/taoensso/roadmap/issues/12) |
| Email | [](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.postal#handler:postal) | - |
| [Graylog](https://graylog.org/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| [Jaeger](https://www.jaegertracing.io/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| [Logstash](https://www.elastic.co/logstash) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| [OpenTelemetry](https://opentelemetry.io/) | [](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.open-telemetry#handler:open-telemetry) | [❤️](https://github.com/taoensso/roadmap/issues/12) |
| [Redis](https://redis.io/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| SQL | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| [Slack](https://slack.com/) | [](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.slack#handler:slack) | - |
| TCP socket | [](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.sockets#handler:tcp-socket) | - |
| UDP socket | [](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.sockets#handler:udp-socket) | - |
| [Zipkin](https://zipkin.io/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
2024-08-20 12:00:45 +00:00
You can also easily [write your own handlers](../../wiki/4-Handlers#writing-handlers).
2024-04-11 09:05:24 +00:00
2024-08-19 16:33:17 +00:00
## Community
My plan for Telemere is to offer a **stable core of limited scope**, then to focus on making it as easy for the **community** to write additional stuff like handlers, transforms, and utils.
2024-08-19 16:33:17 +00:00
See [here](../../wiki/8-Community) for community resources.
2024-04-11 09:05:24 +00:00
## Documentation
2024-04-15 07:57:40 +00:00
- [Wiki][GitHub wiki] (getting started, usage, etc.)
2025-02-20 21:56:30 +00:00
- API reference via [cljdoc][cljdoc]
2024-08-07 10:00:22 +00:00
- Extensive [internal help](#internal-help) (no need to leave your IDE)
2025-11-04 09:18:50 +00:00
- Support via [Slack][] or [GitHub issues][]
2024-05-08 07:54:08 +00:00
- [General observability tips](../../wiki/7-Tips) (advice on building and maintaining observable Clojure/Script systems, and getting the most out of Telemere)
2024-03-06 10:54:51 +00:00
2023-10-16 08:50:36 +00:00
## Funding
2025-06-19 13:01:41 +00:00
You can [help support][sponsor] continued work on this project and [others][my work], thank you!! 🙏
2023-10-16 08:50:36 +00:00
## License
2025-02-28 09:53:11 +00:00
Copyright &copy; 2023-2025 [Peter Taoussanis][].
2023-10-16 08:50:36 +00:00
Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).
<!-- Common -->
[GitHub releases]: ../../releases
[GitHub issues]: ../../issues
[GitHub wiki]: ../../wiki
2025-11-04 09:18:50 +00:00
[Slack]: https://www.taoensso.com/telemere/slack
2023-10-16 08:50:36 +00:00
[Peter Taoussanis]: https://www.taoensso.com
[sponsor]: https://www.taoensso.com/sponsor
2025-06-19 13:01:41 +00:00
[my work]: https://www.taoensso.com/clojure-libraries
2023-10-16 08:50:36 +00:00
<!-- Project -->
2025-02-20 21:56:30 +00:00
[cljdoc]: https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere
2023-10-16 08:50:36 +00:00
[Clojars SVG]: https://img.shields.io/clojars/v/com.taoensso/telemere.svg
[Clojars URL]: https://clojars.org/com.taoensso/telemere
2025-06-20 09:01:15 +00:00
[Clj tests SVG]: https://github.com/taoensso/telemere/actions/workflows/clj-tests.yml/badge.svg
[Clj tests URL]: https://github.com/taoensso/telemere/actions/workflows/clj-tests.yml
[Cljs tests SVG]: https://github.com/taoensso/telemere/actions/workflows/cljs-tests.yml/badge.svg
[Cljs tests URL]: https://github.com/taoensso/telemere/actions/workflows/cljs-tests.yml
2023-10-16 08:50:36 +00:00
[Graal tests SVG]: https://github.com/taoensso/telemere/actions/workflows/graal-tests.yml/badge.svg
2025-08-22 08:25:27 +00:00
[Graal tests URL]: https://github.com/taoensso/telemere/actions/workflows/graal-tests.yml