luciano/telemere
1
0
Fork 0
mirror of https://github.com/taoensso/telemere.git synced 2025-12-25 21:08:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
telemere/README.md

359 lines
22 KiB
Markdown
Raw Blame History

<a href="https://www.taoensso.com/clojure" title="More stuff by @ptaoussanis at www.taoensso.com"><img src="https://www.taoensso.com/open-source.png" alt="Taoensso open source" width="340"/></a>
[**API**][cljdoc docs] | [**Wiki**][GitHub wiki] | [Latest releases](#latest-releases) | [Slack channel][]
# <img src="https://raw.githubusercontent.com/taoensso/telemere/master/imgs/telemere-logo.svg" alt="Telemere logo" width="360"/>
### Structured telemetry library for Clojure/Script
**Telemere** is a **pure Clojure/Script library** that offers an elegant and simple **unified API** to cover:
- **Traditional logging** (string messages)
- **Structured logging** (rich Clojure data types and structures)
- **Events** (named thing happened, with optional data)
- **Tracing** (nested flow tracking, with optional data)
- Basic **performance monitoring** (nested form runtimes)
- Any combination of the above
It's small, *super* fast, easy to learn, easy to use, and **absurdly flexible**.
It helps enable Clojure/Script systems that are easily **observable**, **robust**, and **debuggable** - and it represents the refinement and culmination of ideas brewing over 12+ years in [Timbre](https://www.taoensso.com/timbre), [Tufte](https://www.taoensso.com/tufte) and [Truss](https://www.taoensso.com/truss).
See [here](../../wiki/1-Getting-started) for **full introduction** (concepts, terminology, getting started).
## Latest release/s
- `2024-12-24` `v1.0.0-RC2`: [release info](../../releases/tag/v1.0.0-RC2)
[![Main tests][Main tests SVG]][Main tests URL]
[![Graal tests][Graal tests SVG]][Graal tests URL]
<!--See [here][GitHub releases] for earlier releases.-->
## Next-gen observability
A key hurdle in building **observable systems** is that it's often inconvenient and costly to get out the kind of **detailed info** that we need when debugging.
Telemere's strategy to address this is to:
1. Provide **lean, low-fuss syntax** to let you conveniently convey program state.
2. Use the unique power of **Lisp macros** to let you **dynamically filter costs as you filter signals** (pay only for what you need, when you need it).
3. For those signals that *do* pass filtering: move costs from the callsite to a/sync handlers with explicit [threading and back-pressure semantics](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:handler-dispatch-options) and [performance monitoring](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#get-handlers-stats).
The effect is more than impressive micro-benchmarks. This approach enables a fundamental (qualitative) change in one's approach to observability.
It enables you to write code that is **information-verbose by default**.
## Quick examples
> (Or see [examples.cljc](https://github.com/taoensso/telemere/blob/master/examples.cljc) for REPL-ready snippets)
<details open><summary>Create signals</summary><br/>
```clojure
(require '[taoensso.telemere :as t])
;; (Just works / no config necessary for typical use cases)
;; Without structured data
(t/log! :info "Hello world!") ; %> Basic log signal (has message)
(t/event! ::my-id :debug) ; %> Basic event signal (just id)
;; With structured data
(t/log! {:level :info, :data {...}} "Hello again!")
(t/event! ::my-id {:level :debug, :data {...}})
;; Trace (can interop with OpenTelemetry)
;; Tracks form runtime, return value, and (nested) parent tree
(t/trace! {:id ::my-id :data {...}}
(do-some-work))
;; Check resulting signal content for debug/tests
(t/with-signal (t/event! ::my-id)) ; => {:keys [ns level id data msg_ ...]}
;; Getting fancy (all costs are conditional!)
(t/log!
{:level :debug
:sample-rate 0.75 ; 75% sampling (noop 25% of the time)
:when (my-conditional)
:rate-limit {"1 per sec" [1 1000]
"5 per min" [5 60000]}
:rate-limit-by my-user-ip-address ; Optional rate-limit scope
:do (inc-my-metric!)
:let
[diagnostics (my-expensive-diagnostics)
formatted (my-expensive-format diagnostics)]
:data
{:diagnostics diagnostics
:formatted formatted
:local-state *my-dynamic-context*}}
;; Message string or vector to join as string
["Something interesting happened!" formatted])
```
</details>
<details><summary>Filter signals</summary><br/>
```clojure
;; Set minimum level
(t/set-min-level! :warn) ; For all signals
(t/set-min-level! :log :debug) ; For `log!` signals only
;; Set id and namespace filters
(t/set-id-filter! {:allow #{::my-particular-id "my-app/*"}})
(t/set-ns-filter! {:disallow "taoensso.*" :allow "taoensso.sente.*"})
;; SLF4J signals will have their `:ns` key set to the logger's name
;; (typically a source class)
(t/set-ns-filter! {:disallow "com.noisy.java.package.*"})
;; Set minimum level for `event!` signals for particular ns pattern
(t/set-min-level! :event "taoensso.sente.*" :warn)
;; Use middleware to:
;; - Transform signals
;; - Filter signals by arb conditions (incl. data/content)
(t/set-middleware!
(fn [signal]
(if (-> signal :data :skip-me?)
nil ; Filter signal (don't handle)
(assoc signal :passed-through-middleware? true))))
(t/with-signal (t/event! ::my-id {:data {:skip-me? true}})) ; => nil
(t/with-signal (t/event! ::my-id {:data {:skip-me? false}})) ; => {...}
;; See `t/help:filters` docstring for more filtering options
```
</details>
<details><summary>Add handlers</summary><br/>
```clojure
;; Add your own signal handler
(t/add-handler! :my-handler
(fn
([signal] (println signal))
([] (println "Handler has shut down"))))
;; Use `add-handler!` to set handler-level filtering and back-pressure
(t/add-handler! :my-handler
(fn
([signal] (println signal))
([] (println "Handler has shut down")))
{:async {:mode :dropping, :buffer-size 1024, :n-threads 1}
:priority 100
:sample-rate 0.5
:min-level :info
:ns-filter {:disallow "taoensso.*"}
:rate-limit {"1 per sec" [1 1000]}
;; See `t/help:handler-dispatch-options` for more
})
;; See current handlers
(t/get-handlers) ; => {<handler-id> {:keys [handler-fn handler-stats_ dispatch-opts]}}
;; Add console handler to print signals as human-readable text
(t/add-handler! :my-handler
(t/handler:console
{:output-fn (t/format-signal-fn {})}))
;; Add console handler to print signals as edn
(t/add-handler! :my-handler
(t/handler:console
{:output-fn (t/pr-signal-fn {:pr-fn :edn})}))
;; Add console handler to print signals as JSON
;; Ref. <https://github.com/metosin/jsonista> (or any alt JSON lib)
#?(:clj (require '[jsonista.core :as jsonista]))
(t/add-handler! :my-handler
(t/handler:console
{:output-fn
#?(:cljs :json ; Use js/JSON.stringify
:clj jsonista/write-value-as-string)}))
```
</details>
## Why Telemere?
### Ergonomics
- Elegant, lightweight API that's **easy to use**, **easy to configure**, and **deeply flexible**.
- **Sensible defaults** to make getting started **fast and easy**.
- Extensive **beginner-oriented** [documentation][GitHub wiki], [docstrings](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere), and error messages.
### Interop
- 1st-class **out-the-box interop** with [tools.logging](../../wiki/3-Config#toolslogging), [Java logging via SLF4J v2](../../wiki/3-Config#java-logging), [OpenTelemetry](../../wiki/3-Config#opentelemetry), and [Tufte](../../wiki/3-Config#tufte).
- Included [shim](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.timbre) for easy/gradual [migration from Timbre](../../wiki/5-Migrating).
- Extensive set of [handlers](../../wiki/4-Handlers#included-handlers) included out-the-box.
### Scaling
- Hyper-optimized and **blazing fast**, see [performance](#performance).
- An API that **scales comfortably** from the smallest disposable code, to the most massive and complex real-world production environments.
- Auto [handler stats](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#get-handlers-stats) for debugging performance and other issues at scale.
### Flexibility
- Config via plain **Clojure vals and fns** for easy customization, composition, and REPL debugging.
- Unmatched [environmental config](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:environmental-config) support: JVM properties, environment variables, or classpath resources. Per platform, or cross-platform.
- Unmatched [filtering](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:filters) support: by namespace, id pattern, level, level by namespace pattern, etc. At runtime and compile-time.
- Fully [configurable](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:handler-dispatch-options) **a/sync dispatch support**: blocking, dropping, sliding, etc.
- Turn-key **sampling**, **rate-limiting**, and **back-pressure monitoring** with sensible defaults.
## Comparisons
- Telemere [compared](../../wiki/5-Migrating#from-timbre) to [Timbre](https://www.taoensso.com/timbre) (Telemere's predecessor)
- Telemere [compared](../../wiki/6-FAQ#how-does-telemere-compare-to-mulog) to [Mulog](https://github.com/BrunoBonacci/mulog) (Structured micro-logging library)
## Videos
### Lightning intro (7 mins):
<a href="https://www.youtube.com/watch?v=lOaZ0SgPVu4" target="_blank">
<img src="https://img.youtube.com/vi/lOaZ0SgPVu4/maxresdefault.jpg" alt="Telemere lightning intro" width="480" border="0" />
</a>
### REPL demo (24 mins):
<a href="https://www.youtube.com/watch?v=-L9irDG8ysM" target="_blank">
<img src="https://img.youtube.com/vi/-L9irDG8ysM/maxresdefault.jpg" alt="Telemere demo video" width="480" border="0" />
</a>
## API overview
See relevant docstrings (links below) for usage info-
### Creating signals
| Name | Kind | Args | Returns |
| :---------------------------------------------------------------------------------------------------------- | :--------- | :--------------- | :--------------------------- |
| [`signal!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#signal!) | `:generic` | `opts` | Depends on opts |
| [`event!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#event!) | `:event` | `id` + `?level` | Signal allowed? |
| [`log!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#log!) | `:log` | `?level` + `msg` | Signal allowed? |
| [`trace!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#trace!) | `:trace` | `?id` + `run` | Form result |
| [`spy!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#spy!) | `:spy` | `?level` + `run` | Form result |
| [`error!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#error!) | `:error` | `?id` + `error` | Given error |
| [`catch->error!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#catch-%3Eerror!) | `:error` | `?id` | Form value or given fallback |
### Internal help
Detailed help is available without leaving your IDE:
| Var | Help with |
| :---------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------- |
| [`help:signal-creators`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-creators) | Creating signals |
| [`help:signal-options`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-options) | Options when creating signals |
| [`help:signal-content`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-content) | Signal content (map given to middleware/handlers) |
| [`help:filters`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:filters) | Signal filtering and transformation |
| [`help:handlers`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:handlers) | Signal handler management |
| [`help:handler-dispatch-options`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:handler-dispatch-options) | Signal handler dispatch options |
| [`help:environmental-config`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:environmental-config) | Config via JVM properties, environment variables, or classpath resources |
## Performance
Telemere is **highly optimized** and offers great performance at any scale, handling up to **4.2 million filtered signals/sec** on a 2020 Macbook Pro M1.
Signal call benchmarks (per thread):
| Compile-time filtering? | Runtime filtering? | Profile? | Trace? | nsecs / call |
| :---------------------: | :----------------: | :------: | :----: | -----------: |
| ✓ (elide) | - | - | - | 0 |
| - | ✓ | - | - | 350 |
| - | ✓ | ✓ | - | 450 |
| - | ✓ | ✓ | ✓ | 1000 |
- Nanoseconds per signal call ~ **milliseconds per 1e6 calls**
- Times exclude [handler runtime](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#get-handlers-stats) (which depends on handler/s, is usually async)
- Benched on a 2020 Macbook Pro M1, running Clojure v1.12 and OpenJDK v22
### Performance philosophy
Telemere is optimized for *real-world* performance. This means **prioritizing flexibility** and realistic usage over synthetic micro-benchmarks.
Large applications can produce absolute *heaps* of data, not all equally valuable. Quickly processing infinite streams of unmanageable junk is an anti-pattern. As scale and complexity increase, it becomes more important to **strategically plan** what data to collect, when, in what quantities, and how to manage it.
Telemere is designed to help with all that. It offers [rich data](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-content) and unmatched [filtering](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:filters) support - including per-signal and per-handler **sampling** and **rate-limiting**, and zero cost compile-time filtering.
Use these to ensure that you're not capturing useless/low-value/high-noise information in production! With appropriate planning, Telemere is designed to scale to systems of any size and complexity.
See [here](../../wiki/7-Tips) for detailed tips on real-world usage.
## Included handlers
See ✅ links below for **features and usage**,
See ❤️ links below to **vote on future handlers**:
| Target (↓) | Clj | Cljs |
| :--------------------------------------------- | :-----------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------: |
| [Apache Kafka](https://kafka.apache.org/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| [AWS Kinesis](https://aws.amazon.com/kinesis/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| Console | [✅](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console) | [✅](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console) |
| Console (raw) | - | [✅](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console-raw) |
| [Datadog](https://www.datadoghq.com/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | [❤️](https://github.com/taoensso/roadmap/issues/12) |
| Email | [✅](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.postal#handler:postal) | - |
| [Graylog](https://graylog.org/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| [Jaeger](https://www.jaegertracing.io/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| [Logstash](https://www.elastic.co/logstash) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| [OpenTelemetry](https://opentelemetry.io/) | [✅](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.open-telemetry#handler:open-telemetry) | [❤️](https://github.com/taoensso/roadmap/issues/12) |
| [Redis](https://redis.io/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| SQL | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
| [Slack](https://slack.com/) | [✅](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.slack#handler:slack) | - |
| TCP socket | [✅](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.sockets#handler:tcp-socket) | - |
| UDP socket | [✅](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.sockets#handler:udp-socket) | - |
| [Zipkin](https://zipkin.io/) | [❤️](https://github.com/taoensso/roadmap/issues/12) | - |
You can also easily [write your own handlers](../../wiki/4-Handlers#writing-handlers).
## Community
My plan for Telemere is to offer a **stable core of limited scope**, then to focus on making it as easy for the **community** to write additional stuff like handlers, middleware, and utils.
See [here](../../wiki/8-Community) for community resources.
## Documentation
- [Wiki][GitHub wiki] (getting started, usage, etc.)
- API reference via [cljdoc][cljdoc docs] or [Codox][Codox docs]
- Extensive [internal help](#internal-help) (no need to leave your IDE)
- Support via [Slack channel][] or [GitHub issues][]
- [General observability tips](../../wiki/7-Tips) (advice on building and maintaining observable Clojure/Script systems, and getting the most out of Telemere)
## Funding
You can [help support][sponsor] continued work on this project, thank you!! 🙏
## License
Copyright &copy; 2023-2024 [Peter Taoussanis][].
Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).
<!-- Common -->
[GitHub releases]: ../../releases
[GitHub issues]: ../../issues
[GitHub wiki]: ../../wiki
[Slack channel]: https://www.taoensso.com/telemere/slack
[Peter Taoussanis]: https://www.taoensso.com
[sponsor]: https://www.taoensso.com/sponsor
<!-- Project -->
[Codox docs]: https://taoensso.github.io/telemere/
[cljdoc docs]: https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere
[Clojars SVG]: https://img.shields.io/clojars/v/com.taoensso/telemere.svg
[Clojars URL]: https://clojars.org/com.taoensso/telemere
[Main tests SVG]: https://github.com/taoensso/telemere/actions/workflows/main-tests.yml/badge.svg
[Main tests URL]: https://github.com/taoensso/telemere/actions/workflows/main-tests.yml
[Graal tests SVG]: https://github.com/taoensso/telemere/actions/workflows/graal-tests.yml/badge.svg
[Graal tests URL]: https://github.com/taoensso/telemere/actions/workflows/graal-tests.yml
Reference in a new issue View git blame Copy permalink