Compare commits

...

5 commits

Author SHA1 Message Date
Peter Taoussanis
37eed1ef08 v1.0.0-beta4 (2024-04-29) 2024-04-29 09:28:44 +02:00
Peter Taoussanis
f348c06a75 [doc] Documentation improvements 2024-04-29 09:21:36 +02:00
Peter Taoussanis
28618f2115 [mod] Don't include user-level kvs in default signal content handler 2024-04-29 09:21:36 +02:00
Peter Taoussanis
2c7ee74901 [new] Add postal (email) handler 2024-04-29 09:21:36 +02:00
Peter Taoussanis
838683204f [nop] Misc housekeeping 2024-04-29 09:18:02 +02:00
26 changed files with 520 additions and 234 deletions

View file

@ -2,6 +2,36 @@ This project uses [**Break Versioning**](https://www.taoensso.com/break-versioni
---
# `v1.0.0-beta4` (2024-04-29)
> **Dep/s**: [Telemere](https://clojars.org/com.taoensso/telemere/versions/1.0.0-beta4) and [Telemere SLF4J provider](https://clojars.org/com.taoensso/slf4j-telemere/versions/1.0.0-beta4) are on Clojars.
> **Versioning**: Telemere uses [Break Versioning](https://www.taoensso.com/break-versioning).
This is a **maintenance pre-release** intended to fix issues that have come up during the beta. See below for details, and please **report any unexpected problems** on [GitHub](https://github.com/taoensso/telemere/issues) or the [Slack channel](https://www.taoensso.com/telemere/slack), thank you! 🙏
\- Peter Taoussanis
## Changes since `v1.0.0-beta1`
* d0a15bac [mod] Don't auto add OpenTelemetry handler
* 6d545dfc [mod] Move (simplify) OpenTelemetry ns
* 28618f21 [mod] Don't include user-level kvs in default signal content handler
* d3c63e17 [mod] Rename `clojure.tools.logging` sys val
## Fixes since `v1.0.0-beta1`
* ffea1a30 [fix] Fix broken AOT support, add AOT tests
* e222297a [fix] SLF4J broken timestamps, add tests
## New since `v1.0.0-beta1`
* 2c7ee749 [new] Add postal (email) handler
* Handlers will now drain their signal queues on shutdown (configurable)
* Rate limiter performance improvements (via Encore)
* Doc improvements based on questions that've come up on Slack, etc.
---
# `v1.0.0-beta3` (2024-04-23)
> **Dep/s**: [Telemere](https://clojars.org/com.taoensso/telemere/versions/1.0.0-beta3) and [Telemere SLF4J provider](https://clojars.org/com.taoensso/slf4j-telemere/versions/1.0.0-beta3) are on Clojars.

View file

@ -13,7 +13,7 @@ It helps enable Clojure/Script systems that are **observable**, **robust**, and
## Latest release/s
- `v1.0.0-beta3`: [release info](../../releases/tag/v1.0.0-beta3)
- `v1.0.0-beta4`: [release info](../../releases/tag/v1.0.0-beta4)
[![Main tests][Main tests SVG]][Main tests URL]
[![Graal tests][Graal tests SVG]][Graal tests URL]
@ -41,7 +41,7 @@ See [here][GitHub releases] for earlier releases.
#### Flexibility
- Config via plain **Clojure vals and fns** for easy customization, composition, and REPL debugging.
- Unmatched support for **system-level config** (JVM props, ENV vars, classpath resources).
- Unmatched **environmental config** support (JVM properties, environment variables, or classpath resources).
- Expressive **per-call** and **per-handler** filtering at both **runtime** and **compile-time**.
- Filter by namespace and id pattern, level, **level by namespace pattern**, etc.
- **Sampling**, **rate-limiting**, and **back-pressure monitoring**.

View file

@ -160,7 +160,8 @@
(defn handler:my-handler ; Note naming convention
"Returns a (fn handler [signal] that:
- Does something.
- Takes a Telemere signal.
- Does something with it.
Options:
`:option1` - Description
@ -169,8 +170,9 @@
([] (handler:my-handler nil)) ; Use default opts
([{:as constructor-opts}]
;; Do expensive prep outside returned handler fn whenever possible -
;; i.e. at (one-off) construction time rather than handling time.
;; Do option validation and expensive prep *outside* returned handler
;; fn whenever possible - i.e. at (one-off) construction time rather than
;; at every handler call.
(let []
(fn a-handler:my-handler ; Note naming convention

View file

@ -1,4 +1,4 @@
(defproject com.taoensso/telemere "1.0.0-beta3"
(defproject com.taoensso/telemere "1.0.0-beta4"
:author "Peter Taoussanis <https://www.taoensso.com>"
:description "Structured telemetry library for Clojure/Script"
:url "https://www.taoensso.com/telemere"
@ -8,16 +8,16 @@
:url "https://www.eclipse.org/legal/epl-v10.html"}
:dependencies
[[com.taoensso/encore "3.104.1"]]
[[com.taoensso/encore "3.105.0"]]
:test-paths ["test" #_"src"]
:profiles
{;; :default [:base :system :user :provided :dev]
:provided {:dependencies [[org.clojure/clojurescript "1.11.132"]
[org.clojure/clojure "1.11.2"]]}
[org.clojure/clojure "1.11.3"]]}
:c1.12 {:dependencies [[org.clojure/clojure "1.12.0-alpha9"]]}
:c1.11 {:dependencies [[org.clojure/clojure "1.11.2"]]}
:c1.11 {:dependencies [[org.clojure/clojure "1.11.3"]]}
:c1.10 {:dependencies [[org.clojure/clojure "1.10.1"]]}
:graal-tests
@ -45,12 +45,13 @@
[[org.clojure/test.check "1.1.1"]
[org.clojure/tools.logging "1.3.0"]
[org.slf4j/slf4j-api "2.0.13"]
[com.taoensso/slf4j-telemere "1.0.0-beta3"]
;; [org.slf4j/slf4j-simple "2.0.13"]
;; [org.slf4j/slf4j-nop "2.0.13"]
[io.opentelemetry/opentelemetry-api "1.37.0"]
[io.opentelemetry/opentelemetry-sdk-extension-autoconfigure "1.37.0"]
[io.opentelemetry/opentelemetry-exporter-otlp "1.37.0"]]
[com.taoensso/slf4j-telemere "1.0.0-beta4"]
#_[org.slf4j/slf4j-simple "2.0.13"]
#_[org.slf4j/slf4j-nop "2.0.13"]
[com.draines/postal "2.0.5"]
[io.opentelemetry/opentelemetry-api "1.37.0"]
#_[io.opentelemetry/opentelemetry-sdk-extension-autoconfigure "1.37.0"]
#_[io.opentelemetry/opentelemetry-exporter-otlp "1.37.0"]]
:plugins
[[lein-pprint "1.3.2"]

View file

@ -0,0 +1,52 @@
Environmental filter config includes:
Kind filter:
JVM property: `taoensso.telemere.rt-kind-filter.edn`
Env variable: `TAOENSSO_TELEMERE_RT_KIND_FILTER_EDN`
Classpath resource: `taoensso.telemere.rt-kind-filter.edn`
Namespace filter:
JVM property: `taoensso.telemere.rt-ns-filter.edn`
Env variable: `TAOENSSO_TELEMERE_RT_NS_FILTER_EDN`
Classpath resource: `taoensso.telemere.rt-ns-filter.edn`
Id filter:
JVM property: `taoensso.telemere.rt-id-filter.edn`
Env variable: `TAOENSSO_TELEMERE_RT_ID_FILTER_EDN`
Classpath resource: `taoensso.telemere.rt-id-filter.edn`
Minimum level:
JVM property: `taoensso.telemere.rt-min-level.edn`
Env variable: `TAOENSSO_TELEMERE_RT_MIN_LEVEL_EDN`
Classpath resource: `taoensso.telemere.rt-min-level.edn`
Examples:
`taoensso.telemere.rt-min-level.edn` -> ":info"
`TAOENSSO_TELEMERE_RT_NS_FILTER_EDN` -> "{:deny \"taoensso.*\"}"
`taoensso.telemere.rt-id-filter.cljs.edn` -> "#{:my-id1 :my-id2}"
`TAOENSSO_TELEMERE_RT_KIND_FILTER_CLJ_EDN` -> "nil"
Tips:
- The above ids are for runtime filters (the most common).
For compile-time filters, change `rt`->`ct` / `RT`->`CT`.
- The above ids will affect both Clj AND Cljs.
For platform-specific filters, use
".clj.edn" / "_CLJ_EDN" or
".cljs.edn" / "_CLJS_EDN" suffixes instead.
- ".edn" / "_EDN" suffixes are optional.
- Config values should be edn. To get the right syntax, first set
your runtime filters using the standard utils (`set-min-level!`,
etc.). Then call `get-filters` and serialize the relevant parts
to edn with `pr-str`.
- All environmental config uses `get-env` underneath.
See the `get-env` docstring for more/advanced details.
- Classpath resources are files accessible on your project's
classpath. This usually includes files in your project's
`resources/` dir.

View file

@ -28,7 +28,9 @@ Default signal keys:
`:file` -------- ?str filename of signal creator callsite, same as (:file location)
`:sample-rate` - ?rate ∈ℝ[0,1] for combined signal AND handler sampling (0.75 => allow 75% of signals, nil => allow all)
<kvs> ---------- Arb other user-level ?kvs given to signal creator
<kvs> ---------- Other arb user-level ?kvs given to signal creator. Typically NOT included
in handler output, so a great way to provide custom data/opts for use
(only) by custom middleware/handlers.
If anything is unclear, please ping me (@ptaoussanis) so that I can improve these docs!

View file

@ -1,4 +1,5 @@
A signal will be provided to a handler iff ALL of the following are true:
1. Signal (creation) is allowed by compile-time filters
2. Signal (creation) is allowed by runtime filters
3. Signal (handling) is allowed by handler filters
@ -7,18 +8,40 @@ A signal will be provided to a handler iff ALL of the following are true:
5. Handler middleware does not suppress the signal (return nil)
For 1-3, filtering may depend on (in order):
Sample rate → namespace → kind → id → level → when form/fn → rate limit
Note that sample rates are multiplicative:
If a signal is created with 20% sampling and a handler handles 50%
of given signals, then 10% of possible signals will be handled.
Compile-time vs runtime filtering:
This multiplicative rate is helpfully reflected in each signal's final
Compile-time filtering is an advanced feature that can be tricky to set
and use correctly. Most folks will want ONLY runtime filtering.
Compile-time filtering works by eliding (completely removing the code for)
disallowed signals. This means zero performance cost for these signals,
but also means that compile-time filtering is PERMANENT once applied.
So if you set `:info` as the compile-time minimum level, that'll REMOVE
CODE for every signal call below `:info` level. To decrease that minimum
level, you'll need to rebuild.
Compile-time filtering can be set ONLY with environmental config
(JVM properties, environment variables, or classpath resources).
Signal and handler sampling is multiplicative:
Both signals and handlers can have independent sample rates, and these
MULTIPLY!
If a signal is created with 20% sampling and a handler handles 50%
of received signals, then 10% of possible signals will be handled
(50% of 20%).
The multiplicative rate is helpfully reflected in each signal's final
`:sample-rate` value.
For a visual flowchart, see: Ref. <https://www.taoensso.com/telemere/flow>
For more info:
- Signal visual flowchart, Ref. <https://www.taoensso.com/telemere/flow>
- On signal filters, see: `help:signal-filters` docstring
- On handler filters, see: `help:signal-handlers` docstring

View file

@ -18,14 +18,18 @@ Signal options (shared by all signal creators):
`:parent` ------ Custom ?{:keys [id uid]} to override auto (dynamic) parent signal info in signal
`:location` ---- Custom ?{:keys [ns line column file]} to override auto signal creator callsite location
`:elidable?` --- Should signal be subject to compile-time elision? (Default: true).
`:elidable?` --- Should signal be subject to compile-time elision? (Default: true)
`:sample-rate` - ?rate ∈ℝ[0,1] for signal sampling (0.75 => allow 75% of signals, nil => allow all)
`:when` -------- Arb ?form; when present, form must return truthy to allow signal
`:rate-limit` -- ?spec as given to `taoensso.telemere/rate-limiter`, see its docstring for details
`:middleware` -- ?[(fn [signal])=>modified-signal ...] signal middleware
`:trace?` ------ Should tracing be enabled for `:run` form?
<kvs> ---------- Arb other user-level ?kvs to incl. in signal
<kvs> ---------- Other arb user-level ?kvs to incl. in signal. Typically NOT included in
handler output, so a great way to provide custom data/opts for use
(only) by custom middleware/handlers.
handler-specific data that can just be ignored by other handlers
If anything is unclear, please ping me (@ptaoussanis) so that I can improve these docs!

View file

@ -1,7 +1,7 @@
{;;:lein true
:source-paths ["src" "test"]
:dependencies
[[com.taoensso/encore "3.104.1"]
[[com.taoensso/encore "3.105.0"]
[cider/cider-nrepl "0.47.0"]
[binaryage/devtools "1.0.7"]]

View file

@ -1,4 +1,4 @@
(defproject com.taoensso/slf4j-telemere "1.0.0-beta3"
(defproject com.taoensso/slf4j-telemere "1.0.0-beta4"
:author "Peter Taoussanis <https://www.taoensso.com>"
:description "Telemere backend/provider for SLF4J API v2"
:url "https://www.taoensso.com/telemere"
@ -14,9 +14,9 @@
:profiles
{:provided
{:dependencies
[[org.clojure/clojure "1.11.2"]
[[org.clojure/clojure "1.11.3"]
[org.slf4j/slf4j-api "2.0.13"]
[com.taoensso/telemere "1.0.0-beta3"]]}
[com.taoensso/telemere "1.0.0-beta4"]]}
:dev
{:plugins

View file

@ -11,9 +11,9 @@
[taoensso.encore.signals :as sigs]
[taoensso.telemere.impl :as impl]
[taoensso.telemere.utils :as utils]
#?(:clj [taoensso.telemere.streams :as streams])
#?(:default [taoensso.telemere.console-handlers :as ch])
#?(:clj [taoensso.telemere.file-handler :as fh]))
#?(:default [taoensso.telemere.consoles :as consoles])
#?(:clj [taoensso.telemere.streams :as streams])
#?(:clj [taoensso.telemere.files :as files]))
#?(:cljs
(:require-macros
@ -32,10 +32,9 @@
(remove-ns 'taoensso.telemere)
(:api (enc/interns-overview)))
(enc/assert-min-encore-version [3 104 1])
(enc/assert-min-encore-version [3 105 0])
;;;; TODO
;; - Add email handler
;; - Native OpenTelemetry traces and spans
;; - Update Tufte (signal API, config API, signal keys, etc.)
;; - Update Timbre (signal API, config API, signal keys, backport improvements)
@ -48,22 +47,8 @@
:ct-sig-filter impl/ct-sig-filter
:*rt-sig-filter* impl/*rt-sig-filter*
:*sig-handlers* impl/*sig-handlers*
:sig-filter-system-vals-info
"These include:
Compile-time:
ns-filter: (get-env {:as :edn} :taoensso.telemere/ct-ns-filter<.platform><.edn>)
id-filter: (get-env {:as :edn} :taoensso.telemere/ct-id-filter<.platform><.edn>)
min-level: (get-env {:as :edn} :taoensso.telemere/ct-min-level<.platform><.edn>)
Runtime:
ns-filter: (get-env {:as :edn} :taoensso.telemere/rt-ns-filter<.platform><.edn>)
id-filter: (get-env {:as :edn} :taoensso.telemere/rt-id-filter<.platform><.edn>)
min-level: (get-env {:as :edn, :default :info} :taoensso.telemere/rt-min-level<.platform><.edn>)
See `get-env` for details."})
:sig-filter-env-config-help
(impl/signal-docstring :filter-env-config)})
(comment help:filters)
@ -181,30 +166,6 @@
See `*middleware*` for details."
[init-val form] `(binding [*middleware* ~init-val] ~form)))
;;;; Encore integration
(do
(enc/set-var-root! sigs/*default-handler-error-fn*
(fn [{:keys [error] :as m}]
(impl/signal!
{:kind :error
:level :error
:error error
:location {:ns "taoensso.encore.signals"}
:id :taoensso.encore.signals/handler-error
:msg "Error executing wrapped handler fn"
:data (dissoc m :error)})))
(enc/set-var-root! sigs/*default-handler-backp-fn*
(fn [data]
(impl/signal!
{:kind :event
:level :warn
:location {:ns "taoensso.encore.signals"}
:id :taoensso.encore.signals/handler-back-pressure
:msg "Back pressure on wrapped handler fn"
:data data}))))
;;;; Signal creators
;; - signal! [ opts] ; => allowed? / run result (value or throw)
;; - event! [id ] [id level-or-opts] ; id + ?level => allowed? ; Sole signal with descending main arg!
@ -376,9 +337,39 @@
;;;; Handlers
(enc/defaliases
#?(:default ch/handler:console)
#?(:cljs ch/handler:console-raw)
#?(:clj fh/handler:file))
#?(:default consoles/handler:console)
#?(:cljs consoles/handler:console-raw)
#?(:clj files/handler:file))
;;;; Init
(impl/on-init
(enc/set-var-root! sigs/*default-handler-error-fn*
(fn [{:keys [error] :as m}]
(impl/signal!
{:kind :error
:level :error
:error error
:location {:ns "taoensso.encore.signals"}
:id :taoensso.encore.signals/handler-error
:msg "Error executing wrapped handler fn"
:data (dissoc m :error)})))
(enc/set-var-root! sigs/*default-handler-backp-fn*
(fn [data]
(impl/signal!
{:kind :event
:level :warn
:location {:ns "taoensso.encore.signals"}
:id :taoensso.encore.signals/handler-back-pressure
:msg "Back pressure on wrapped handler fn"
:data data})))
(add-handler! :default/console (handler:console))
#?(:clj (enc/catching (require '[taoensso.telemere.tools-logging])))
#?(:clj (enc/catching (require '[taoensso.telemere.slf4j]))))
;;;; Flow benchmarks
@ -413,14 +404,6 @@
;;;;
(impl/on-init
(add-handler! :default/console (handler:console))
#?(:clj (enc/catching (require '[taoensso.telemere.tools-logging])))
#?(:clj (enc/catching (require '[taoensso.telemere.slf4j]))))
;;;;
(comment
(with-handler :hid1 (handler:console) {} (log! "Message"))

View file

@ -1,22 +1,28 @@
(ns ^:no-doc taoensso.telemere.console-handlers
(ns ^:no-doc taoensso.telemere.consoles
"Private ns, implementation detail.
Core console handlers."
Core console handlers, aliased in main Telemere ns."
(:require
[taoensso.encore :as enc :refer [have have?]]
[taoensso.telemere.utils :as utils]))
(comment
(remove-ns 'taoensso.telemere.console-handlers)
(require '[taoensso.telemere :as tel])
(remove-ns 'taoensso.telemere.consoles)
(:api (enc/interns-overview)))
;;;; Handlers
#?(:clj
(defn ^:public handler:console
"Experimental, subject to change.
"Experimental, subject to change. Feedback welcome!
Returns a (fn handler [signal]) that:
- Takes a Telemere signal.
- Writes a formatted signal string to stream.
A general-purpose `println`-style handler that's well suited for outputting
signals formatted as edn, JSON, or human-readable strings.
Options:
`:format-signal-fn` - (fn [signal]) => output, see `help:signal-formatters`
@ -42,12 +48,15 @@
:cljs
(defn ^:public handler:console
"Experimental, subject to change.
"Experimental, subject to change. Feedback welcome!
If `js/console` exists, returns a (fn handler [signal]) that:
- Takes a Telemere signal.
- Writes a formatted signal string to JavaScript console.
A general-purpose `println`-style handler that's well suited for outputting
signals formatted as edn, JSON, or human-readable strings.
Options:
`:format-signal-fn` - (fn [signal]) => output, see `help:signal-formatters`"
@ -77,7 +86,7 @@
#?(:cljs
(defn ^:public handler:console-raw
"Experimental, subject to change.
"Experimental, subject to change. Feedback welcome!
If `js/console` exists, returns a (fn handler [signal]) that:
- Takes a Telemere signal.
@ -87,10 +96,10 @@
Ref. <https://github.com/binaryage/cljs-devtools>."
([] (handler:console-raw nil))
([{:keys [format-signal-prelude-fn format-nsecs-fn] :as opts
([{:keys [format-signal->prelude-fn format-nsecs-fn] :as opts
:or
{format-signal-prelude-fn (utils/format-signal-prelude-fn) ; (fn [signal])
format-nsecs-fn (utils/format-nsecs-fn) ; (fn [nanosecs])
{format-signal->prelude-fn (utils/format-signal->prelude-fn) ; (fn [signal])
format-nsecs-fn (utils/format-nsecs-fn) ; (fn [nanosecs])
}}]
(when (and (exists? js/console) (exists? js/console.group))
@ -108,7 +117,7 @@
logger (js-console-logger level)]
;; Unfortunately groups have no level
(.group js/console (format-signal-prelude-fn signal))
(.group js/console (format-signal->prelude-fn signal))
(signal-content-handler signal (logger-fn logger) identity)
(when-let [stack (and error (.-stack (enc/ex-root error)))]

View file

@ -1,12 +1,13 @@
(ns ^:no-doc taoensso.telemere.file-handler
(ns ^:no-doc taoensso.telemere.files
"Private ns, implementation detail.
Core archiving file handler."
Core file handler, aliased in main Telemere ns."
(:require
[taoensso.encore :as enc :refer [have have?]]
[taoensso.telemere.utils :as utils]))
(comment
(remove-ns 'taoensso.telemere.file-handler)
(require '[taoensso.telemere :as tel])
(remove-ns 'taoensso.telemere.files)
(:api (enc/interns-overview)))
;;;; Implementation
@ -265,7 +266,7 @@
;;;; Handler
(defn ^:public handler:file
"Experimental, subject to change.
"Experimental, subject to change. Feedback welcome!
Returns a (fn handler [signal]) that:
- Takes a Telemere signal.

View file

@ -34,29 +34,35 @@
;;;; Config
#?(:clj
(let [base (enc/get-env {:as :edn} :taoensso.telemere/ct-filters<.platform><.edn>)
ns-filter (enc/get-env {:as :edn} :taoensso.telemere/ct-ns-filter<.platform><.edn>)
id-filter (enc/get-env {:as :edn} :taoensso.telemere/ct-id-filter<.platform><.edn>)
min-level (enc/get-env {:as :edn} :taoensso.telemere/ct-min-level<.platform><.edn>)]
(let [base (enc/get-env {:as :edn} :taoensso.telemere/ct-filters<.platform><.edn>)
kind-filter (enc/get-env {:as :edn} :taoensso.telemere/ct-kind-filter<.platform><.edn>)
ns-filter (enc/get-env {:as :edn} :taoensso.telemere/ct-ns-filter<.platform><.edn>)
id-filter (enc/get-env {:as :edn} :taoensso.telemere/ct-id-filter<.platform><.edn>)
min-level (enc/get-env {:as :edn} :taoensso.telemere/ct-min-level<.platform><.edn>)]
(enc/defonce ct-sig-filter
"`SigFilter` used for compile-time elision, or nil."
(sigs/sig-filter
{:ns-filter (or ns-filter (get base :ns-filter))
:id-filter (or id-filter (get base :id-filter))
:min-level (or min-level (get base :min-level))}))))
{:kind-filter (or kind-filter (get base :kind-filter))
:ns-filter (or ns-filter (get base :ns-filter))
:id-filter (or id-filter (get base :id-filter))
:min-level (or min-level (get base :min-level))}))))
(let [base (enc/get-env {:as :edn} :taoensso.telemere/rt-filters<.platform><.edn>)
ns-filter (enc/get-env {:as :edn} :taoensso.telemere/rt-ns-filter<.platform><.edn>)
id-filter (enc/get-env {:as :edn} :taoensso.telemere/rt-id-filter<.platform><.edn>)
min-level (enc/get-env {:as :edn, :default :info} :taoensso.telemere/rt-min-level<.platform><.edn>)]
(let [base (enc/get-env {:as :edn} :taoensso.telemere/rt-filters<.platform><.edn>)
kind-filter (enc/get-env {:as :edn} :taoensso.telemere/rt-kind-filter<.platform><.edn>)
ns-filter (enc/get-env {:as :edn} :taoensso.telemere/rt-ns-filter<.platform><.edn>)
id-filter (enc/get-env {:as :edn} :taoensso.telemere/rt-id-filter<.platform><.edn>)
min-level (enc/get-env {:as :edn, :default :info} :taoensso.telemere/rt-min-level<.platform><.edn>)]
(enc/defonce ^:dynamic *rt-sig-filter*
"`SigFilter` used for runtime filtering, or nil."
(sigs/sig-filter
{:ns-filter (or ns-filter (get base :ns-filter))
:id-filter (or id-filter (get base :id-filter))
:min-level (or min-level (get base :min-level))})))
{:kind-filter (or kind-filter (get base :kind-filter))
:ns-filter (or ns-filter (get base :ns-filter))
:id-filter (or id-filter (get base :id-filter))
:min-level (or min-level (get base :min-level))})))
(comment (enc/get-env {:as :edn, :return :explain} :taoensso.telemere/rt-filters<.platform><.edn>))
;;;; Context (optional arb app-level state)
;; taoensso.telemere/*ctx*
@ -396,8 +402,8 @@
;;;; Signal API helpers
#?(:clj (defmacro signal-docstring [rname] (enc/slurp-resource (str "signal-docstrings/" (name rname) ".txt"))))
#?(:clj (defmacro defhelp [sym rname] `(enc/def* ~sym {:doc ~(eval `(signal-docstring ~rname))} "See docstring")))
#?(:clj (defmacro signal-docstring [ rname] (enc/slurp-resource (str "signal-docstrings/" (name rname) ".txt"))))
#?(:clj (defmacro defhelp [sym rname] `(enc/def* ~sym {:doc ~(eval `(signal-docstring ~rname))} "See docstring")))
#?(:clj
(defn signal-arglists [macro-id]
@ -693,7 +699,7 @@
(and (not elide?) allow?))))
;;;; Intake
;;;; Intakes
#?(:clj
(do
@ -722,6 +728,3 @@
(with-signal :raw :trap (test-fn msg)))]
(= (force (get signal :msg_)) msg)))))

View file

@ -1,9 +1,6 @@
(ns taoensso.telemere.open-telemetry
"Core OpenTelemetry handler and utils.
Needs `OpenTelemetry Java`,
"OpenTelemetry handler using `opentelemetry-java`,
Ref. <https://github.com/open-telemetry/opentelemetry-java>."
(:require
[clojure.string :as str]
[taoensso.encore :as enc :refer [have have?]]
@ -165,40 +162,53 @@
attrs-map))
(defn get-default-logger-provider
"Experimental, subject to change!! Feedback very welcome!
(defn default-logger-provider
"Experimental, subject to change. Feedback welcome!
Returns `io.opentelemetry.api.logs.LoggerProvider` via:
`AutoConfiguredOpenTelemetrySdk` when possible, or
`GlobalOpenTelemetry` otherwise."
`GlobalOpenTelemetry` otherwise.
See the relevant `opentelemetry-java` docs for details."
^LoggerProvider []
(or
;; Without Java agent
(enc/compile-when
io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk
(enc/catching :common
(let [builder (io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk/builder)]
(.getSdkLoggerProvider (.getOpenTelemetrySdk (.build builder))))))
;; With Java agent
(.getLogsBridge (GlobalOpenTelemetry/get))))
;;;; Handler
(defn handler:open-telemetry-logger
"Experimental, subject to change!! Feedback very welcome!
"Experimental, subject to change. Feedback welcome!
Needs `opentelemetry-java`,
Ref. <https://github.com/open-telemetry/opentelemetry-java>.
Returns a (fn handler [signal]) that:
- Takes a Telemere signal.
- Emits signal content to the `io.opentelemetry.api.logs.Logger`
returned by given `io.opentelemetry.api.logs.LoggerProvider`."
returned by given `io.opentelemetry.api.logs.LoggerProvider`.
Options:
`:logger-provider` - `io.opentelemetry.api.logs.LoggerProvider`
Defaults to the LoggerProvider returned by (default-logger-provider),
see that docstring for details."
([] (handler:open-telemetry-logger nil))
([{:keys
[^LoggerProvider logger-provider
attrs-key ; Advanced, undocumented
attrs-signal-key ; Advanced, undocumented
]
:or
{logger-provider (get-default-logger-provider)
attrs-key :open-telemetry-attrs}}]
{logger-provider (default-logger-provider)
attrs-signal-key :open-telemetry/attrs}}]
(let []
(fn a-handler:open-telemetry-logger
@ -208,7 +218,7 @@
logger (.get logger-provider (or ns "default"))
severity (level->severity level)
msg (force msg_)
attrs-map (signal->attrs-map attrs-key signal)
attrs-map (signal->attrs-map attrs-signal-key signal)
attrs (as-attrs attrs-map)]
(.emit
@ -217,5 +227,3 @@
(.setSeverity severity)
(.setBody msg)
(.setAllAttributes attrs)))))))))

View file

@ -0,0 +1,139 @@
(ns taoensso.telemere.postal
"Email handler using `postal`,
Ref. <https://github.com/drewr/postal>."
(:require
[taoensso.encore :as enc :refer [have have?]]
[taoensso.telemere.utils :as utils]
[postal.core :as postal]))
(comment
(require '[taoensso.telemere :as tel])
(remove-ns 'taoensso.telemere.postal)
(:api (enc/interns-overview)))
;;;; Implementation
(defn format-signal->subject-fn
"Experimental, subject to change.
Returns a (fn format [signal]) that:
- Takes a Telemere signal.
- Returns a formatted email subject like:
\"INFO EVENT :taoensso.telemere.postal/ev-id1 - msg\""
([] (format-signal->subject-fn nil))
([{:keys [max-len subject-signal-key]
:or
{max-len 128
subject-signal-key :postal/subject}}]
(fn format-signal->subject [signal]
(or
(get signal subject-signal-key) ; Custom subject
;; Simplified `format-signal->prelude-fn`
(let [{:keys [level kind #_ns id msg_]} signal
sb (enc/str-builder)
s+spc (enc/sb-appender sb " ")]
(when level (s+spc (utils/format-level level)))
(when kind (s+spc (utils/upper-qn kind)))
(when id (s+spc (utils/format-id nil id)))
(when-let [msg (force msg_)] (s+spc "- " msg))
(enc/get-substr-by-len (str sb) 0 max-len))))))
(comment
((format-signal->subject-fn)
(tel/with-signal (tel/event! ::ev-id1 #_{:postal/subject "My subject"}))))
;;;; Handler
(defn handler:postal
"Experimental, subject to change. Feedback welcome!
Needs `postal`,
Ref. <https://github.com/drewr/postal>.
Returns a (fn handler [signal]) that:
- Takes a Telemere signal.
- Sends an email with formatted signal content to the configured recipient.
Useful for emailing important alerts to admins, etc.
NB can incur financial costs!!
See tips section re: protecting against unexpected costs.
Options:
`:postal/conn-opts` - Map of connection opts provided to `postal`
Examples:
{:host \"mail.isp.net\", :user \"jsmith\", :pass \"a-secret\"},
{:host \"smtp.gmail.com\", :user \"jsmith@gmail.com\", :pass \"a-secret\" :port 587 :tls true},
{:host \"email-smtp.us-east-1.amazonaws.com\", :port 587, :tls true
:user \"AKIAIDTP........\" :pass \"AikCFhx1P.......\"}
`:postal/msg-opts` - Map of message options
Examples:
{:from \"foo@example.com\", :to \"bar@example.com\"},
{:from \"Alice <foo@example.com\", :to \"Bob <bar@example.com>\"},
{:from \"no-reply@example.com\", :to [\"first-responders@example.com\",
\"devops@example.com\"],
:cc \"engineering@example.com\"
:X-MyHeader \"A custom header\"}
`:format-signal-fn` - (fn [signal]) => output, see `help:signal-formatters`
`:format-signal->subject-fn` - (fn [signal]) => email subject string
Tips:
- Sending emails can incur financial costs!
Use appropriate dispatch filtering options when calling `add-handler!` to prevent
handler from sending unnecessary emails!
At least ALWAYS set an appropriate `:rate-limit` option, e.g.:
(add-handler! :my-postal-handler (handler:postal {<my-handler-opts})
{:rate-limit {\"Max 1 per min\" [1 (enc/msecs :mins 1)]
\"Max 3 per 15 mins\" [3 (enc/msecs :mins 15)]
\"Max 5 per hour\" [5 (enc/msecs :hours 1)]}, ...}), etc.
- Sending emails is slow!
Use appropriate async dispatch options when calling `add-handler!` to prevent
handler from blocking signal creator calls, e.g.:
(add-handler! :my-postal-handler (handler:postal {<my-handler-opts>})
{:async {:mode :dropping, :buffer-size 128, :n-threads 4} ...}), etc.
- Ref. <https://github.com/drewr/postal> for more info on `postal` options."
([] (handler:postal nil))
([{:keys
[postal/conn-opts
postal/msg-opts
format-signal-fn
format-signal->subject-fn]
:or
{format-signal-fn (utils/format-signal->str-fn)
format-signal->subject-fn (format-signal->subject-fn)}}]
(when-not conn-opts (throw (ex-info "No `:postal/conn-opts` was provided" {})))
(when-not msg-opts (throw (ex-info "No `:postal/msg-opts` was provided" {})))
(let []
(defn a-handler:postal
([]) ; Shut down (no-op)
([signal]
(let [msg
(assoc msg-opts
:subject (format-signal->subject-fn)
:body
[{:type "text/plain; charset=utf-8"
:content (format-signal-fn signal)}])
[result ex]
(try
[(postal/send-message conn-opts msg) nil]
(catch Exception ex [nil ex]))
success? (= (get result :code) 0)]
(when-not success?
(throw (ex-info "Failed to send email" result ex)))))))))

View file

@ -85,7 +85,7 @@
(let [monitor (Object.)]
(defn ^:public streams->reset!
"Experimental, subject to change without notice!
"Experimental, subject to change.
Resets `System/out` and `System/err` to their original value (prior to any
`streams->telemere!` call)."
[]
@ -107,7 +107,7 @@
(boolean (or orig-out orig-err))))
(defn ^:public streams->telemere!
"Experimental, subject to change without notice!
"Experimental, subject to change.
When given `out`, sets JVM's `System/out` to flush to Telemere signals with those opts.
When given `err`, sets JVM's `System/err` to flush to Telemere signals with those opts.

View file

@ -39,10 +39,10 @@
(defn tools-logging->telemere!
"Configures `clojure.tools.logging` to use Telemere as its logging implementation.
Called automatically if the following is true:
(get-env {:as :bool} :clojure.tools.logging/to-telemere)
See `get-env` for details."
Called automatically if one of the following is \"true\":
JVM property: `clojure.tools.logging.to-telemere`
Env variable: `CLOJURE_TOOLS_LOGGING_TO_TELEMERE`
Classpath resource: `clojure.tools.logging.to-telemere`"
[]
(impl/signal!
{:kind :event

View file

@ -46,7 +46,9 @@
(str x))
(str x)))))
(comment (format-id (str *ns*) ::id1))
(comment
(format-id (str *ns*) ::id1)
(format-id nil ::id1))
;;;; Public misc
@ -123,6 +125,15 @@
(comment (error-in-signal->maps {:level :info :error (ex-info "Ex" {})}))
(defn remove-kvs
"Returns the given signal without user-level kvs."
[signal]
(if-let [kvs (get signal :kvs)]
(reduce-kv (fn [m k _v] (dissoc m k)) (dissoc signal :kvs) kvs)
signal))
(comment (remove-kvs {:a :A, :b :B, :kvs {:a :A}}))
(defn minify-signal
"Experimental, subject to change.
Returns minimal signal map, removing:
@ -130,6 +141,12 @@
- Keys with redundant values (`:kvs`, `:location`, `:file`).
Useful when serializing signals to edn/JSON/etc."
;; Note that while handlers typically don't include user-level kvs, we
;; DO retain these here since signal serialization often implies transit
;; to some other system that may still need/want this info before final
;; processing/storage/etc.
[signal]
(reduce-kv
(fn [m k v]
@ -174,16 +191,20 @@
#?(:clj
(defn file-writer
"Experimental, subject to change!!
"Experimental, subject to change. Feedback welcome!
Opens the specified file and returns a stateful fn of 2 arities:
[content] => Writes given content to file, or no-ops if closed.
[] => Closes the writer.
Thread safe. Automatically creates file and parent dirs as necessary.
Writers MUST ALWAYS be manually closed after use!
Useful for basic handlers that write to a file, etc.
Notes:
- Automatically creates file and parent dirs as necessary.
- Writer should be manually closed after use (with zero-arity call).
- Flushes after every write.
- Thread safe, locks on single file stream."
Useful for handlers that write to files, etc."
[file append?]
(let [file (writeable-file! file)
stream_ (volatile! (file-stream file append?))
@ -205,7 +226,7 @@
true)
write-ba!
(fn [^bytes ba-content retrying?]
(fn [^bytes ba-content]
(when-let [^java.io.FileOutputStream stream (.deref stream_)]
(.write stream ba-content)
(.flush stream)
@ -219,24 +240,23 @@
lock (Object.)]
(fn file-writer
(fn a-file-writer
([] (when (open?_) (locking lock (close!))))
([content-or-action]
(case content-or-action ; Undocumented
(case content-or-action ; Undocumented, for dev/testing
:writer/open? (open?_)
:writer/file file
:writer/stream (.deref stream_)
:writer/reset! (locking lock (reset!))
:writer/state {:file file, :stream (.deref stream_)}
(when (open?_)
(let [content content-or-action
ba (.getBytes (str content) java.nio.charset.StandardCharsets/UTF_8)]
(locking lock
(try
(file-exists!)
(write-ba! ba false)
(catch java.io.IOException _
(write-ba! ba)
(catch java.io.IOException _ ; Retry once
(reset!)
(write-ba! ba true))))))))))))
(write-ba! ba))))))))))))
(comment (def fw1 (file-writer "test.txt" true)) (fw1 "x") (fw1))
@ -304,17 +324,17 @@
(do (enc/ex-map (ex-info "Ex2" {:k2 "v2"} (ex-info "Ex1" {:k1 "v1"}))))
(println (str "--\n" ((format-error-fn) (ex-info "Ex2" {:k2 "v2"} (ex-info "Ex1" {:k1 "v1"}))))))
(defn format-signal-prelude-fn
(defn format-signal->prelude-fn
"Experimental, subject to change.
Returns a (fn format [signal]) that:
- Takes a Telemere signal.
- Returns a formatted prelude string like:
\"2024-03-26T11:14:51.806Z INFO EVENT Hostname taoensso.telemere(2,21) ::ev-id - msg\""
([] (format-signal-prelude-fn nil))
([] (format-signal->prelude-fn nil))
([{:keys [format-inst-fn]
:or {format-inst-fn (format-inst-fn)}}]
(fn format-signal-prelude [signal]
(fn format-signal->prelude [signal]
(let [{:keys [inst level kind ns id msg_]} signal
sb (enc/str-builder)
s+spc (enc/sb-appender sb " ")]
@ -338,12 +358,12 @@
(when-let [msg (force msg_)] (s+spc "- " msg))
(str sb)))))
(comment ((format-signal-prelude-fn) (tel/with-signal (tel/event! ::ev-id))))
(comment ((format-signal->prelude-fn) (tel/with-signal (tel/event! ::ev-id))))
(defn ^:no-doc signal-content-handler
"Private, don't use.
Returns a (fn handle [signal handle-fn value-fn]) for internal use.
Content equivalent to `format-signal-prelude-fn`."
Content equivalent to `format-signal->prelude-fn`."
([] (signal-content-handler nil))
([{:keys [format-nsecs-fn format-error-fn raw-error?]
:or
@ -355,12 +375,12 @@
err-stop (str newline ">>> error >>>")]
(fn a-signal-content-handler [signal hf vf]
(let [{:keys [uid parent data kvs ctx sample-rate]} signal]
(let [{:keys [uid parent data #_kvs ctx sample-rate]} signal]
(when sample-rate (hf "sample: " (vf sample-rate)))
(when uid (hf " uid: " (vf uid)))
(when parent (hf "parent: " (vf parent)))
(when data (hf " data: " (vf data)))
(when kvs (hf " kvs: " (vf kvs)))
#_(when kvs (hf " kvs: " (vf kvs))) ; Don't auto include in output
(when ctx (hf " ctx: " (vf ctx))))
(let [{:keys [run-form error]} signal]
@ -435,12 +455,12 @@
- Takes a Telemere signal.
- Returns a formatted string intended for text consoles, etc."
([] (format-signal->str-fn nil))
([{:keys [format-signal-prelude-fn
([{:keys [format-signal->prelude-fn
format-nsecs-fn format-error-fn]
:or
{format-signal-prelude-fn (format-signal-prelude-fn) ; (fn [signal])
format-nsecs-fn (format-nsecs-fn) ; (fn [nanosecs])
format-error-fn (format-error-fn) ; (fn [error])
{format-signal->prelude-fn (format-signal->prelude-fn) ; (fn [signal])
format-nsecs-fn (format-nsecs-fn) ; (fn [nanosecs])
format-error-fn (format-error-fn) ; (fn [error])
}}]
(let [signal-content-handler ; (fn [signal hf vf]
@ -453,7 +473,7 @@
s+ (partial enc/sb-append sb)
s++ (partial enc/sb-append sb (str newline " "))]
(when-let [ff format-signal-prelude-fn] (s+ (ff signal))) ; Prelude
(when-let [ff format-signal->prelude-fn] (s+ (ff signal))) ; Prelude
(signal-content-handler signal s++ enc/pr-edn*) ; Content
(str sb))))))

View file

@ -14,7 +14,7 @@
#_[taoensso.telemere.streams :as streams]
#?(:clj [taoensso.telemere.slf4j :as slf4j])
#?(:clj [taoensso.telemere.open-telemetry :as otel])
#?(:clj [taoensso.telemere.file-handler :as fh])
#?(:clj [taoensso.telemere.files :as files])
#?(:clj [clojure.tools.logging :as ctl])))
(comment
@ -626,7 +626,8 @@
(is (= (utils/format-level 8) "LEVEL:8"))
(is (= (utils/format-id "foo.bar" :foo.bar/qux) "::qux"))
(is (= (utils/format-id "foo.baz" :foo.bar/qux) ":foo.bar/qux"))])
(is (= (utils/format-id "foo.baz" :foo.bar/qux) ":foo.bar/qux"))
(is (= (utils/format-id nil :foo.bar/qux) ":foo.bar/qux"))])
(testing "error-signal?"
[(is (= (utils/error-signal? {:error nil}) false))
@ -670,7 +671,7 @@
:cljs " Root: cljs.core/ExceptionInfo - Ex1\n data: {:k1 \"v1\"}\n\nCaused: cljs.core/ExceptionInfo - Ex2\n data: {:k2 \"v2\"}\n\nRoot stack trace:\n")))
(let [sig (with-sig (tel/event! ::ev-id {:inst t0}))
prelude ((utils/format-signal-prelude-fn) sig)] ; "2024-06-09T21:15:20.170Z INFO EVENT taoensso.telemere-tests(592,35) ::ev-id"
prelude ((utils/format-signal->prelude-fn) sig)] ; "2024-06-09T21:15:20.170Z INFO EVENT taoensso.telemere-tests(592,35) ::ev-id"
[(is (enc/str-starts-with? prelude "2024-06-09T21:15:20.170Z INFO EVENT"))
(is (enc/str-ends-with? prelude "::ev-id"))
(is (string? (re-find #"taoensso.telemere-tests\(\d+,\d+\)" prelude)))])
@ -707,44 +708,44 @@
#?(:clj
(deftest _file-names
[(is (= (fh/get-file-name "/logs/app.log" nil nil false) "/logs/app.log"))
(is (= (fh/get-file-name "/logs/app.log" nil nil true) "/logs/app.log"))
(is (= (fh/get-file-name "/logs/app.log" "ts" nil true) "/logs/app.log-ts"))
(is (= (fh/get-file-name "/logs/app.log" "ts" 1 false) "/logs/app.log-ts.1"))
(is (= (fh/get-file-name "/logs/app.log" "ts" 1 true) "/logs/app.log-ts.1.gz"))
(is (= (fh/get-file-name "/logs/app.log" nil 1 false) "/logs/app.log.1"))
(is (= (fh/get-file-name "/logs/app.log" nil 1 true) "/logs/app.log.1.gz"))]))
[(is (= (files/get-file-name "/logs/app.log" nil nil false) "/logs/app.log"))
(is (= (files/get-file-name "/logs/app.log" nil nil true) "/logs/app.log"))
(is (= (files/get-file-name "/logs/app.log" "ts" nil true) "/logs/app.log-ts"))
(is (= (files/get-file-name "/logs/app.log" "ts" 1 false) "/logs/app.log-ts.1"))
(is (= (files/get-file-name "/logs/app.log" "ts" 1 true) "/logs/app.log-ts.1.gz"))
(is (= (files/get-file-name "/logs/app.log" nil 1 false) "/logs/app.log.1"))
(is (= (files/get-file-name "/logs/app.log" nil 1 true) "/logs/app.log.1.gz"))]))
#?(:clj
(deftest _file-timestamps
[(is (= (fh/format-file-timestamp :daily (fh/udt->edy udt0)) "2024-06-09d"))
(is (= (fh/format-file-timestamp :weekly (fh/udt->edy udt0)) "2024-06-03w"))
(is (= (fh/format-file-timestamp :monthly (fh/udt->edy udt0)) "2024-06-01m"))]))
[(is (= (files/format-file-timestamp :daily (files/udt->edy udt0)) "2024-06-09d"))
(is (= (files/format-file-timestamp :weekly (files/udt->edy udt0)) "2024-06-03w"))
(is (= (files/format-file-timestamp :monthly (files/udt->edy udt0)) "2024-06-01m"))]))
(comment (fh/manage-test-files! :create))
(comment (files/manage-test-files! :create))
#?(:clj
(deftest _file-handling
[(is (boolean (fh/manage-test-files! :create)))
[(is (boolean (files/manage-test-files! :create)))
(testing "`scan-files`"
;; Just checking basic counts here, should be sufficient
[(is (= (count (fh/scan-files "test/logs/app1.log" nil nil :sort)) 1) "1 main, 0 parts")
(is (= (count (fh/scan-files "test/logs/app1.log" :daily nil :sort)) 0) "0 stamped")
(is (= (count (fh/scan-files "test/logs/app2.log" nil nil :sort)) 6) "1 main, 5 parts (+gz)")
(is (= (count (fh/scan-files "test/logs/app3.log" nil nil :sort)) 6) "1 main, 5 parts (-gz")
(is (= (count (fh/scan-files "test/logs/app4.log" nil nil :sort)) 11) "1 main, 5 parts (+gz) + 5 parts (-gz)")
(is (= (count (fh/scan-files "test/logs/app5.log" nil nil :sort)) 1) "1 main, 0 unstamped")
(is (= (count (fh/scan-files "test/logs/app5.log" :daily nil :sort)) 5) "5 stamped")
(is (= (count (fh/scan-files "test/logs/app6.log" nil nil :sort)) 1) "1 main, 0 unstamped")
(is (= (count (fh/scan-files "test/logs/app6.log" :daily nil :sort)) 25) "5 stamped * 5 parts")
(is (= (count (fh/scan-files "test/logs/app6.log" :weekly nil :sort)) 5) "5 stamped")])
[(is (= (count (files/scan-files "test/logs/app1.log" nil nil :sort)) 1) "1 main, 0 parts")
(is (= (count (files/scan-files "test/logs/app1.log" :daily nil :sort)) 0) "0 stamped")
(is (= (count (files/scan-files "test/logs/app2.log" nil nil :sort)) 6) "1 main, 5 parts (+gz)")
(is (= (count (files/scan-files "test/logs/app3.log" nil nil :sort)) 6) "1 main, 5 parts (-gz")
(is (= (count (files/scan-files "test/logs/app4.log" nil nil :sort)) 11) "1 main, 5 parts (+gz) + 5 parts (-gz)")
(is (= (count (files/scan-files "test/logs/app5.log" nil nil :sort)) 1) "1 main, 0 unstamped")
(is (= (count (files/scan-files "test/logs/app5.log" :daily nil :sort)) 5) "5 stamped")
(is (= (count (files/scan-files "test/logs/app6.log" nil nil :sort)) 1) "1 main, 0 unstamped")
(is (= (count (files/scan-files "test/logs/app6.log" :daily nil :sort)) 25) "5 stamped * 5 parts")
(is (= (count (files/scan-files "test/logs/app6.log" :weekly nil :sort)) 5) "5 stamped")])
(testing "`archive-main-file!`"
[(is (= (let [df (fh/debugger)] (fh/archive-main-file! "test/logs/app1.log" nil nil 2 :gz df) (df))
[(is (= (let [df (files/debugger)] (files/archive-main-file! "test/logs/app1.log" nil nil 2 :gz df) (df))
[[:rename "test/logs/app1.log" "test/logs/app1.log.1.gz"]]))
(is (= (let [df (fh/debugger)] (fh/archive-main-file! "test/logs/app2.log" nil nil 2 :gz df) (df))
(is (= (let [df (files/debugger)] (files/archive-main-file! "test/logs/app2.log" nil nil 2 :gz df) (df))
[[:delete "test/logs/app2.log.5.gz"]
[:delete "test/logs/app2.log.4.gz"]
[:delete "test/logs/app2.log.3.gz"]
@ -752,7 +753,7 @@
[:rename "test/logs/app2.log.1.gz" "test/logs/app2.log.2.gz"]
[:rename "test/logs/app2.log" "test/logs/app2.log.1.gz"]]))
(is (= (let [df (fh/debugger)] (fh/archive-main-file! "test/logs/app3.log" nil nil 2 :gz df) (df))
(is (= (let [df (files/debugger)] (files/archive-main-file! "test/logs/app3.log" nil nil 2 :gz df) (df))
[[:delete "test/logs/app3.log.5"]
[:delete "test/logs/app3.log.4"]
[:delete "test/logs/app3.log.3"]
@ -760,7 +761,7 @@
[:rename "test/logs/app3.log.1" "test/logs/app3.log.2"]
[:rename "test/logs/app3.log" "test/logs/app3.log.1.gz"]]))
(is (= (let [df (fh/debugger)] (fh/archive-main-file! "test/logs/app6.log" :daily "2021-01-01d" 2 :gz df) (df))
(is (= (let [df (files/debugger)] (files/archive-main-file! "test/logs/app6.log" :daily "2021-01-01d" 2 :gz df) (df))
[[:delete "test/logs/app6.log-2021-01-01d.5.gz"]
[:delete "test/logs/app6.log-2021-01-01d.4.gz"]
[:delete "test/logs/app6.log-2021-01-01d.3.gz"]
@ -769,15 +770,15 @@
[:rename "test/logs/app6.log" "test/logs/app6.log-2021-01-01d.1.gz"]]))])
(testing "`prune-archive-files!`"
[(is (= (let [df (fh/debugger)] (fh/prune-archive-files! "test/logs/app1.log" nil 2 df) (df)) []))
(is (= (let [df (fh/debugger)] (fh/prune-archive-files! "test/logs/app2.log" nil 2 df) (df)) []))
(is (= (let [df (fh/debugger)] (fh/prune-archive-files! "test/logs/app5.log" nil 2 df) (df)) []))
(is (= (let [df (fh/debugger)] (fh/prune-archive-files! "test/logs/app5.log" :daily 2 df) (df))
[(is (= (let [df (files/debugger)] (files/prune-archive-files! "test/logs/app1.log" nil 2 df) (df)) []))
(is (= (let [df (files/debugger)] (files/prune-archive-files! "test/logs/app2.log" nil 2 df) (df)) []))
(is (= (let [df (files/debugger)] (files/prune-archive-files! "test/logs/app5.log" nil 2 df) (df)) []))
(is (= (let [df (files/debugger)] (files/prune-archive-files! "test/logs/app5.log" :daily 2 df) (df))
[[:delete "test/logs/app5.log-2020-01-01d"]
[:delete "test/logs/app5.log-2020-01-02d"]
[:delete "test/logs/app5.log-2020-02-01d"]]))
(is (= (let [df (fh/debugger)] (fh/prune-archive-files! "test/logs/app6.log" :daily 2 df) (df))
(is (= (let [df (files/debugger)] (files/prune-archive-files! "test/logs/app6.log" :daily 2 df) (df))
[[:delete "test/logs/app6.log-2020-01-01d.5.gz"]
[:delete "test/logs/app6.log-2020-01-01d.4.gz"]
[:delete "test/logs/app6.log-2020-01-01d.3.gz"]
@ -798,7 +799,7 @@
"Prune oldest 3 intervals, with 5 parts each")])
(is (boolean (fh/manage-test-files! :delete)))]))
(is (boolean (files/manage-test-files! :delete)))]))
;;;; Other handlers

View file

@ -188,11 +188,11 @@ Quick examples of some basic filtering:
```
- Filtering is always O(1), except for rate limits which are O(n_windows).
- Sample rates are *multiplicative*: if a signal is created with *20%* sampling and a handler handles *50%* of given signals, then *10%* of possible signals will be handled. This multiplicative rate is helpfully reflected in each signal's final `:sample-rate` value.
- Signal and handler sampling is *multiplicative*: if a signal is created with *20%* sampling and a handler handles *50%* of received signals, then *10%* of possible signals will be handled. This multiplicative rate is helpfully reflected in each signal's final `:sample-rate` value.
- See [`help:signal-flow`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-flow) for internal docs on signal flow.
- See section [2-Architecture](./2-Architecture) for a flowchart / visual aid.
Runtime signal filters can be configured with:
Runtime signal filters can be set with:
| Global | Dynamic | Filters by |
| :-------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------- |
@ -202,7 +202,7 @@ Runtime signal filters can be configured with:
| [`set-min-level`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#set-min-level) | [`with-min-level`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#with-min-level) | Signal level (minimum can be specified by kind and/or ns) |
- See relevant docstrings (links above) for usage info.
- Compile-time filters are controlled by system-level config, see section [3-Config](./3-Config).
- Compile-time filters are set with environmental config, see section [3-Config](./3-Config).
# Internal help

View file

@ -13,7 +13,7 @@ A signal will be provided to a handler iff ALL of the following are true:
For 1-3, filtering may depend on (in order):
Sample rate → namespace → kind → id → level → when form/fn → rate limit
- See [`help:signal-filters`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-filters) for info on signal creation filters.
- See [`help:signal-filters`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-filters) for info on signal creation filters, **environmental config**, etc.
- See [`add-handler!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#add-handler!) for info on signal handler filters.
# Signal handlers
@ -29,7 +29,7 @@ See section [4-Handlers](./4-Handlers).
To do this:
1. Ensure that you have the `clojure.tools.logging` dependency, and
2. Call [`tools-logging->telemere!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.tools-logging#tools-logging-%3Etelemere!), or set the relevant system config as described in its docstring.
2. Call [`tools-logging->telemere!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.tools-logging#tools-logging-%3Etelemere!), or set the relevant environmental config as described in its docstring.
Verify successful intake with [`check-intakes`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#check-intakes):

View file

@ -2,18 +2,22 @@ Signal handlers process created signals to *do something with them* (analyse the
# Included handlers
The following handlers are included out-the-box:
The following signal handlers are currently included out-the-box:
| Name | Platform | Writes signals to | Writes signals as |
| :------------------------------------------------------------------------------------------------------------------------------------------------------- | :------- | :------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------- |
| [`handler:console`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console) | Clj | `*out*` or `*err*` | String ([edn](https://github.com/edn-format/edn), JSON, formatted, etc.) |
| [`handler:console`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console) | Cljs | Browser console | String ([edn](https://github.com/edn-format/edn), JSON, formatted, etc.) |
| [`handler:console-raw`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console-raw) | Cljs | Browser console | Raw data (for [cljs-devtools](https://github.com/binaryage/cljs-devtools), etc.) |
| [`handler:file`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:file) | Clj | File/s on disk | String ([edn](https://github.com/edn-format/edn), JSON, formatted, etc.) |
| [`handler:open-telemetry-logger`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.open-telemetry#handler:open-telemetry-logger) | Clj | [OpenTelemetry](https://opentelemetry.io/) [Java client](https://github.com/open-telemetry/opentelemetry-java) | [LogRecord](https://opentelemetry.io/docs/specs/otel/logs/data-model/) |
| Name | Platform | Output target | Output format |
| :------------------------------------------------------------------------------------------------------------------------------------------------------- | :------- | :------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------- |
| [`handler:console`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console) | Clj | `*out*` or `*err*` | Formatted string [1] |
| [`handler:console`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console) | Cljs | Browser console | Formatted string [1] |
| [`handler:console-raw`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:console-raw) | Cljs | Browser console | Raw signal data [2] |
| [`handler:file`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#handler:file) | Clj | File/s on disk | Formatted string [1] |
| [`handler:postal`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.postal#handler:postal) | Clj | Email (via [postal](https://github.com/drewr/postal)) | Formatted string [1] |
| [`handler:open-telemetry-logger`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere.open-telemetry#handler:open-telemetry-logger) | Clj | [OpenTelemetry](https://opentelemetry.io/) [Java client](https://github.com/open-telemetry/opentelemetry-java) | [LogRecord](https://opentelemetry.io/docs/specs/otel/logs/data-model/) |
- See relevant docstrings (links above) for more info.
- See section [8-Community](8-Community.md) for additional handlers.
- \[1] [Configurable](https://cljdoc.org/d/com.taoensso/telemere/1.0.0-beta3/api/taoensso.telemere#help:signal-formatters): human-readable (default), [edn](https://github.com/edn-format/edn), [JSON](https://www.json.org/), etc.
- \[2] For use with browser formatting tools like [cljs-devtools](https://github.com/binaryage/cljs-devtools).
- See relevant docstrings (links above) for features, usage, etc.
- See section [8-Community](8-Community.md) for more (community-supported) handlers.
- If there's other handlers you'd like to see, feel free to [ping me](https://github.com/taoensso/telemere/issues), or ask on the [`#telemere` Slack channel](https://www.taoensso.com/telemere/slack). It helps to know what people most need!
# Configuring handlers
@ -97,11 +101,11 @@ Want to add or remove a particular handler when your application starts?
Just make an appropriate call to [`add-handler!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#add-handler!) or [`remove-handler!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#remove-handler!).
## System-level config
## Environmental config
If you want to manage handlers **conditionally** based on **system-level config** (e.g. JVM prop, ENV var, or classpath resource) - Telemere provides the highly flexible [`get-env`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#get-env) util.
If you want to manage handlers **conditionally** based on **environmental config** (JVM properties, environment variables, or classpath resources) - Telemere provides the highly flexible [`get-env`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#get-env) util.
Use this to easily check your own cross-platform system config, and make whatever conditional handler management decisions you'd like.
Use this to easily define your own arbitrary cross-platform config, and make whatever conditional handler management decisions you'd like.
# Writing handlers
@ -120,7 +124,8 @@ For more complex cases, or for handlers that you want to make available for use
```clojure
(defn handler:my-handler ; Note naming convention
"Returns a (fn handler [signal] that:
- Does something.
- Takes a Telemere signal.
- Does something with it.
Options:
`:option1` - Description
@ -129,8 +134,9 @@ For more complex cases, or for handlers that you want to make available for use
([] (handler:my-handler nil)) ; Use default opts
([{:as constructor-opts}]
;; Do expensive prep outside returned handler fn whenever possible -
;; i.e. at (one-off) construction time rather than handling time.
;; Do option validation and expensive prep *outside* returned handler
;; fn whenever possible - i.e. at (one-off) construction time rather than
;; at every handler call.
(let []
(fn a-handler:my-handler ; Note naming convention
@ -181,4 +187,4 @@ Chrome console, with [cljs-devtools](https://github.com/binaryage/cljs-devtools)
MacOS terminal:
<img src="https://raw.githubusercontent.com/taoensso/telemere/master/imgs/handler-output-clj-file.png" alt="Default Clojure file handler output" width="640"/>
<img src="https://raw.githubusercontent.com/taoensso/telemere/master/imgs/handler-output-clj-file.png" alt="Default Clojure file handler output" width="640"/>

View file

@ -26,7 +26,7 @@ If not, you may need to [write something yourself](./4-Handlers#writing-handlers
This may be easier than it sounds. Remember that signals are just plain Clojure/Script [maps](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#help:signal-content), and handlers just plain Clojure/Script functions that do something with those maps.
Feel free to [ping me](https://github.com/taoensso/telemere/issues) for assistance, or ask on the [`#telemere` Slack channel](https://clojurians.slack.com/archives/C06ALA6EEUA).
Feel free to [ping me](https://github.com/taoensso/telemere/issues) for assistance, or ask on the [`#telemere` Slack channel](https://www.taoensso.com/telemere/slack).
### 2. Imports

View file

@ -72,7 +72,7 @@ Consider the [differences](https://www.youtube.com/watch?v=oyLBGkS5ICk) between
- [`log!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#log!) and [`event!`](https://cljdoc.org/d/com.taoensso/telemere/CURRENT/api/taoensso.telemere#event!) are both **good general-purpose** signal creators.
- Try **always provide an id** for all signals you create.
- **Provide an id** for all signals you create.
Qualified keywords are perfect! Downstream behaviour (e.g. alerts) can then look for these ids rather than messages (which are harder to match and more likely to change).
@ -84,11 +84,13 @@ Consider the [differences](https://www.youtube.com/watch?v=oyLBGkS5ICk) between
The result of signal middleware is cached and *shared between all handlers* making it an efficient place to transform signals. For this reason - prefer signal middleware to handler middleware when possible/convenient.
- **Signal sampling** and **handler sampling** are **multiplicative**.
- Signal and handler **sampling is multiplicative**.
If a signal is created with *20%* sampling and a handler handles *50%* of given signals, then *10%* of possible signals will be handled.
Both signals and handlers can have independent sample rates, and these MULTIPLY!
This multiplicative rate is helpfully reflected in each signal's final `:sample-rate` value, making it possible to estimate unsampled cardinalities in relevant cases.
If a signal is created with *20%* sampling and a handler handles *50%* of received signals, then *10%* of possible signals will be handled (50% of 20%).
This multiplicative rate is helpfully reflected in each signal's final `:sample-rate` value, making it possible to estimate *unsampled* cardinalities in relevant cases.
So for `n` randomly sampled signals matching some criteria, you'd have seen an estimated `Σ(1.0/sample-rate_i)` such signals _without_ sampling, etc.
@ -117,7 +119,7 @@ Consider the [differences](https://www.youtube.com/watch?v=oyLBGkS5ICk) between
Note that all user kvs will *also* be available *together* under the signal's `:kvs` key.
User kvs are a great way of controlling the per-signal behaviour of custom/advanced handlers.
User kvs are typically *not* included in handler output, so are a great way of providing custom data/opts for use (only) by custom middleware or handlers.
- Signal `kind` can be useful in advanced cases.

View file

@ -4,7 +4,7 @@ If there's demand, additional stuff can then be authored by Telemere's *communit
**PRs very welcome** to add links to this page for:
- Handlers (see [Writing handlers](./4-Handlers#writing-handlers))
- Handler libraries or examples (see [Writing handlers](./4-Handlers#writing-handlers))
- Handler utils (formatters, etc.)
- Middleware
- Tutorials / demos / etc.
@ -12,9 +12,9 @@ If there's demand, additional stuff can then be authored by Telemere's *communit
If you spot issues with any linked resources, please **contact the relevant authors** to let them know! Thank you! 🙏
| Contributor | Link | Description |
| :--------------------------------------------- | :-------------------------------------------------------------------------- | :------------------------------------------------------------ |
| [@ptaoussanis](https://github.com/ptaoussanis) | [Official Slack channel](https://clojurians.slack.com/archives/C06ALA6EEUA) | For questions, support, etc. |
| [@ptaoussanis](https://github.com/ptaoussanis) | [GitHub issues](https://github.com/taoensso/telemere/issues) | For questions, support, bug reports, PRs, etc. |
| _ | _ | Your link here? [PRs](../wiki#contributions-welcome) welcome! |
| [@username](https://github.com/username) | [Project](https://github.com/username/project) | Short description of resource |
| Contributor | Link | Description |
| :--------------------------------------------- | :---------------------------------------------------------------- | :------------------------------------------------------------ |
| [@ptaoussanis](https://github.com/ptaoussanis) | [Official Slack channel](https://www.taoensso.com/telemere/slack) | For questions, support, etc. |
| [@ptaoussanis](https://github.com/ptaoussanis) | [GitHub issues](https://github.com/taoensso/telemere/issues) | For questions, support, bug reports, PRs, etc. |
| _ | _ | Your link here? [PRs](../wiki#contributions-welcome) welcome! |
| [@username](https://github.com/username) | [Project](https://github.com/username/project) | Short description of resource |