Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds missing schemas + data for metrics to snowplow stats ping #48476

Merged
merged 37 commits into from
Oct 15, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
c8810ec
adds data + schema for metrics stats ping
escherize Oct 8, 2024
b628c4a
remove comment
escherize Oct 8, 2024
b3f5ef3
annotate todos
escherize Oct 8, 2024
4118cd2
fill in the rest of the metrics values
escherize Oct 8, 2024
4d82879
fix some definitions + use a single timestamp
escherize Oct 8, 2024
a081ea4
shuffle stuff around to appease the namespace linter
escherize Oct 8, 2024
a38fe1b
Merge branch 'master' into add-metrics-to-snowplow-ping
escherize Oct 8, 2024
07248f4
update docstring
escherize Oct 8, 2024
14b7a27
fix jsonschema, maybe
escherize Oct 8, 2024
0049acb
review responses
escherize Oct 9, 2024
1ffd3b6
remove a footgun
escherize Oct 9, 2024
016b435
Merge branch 'master' into add-metrics-to-snowplow-ping
escherize Oct 9, 2024
9dda2c7
Merge branch 'add-metrics-to-snowplow-ping' of github.com:metabase/me…
escherize Oct 9, 2024
d68ccac
add tags to metrics,
escherize Oct 9, 2024
5e6672e
indent
escherize Oct 9, 2024
1326fd0
cljfmt
escherize Oct 9, 2024
4e6afe0
Merge branch 'master' into add-metrics-to-snowplow-ping
escherize Oct 9, 2024
64cda71
version should match filename
escherize Oct 10, 2024
3cd6980
Merge branch 'add-metrics-to-snowplow-ping' of github.com:metabase/me…
escherize Oct 10, 2024
0df7a00
update instance stats 1-0-1 schema
escherize Oct 10, 2024
fd4328e
require `tags` in grouped_metric + snowcat comment
escherize Oct 10, 2024
65f5726
Merge branch 'master' into add-metrics-to-snowplow-ping
escherize Oct 10, 2024
e8eceb1
fix formatting noise
escherize Oct 10, 2024
520c91f
update schema to make it valid
escherize Oct 10, 2024
e09d74f
remove grouped_metrics from instance_stats schema
escherize Oct 10, 2024
80dee46
cljfmt appeasement
escherize Oct 10, 2024
88b6166
Merge branch 'master' into add-metrics-to-snowplow-ping
escherize Oct 10, 2024
35f5b7b
unbin cache_num_queries_cached value
escherize Oct 11, 2024
7ff4ba7
we can now guarantee metric values will be ints
escherize Oct 11, 2024
f8397bb
jsonschema for instance_uuid, settings, and grouped_metrics
escherize Oct 14, 2024
ddf7c64
add analytics_uuid and make it required
escherize Oct 14, 2024
19b4ff8
lint + alphabetize instance stats json schema
escherize Oct 14, 2024
1961f50
update setting key type
escherize Oct 14, 2024
bdd3abe
lint jsonschema
escherize Oct 14, 2024
fe1db2f
Bump instance stats to 2-0-0
escherize Oct 15, 2024
27e4e0e
adds a grouped-metric to stats ping
escherize Oct 15, 2024
df957a6
cljfmt
escherize Oct 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
shuffle stuff around to appease the namespace linter
- we aren't reaching into the api API, so the linter is more of a
  formality here.
  • Loading branch information
escherize committed Oct 8, 2024
commit a081ea4351c2b7ca41cc2b2e8249b98b9bb57e88
1 change: 1 addition & 0 deletions .clj-kondo/config.edn
Original file line number Diff line number Diff line change
Expand Up @@ -595,6 +595,7 @@
metabase.driver.sql-jdbc.test-util sql-jdbc.tu
metabase.driver.sql.parameters.substitute sql.params.substitute
metabase.driver.sql.parameters.substitution sql.params.substitution
metabase.eid-translation eid-translation
metabase.email-test et
metabase.email.messages messages
metabase.formatter formatter
Expand Down
22 changes: 18 additions & 4 deletions src/metabase/analytics/stats.clj
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,11 @@
[java-time.api :as t]
[medley.core :as m]
[metabase.analytics.snowplow :as snowplow]
#_:clj-kondo/ignore
[metabase.api.embed.common :as embed.common]
[metabase.config :as config]
[metabase.db :as db]
[metabase.db.query :as mdb.query]
[metabase.driver :as driver]
[metabase.eid-translation :as eid-translation]
[metabase.email :as email]
[metabase.embed.settings :as embed.settings]
[metabase.integrations.google :as google]
Expand Down Expand Up @@ -578,10 +577,25 @@
:has_activation_signals_completed (completed-activation-signals?)})]
(m->kv-vec instance-attributes)))


(mu/defn- get-translation-count!
:- [:map [:ok :int] [:not-found :int] [:invalid-format :int] [:total :int]]
"Get and clear the entity-id translation counter. This is meant to be called during the daily stats collection process."
[]
(let [counter (setting/get-value-of-type :json :entity-id-translation-counter)]
(merge counter {:total (apply + (vals counter))})))

(mu/defn- clear-translation-count!
"We want to reset the eid translation count on every stat ping, so we do it here."
[]
(setting/set-value-of-type! :json eid-translation/default-counter))

(defn- ->snowplow-metric-info
"Collects Snowplow metrics data that is not in the legacy stats format."
[]
(let [one-day-ago (t/minus (t/offset-date-time) (t/days 1))]
(let [one-day-ago (t/minus (t/offset-date-time) (t/days 1))
total-translation-count (:total (get-translation-count!))
_ (clear-translation-count!)]
{:models (t2/count :model/Card :type :model :archived false)
:new_embedded_dashboards (t2/count :model/Dashboard
:enable_embedding true
escherize marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -592,7 +606,7 @@
:date_joined [:>= one-day-ago])
:pivot_tables (t2/count :model/Card :display :pivot :archived false)
:query_executions_last_24h (t2/count :model/QueryExecution :started_at [:>= one-day-ago])
:entity_id_translations_last_24h (:total (embed.common/get-and-clear-translation-count!))}))
:entity_id_translations_last_24h total-translation-count}))

(mu/defn- snowplow-metrics
[stats metric-info :- [:map
Expand Down
29 changes: 5 additions & 24 deletions src/metabase/api/embed/common.clj
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
[metabase.api.dashboard :as api.dashboard]
[metabase.api.public :as api.public]
[metabase.driver.common.parameters.operators :as params.ops]
[metabase.eid-translation :as eid-translation]
[metabase.models.card :as card]
[metabase.models.params :as params]
[metabase.models.setting :as setting :refer [defsetting]]
Expand Down Expand Up @@ -325,49 +326,29 @@
"A Malli schema for a map of model names to a sequence of entity ids."
(mc/schema [:map-of ApiName [:sequential :string]]))

(def ^:private EidTranslationStatus
[:enum :ok :not-found :invalid-format])

;; -------------------- Entity Id Translation Analytics --------------------

(def ^:private
default-eid-translation-counter
(zipmap (rest EidTranslationStatus) (repeat 0)))

(defsetting entity-id-translation-counter
(deferred-tru "A counter for tracking the number of entity_id -> id translations. Whenever we call [[model->entity-ids->ids]], we increment this counter by the number of translations.")
:encryption :no
:visibility :internal
:export? false
:audit :never
:type :json
:default default-eid-translation-counter)

(defn- compute-result [processed-result current-count]
(merge-with + processed-result current-count))
:default eid-translation/default-counter)

(mu/defn update-translation-count!
"Update the entity-id translation counter with the results of a batch of entity-id translations."
[results :- [:sequential EidTranslationStatus]]
[results :- [:sequential eid-translation/Status]]
(let [processed-result (frequencies results)]
(entity-id-translation-counter!
(compute-result processed-result (entity-id-translation-counter)))))

(defn- add-total [counter]
(merge counter {:total (apply + (vals counter))}))

(mu/defn get-and-clear-translation-count!
:- [:map [:ok :int] [:not-found :int] [:invalid-format :int] [:total :int]]
"Get and clear the entity-id translation counter. This is meant to be called during the daily stats collection process."
[]
(u/prog1 (add-total (entity-id-translation-counter))
(entity-id-translation-counter! default-eid-translation-counter)))
(merge-with + processed-result (entity-id-translation-counter)))))

(mu/defn- entity-ids->id-for-model :- [:sequential [:tuple
;; We want to pass incorrectly formatted entity-ids through here,
;; but this is assumed to be an entity-id:
:string
[:map [:status EidTranslationStatus]]]]
[:map [:status eid-translation/Status]]]]
"Given a model and a sequence of entity ids on that model, return a pairs of entity-id, id."
[api-name eids]
(let [model (->model api-name) ;; This lookup is safe because we've already validated the api-names
Expand Down
9 changes: 9 additions & 0 deletions src/metabase/eid_translation.clj
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
(ns metabase.eid-translation)

(def Status
"Malli enum for possible statuses for entity_id -> id translations."
[:enum :ok :not-found :invalid-format])

(def default-counter
"The empty counter for tracking the number of entity_id -> id translations."
(zipmap (rest Status) (repeat 0)))
Loading