feat: Add pipeline run outcome metrics

[morgan-wowk]

Pipeline Metrics Instrumentation

Tracks pipeline lifecycle from creation to completion with labels for status and user.

Metrics added:

• pipeline_runs_total: Counter tracking pipeline runs by status (running/succeeded/failed/cancelled) and created_by
• pipeline_run_duration_seconds: Histogram tracking total pipeline duration by final status

These metrics provide visibility into pipeline success rates, completion times, and usage patterns per user.
Durations measure total lifecycle time from creation to terminal state (including queue and execution time).

[morgan-wowk]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat: Add orchestrator error tracking by type #81
• feat: Add database performance monitoring #80
• feat: Add container performance metrics #79
• feat: Add execution outcome and cache metrics #78
• feat: Add orchestrator queue health metrics #77
• feat: Add pipeline run outcome metrics #76 👈 (View in Graphite)
• feat: Add HTTP request metrics instrumentation #75
• feat: API-Server - Added OTel trace id auto-instrumentation for FastAPI #82 : 1 other dependent PR (#101 )
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Ark-kun]

I'm not fully sure we should be doing this on the app side.
Backend can be restarted at any time (e.g. new version is deployed). With current implementation in the PR, this seems to affect the metrics. But I think it shouldn't.

I'm not sure about reporting pipeline run times like this. Tracking execution durations might be more OK.

I'm not sure the backend should be aggregating run's execution status statistics into a single status. This single status will likely be not useful for many of the users. I think it's better when the users can get the full status stats and UI can provide derivatives.

[morgan-wowk]

I'm not fully sure we should be doing this on the app side. Backend can be restarted at any time (e.g. new version is deployed). With current implementation in the PR, this seems to affect the metrics. But I think it shouldn't.

I'm not sure about reporting pipeline run times like this. Tracking execution durations might be more OK.

I'm not sure the backend should be aggregating run's execution status statistics into a single status. This single status will likely be not useful for many of the users. I think it's better when the users can get the full status stats and UI can provide derivatives.

Thanks for jumping in to these draft PRs and giving some early feedback. Especially this one. After all our discussion, that does sound like the best thing to do.

[Ark-kun]

Overall idea: We want to leverage the telemetry processing systems and dashboards as much as possible to avoid re-inventing the wheel.

Ideal case: We report events via OTel and the dedicated metrics system (which we do not re-invent) creates metrics from those events.
Realistic compromise case: In addition to events, we report some simple metrics that we get anyways (or are cheap to get). E.g. time to process a queued/running execution.
Case we want to avoid (at least for now): Calculating and tracking complex metrics ourselves on the backend side.

[morgan-wowk]

Overall idea: We want to leverage the telemetry processing systems and dashboards as much as possible to avoid re-inventing the wheel.

Ideal case: We report events via OTel and the dedicated metrics system (which we do not re-invent) creates metrics from those events. Realistic compromise case: In addition to events, we report some simple metrics that we get anyways (or are cheap to get). E.g. time to process a queued/running execution. Case we want to avoid (at least for now): Calculating and tracking complex metrics ourselves on the backend side.

It sounds great to me! Let's do this

[morgan-wowk]

I feel like if we are ever calculating complex / non-cheap metrics ourselves, it's because it's a feature of Tangle, not because we need that metric for reporting. Otherwise, the observation platform should be ingesting simple, non-calculated events