feat: API-Server - Added OTel trace id auto-instrumentation for FastAPI by morgan-wowk · Pull Request #82 · TangleML/tangle

morgan-wowk · 2026-02-03T06:33:44Z

Add OpenTelemetry tracing support

TL;DR

Adds OpenTelemetry (OTel) distributed tracing capabilities to the API server for improved observability.

What changed?

Created a new module otel_tracing.py that configures OpenTelemetry tracing for FastAPI applications
Added OTel tracing initialization to both api_server_main.py and start_local.py
Added required OpenTelemetry dependencies to pyproject.toml
The implementation supports both gRPC and HTTP protocols for exporting traces

How to test?

Run an OTel collector locally (see https://hub.docker.com/r/otel/opentelemetry-collector)
Set the following environment variables:
- TANGLE_OTEL_EXPORTER_ENDPOINT: URL for the OTLP collector (e.g., "http://localhost:4317")
- TANGLE_ENV: Environment name (defaults to "development")
- TANGLE_OTEL_EXPORTER_PROTOCOL: Protocol to use (defaults to "grpc", can be "http")
Start the application
Make API requests and verify traces are being sent to your collector

Why make this change?

Distributed tracing provides deeper insights into request flows through the system, making it easier to:

Debug performance issues
Understand service dependencies
Monitor request latency across components
Identify bottlenecks in the application

This implementation uses the OpenTelemetry standard, which is vendor-neutral and widely supported by observability platforms.

morgan-wowk · 2026-02-03T06:38:23Z

feat: API-Server - Added OTel trace id auto-instrumentation for FastAPI #82 : 2 dependent PRs (#75 , #101 ) 👈 (View in Graphite)
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Ark-kun

Thank you for implementing this.

Ark-kun · 2026-02-06T08:27:01Z

cloud_pipelines_backend/instrumentation/otel_tracing.py

+        )
+
+    except Exception as e:
+        logger.error(f"Failed to configure OpenTelemetry tracing: {e}", exc_info=True)


Nit: Can just use logger.exception. It sets exc_info=True automatically.

Nice! Changed

Ark-kun · 2026-02-06T08:27:26Z

cloud_pipelines_backend/instrumentation/otel_tracing.py

+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+
+logger = logging.getLogger(__name__)


_logger (to make it private)

Python knowledge +1

Ark-kun · 2026-02-06T08:29:25Z

cloud_pipelines_backend/instrumentation/otel_tracing.py

+from fastapi import FastAPI
+from opentelemetry import trace
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
+    OTLPSpanExporter as GRPCSpanExporter,


Just curious - Why are we renaming this import?

It's renamed for clarity because OTel provides two versions of OTLPSpanExporter, one for http and one for grpc. It was not entirely needed previously, but now that I have added http support, it is helpful for distinction.

Update: Now I am importing modules and not classes directly. This has changed

Ark-kun · 2026-02-06T08:45:12Z

cloud_pipelines_backend/instrumentation/otel_tracing.py

+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
+    OTLPSpanExporter as GRPCSpanExporter,
+)
+from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor


Style: Let's import modules, not particular classes/functions. https://google.github.io/styleguide/pyguide.html#22-imports
Usage then becomes: module1.Class1 or module1.function1.
Ambiguous module names can be renamed.

import fastapi #? from opentelemetry.exporter.otlp.proto.grpc import trace_exporter from opentelemetry import trace as otel_trace from opentelemetry.sdk import resources as otel_sdk_resources from opentelemetry.sdk import trace as otel_sdk_trace from opentelemetry.sdk.trace import export as otel_sdk_trace_export

TBH, I dislike the module naming conventions of opentelemetry SDK. Ambiguity like .trace vs .sdk.trace. I also kind of dislike the module names that look like verbs: trace, export and single nouns like status. These names clash with function names and local variables.

So if the result looks too ugly to you, feel free to skip.

Understood! Thanks for the style guide. +1 existing style awareness!

Ark-kun · 2026-02-06T08:47:11Z

cloud_pipelines_backend/instrumentation/otel_tracing.py

+        resource = Resource(attributes={SERVICE_NAME: service_name})
+
+        # Create the OTLP exporter
+        otlp_exporter = GRPCSpanExporter(


Is GRPCSpanExporter the only exporter that would work good for Tangle API telemetry?

We can actually support multiple. Like HTTP. Good point.

I'll change this to be configurable and support multiple exporters

I do not strictly require to make this configurable.
But just wanted to know whether there is only a single exporter taht makes sense for Tangle or there are more.

I think it is a great addition to have this configurable. I have added support for both http and grpc

Ark-kun · 2026-02-06T08:47:56Z

cloud_pipelines_backend/instrumentation/otel_tracing.py

+
+    try:
+        # Build service name with environment suffix
+        app_env = os.environ.get("APP_ENV", "development")


Isn't environement usually ENV?

That's a tough one. In my Laravel time, they used APP_ENV, but personally I prefer prefixing application variables with TANGLE_.

I've changed this to TANGLE_ENV but am open to going with whatever you prefer. If it's ENV that's totally cool. The reason I like application prefixes is to avoid side effects in other software running on the same server.

Ark-kun · 2026-02-06T08:51:39Z

api_server_main.py

 )

+# Configure OpenTelemetry tracing
+otel_tracing.setup_api_tracing(app)


Thank you for making it non-invasive.

Ark-kun · 2026-02-06T09:13:03Z

cloud_pipelines_backend/instrumentation/otel_tracing.py

+
+        # Instrument the FastAPI application
+        # This automatically creates spans for all incoming HTTP requests
+        FastAPIInstrumentor.instrument_app(app)


What kind of data will this capture and export?

It is exclusive to API request lifecycle

Here is an example trace: https://docs.google.com/document/d/1h5RrkD2aR77eed6HXGCePHAMivX92HSesVjWeRo17z4/edit?usp=sharing

Volv-G

Given Alexey's comments this looks good to me

morgan-wowk · 2026-02-19T20:14:25Z

@Ark-kun Please feel free to give it a final pass when you have a moment.

After this, we will be able to take the next step which is metrics 😃

morgan-wowk · 2026-02-20T01:13:00Z

Note to self:

Before merging, try running the collector, and then shut down the collector while Tangle has already sent some successful logs. Let's see what happens when we can no longer report trace data to the collector.

morgan-wowk · 2026-02-20T01:51:53Z

Note to self:

Before merging, try running the collector, and then shut down the collector while Tangle has already sent some successful logs. Let's see what happens when we can no longer report trace data to the collector.

I have verified that if an OTel collector experiences downtime while Tangle has already started, it will not cause downtime for Tangle.

Some logs:

2026-02-19 17:50:28,547 [WARNING] opentelemetry.exporter.otlp.proto.grpc.exporter: Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 0.81s.
2026-02-19 17:50:35,099 [ERROR] opentelemetry.exporter.otlp.proto.grpc.exporter: Failed to export traces to localhost:4317, error code: StatusCode.UNAVAILABLE
2026-02-19 17:50:41,868 [INFO] uvicorn.access: 127.0.0.1:0 - "GET /services/ping HTTP/1.0" 200
2026-02-19 17:50:41,893 [INFO] uvicorn.access: 127.0.0.1:0 - "GET /api/pipeline_runs/?include_pipeline_names=true&include_execution_stats=true HTTP/1.0" 200
2026-02-19 17:50:45,110 [WARNING] opentelemetry.exporter.otlp.proto.grpc.exporter: Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 0.92s.
2026-02-19 17:50:51,908 [ERROR] opentelemetry.exporter.otlp.proto.grpc.exporter: Failed to export traces to localhost:4317, error code: StatusCode.UNAVAILABLE

morgan-wowk requested a review from Ark-kun as a code owner February 3, 2026 06:33

morgan-wowk force-pushed the otel-api-tracing branch from 567fbc0 to bfea644 Compare February 3, 2026 06:38

morgan-wowk force-pushed the otel-api-tracing branch from bfea644 to fd231ed Compare February 3, 2026 06:53

Ark-kun changed the title ~~feat: Add OTel trace id auto-instrumentation for FastAPI~~ feat: API-Server - Added OTel trace id auto-instrumentation for FastAPI Feb 6, 2026

Ark-kun reviewed Feb 6, 2026

View reviewed changes

Ark-kun requested changes Feb 17, 2026

View reviewed changes

morgan-wowk force-pushed the otel-api-tracing branch 4 times, most recently from 3a0b981 to 6041ed7 Compare February 19, 2026 00:32

morgan-wowk mentioned this pull request Feb 19, 2026

feat: Add development observability stack for testing OpenTelemetry with Jaeger, and Prometheus #101

Open

morgan-wowk requested a review from Ark-kun February 19, 2026 00:45

Volv-G approved these changes Feb 19, 2026

View reviewed changes

morgan-wowk added the observability label Feb 19, 2026

chore: API Server - Added more logging for run cancellation

a12aa86

morgan-wowk force-pushed the otel-api-tracing branch from 6041ed7 to a12aa86 Compare February 19, 2026 04:11

Ark-kun approved these changes Feb 20, 2026

View reviewed changes

morgan-wowk merged commit f7913db into master Feb 20, 2026
2 checks passed

Comments

Conversation

morgan-wowk commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add OpenTelemetry tracing support

TL;DR

What changed?

How to test?

Why make this change?

Uh oh!

morgan-wowk commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ark-kun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Volv-G left a comment

Choose a reason for hiding this comment

Uh oh!

morgan-wowk commented Feb 19, 2026

Uh oh!

morgan-wowk commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

morgan-wowk commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

morgan-wowk commented Feb 3, 2026 •

edited

Loading

morgan-wowk commented Feb 3, 2026 •

edited

Loading

morgan-wowk commented Feb 20, 2026 •

edited

Loading

morgan-wowk commented Feb 20, 2026 •

edited

Loading