make otel runtime default for system/metrics

[leehinman]

What does this PR do?

Changes the default runtime for the "system/metrics" metricbeat input to be "otel"

Why is it important?

Part of conversion to beat receivers. "system/metrics" has been tested with otel runtime and has been deemed ready to be the default.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

One less agentbeat process will be running with the system integration and elastic-agent otel will collect the data using a beat receiver.

Users should see no other change than the drop in processes.

How to test this PR locally

Integration test

cd pkg/component
go test .

Manual testing

Install agent with system integration in policy.

elastic-agent status --output full will be similar to:

ubuntu@test:~/elastic-agent-9.3.0-SNAPSHOT-linux-arm64$ sudo elastic-agent status --output full
┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   ├─ info
   │  ├─ id: 2734d015-24ec-43a4-9017-12017fbbb3ce
   │  ├─ version: 9.3.0
   │  └─ commit: a85ca5c5752d8e0df404eb4e3b5fbda2649e5b7a
   ├─ beat/metrics-monitoring
   │  ├─ status: (HEALTHY) Healthy
   │  ├─ beat/metrics-monitoring
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ beat/metrics-monitoring-metrics-monitoring-beats
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   ├─ filestream-monitoring
   │  ├─ status: (HEALTHY) Healthy
   │  ├─ filestream-monitoring
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ filestream-monitoring-filestream-monitoring-agent
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   ├─ http/metrics-monitoring
   │  ├─ status: (HEALTHY) Healthy
   │  ├─ http/metrics-monitoring
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ http/metrics-monitoring-metrics-monitoring-agent
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   ├─ log-default
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '1752'
   │  ├─ log-default
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ log-default-logfile-system-e3b5495e-768f-4798-ac7b-0384efb79ad2
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   ├─ prometheus/metrics-monitoring
   │  ├─ status: (HEALTHY) Healthy
   │  ├─ prometheus/metrics-monitoring
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ prometheus/metrics-monitoring-metrics-monitoring-collector
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   └─ system/metrics-default
      ├─ status: (HEALTHY) Healthy
      ├─ system/metrics-default
      │  ├─ status: (HEALTHY) Healthy
      │  └─ type: OUTPUT
      └─ system/metrics-default-system/metrics-system-e3b5495e-768f-4798-ac7b-0384efb79ad2
         ├─ status: (HEALTHY) Healthy
         └─ type: INPUT

Can be overridden using #11186

Related issues

• Relates https://github.com/elastic/ingest-dev/issues/6515
• Relates https://github.com/elastic/ingest-dev/issues/6516
• Relates https://github.com/elastic/ingest-dev/issues/6517

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @leehinman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: fe90881

Failed CI Steps

• 📦 amd64: Containers
• 📦 arm64: OTEL_COMPONENT Containers

History

• 💚 Build #31520 succeeded 3795567
• 💔 Build #31509 failed 0871d6c
• 💔 Build #31494 failed 9bf6f18

cc @leehinman

[ycombinator]

I'm not following why these lines were removed.

[swiatekm]

@leehinman I think it'd be better if you explicitly set system/metrics to run in the process runtime in this test. We want to have a non-otel component in it to verify the env variable doesn't affect it, as far as I understand. @mauri870 is that correct?

[mauri870]

Agreed. The way I did it here is not the best, but the goal was to confirm the env changes the runtime of monitoring components but keeps non monitoring components unnafected.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)