OpenTelemetry with Newrelic
Observability could be defined as the ability to measure a system current state based on the information it generates, such as metrics, traces and logs. But it's not only generates a lot of data, we need to create data following an observability standard and provide access to this data in some way to be analyzed with specialized tools.
As the adoption of distributed systems has growing, in this context, observability becomes a mandatory requirement because a simple operation on a system can be processed by several services and if one of these is in an undesired stated, it can generate serious problems, we have been using observability approach to monitoring the applications stats and ensure all services will be in the desired stated in most of the time.
In the next sections we will discuss about an observability standard called OpenTelemetry and how we can delivery the data generated to be analyzed in the future.
OpenTelemetry
As was mentioned ours applications need to generated observarbilty informations to allows us to monitoring and ensure the desired states, to do that we need to instrument our applications to create this kind of data. When our applications are instrumented properly, it will generated and send the data to an observability back-end, there are a number of observability back-ends out there, ranging from self-hosted, open source tools, SaaS platforms, and so on.
Here we are in front of a big problem, each observability back-end have they specific standard for observability data and if we want to exchange to one to another this will cost a lot of effort to do so, because we will need to change all the instrumentation we mad in our applications. Example, if we have been using Sentry instrumentation and for some reason we will need to change to the Datadog, we will need to change the instrumentation library and the instrumentation approach used in the applications.
To solve this problem and others OpenTelemetry was created. OpenTelemetry is a standard maintained by the Cloud Native Computing Foundation that create the idea of Instrument once and send to everywhere. To allow this OpenTelemetry standard is made up of three main components, SDK, Collectors and Exporters. The SDK is platform agnostic, this a simple way to create our metrics, tracing and logs using only the OpenTelemetry library. When we create the data using the SDK we need to collect this data using the Collectors, there are a lot of ways to collect the data generated and we can even convert the data to other standard like Open Census and Jaeger. After the data was collect they are ready to be exported, we can use a lot of kind of exported but the default choice is the Open Telemetry Protocol (OTEL).
Using the OpenTelemetry we can instrument our application just once, and only change the Collector and the Exporter we can send our data to everywhere.
Newrelic
Newrelic is an a well know Application Monitoring (APM) that we can send ours observability data, as all the others APM in the market we can use the Newrelic instrument library or we can use the OpenTelemetry standard.
How can I use OpenTelemetry
To show haw easy can be to use OpenTelemetry, I have created a basic example using Rust to collect some metrics and tracing and send to Newrelic. The example could be found here.
The purpose of this article is not to explain technically how this example was created but to present the results of metrics and tracing collected and sent to Newrelic.
The project is basically two applications, a HTTP Server and a RabbitMQ Consumer, there are a few HTTP endpoint's and when we send a request in the create endpoint the data will be register in the database and delivery to the RabbitMQ to be consumed by the Consumer.
To record the traces properly, each time an HTTP request is received, a middleware searches the request header for information called traceparent, if has one, this information contains the ID of the trace that started the operation, in this way we can monitor the operation in all the services in which this operation travels, if there is no traceparent, is create a new one. The same approach was used in the RabbitMQ Consumer. With this approach we can see the following results
In each application was created health metrics to register if the application is running and a metric counter to register how many requests was made in the HTTP Server and how many messages was processed by the Consumer. With this metrics I was able to create a custom Dashboard to monitoring the applications Health and to show how many requests our applications processed.
Conclusion
Observability is an extremely important practice in applications today and not paying attention to it today can be considered an anti-pattern. We can see in the article that OpenTelemetry is a very simple pattern to use and that generates great results. In the future I will create other articles more technical to explain in deep about OpenTelemetry.
Thanks for reading and if it was useful please leave a clap.
