OpenTelemetry: A beginner’s handbook to instrument your application !!!
Recently I completed reading “Observability Engineering” by Honeycomb team Harity Majors, George Miranda, and Liz Fong-Jones. I have been a big follower of Liz since her google days when they released “Site Reliability Engineering: Measuring and Managing Reliability” course which as such was my first interaction with SRE Concepts.Coming back to the read, the book does really really well in terms of differentiating Observability from Monitoring and why you should enable telemetry signals in your application to capture essential insights.
A tad bit about Observability!
To put in layman terms observability has everything to do with your application and as applications have become the centre of universe importance of observability has multi folded over the years.While the term Observability was first coined by twitter back in 2013 (https://blog.twitter.com/engineering/en_us/a/2013/observability-at-twitter) it has come a long way since.With today’s distributed systems, application teams can’t just relay on their knowledge of the systems and known issues but be prepared for unknown of unknowns and that’s exactly where observability can help.
So where does Open-telemetry fit in?
Open-telemetry is vendor agnostic, opensource and probably the most popular observability framework at the moment.It’s been developed by merger of two independent projects Opentracing and Opencensus and is a sandbox project under CNCF. The merger created a new unified set of libraries and specifications for Observability telemetry and as if today it has SDK support for most of the languages like Java, go, python, node. js, c++.The ultimate goal for OpenTelemetry is to ensure that the telemetry data(Logs, traces, metrics)is a built-in feature of cloud-native software. This means that libraries, frameworks, and SDKs should emit this telemetry data without requiring end-users to proactively instrument their code. To accomplish this, OpenTelemetry is producing a single set of system components and language-specific libraries that can create, collect, and capture these sources of telemetry data and export them to analysis tools through an exporter.
Okay, So how does OTel emit telemetry signals?
Before understanding how oTel emit signals it’s important to understand the different components it consists of. Briefly speaking there are three major parts of OTel.
- The Opentelemetry Specification — Describes the cross-language requirements and expectations for all OpenTelemetry implementations. This allows the full stack to speak the same language, gather data in the same format, and ultimately transport that data for processing.
- Opentelemery APIs and SDKs- API library and SDK library written natively in every major programming language which are the implementations of the OpenTelemetry specification.This gives freedom to the developer selecting the libraries from which they would like to collect telemertry data, enrich that data and export that data to collector.
- Opentelemetry Collector- The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process and export telemetry data.. It supports receiving telemetry data in multiple formats (e.g., OTLP, Jaeger, Prometheus, as well as many commercial/proprietary tools) and sending data to one or more backends. It also supports processing and filtering telemetry data before it gets exported.
Now as we discussed various components let’s take a look at how instrumentation works with an example.
Our demo go application maintains employee records and allows operations like get employee list, get specific employee by ID, add new employee and delete employee entry.
package client
import (
"net/http"
"github.com/gin-gonic/gin"
)
// employee represents data about a an employee record.
type employee struct {
ID string `json:"id"`
Name string `json:"name"`
Role string `json:"role"`
Department string `json:"department"`
Gendor string `json:"gendor"`
}
// employees slice to seed record album data.
var employees = []employee{
{ID: "P0P1", Role: "Associate", Name: "Aakash", Department: "R&D",Gendor: "Male"},
{ID: "P0P2", Role: "Consultant", Name: "Divit", Department: "R&D", Gendor: "Male"},
{ID: "P0P3", Role: "Manager", Name: "Dhriti", Department: "R&D", Gendor: "Female"},
}
// getEmployees responds with the list of all albums as JSON.
func GetEmployees(c *gin.Context) {
c.IndentedJSON(http.StatusOK, employees)
}
// postEmployees adds an album from JSON received in the request body.
func CreateEmployees(c *gin.Context) {
var newemployee employee
// Call BindJSON to bind the received JSON to
// newEmployee.
if err := c.BindJSON(&newemployee); err != nil {
return
}
// Add the new employee to the slice.
employees = append(employees, newemployee)
c.IndentedJSON(http.StatusCreated, newemployee)
}
// getEmployeeByID locates the employee whose ID value matches the id
// parameter sent by the client, then returns that employee as a response.
func GetEmployeeByID(c *gin.Context) {
id := c.Param("id")
// Loop through the list of employees, looking for
// an employee whose ID value matches the parameter.
for _, a := range employees {
if a.ID == id {
c.IndentedJSON(http.StatusOK, a)
return
}
}
c.IndentedJSON(http.StatusNotFound, gin.H{"message": "employee not found"})
}
// Delete a particular employee record by ID
func DeleteEmployeeByID(c *gin.Context) {
id := c.Param("id")
// Loop through the list of employees, looking for
// an employee whose ID value matches the parameter.
for _, a := range employees {
if a.ID == id {
c.IndentedJSON(http.StatusOK, a)
return
}
}
c.IndentedJSON(http.StatusNotFound, gin.H{"message": "employee records deleted"})
}To enable oTel instrumentation in main.go import oTel packages.
import(
"github.com/gin-gonic/gin"
"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
)Declare following variables to configure oTel.
var (
serviceName = os.Getenv("SERVICE_NAME")
collectorURL = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
insecure = os.Getenv("INSECURE_MODE")
)Define startTracer function to extract oTel telemetry data from application.
func startTracer() func(context.Context) error {
secureOption := otlptracegrpc.WithTLSCredentials(credentials.NewClientTLSFromCert(nil, ""))
if len(insecure) > 0 {
secureOption = otlptracegrpc.WithInsecure()
}
exporter, err := otlptrace.New(
context.Background(),
otlptracegrpc.NewClient(
secureOption,
otlptracegrpc.WithEndpoint(collectorURL),
),
)
if err != nil {
log.Fatal(err)
}
resources, err := resource.New(
context.Background(),
resource.WithAttributes(
attribute.String("service.name", serviceName),
attribute.String("library.language", "go"),
),
)
if err != nil {
log.Printf("Could not set resources: ", err)
}
otel.SetTracerProvider(
sdktrace.NewTracerProvider(
sdktrace.WithSampler(sdktrace.AlwaysSample()),
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(resources),
),
)
return exporter.Shutdown
}In the main.go main function initialise the tracer.
func main() {
init_trace := startTracer()
defer init_trace(context.Background())
}Here is the entire main.go.
package main
import (
"context"
"log"
"os"
"google.golang.org/grpc/credentials"
"sample-golang-app/client"
"github.com/gin-gonic/gin"
"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
)
var (
serviceName = os.Getenv("SERVICE_NAME")
collectorURL = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
insecure = os.Getenv("INSECURE_MODE")
)
func startTracer() func(context.Context) error {
secureOption := otlptracegrpc.WithTLSCredentials(credentials.NewClientTLSFromCert(nil, ""))
if len(insecure) > 0 {
secureOption = otlptracegrpc.WithInsecure()
}
exporter, err := otlptrace.New(
context.Background(),
otlptracegrpc.NewClient(
secureOption,
otlptracegrpc.WithEndpoint(collectorURL),
),
)
if err != nil {
log.Fatal(err)
}
resources, err := resource.New(
context.Background(),
resource.WithAttributes(
attribute.String("service.name", serviceName),
attribute.String("library.language", "go"),
),
)
if err != nil {
log.Printf("Could not set resources: ", err)
}
otel.SetTracerProvider(
sdktrace.NewTracerProvider(
sdktrace.WithSampler(sdktrace.AlwaysSample()),
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(resources),
),
)
return exporter.Shutdown
}
func main() {
init_trace := startTracer()
defer init_trace(context.Background())
r := gin.Default()
r.Use(otelgin.Middleware(serviceName))
// Routes
r.GET("/employees", client.GetEmployees)
r.GET("/employees/:id", client.GetEmployeeByID)
r.POST("/employees", client.CreateEmployees)
r.DELETE("/employees/:id", client.DeleteEmployeeByID)
// Run the server
r.Run(":8090")
}Now as we have enabled instrumentation let’s define a backend to receive and visualise this data. For our example we would use SigNoz. SigNoz is an open-source observability platform which enables visualisation of metrics, logs and traces on a single dashboard.
Install SigNoz . You need to have docker installed in your system as a pre requisite.
git clone -b main https://github.com/SigNoz/signoz.git
cd signoz/deploy/
./install.shWhen you are done installing SigNoz, you can access the UI at http://localhost:3301.
As the dashboard is up lets set the environment variables to pump in data to oTel Collector. Run the following command to send application telemetry data to SigNoz.
SERVICE_NAME=oTelgoApp INSECURE_MODE=true OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 go run main.goOpen http://localhost:8090/employees and refresh page few times to generate some traffic.Now we can see the app being visible in SigNoz dashboard.
Click on the app which takes you to the metrics page where you can find no of requests, response status and latency.You can also get details about db and external calls, logs and traces.
As we saw above how effective it is to instrument your app with OpenTelemetry to achieve observability. In addition to auto generating telemetry data oTel also allows custom instrumentation which can be curated in accordance with the need of an application. We would discuss about custom instrumentation in subsequent blogs. So does that mean oTel is all you need for your application monitoring. May be not!!Like any other tools oTel has it’s own set up limitations as well.
What’s not so good?
Infra Metrics: oTel can’t emit infra metrics for which you need to use tools like prometheus to get important insights around infrastructure.
Lacks storage backend and visualisation layer: oTel does not recommend a storage backend or a dashboard. For which you need to depend upon proprietary tools or open source platform’s like SigNoz. The storage of high volume of telemetry data might become an expensive affair.
Documentation and Support: The project is relatively new so lots of room for documentation and support improvements.
The last say!!!
OpenTelemetry is setting the stage for cloud native apps to be developed and managed effectively. The project has tremendous scope and is backed by a huge community of contributors. It won’t take a long for us to see more and more organisations adopting oTel to manage their applications and certainly it’s time to invest some effort exploring it.