Send a custom span event

lab

This procedure is part of a lab that teaches you how to instrument your application with OpenTelemetry.

Each procedure in the lab builds upon the last, so make sure you've completed the last procedure, View your OpenTelemetry data in New Relic, before starting this one.

You've decided you don't want the Python OpenTelemetry SDK to automatically record exceptions as exception span events, because they're not really errors in the database application. These are expected exceptions based on user behavior. Here, you modify your code to record a custom span event, rather than automatically collecting an exception span event.

Modify your instrumentation

Step 1 of 5

In the terminal window that's running your simulator, press <CTRL-C>.

You should see your simulator shut down. Now you can update your app logic to add custom span events.

Step 2 of 5

In db.py, modify your span context managers to not record exceptions as span events:

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8

9
provider = TracerProvider(
10
    resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
    BatchSpanProcessor(
14
        OTLPSpanExporter(compression=Compression.Gzip)
15
    )
16
)
17
trace.set_tracer_provider(provider)
18

19
tracer = trace.get_tracer(__name__)
20

21
class DuplicateKeyError(Exception):
22
    pass
23

24
class KeyDoesNotExistError(Exception):
25
    pass
26

27
db = {}
28

29
def read(key):
30
    """Read key from the database."""
31
    global db
32

33
    with tracer.start_as_current_span("read", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
34
        try:
35
            value = db[key]
36
            logging.debug("Successful read")
37
            span.set_attribute("key", key)
38
            return value
39
        except KeyError as ke:
40
            msg = f"Key `{key}` doesn't exist"
41
            logging.debug(msg)
42
            raise KeyDoesNotExistError(msg)
43

44
def create(key, value):
45
    """Write key:value to the database."""
46
    global db
47

48
    with tracer.start_as_current_span("create", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
49
        if key in db:
50
            msg = f"Key `{key}` already exists"
51
            logging.debug(msg)
52
            raise DuplicateKeyError(msg)
53

54
        db[key] = value
55
        logging.debug("Successful create")
56
        span.set_attribute("key", key)
57
        return value
58

59
def update(key, value):
60
    """Update key in the database."""
61
    global db
62

63
    with tracer.start_as_current_span("update", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
64
        if key in db:
65
            db[key] = value
66
            logging.debug("Successful update")
67
            span.set_attribute("key", key)
68
            return value
69

70
        msg = f"Key `{key}` doesn't exist"
71
        logging.debug(msg)
72
        raise KeyDoesNotExistError(msg)
73

74
def delete(key):
75
    """Delete key from the database."""
76
    global db
77

78
    with tracer.start_as_current_span("delete", kind=trace.SpanKind.SERVER) as span:
79
        if key in db:
80
            del db[key]
81
            logging.debug("Successful delete")
82
            span.set_attribute("key", key)
83
            return True
84

85
        return False

db.py

Your code will no longer save an exception span event on your spans. However, you still want to know how many times your users attempt to perform these actions against your database. To do this, record your own custom span events.

Tip

Notice that you didn't update the context manager for delete(). This is because the logic in this function doesn't use an exception to indicate the state of the database. Any exception that is raised in the underlying code here, will be a real error. It still makes sense to let the SDK manage real errors.

Step 3 of 5

Record a span event for read():

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8

9
provider = TracerProvider(
10
    resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
    BatchSpanProcessor(
14
        OTLPSpanExporter(compression=Compression.Gzip)
15
    )
16
)
17
trace.set_tracer_provider(provider)
18

19
tracer = trace.get_tracer(__name__)
20

21
class DuplicateKeyError(Exception):
22
    pass
23

24
class KeyDoesNotExistError(Exception):
25
    pass
26

27
db = {}
28

29
def read(key):
30
    """Read key from the database."""
31
    global db
32

33
    with tracer.start_as_current_span("read", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
34
        try:
35
            value = db[key]
36
            logging.debug("Successful read")
37
            span.set_attribute("key", key)
38
            return value
39
        except KeyError as ke:
40
            msg = f"Key `{key}` doesn't exist"
41
            logging.debug(msg)
42
            span.add_event("read_key_dne", {"key": key})
43
            raise KeyDoesNotExistError(msg)
44

45
def create(key, value):
46
    """Write key:value to the database."""
47
    global db
48

49
    with tracer.start_as_current_span("create", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
50
        if key in db:
51
            msg = f"Key `{key}` already exists"
52
            logging.debug(msg)
53
            raise DuplicateKeyError(msg)
54

55
        db[key] = value
56
        logging.debug("Successful create")
57
        span.set_attribute("key", key)
58
        return value
59

60
def update(key, value):
61
    """Update key in the database."""
62
    global db
63

64
    with tracer.start_as_current_span("update", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
65
        if key in db:
66
            db[key] = value
67
            logging.debug("Successful update")
68
            span.set_attribute("key", key)
69
            return value
70

71
        msg = f"Key `{key}` doesn't exist"
72
        logging.debug(msg)
73
        raise KeyDoesNotExistError(msg)
74

75
def delete(key):
76
    """Delete key from the database."""
77
    global db
78

79
    with tracer.start_as_current_span("delete", kind=trace.SpanKind.SERVER) as span:
80
        if key in db:
81
            del db[key]
82
            logging.debug("Successful delete")
83
            span.set_attribute("key", key)
84
            return True
85

86
        return False

db.py

If you try to read a key that does not exist, your application adds a span event on the span.

Step 4 of 5

Record a span event for create():

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8

9
provider = TracerProvider(
10
    resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
    BatchSpanProcessor(
14
        OTLPSpanExporter(compression=Compression.Gzip)
15
    )
16
)
17
trace.set_tracer_provider(provider)
18

19
tracer = trace.get_tracer(__name__)
20

21
class DuplicateKeyError(Exception):
22
    pass
23

24
class KeyDoesNotExistError(Exception):
25
    pass
26

27
db = {}
28

29
def read(key):
30
    """Read key from the database."""
31
    global db
32

33
    with tracer.start_as_current_span("read", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
34
        try:
35
            value = db[key]
36
            logging.debug("Successful read")
37
            span.set_attribute("key", key)
38
            return value
39
        except KeyError as ke:
40
            msg = f"Key `{key}` doesn't exist"
41
            logging.debug(msg)
42
            span.add_event("read_key_dne", {"key": key})
43
            raise KeyDoesNotExistError(msg)
44

45
def create(key, value):
46
    """Write key:value to the database."""
47
    global db
48

49
    with tracer.start_as_current_span("create", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
50
        if key in db:
51
            msg = f"Key `{key}` already exists"
52
            logging.debug(msg)
53
            span.add_event("create_key_exists", {"key": key})
54
            raise DuplicateKeyError(msg)
55

56
        db[key] = value
57
        logging.debug("Successful create")
58
        span.set_attribute("key", key)
59
        return value
60

61
def update(key, value):
62
    """Update key in the database."""
63
    global db
64

65
    with tracer.start_as_current_span("update", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
66
        if key in db:
67
            db[key] = value
68
            logging.debug("Successful update")
69
            span.set_attribute("key", key)
70
            return value
71

72
        msg = f"Key `{key}` doesn't exist"
73
        logging.debug(msg)
74
        raise KeyDoesNotExistError(msg)
75

76
def delete(key):
77
    """Delete key from the database."""
78
    global db
79

80
    with tracer.start_as_current_span("delete", kind=trace.SpanKind.SERVER) as span:
81
        if key in db:
82
            del db[key]
83
            logging.debug("Successful delete")
84
            span.set_attribute("key", key)
85
            return True
86

87
        return False

db.py

If you try to create a key that already exists, your application adds a span event on the span.

Step 5 of 5

Record a span event for update():

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8

9
provider = TracerProvider(
10
    resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
    BatchSpanProcessor(
14
        OTLPSpanExporter(compression=Compression.Gzip)
15
    )
16
)
17
trace.set_tracer_provider(provider)
18

19
tracer = trace.get_tracer(__name__)
20

21
class DuplicateKeyError(Exception):
22
    pass
23

24
class KeyDoesNotExistError(Exception):
25
    pass
26

27
db = {}
28

29
def read(key):
30
    """Read key from the database."""
31
    global db
32

33
    with tracer.start_as_current_span("read", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
34
        try:
35
            value = db[key]
36
            logging.debug("Successful read")
37
            span.set_attribute("key", key)
38
            return value
39
        except KeyError as ke:
40
            msg = f"Key `{key}` doesn't exist"
41
            logging.debug(msg)
42
            span.add_event("read_key_dne", {"key": key})
43
            raise KeyDoesNotExistError(msg)
44

45
def create(key, value):
46
    """Write key:value to the database."""
47
    global db
48

49
    with tracer.start_as_current_span("create", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
50
        if key in db:
51
            msg = f"Key `{key}` already exists"
52
            logging.debug(msg)
53
            span.add_event("create_key_exists", {"key": key})
54
            raise DuplicateKeyError(msg)
55

56
        db[key] = value
57
        logging.debug("Successful create")
58
        span.set_attribute("key", key)
59
        return value
60

61
def update(key, value):
62
    """Update key in the database."""
63
    global db
64

65
    with tracer.start_as_current_span("update", kind=trace.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
66
        if key in db:
67
            db[key] = value
68
            logging.debug("Successful update")
69
            span.set_attribute("key", key)
70
            return value
71

72
        msg = f"Key `{key}` doesn't exist"
73
        logging.debug(msg)
74
        span.add_event("update_key_dne", {"key": key})
75
        raise KeyDoesNotExistError(msg)
76

77
def delete(key):
78
    """Delete key from the database."""
79
    global db
80

81
    with tracer.start_as_current_span("delete", kind=trace.SpanKind.SERVER) as span:
82
        if key in db:
83
            del db[key]
84
            logging.debug("Successful delete")
85
            span.set_attribute("key", key)
86
            return True
87

88
        return False

db.py

If you try to update a key that does not exist, your application adds a span event on the span.

Restart your simulator

Now that you've changed the application logic, you need to restart your simulator. Make sure you do this in the same terminal window where you set your environment variables.

Restart your simulator:

bash

$python simulator.py

Now, your simulator is running again.

You've instrumented your application to send custom events with your spans. You've also restarted your simulator. Now, it's time to view your new data.

View your new data in New Relic

Step 1 of 6

Log into New Relic.

Step 2 of 6

Go back to your OpenTelemetry service:

Click database service

Notice that your errors have dropped to zero:

No errors

You won't be able to use error counts to know how many times your users try to read keys that aren't there or create keys that already are, so use the query builder to figure it out.

Tip

If you don't see a change in your data, compare your code to ours.

Step 3 of 6

Click Query your data:

Query your data

With our query builder, you can write arbitrary queries against our database.

Step 4 of 6

Click Query builder and enter the following query:

FROM SpanEvent SELECT count(*) FACET name

Query

This query counts all the span events from your account and groups them by their name.

Step 5 of 6

Click Run to see your data:

Run

Step 6 of 6

Change your chart type to Table for better readability:

Table view of your span events

Here, you see the number of times your application has seen each span event, organized into a nice table.

Tip

You can even add this table to a dashboard if you want to:

Add your chart to a dashboard

You've updated your app to stop reporting normal user actions as errors. At the same time, you've maintained the ability to query the number of occurences of these actions.

Summary

As the developer of speedeedeebee, you've now instrumented your application with OpenTelemetry to send manually-collected traces to New Relic. And because you've instrumented your app with OpenTelemetry instead of our Python agent, you have more flexibility in how you can use your data. For example, if you want to add additional backend data sources besides New Relic, you can easily change that without having to add another vendor-specific agent.

Homework

Now that you know how to instrument a Python application with OpenTelemetry and send that data to New Relic, here are some things you can do next to familiarize yourself even more with New Relic and OpenTelemetry:

Check out our repository of OpenTelemetry examples
Learn more about OpenTelemetry's Python implementation
Read our documentation on New Relic + OpenTelemetry