Eshopworld.Telemetry
A telemetry abstraction, using an Application Insights based implementation through a Reactive Extensions pipeline.
How does it work?
You instanciate BigBrother
and then you use it to Publish
events.
Ideally you store a singleton instance of BigBrother
in your DI container and reference it by
it's interface IBigBrother
.
What are these events that I "publish"?
Events are strongly typed structures of data, written as
POCOs.
They inherit from any of the base event types, except BbEvent
. Currently the following event types are supported:
BigDataEvent
isn't related to telemetry, it's purely here for showing the entire class structure. This special event type should only be used for events that aren't telemetry events but that should end up in the Data Lake store.
BbAnonymousEvent
is an internal event that is used when calling Publish
using an anonymous class. Internally we create one BbAnonymousEvent
and attach the anonymous payload to is before we stream it down the pipeline.
Here's a few examples:
public class MyExceptionEvent : BbExceptionEvent
{
public string WhinerMessage { get; set; }
public string Message { get; set; }
}
public class PaymentEvent : BbTelemetryEvent
{
public DateTime ProcessedOn { get; set; }
public float Ammount { get; set; }
public string Currency { get; set; }
}
public class PaymentAttemptEvent : BbTimedEvent
{
public float Ammount { get; set; }
public string Currency { get; set; }
}
Now that you have some events, publishing them is just:
bb.Publish(new PaymentEvent
{
ProcessedOn = DateTime.Now,
Ammount = 100,
Currency = "USD"
});
Optionally, and especially useful during prototyping phases, you can publish anonymous classes:
bb.Publish(
new
{
ProcessedOn = DateTime.Now,
Ammount = 100,
Currency = "USD"
});
The event will be named from the method name where the Publish was called from.
Timed Events
Timed events, besides tracking the normal event custom dimensions will also measure a ProcessingTime
metric and push it
to ApplicationInsights. The time window starts when the event is instantiated and finishes when the event is Published.
// Time starts being tracked here
var event = new PaymentAttemptEvent
{
Ammount = 100,
Currency = "USD"
});
// Do something that takes a while
Task.Delay(TimeSpan.FromMinutes(5)).Wait();
bb.Publish(event); // Time frame stops here
How do I correlate events?
BigBrother
plugs into the application insights correlation pipeline, so custom events will have operation_id
but won't have id
, this will be blank.
To facilitate even further and to plug into MVC better, we have a constructor that takes a TelemetryClient
that in ASP.NET Core will be a singleton:
/// <summary>
/// Initializes a new instance of <see cref="BigBrother"/>.
/// Used to leverage an existing <see cref="TelemetryClient"/> to track correlation.
/// This constructor does a bit of work, so if you're mocking this, mock the <see cref="IBigBrother"/> contract instead.
/// </summary>
/// <param name="client">The application's existing <see cref="TelemetryClient"/>.</param>
/// <param name="internalKey">The devops internal telemetry Application Insights instrumentation key.</param>
public BigBrother([NotNull]TelemetryClient client, [NotNull]string internalKey)
{
}
With latest .NET Core, it is now recommended to follow the pattern outlined in https://github.com/microsoft/ApplicationInsights-dotnet/issues/1152. Telemetry package offers TelemetryModule, which resolves TelemetryClient from DI as well as Eshopworld.Telemetry.Configuration.TelemetrySettings and exposes IBigBrother singleton instance. Outer and internal instrumentation keys are retrieved from the settings instance.
Sample instrumentation
builder.RegisterInstance(_telemetrySettings).SingleInstance();
builder.RegisterModule<TelemetryModule>();
It may be necessary to construct BigBrother outside of DI e.g. when you need to track container related exceptions during start up time. In these cases, the full initializer set is not necessary/relevant. We provide default BigBrother instance with relevant initializers
- OperationCorrelationTelemetryInitializer
- EnvironmentDetailsTelemetryInitializer
BigBrother.CreateDefault(_telemetrySettings.InstrumentationKey, _telemetrySettings.InternalKey);
EventSource and Trace sinks
The package now supports trace and ETW sinks to all BbExceptionEvents
. To set it up just use the fluent API:
bb.UseEventSourceSink().ForExceptions();
bb.UseTraceSink().ForExceptions();
You can also sink to both EventSource
and Trace
before you reach the point where you can instanciate BigBrother
by using the static method Error
BigBrother.PublishError(new Exception());
Internally the Exception will be placed inside a BbExceptionEvent
and that will be written to an EventSource
and a Trace
. BigBrother
will also replay anything sent through PublishError
when it gets instanciated
so that you'll publish to Application Insights events raised before getting to the point of instanciating
BigBrother
.
Telemetry processors in the package
RoleNameSetter
Sets the RoleName if not already set to be the entry point assembly full name. Useful in scenarios where the out-if-the-box interceptors won't set this for you, like WebJobs. Here's an example on how to set it up:
var builder = TelemetryConfiguration.Active.TelemetryProcessorChainBuilder;
// optional, will default to entry assembly name if not specified
RoleNameSetter.RoleName = "myCustomAppName";
builder.Use((next) => new RoleNameSetter(next));
builder.Build();
TelemetryFilterProcessor
Filters out health probe requests. Example:
public class Startup
{
public IServiceProvider ConfigureServices (IServiceCollection services)
{
// ...
services.AddApplicationInsightsTelemetryProcessor<TelemetryFilterProcessor>();
// ...
}
}
public class Bootstrap: Module
{
protected override void Load (ContainerBuilder builder)
{
// Add this if you don't have configure health checks middleware
builder.RegisterType<SuccessfulProbeFilterCriteria>()
.As<ITelemetryFilterCriteria>();
// Add this if you have configure health checks middleware and
// pass the health check path as parameter ex: '/probe' in this sample
builder.RegisterType<SuccessfulProbeFilterCriteria1>()
.As<ITelemetryFilterCriteria>()
.WithParameter(new TypedParameter(typeof(string), "/probe"));
// only add this if you don't have an instance of FilterCriteria registered
builder.RegisterType<DefaultTelemetryFilterCriteria>()
.As<ITelemetryFilterCriteria>()
.IfNotRegistered(typeof(ITelemetryFilterCriteria));
// ...
}
}
Telemetry initializers in the package
Environment Details
Adds additional environment values(tenant/region/etc) to telemetry for tracing purposes. Very useful when trying to diagnose issues in AI logs
using Microsoft.ApplicationInsights.Extensibility;
public void ConfigureServices(IServiceCollection services)
{
services.AddSingleton<ITelemetryInitializer, Eshopworld.Telemetry.Initializers.EnvironmentDetailsTelemetryInitializer>();
}
What can I also do with it?
You can force a Flush of the AI client, which will send all events right away:
bb.Flush();
You can also set the AI client to use a channel in DeveloperMode, this will push events right away without streaming them in memory. If you're getting a release package (from nuget.org), this doesn't do anything this is to force deployments to never have channels in DeveloperMode even if developers forget to remove the code.
bb.DeveloperMode();
Do I just include this package for instrumentation?
Depends on what type of application you're writting. This only includes the core Application Insights package, so if your application can leverage other packages it should also include any of the top level AI packages. For example, web applications should also include the Application Insights for Web package.
What is this AI key for inner telemetry?
This is the Application Insights account where BigBrother
pushes internal telemetry.
By design, BigBrother
will never throw, but everytime an exception is raised,
it is stream to the inner telemetry account.
We also track certain events, to see if we have wrong usage of BigBrother,
for example we track all calls to Flush
, so if you're pushing events and using Flush
right after,
we know about it!
I want to write some tests now, but I don't want to send events
BigBrother
does a bit of heavy lifting on both it's constructor and the class static constructor,
so you should always stub IBigBrother
instead, to avoid the heavy lifting done on the constructors.
BigBrother
Can also be deconstructed to gain access to the internal Observables and Observers:
public void Deconstruct(out IObservable<BbEvent> telemetryObservable, out IObserver<BbEvent> telemetryObserver, out IObservable<BbEvent> internalObservable)
This can be used in Unit Tests to instead of verifying Publish calls, the test just subscribes to the internal streams and asserts in the scope of that subscription. If you go down this route, be carefull with parallel tests and multiple subscriptions and make sure you always dispose of subscriptions to those Observables at the end of the test.
Kusto
Telemetry events can be sent to Data Explorer (Kusto). To wire it up just specify cluster connection defails and ingestion strategy/type combination (or fallback defaults).
Ingestion strategies:
-
Queued & buffered - uses local application buffer, and also Kusto's aggregation buffer (Data Management Cluster) before flushing the data to Kusto database (Data Explorer Engine). Much safer and reliable approach since internal Service Bus is used to offer better delivery guarantees
Note: local and Kusto buffers are fully configurable, by max buffer interval, massage count and/or size
When to use: lots of messages per second (100s or more), higher delivery reliability required, many services sending messages in parallel to one database
Local app buffer defaults: flush every 1 second or 100 messages, whatever comes first -
Direct - sends telemetry event directly to Kusto Engine. Less reliable but lower latency
When to use: message per second count not too high (up to ~50)
BibBrother first checks if there's a registered type per strategy, and then falls back to default strategy.
Few examples:
var bb = new BigBrother()
.UseKusto()
.WithCluster(engineName, region, database, tenantId)
.WithQueuedClient<FooTelemetryEvent>()
.WithDirectClient<BarTelemetryEvent>()
.WithFallbackQueuedClient() // all other types are going to queued client strategy
.Build();
// or set queued strategy defaults (works on both per type and fallback defaults):
var bb = new BigBrother()
.UseKusto()
.WithCluster(engineName, region, database, tenantId)
.WithQueuedClient<FooTelemetryEvent>(
new BufferedClientOptions { IngestionInterval = TimeSpan.FromSeconds(5), BufferSizeItems = 500 })
.Build();
// or the simplest configuration:
var bb = new BigBrother()
.UseKusto()
.WithCluster(engineName, region, database, tenantId)
.WithFallbackDirectClient()
.Build();
Don't forget to call .Build() at the end of configuration!