How do you capture Business KPIs/Metrics in your systems (bonus for microservices)?

Hi everyone,

I'm curious about recommended strategies for capturing business metrics in applications, especially in highly distributed / microservice-based systems. The ideal world is that we can come up with business metrics/KPIs and then create dashboards to reflect those as we build to measure against the KPIs.

I'll try to explain through an example: Let's say it's an application that supports several task-oriented workflows. Metrics like the number of tasks created, time from task creation to completion, etc. Then, each workflow might have its own domain-centric metrics based on the data gathered during the tasks/workflow. Each domain's workflow/tasks interact with various services that support those specific domains. aka it's very distributed and the process might start in Service A, but end up in Service D, E, F based on the workflow.

The specific example isn't too important as is the concept that it's a very distributed application with related but different domains and the start/finish can be highly variable.

Added context: The system in question is very backend process-heavy, so there's a lot going on in the background. A given process can also have many touchpoints, many different users, and downtime between processes. So, it can't just be measured from Point A to Point B on the FE is a single session.

Options Explored:

• FE solutions like Google Analytics to capture events will only give a small piece of the overall process since much of it is BE processing, events, etc.
• Datadog, Dynatrace type services have custom metrics. This is the closest thing I've found, but it still feels like it's not set up to handle metrics across services. Where the metric might start in Service A, but end in Service D, E, or F.

Any suggestions are appreciated!