# CloudWatch Omni rollouts are decided before the first team signs in

By MarketScale Newsroom · Published 2026-11-19 · Software & Technology on MarketScale
Canonical: https://www.marketscale.com/industries/software-and-technology/cloudwatch-omni-rollouts-are-decided-before-the-first-team-signs-in

> AWS's CloudWatch Omni guide says how well it serves an enterprise depends on decisions made before the first team signs in.

## Key points

- Omni centralization rules capture only telemetry that arrives after they are created, so the date the rules go live sets how far back incident teams can look.
- Omni does not detect or redact personal data on its own; capture-time redaction is the only layer that meets the requirement that data never be exported and it cannot be reversed.
- Organizations with data residency rules should settle their IAM Identity Center primary Region before creating the Omni domain, because the domain has to stay in that Region.

A support agent starts timing out. The real cause is a downstream service that slowed down, and the evidence is spread across several AWS accounts, so the context ends up in screenshots. AWS uses that scenario to open its guide to deploying Amazon CloudWatch Omni at enterprise scale, on the AWS Cloud Operations Blog. Omni's answer is to put application and AI agent observability behind one sign-in URL backed by AWS IAM Identity Center.

The more useful part of the guide is a warning. AWS argues that how well Omni works for a large organization depends on decisions made before the first team signs in, and several of those decisions are hard to change later. Draw a boundary too wide and teams see data they shouldn't. Draw it too narrow and one incident gets split across spaces that can't be queried together. For the platform engineering lead who owns the rollout, the setup order is the governance design.

## Five parts, one hard boundary

Omni is built from five pieces. The domain serves as the organization's front door: its name forms the sign-in URL, in the form https://<domain-name>.cloudwatch-omni.global.app.aws, and the identity provider is connected at the domain rather than in each space. Centralization rules copy telemetry from source accounts and Regions into a destination monitoring account. Teams do their work in a space, which holds dashboards, queries, alerts, investigations and the application map. Every space is tied to exactly one AWS account and Region, and no account and Region pair can hold more than one space.

The remaining two pieces operate inside a space. A CloudWatch Dataset puts logs, metrics and traces in one place where they can be queried and correlated, while ingestion and storage remain on existing CloudWatch resources such as log groups. Enabling Omni creates one Dataset for the space. Access grants assign a permission level to an IAM role, an IAM Identity Center group or an access profile. A grant can also carry an optional data scope that limits which log and trace rows the holder can read.

That split is the design principle. Spaces draw the hard lines between business units and Regions. Grants draw the softer lines between teams inside one space. Omni doesn't aggregate telemetry across accounts or Regions on its own, so a space sees only what centralization rules have delivered into its account and Region.

Block Field

## Region first, then the start date

AWS uses a made-up company, AnyCompany, as its example. It has hundreds of accounts in one AWS Organizations organization, plus some workloads in other clouds and on-premises. Three business units own those accounts. BU1 and BU2 both operate in us-east-1. BU3, which serves European customers, runs from eu-west-2 and must keep its telemetry inside the EU. Before rollout, the platform team settled on six requirements, ranging from a single sign-in URL for everyone to restricting prompts from sensitive agents to a small group of reviewers.

The first fixed decision is the domain's location. The domain is created from the management account in the IAM Identity Center primary Region and cannot move from there, so AWS advises treating the primary Region as a fixed design decision. AnyCompany runs IAM Identity Center in us-east-1, so that is where its domain lives. Before a space can be set up in any other Region, IAM Identity Center must be replicated to it, or sign-in can't be configured. Because BU3's space is in eu-west-2, AnyCompany's first step is to turn on IAM Identity Center multi-Region replication for that Region.

The payoff for getting this right is that every member account that later creates a space links to the domain without extra configuration, and teams keep one URL however many spaces are added. A domain can accept IAM, IAM Identity Center or both. For sign-in, AnyCompany configures the domain to use IAM Identity Center alone, which it federates with its external identity provider. It then applies AWS Organizations policies that block member accounts from setting up domains of their own, which keeps that single front door from quietly sprouting side entrances.

One detail matters to whoever runs access reviews. When a space is created, the creating identity is made its first Space Admin, and as long as no other admin grant exists, that one can't be removed. AWS recommends building every space with a dedicated break-glass IAM role and adding the Identity Center admin groups afterward. That way the admin grant that can't be removed sits with an emergency role created for that purpose.

The second fixed decision is timing. In Omni, history starts the day the centralization rule does: telemetry that arrived before it is not backfilled. Trace centralization also depends on CloudWatch Transaction Search being turned on in each source account. If destination log groups use a customer managed AWS KMS key, the key needs the tag LogsManaged = true before the centralization service can use it.

AnyCompany routes by organizational unit and lets residency pick the destination. BU1 and BU2 replicate into their own monitoring accounts in us-east-1. BU3 replicates into bu3-monitoring in eu-west-2. Each BU monitoring account gets one space. Put another way, the residency rule is enforced when the destination account is chosen, and the space follows from that choice.

Block Field

## Where prompt data gets filtered

Requirement six is the hard one, because agent telemetry carries content. AWS's Omni documentation on sensitive data says prompts and responses often include user messages, retrieved documents and personal data. It also says plainly that Omni does not detect or redact personally identifiable information on its own. Nothing is removed unless someone configures it. Teams are also asked to keep details such as customer email addresses out of tags and the free-form names on spaces, grants, access profiles, dashboards and alerts, because those fields can end up in billing or diagnostic logs.

The documentation lists five places to apply controls, from most restrictive to least. Each one trades data removal against what's left for debugging:

1. Capture time: redact inside the application using OpenTelemetry mechanisms, such as the AWS\_REDACT\_SPAN\_ATTRIBUTES variable (which accepts wildcards) or the OpenInference flags that hide inputs and outputs. This is the only layer that meets a requirement that data never be exported.
2. Collector: OpenTelemetry Collector processors (attributes, redaction, transform, filter) as a second safety net after data has left the process.
3. Ingestion: CloudWatch Logs data protection masks patterns such as Social Security and credit card numbers in log data.
4. At rest: encryption of stored telemetry.
5. Read time: row-level data scopes on access grants control who reads which rows.

The capture-time tools differ in what they leave behind. AWS\_REDACT\_SPAN\_ATTRIBUTES replaces matching values in spans, span events and span links with REDACTED, and it covers every span attribute, not only those from AWS Distro for OpenTelemetry (ADOT) instrumentation. The OpenInference flags work earlier still, swapping prompt and response content for \_\_REDACTED\_\_ before spans are even created.

The strictest layer has a cost. Capture-time redaction is permanent, so redacted content isn't available for root-cause analysis later. AWS says a trace with unexpectedly blank inputs and outputs should send teams to these flags first. Redaction can also break other integrations that rely on those attribute values. The runtime matters too. If an agent runs on Amazon Bedrock AgentCore Runtime, customers do not manage a collector there, so AWS says to redact at capture time instead. Python teams can write a custom span processor, which needs opentelemetry-sdk 1.39.0 or later. JavaScript has no equivalent, so AWS recommends the environment variable there.

The collector layer splits the work four ways: attributes removes or modifies specific attributes, redaction drops anything not on an allow list, transform replaces values using regular expressions, and filter drops entire spans that match a condition. It is available wherever teams run their own collector, for example on Amazon EC2, Amazon ECS or Amazon EKS. AWS frames it as a safety net, not a substitute, for organizations whose compliance model requires that data never leave their infrastructure.

The CloudWatch Logs documentation sets out how ingestion masking works. Masked values stay hidden at every exit point, including Logs Insights, metric filters and subscription filters, and only principals with the logs:Unmask IAM permission see the originals. A policy can cover the whole account, including log groups created later, or a single log group, and both apply where they overlap. Detection happens at ingestion, so events that arrived before a policy existed stay unmasked. Each match sends a free LogEventsWithFindings metric to the AWS/Logs namespace, and teams can set alarms on it.

Detection runs on managed data identifiers covering credentials, financial information, PII, protected health information and device identifiers such as IP and MAC addresses, using pattern matching and machine learning models, and teams can add custom identifiers of their own. Each log group takes one group-level policy of up to 30,720 characters, which can list many identifiers. Because those identifiers target known data types, ingestion masking looks best suited to catching stray card numbers or keys in logs, while whole prompt and response attributes are a job for the capture-time controls above.

Masking has the same timing problem as centralization: a policy written after launch protects only new data. It also adds a second permission list. Read-time data scopes on Omni grants decide who sees sensitive agent rows. Separately, logs:Unmask decides who sees masked log values. For a company meeting a rule like AnyCompany's sixth, those two lists need the same owner, or they will drift apart.

## When separate spaces cost too much

Separate spaces have a cost, and AWS names it: split them too finely and an incident gets scattered across views that can't be queried together. AnyCompany's slow support agent would hit that wall if the downstream service lived in another business unit's monitoring account. One space per BU suits an organization whose incidents mostly stay inside one unit. If services regularly call across BU lines, there's a better case for wider spaces, with grants keeping teams apart inside them.

Data platform teams will know this tension. Databricks' operational excellence guidance, last updated November 5, 2026, recommends separate workspaces for each business unit so admins and assets aren't shared by accident. It also warns that notebook collaboration doesn't work across workspaces and that running many workspaces needs fully automated setup. Omni spaces have the same tradeoff for telemetry, plus one constraint that access control can't loosen: BU3's EU data has to land in eu-west-2.

The analogy runs one layer deeper. In Databricks the workspace is the primary computing environment and catalogs are the primary data domain, and binding a catalog to specific workspaces decides what is available there regardless of the object permissions a user holds. Omni's space plays the workspace-and-binding role, fixing which telemetry exists to be queried at all, while grants and data scopes play the permission role inside it.

An organization running in a single Region, with one business unit and no residency rule, can probably fold most of this into one space and a set of grants. The guide is written for the other case: hundreds of accounts, more than one jurisdiction, and agents whose traces carry customer text.

For that case, the first decision may not need a workshop. It could start as a question for whoever runs IAM Identity Center: which Region is primary, and is the organization comfortable with Omni's sign-in domain staying in it?

Block Field

## Sources

- [Best practices for deploying Amazon CloudWatch Omni at enterprise scale](https://aws.amazon.com/blogs/mt/best-practices-for-deploying-amazon-cloudwatch-omni-at-enterprise-scale/)
- [Protect sensitive data - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/omni-data-protection.html)
- [Help protect sensitive log data with masking - Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html)
- [Best practices for operational excellence | Databricks on AWS](https://docs.databricks.com/aws/en/lakehouse-architecture/operational-excellence/best-practices)
- [AWS Connected Community - Events](https://aws-experience.com/amer/smb/events/ff098/hands-on-aws-observability-mastering-ai-operations-aiops)

Tags: Amazon CloudWatch Omni, AWS IAM Identity Center, AI agent observability, Data residency, OpenTelemetry, CloudWatch Logs data protection

---
Source: MarketScale, https://www.marketscale.com/industries/software-and-technology/cloudwatch-omni-rollouts-are-decided-before-the-first-team-signs-in. Published for AI indexing and citation; cite the canonical URL. Site guide for agents: https://www.marketscale.com/llms.txt
