StackScopes
StackScopes begins with an accurate, explainable view of cloud resources and configuration. The Multi-Account AWS Discovery Pipeline collects supported AWS metadata across approved accounts and Regions, normalizes service-specific identities, and reports freshness and collection gaps without presenting partial visibility as completeness.
Problem
AWS environments rarely remain inside one account or Region. Production, development, networking, security, observability, and shared services are commonly separated. AWS Organizations can add hierarchy, but application dependencies still cross organizational units through identity, networking, encryption, artifacts, logging, and events.
Discovery becomes difficult for reasons that are easy to hide in a polished dashboard:
- Service APIs use different identifiers, pagination models, and regional behavior.
- Some resources are global, some regional, and some zonal.
- Read permissions intentionally differ across accounts.
- API throttling and temporary failures create partial results.
- Similar resource names can appear in many accounts.
- Resources change while collection is running.
- Disabled Regions and unsupported types create expected gaps.
- A failed role assumption can make an entire account appear empty.
A flat import that ignores these conditions can produce duplicate nodes, merge unrelated resources, miss dependencies, or silently delete resources during a partial scan. Every later analysis—topology, simulation, blast radius, and incident investigation—would inherit those errors.
What We Built
We built a discovery pipeline specifically for distributed AWS estates. It uses approved cross-account role assumptions, enumerates configured scopes, queries supported service APIs, normalizes resource identity and configuration, extracts explicit relationships, and reconciles observations against prior state.
The pipeline treats account, Region, and tenant context as boundaries, not optional tags. A resource identifier is resolved with its provider scope so two similarly named resources do not collapse into one node.
Collection is read-only. Any future controlled remediation capability uses a separate permission path. This separation reduces accidental privilege expansion and makes it possible to reason clearly about what discovery can and cannot do.
The pipeline is fault-aware. A throttled API or denied action creates a visible collection result for that service and scope. It does not become evidence that no resources exist.
How It Works
Establish approved scope
StackScopes records which AWS accounts, Regions, and services are in scope. Cross-account access uses short-lived role sessions and can incorporate an external ID where appropriate. Offboarding removes the authorized relationship without requiring long-lived credentials to be stored.
The pipeline can use AWS Organizations context where available, but organizational hierarchy is not treated as application ownership. A shared-services account may support workloads across several organizational units.
Enumerate safely
Collectors call supported AWS service APIs with pagination, bounded concurrency, throttling controls, and retries appropriate to each source. Errors are isolated so a failure in one Region or service does not discard successful observations elsewhere.
Each run maintains status by account, Region, service, and operation. This allows the final freshness view to distinguish a fully reconciled scope from a partial one.
Normalize identity
The normalization layer combines account ID, partition, Region, service, resource type, provider identifier, and ARN where available. It understands that IAM resources have different scope from RDS instances or Availability Zone resources.
Names and tags become useful attributes but do not replace identity. Logical infrastructure identifiers from Terraform or CloudFormation can be linked to deployed resource identities using evidence rather than assumed from naming similarity.
Preserve service detail
Common fields—identity, ownership context, lifecycle, placement, and tags—are normalized across services. Service-specific configuration remains available because important behavior lives in details such as listener rules, route targets, role trust policies, encryption keys, subnet groups, and event-source mappings.
Reconcile incrementally
Event-driven inputs can reduce latency, while scheduled reconciliation repairs missed changes and verifies current state. The pipeline compares new observations with previous versions, updates freshness, opens new configuration versions, and retires resources only after collection status supports that conclusion.
Incremental work can focus on changed accounts or resources. Periodic broader scans defend against missed events and source inconsistencies.
Extract evidence
Explicit configuration references create initial topology edges. A listener can be connected to its target group, a task definition to its role, a database to its subnet group, and a bucket to its KMS key. Each relationship records the source field, observation time, and scope.
Runtime and declared evidence can enrich the graph later; discovery does not pretend those sources are interchangeable.
Output and Evidence
The pipeline produces more than a resource count. It provides:
- Normalized resource inventory
- Account, Region, and Availability Zone context
- Current configuration versions
- Explicit resource relationships
- Source and observation timestamp
- Freshness by resource and collection scope
- API and permission coverage
- Partial-failure details
- Identity matches to declared infrastructure where supported
- Change records for temporal analysis
Illustrative scenario: Nine production accounts are collected successfully, while one centralized-networking account denies a newly required read action. StackScopes retains the prior known resources for that account with stale-status indicators and reports the exact coverage gap. It does not show the account as empty or infer that cross-account routes disappeared. Engineers can therefore interpret downstream topology with the correct uncertainty.
Evidence remains inspectable. A user can see which service API supplied a configuration, when it was observed, and whether a relationship was explicit or inferred elsewhere.
Why It Matters
Every advanced cloud analysis depends on discovery quality. A failure simulation cannot find a shared endpoint that was never collected. Blast-radius analysis cannot preserve account boundaries if identities were merged. Incident investigation cannot compare state if collection overwrites history or silently drops resources.
Reliable discovery also builds operational trust. Teams can see where visibility is current and where permissions or source behavior limit it. This is more useful than a universal “sync complete” message.
Multi-account normalization makes cross-boundary paths understandable. Central identity, transit networking, shared DNS, artifact repositories, KMS policies, and event buses can be represented without erasing the ownership and security boundaries around them.
The result is a dependable input to the StackScopes graph, not a claim that every AWS service or application dependency is automatically visible.
Operational Guardrails
Discovery is designed for least-privilege, read-only access. StackScopes does not require secret values to model secret references, and it minimizes collection of sensitive payloads.
Account sessions are short-lived, collection activity is auditable, and access removal is part of the connection lifecycle. Tenant-aware authorization applies to stored topology and exports as well as ingestion.
The pipeline applies bounded concurrency and service-aware retries rather than forcing collection through provider limits. Persistent failures are surfaced for review.
Coverage is explicit. Unsupported resource types, disabled telemetry, missing Regions, denied actions, and stale observations remain visible. StackScopes does not claim that read-only access alone guarantees security; encryption, isolation, retention, monitoring, and operational controls are also required.
FAQ
Does discovery require administrator access?
No. StackScopes is designed around an approved read-only role with permissions limited to supported discovery needs. Exact policy scope depends on enabled capabilities and must be reviewed.
How are multiple accounts kept separate?
Account, partition, Region, service, and resource identity are part of the normalized key and authorization context. Cross-account relationships are represented explicitly rather than merging resources.
What happens when an API call fails?
The failure is recorded at its collection scope. Successful results remain usable, prior state can be marked stale, and the gap is exposed instead of being interpreted as resource deletion.
Is discovery real time?
Freshness varies by source, service, event delivery, and reconciliation schedule. StackScopes reports observation time and coverage rather than promising one universal real-time guarantee.
Related StackScopes links
StackScopes keeps operational evidence, timestamps, assumptions, and confidence visible. Production-impacting recovery remains subject to policy and human approval.