Why East-West Traffic Needs a Service Mesh
What “east-west” means in Kubernetes: East-west traffic is service-to-service communication inside the cluster (for example, checkout calling payments). This differs from north-south traffic, which is user-to-cluster traffic typically handled by an Ingress or Gateway. In modern microservice architectures, most requests are east-west, and they often traverse multiple hops, cross namespaces, and involve different teams.
Why Kubernetes primitives are not enough by themselves: Kubernetes Services provide stable virtual IPs and DNS names, and kube-proxy (or eBPF-based data planes) load-balance traffic. However, Kubernetes does not natively provide consistent, per-request identity, uniform traffic policy enforcement, detailed L7 telemetry, or safe progressive delivery controls across all services. You can implement some of these in each application, but that creates duplicated logic, inconsistent behavior, and uneven security posture.
What a service mesh adds: A service mesh is an infrastructure layer for service-to-service communication that provides traffic management, security, and observability features in a consistent way. The key idea is to move cross-cutting concerns out of application code and into the platform, so policies can be applied uniformly and changed centrally.
Core Building Blocks: Data Plane and Control Plane
Data plane: the proxies that handle requests: Most meshes use a per-pod proxy (sidecar) or a node-level proxy (ambient/sidecarless approaches). The proxy intercepts inbound and outbound traffic and can apply routing rules, enforce security, and emit telemetry. In a sidecar model, each workload pod has an additional container (the proxy) in the same network namespace, so it can transparently capture traffic.
Control plane: the brains that configure the proxies: The control plane distributes configuration to proxies (for example, routing rules, mTLS settings, authorization policies) and often provides certificate management. You typically express desired behavior via Kubernetes custom resources (CRDs) or mesh-specific APIs, and the control plane translates those into proxy configuration.
Continue in our app.
You can listen to the audiobook with the screen off, receive a free certificate for this course, and also have access to 5,000 other free online courses.
Or continue reading below...
Download the app
How traffic interception works: In sidecar-based meshes, iptables rules (or eBPF) redirect traffic from the application container to the proxy. Outbound calls from the app go to the proxy first; inbound calls hit the proxy, which then forwards to the app. This is why a mesh can apply policies without changing application code.
Identity, Trust, and mTLS for East-West Security
Service identity as the foundation: For secure east-west traffic, the mesh needs a stable identity for each workload. In Kubernetes, the most common identity anchor is the ServiceAccount, often represented as a SPIFFE ID-like identity (for example, spiffe://cluster.local/ns/shop/sa/checkout). Identity enables authentication (who is calling) and authorization (who is allowed).
mTLS: encryption and authentication between services: Mutual TLS means both client and server present certificates, enabling encryption in transit and strong peer authentication. In a mesh, proxies typically handle mTLS automatically, so applications can keep using plain HTTP inside the pod while traffic on the wire is encrypted and authenticated.
Certificate issuance and rotation: The control plane commonly acts as (or integrates with) a certificate authority. It issues short-lived certificates to proxies and rotates them automatically. Short lifetimes reduce blast radius if a credential is compromised and remove manual certificate management from teams.
Practical note: permissive vs strict modes: Many meshes support a gradual rollout of mTLS. A permissive mode allows both plaintext and mTLS, which helps during migration. Strict mode enforces mTLS only. For east-west control, strict mode is the goal, but permissive can be a stepping stone when not all workloads are meshed yet.
Traffic Management for East-West Control (Without Repeating Reliability Patterns)
Layer 4 vs Layer 7 control: Kubernetes Services load-balance at L4 (TCP/UDP) and do not understand HTTP routes, headers, or methods. A mesh proxy can apply L7 policies: route based on URL path, HTTP method, headers, cookies, or even gRPC service/method. This is crucial for internal APIs where you want fine-grained control over which version handles which requests.
Service discovery and endpoint selection: The mesh usually relies on Kubernetes service discovery (Endpoints/EndpointSlices) but can add richer metadata (workload labels, service accounts, locality). This enables policies like “send traffic only to endpoints with label version=v2” or “prefer same-zone endpoints for lower latency.”
Progressive delivery primitives: For east-west traffic, you often need to shift traffic between versions of a service without changing clients. Mesh routing rules can implement weighted splits (for example, 90% to v1, 10% to v2) and header-based routing (for example, route internal testers to v2). These controls are especially useful for internal dependencies where an Ingress is not involved.
Policy as configuration, not code: A key mesh value is that traffic rules live in declarative configuration. Teams can change routing behavior without redeploying applications, and platform teams can enforce guardrails (for example, only allow certain namespaces to create cross-namespace routes).
Observability: Understanding East-West Flows
Uniform metrics and logs from proxies: Because every request passes through a proxy, the mesh can emit consistent telemetry: request counts, latency distributions, response codes, bytes sent/received, and peer identities. This is especially helpful when services are written in different languages or frameworks.
Distributed tracing context propagation: Many meshes can generate or propagate trace headers (for example, W3C Trace Context) so you can follow a request across multiple internal hops. Even when applications do not instrument tracing consistently, the mesh can provide baseline spans at the proxy layer.
Service graph and dependency mapping: With proxy telemetry, you can build a near-real-time map of which services call which others, including error rates and latency. This helps detect unexpected dependencies (for example, a new service calling a database directly) and supports capacity planning and incident response.
Authorization for East-West: From “Can Reach” to “Is Allowed”
Network reachability is not authorization: In Kubernetes, if a pod can reach another pod’s IP and the Service allows it, the request can be made. NetworkPolicies can restrict traffic, but they are L3/L4 and typically do not understand service identity or HTTP semantics. A mesh can enforce authorization based on authenticated identity and request attributes.
Common authorization dimensions: Mesh authorization policies often include: source identity (service account), source namespace, destination service/workload, request path/method, and sometimes JWT claims for end-user context. This enables rules like “only checkout can call payments on POST /charge” while allowing other services to call GET /health.
Defense in depth with NetworkPolicy: A mesh does not replace NetworkPolicy; it complements it. NetworkPolicy can reduce the blast radius at the network layer, while mesh authorization provides identity-aware, L7 controls. Together they provide stronger east-west security than either alone.
Step-by-Step: Installing a Mesh and Enabling Sidecar Injection
Goal of this walkthrough: You will install a service mesh, enable automatic sidecar injection for a namespace, deploy two simple services, and verify that traffic flows through the proxies. The steps use a commonly used mesh CLI workflow; adapt the exact commands to your chosen mesh distribution.
Step 1: Prepare a namespace for meshed workloads
Create a namespace dedicated to the demo and label it for automatic injection. Many meshes watch for a label such as istio-injection=enabled or linkerd.io/inject=enabled.
kubectl create namespace mesh-demo
kubectl label namespace mesh-demo istio-injection=enabledStep 2: Install the mesh control plane
Install the control plane into its own namespace (often istio-system or linkerd). Use a minimal profile first to reduce moving parts, then add components like ingress gateways or telemetry add-ons as needed.
# Example pattern (exact command varies by mesh)
meshctl install --profile minimal | kubectl apply -f -
kubectl get pods -n istio-systemStep 3: Deploy two services that talk to each other
Deploy a simple “caller” service and a “server” service in the injected namespace. The key is that pods should start with an extra proxy container. You can use any HTTP echo server and a small client that periodically calls it.
kubectl -n mesh-demo apply -f server.yaml
kubectl -n mesh-demo apply -f caller.yaml
kubectl -n mesh-demo get podsWhen injection is working, each pod typically shows 2/2 containers ready (app + proxy). If you see 1/1, injection did not occur, and the mesh will not control that workload’s traffic.
Step 4: Verify that traffic is intercepted by the proxy
Exec into the caller pod and make a request to the server service. Then inspect proxy stats or logs. Many meshes expose an admin port on the proxy container for metrics and configuration inspection.
CALLER_POD=$(kubectl -n mesh-demo get pod -l app=caller -o jsonpath='{.items[0].metadata.name}')
kubectl -n mesh-demo exec -it $CALLER_POD -c caller -- curl -s http://server.mesh-demo.svc.cluster.local:8080/Next, query the proxy’s metrics endpoint (example shown; port and path vary). You should see counters increment for outbound requests.
kubectl -n mesh-demo exec -it $CALLER_POD -c istio-proxy -- curl -s http://localhost:15000/stats | headStep-by-Step: Enabling mTLS for East-West Traffic
Goal of this walkthrough: Turn on mesh mTLS so service-to-service traffic is encrypted and authenticated. The exact resources differ by mesh, but the workflow is similar: start permissive (optional), validate, then enforce strict.
Step 1: Check current mTLS state
Some meshes provide a command to inspect whether workloads are using mTLS. Alternatively, you can inspect proxy connection details or use telemetry labels that indicate tls mode.
# Example inspection command (varies)
meshctl authn tls-check -n mesh-demoStep 2: Apply a namespace-level mTLS policy
Apply a policy that enables mTLS for workloads in the namespace. If you are migrating, start with permissive mode to avoid breaking calls from non-meshed workloads.
# Example resource (Istio-style)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: mesh-demo
spec:
mtls:
mode: PERMISSIVEAfter applying, verify that calls still succeed. Then switch to strict mode once you confirm all callers are meshed.
# Change mode to STRICT when ready
spec:
mtls:
mode: STRICTStep 3: Confirm encryption and identity
To confirm mTLS, inspect proxy connection metadata. Many meshes can show whether a connection is using TLS and what identity the peer presented. You should see source and destination identities tied to Kubernetes service accounts.
# Example (varies by mesh)
meshctl proxy connections -n mesh-demo deploy/callerStep-by-Step: Fine-Grained East-West Authorization
Goal of this walkthrough: Allow only a specific caller identity to access a sensitive endpoint on the server. This demonstrates the shift from “any pod can call” to “only the right workload identity can call,” enforced consistently at the proxy.
Step 1: Ensure distinct service accounts
Create separate ServiceAccounts for the caller and server deployments. This makes identity-based policies meaningful.
kubectl -n mesh-demo create serviceaccount caller-sa
kubectl -n mesh-demo create serviceaccount server-saPatch or update the deployments to use these service accounts.
kubectl -n mesh-demo patch deploy caller -p '{"spec":{"template":{"spec":{"serviceAccountName":"caller-sa"}}}}'
kubectl -n mesh-demo patch deploy server -p '{"spec":{"template":{"spec":{"serviceAccountName":"server-sa"}}}}'Step 2: Apply an authorization policy on the server
Create a policy that allows requests only from caller-sa in the same namespace, and only to a specific path such as /api. The policy language depends on the mesh; below is an Istio-style example.
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: server-allow-caller
namespace: mesh-demo
spec:
selector:
matchLabels:
app: server
action: ALLOW
rules:
- from:
- source:
principals:
- cluster.local/ns/mesh-demo/sa/caller-sa
to:
- operation:
methods: ["GET", "POST"]
paths: ["/api/*"]After applying, test access from the caller to /api (should succeed) and to other paths (should be denied if you also add a default-deny policy). Many meshes require an explicit deny-by-default posture, often implemented by adding an ALLOW policy for known traffic and relying on an implicit deny, or by adding a separate DENY policy depending on the mesh.
Step 3: Validate with an unauthorized client
Deploy a second client with a different service account and attempt the same request. It should fail with a 403 (or similar) generated by the proxy, not by application code.
kubectl -n mesh-demo create serviceaccount intruder-sa
# deploy intruder client using intruder-sa, then:
INTRUDER_POD=$(kubectl -n mesh-demo get pod -l app=intruder -o jsonpath='{.items[0].metadata.name}')
kubectl -n mesh-demo exec -it $INTRUDER_POD -c intruder -- curl -i http://server:8080/api/testTraffic Shaping for Internal APIs: Canary and Header-Based Routing
Why internal canaries matter: Many failures happen in east-west dependencies before users ever hit an Ingress. If you can shift only a small portion of internal calls to a new version of a dependency, you can validate behavior under real traffic while limiting impact.
Weighted routing between versions: A common pattern is to deploy server-v1 and server-v2 as separate Deployments behind the same Kubernetes Service, and then use mesh routing to split traffic by weight. This keeps clients unchanged while you progressively increase the share to v2.
# Example (Istio-style) destination rule + virtual service pattern
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: server-subsets
namespace: mesh-demo
spec:
host: server.mesh-demo.svc.cluster.local
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: server-route
namespace: mesh-demo
spec:
hosts:
- server.mesh-demo.svc.cluster.local
http:
- route:
- destination:
host: server.mesh-demo.svc.cluster.local
subset: v1
weight: 90
- destination:
host: server.mesh-demo.svc.cluster.local
subset: v2
weight: 10Header-based routing for targeted testing: You can route requests with a specific header (for example, x-canary: true) to v2 while keeping everyone else on v1. This is useful for internal QA, synthetic tests, or specific upstream services.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: server-header-route
namespace: mesh-demo
spec:
hosts:
- server.mesh-demo.svc.cluster.local
http:
- match:
- headers:
x-canary:
exact: "true"
route:
- destination:
host: server.mesh-demo.svc.cluster.local
subset: v2
- route:
- destination:
host: server.mesh-demo.svc.cluster.local
subset: v1Operational Considerations: Performance, Scope, and Ownership
Resource overhead and latency: Proxies consume CPU and memory and add a small amount of latency per hop. In high-throughput clusters, this overhead is real. Plan capacity accordingly, set resource requests/limits for proxies, and measure p50/p95 latency changes when enabling the mesh.
Scoping the mesh: You do not need to mesh everything on day one. A practical approach is to start with a single namespace or a set of critical services, then expand. Use permissive mTLS and careful policy rollout to avoid breaking non-meshed callers.
Multi-team governance: East-west traffic policies affect multiple teams. Establish ownership boundaries: platform team owns global defaults (mTLS mode, baseline telemetry), while service teams own service-specific routing and authorization within guardrails. Prefer GitOps workflows for mesh configuration to ensure changes are reviewed and auditable.
Debugging workflow: When a call fails in a mesh, you typically check: (1) is the workload injected, (2) is mTLS mode compatible between peers, (3) are authorization policies allowing the identity, (4) is routing sending traffic to the expected subset, and (5) what do proxy access logs show. Having a standard runbook for these checks reduces mean time to resolution.
