Switching vs Routing in Cloud Networks: Where Each Applies

1) What a switch does (Layer 2)

MAC learning: how the switch builds its “who is where” table

A Layer 2 switch forwards Ethernet frames based on destination MAC address. To do that efficiently, it learns which MAC addresses live behind which switch ports by watching source MAC addresses on incoming frames.

• Learning: When a frame arrives, the switch records source MAC → ingress port in its MAC (CAM) table.
• Forwarding: If the destination MAC is known, the switch sends the frame only out the associated port.
• Flooding (unknown unicast): If the destination MAC is unknown, the switch floods the frame out all ports in the same VLAN (except the port it came in on).
• Aging: Learned entries expire after a timer so the table stays accurate when devices move or reboot.

Forwarding behavior: unicast, broadcast, and ARP

Within a VLAN, devices use ARP to map an IP address to a MAC address. ARP requests are broadcasts (destination MAC ff:ff:ff:ff:ff:ff), so the switch floods them across the VLAN. ARP replies are unicast and are forwarded based on the learned MAC table.

• Unicast frame: One sender to one receiver (most traffic after ARP is resolved).
• Broadcast frame: One sender to everyone in the VLAN (ARP request, DHCP discover, some discovery protocols).
• Multicast frame: One sender to a subscribed group (depends on features like IGMP snooping; often treated like broadcast if not optimized).

Broadcast domains: the key limit of Layer 2

A VLAN is a broadcast domain: broadcasts are forwarded to all ports in that VLAN. Switches do not stop broadcasts; they contain them to the VLAN. If you need to keep broadcast traffic from reaching another group of hosts, you separate them into different VLANs/subnets and use routing between them.

Scenario A: two servers in the same subnet talk directly (Layer 2 + ARP)

Assume Server A is 10.10.1.10/24 and Server B is 10.10.1.20/24, and both are in the same VLAN/subnet.

• Step 1 (decision): Server A checks its subnet mask and sees 10.10.1.20 is local (same /24).
• Step 2 (ARP request): Server A broadcasts “Who has 10.10.1.20?” to ff:ff:ff:ff:ff:ff.
• Step 3 (switch floods): The switch floods the ARP request to all ports in that VLAN.
• Step 4 (ARP reply): Server B replies with its MAC address (unicast to Server A’s MAC).
• Step 5 (switch forwards): The switch forwards the unicast ARP reply using its MAC table (and learns B’s MAC on B’s port).
• Step 6 (data transfer): Server A now sends IP packets encapsulated in Ethernet frames to Server B’s MAC; the switch forwards those frames at Layer 2.

Same-subnet flow (simplified) A (10.10.1.10) --ARP--> broadcast domain --ARP reply--> A A --unicast frames to B's MAC--> Switch forwards to B

2) What a router does (Layer 3)

IP forwarding: moving packets between networks

A Layer 3 router forwards packets based on destination IP address. It consults a routing table to decide the next hop (or outgoing interface) toward the destination network.

Continue in our app.

You can listen to the audiobook with the screen off, receive a free certificate for this course, and also have access to 5,000 other free online courses.

Or continue reading below...

Download the app

• Routing table lookup: The router matches the destination IP to the most specific route (longest prefix match).
• Next hop: The route points to a next-hop IP (another router) or a directly connected network.
• Rewrite at Layer 2: The router changes the Ethernet header for each hop (new source/destination MAC), while the IP packet continues toward the destination (TTL decreases each hop).

Separating broadcast domains

Routers do not forward Layer 2 broadcasts between subnets by default. This is one of the main reasons networks are segmented into multiple subnets: each subnet is its own broadcast domain, and routing controls how traffic moves between them.

Default gateways: where hosts send “non-local” traffic

When a host wants to reach an IP outside its local subnet, it sends the packet to its default gateway (a router interface IP in the same subnet). The host still uses ARP, but it ARPs for the gateway’s IP to learn the gateway’s MAC, then sends frames to the gateway’s MAC.

Scenario B: two servers in different subnets communicate via a gateway (Layer 3)

Assume Server A is 10.10.1.10/24 with default gateway 10.10.1.1, and Server C is 10.10.2.20/24 with default gateway 10.10.2.1. The router has interfaces (or SVIs) in both subnets.

• Step 1 (decision): Server A sees 10.10.2.20 is not in 10.10.1.0/24.
• Step 2 (ARP for gateway): If needed, Server A ARPs for 10.10.1.1 and learns the gateway’s MAC.
• Step 3 (send to gateway): Server A sends an Ethernet frame to the gateway’s MAC, containing an IP packet destined to 10.10.2.20.
• Step 4 (router routes): The router receives the frame, decapsulates to IP, looks up 10.10.2.20, and selects the 10.10.2.0/24 connected route.
• Step 5 (router ARPs on destination subnet): If needed, the router ARPs for 10.10.2.20 to learn Server C’s MAC.
• Step 6 (forward to Server C): The router sends a new Ethernet frame out the 10.10.2.0/24 interface to Server C’s MAC, with the original IP destination intact.
• Step 7 (return traffic): Server C replies to 10.10.1.10 and uses its own default gateway (10.10.2.1) to route back.

Different-subnet flow (simplified) A (10.10.1.10) --frame to GW MAC--> Router (10.10.1.1/10.10.2.1) Router --new frame to C MAC--> C (10.10.2.20)

3) How cloud networks emulate or replace these functions

Layer 2 in the cloud: virtual switching without “your” VLANs

In many public clouds, you don’t manage traditional VLAN trunks, spanning tree, or physical switch ports. Instead, the provider implements a virtualized Layer 2-like forwarding function inside the hypervisor and fabric. Your instance’s virtual NIC connects to a virtual switch/port group conceptually, but you typically interact with it through subnet attachment and security policy rather than VLAN configuration.

• What you still rely on: MAC learning/forwarding behavior exists under the hood so same-subnet instances can exchange frames and ARP can work.
• What is often restricted: You usually cannot run arbitrary Layer 2 adjacency features across subnets, and some clouds limit or abstract away L2 constructs (no customer-controlled VLAN IDs, limited promiscuous mode, limited L2 broadcast scope).
• Practical implication: Treat a cloud subnet as the boundary where “local” traffic can happen directly; assume anything beyond that boundary requires routing and policy.

Layer 3 in the cloud: virtual routers and routing tables as the control surface

Cloud platforms provide a virtual routing function that connects subnets and decides where packets go next. Instead of configuring router interfaces and dynamic routing on a box, you typically manage route tables and attachments (subnet association, peering, gateways, transit hubs).

• VPC/VNet route tables: Define which destination prefixes are reachable and what the next hop is (local, internet gateway, NAT gateway, VPN/ExpressRoute, peering, transit gateway/hub, virtual appliance).
• System routes vs custom routes: There are default “local” routes for intra-VPC/VNet communication and optional user-defined routes that can override or add paths.
• Next-hop targets are cloud constructs: Instead of “interface Gi0/1,” next hop might be “NAT gateway,” “virtual appliance,” or “peering connection.”

Mapping classic concepts to cloud constructs

• Switch port ↔ virtual NIC attachment: An instance NIC attaches to a subnet; the fabric provides L2-like delivery within that subnet.
• VLAN/broadcast domain ↔ subnet boundary: Broadcast and ARP scope is effectively limited to the subnet.
• Router interface (gateway IP) ↔ subnet’s default gateway: Often an implicit gateway address in the subnet that instances use automatically.
• Routing table on a router ↔ VPC/VNet route table: The cloud routing plane decides next hop based on destination prefix.
• ACL/firewall on router/switch ↔ security groups/NSGs + network ACLs: Policy can block traffic even when routes exist.

Scenario walk-through: same subnet in the cloud (why it “just works”)

Two instances in the same cloud subnet usually communicate without you adding routes because the platform includes an automatic “local” route for the subnet/VPC/VNet and provides the L2-like delivery needed for ARP and unicast frames.

• Step 1: Instance A determines Instance B is in the same subnet (based on its IP/mask).
• Step 2: ARP resolves B’s MAC (or an equivalent internal mechanism if the platform virtualizes ARP behavior).
• Step 3: Traffic flows directly within the subnet, subject to security policy (security groups/NSGs, host firewall).

Common failure even in same subnet: Security policy blocks it. For example, B’s inbound rules allow TCP/22 only from a bastion subnet, not from A, so ping/SSH fails even though both instances are healthy and “local.”

Scenario walk-through: crossing subnets in the cloud (routes + policy must align)

Two instances in different subnets can communicate only if (a) the cloud routing plane has routes for both directions and (b) policy allows the traffic.

• Step 1 (host decision): Instance A sees the destination is off-subnet and sends traffic to its default gateway (implicit cloud router).
• Step 2 (route table lookup): The subnet’s associated route table is consulted for the destination prefix.
• Step 3 (next hop): If the destination is another subnet in the same VPC/VNet, the “local” route typically covers it; if it’s in a peered VPC/VNet or on-prem, a specific route must exist.
• Step 4 (return path): The destination subnet must also have a route back (often automatic for local, but not for custom paths like appliances or asymmetric designs).
• Step 5 (policy enforcement): Security groups/NSGs and NACLs must allow the relevant protocol/ports in both directions (stateful vs stateless behavior matters).

Cloud inter-subnet checklist 1) Source subnet route table has destination prefix → correct next hop 2) Destination subnet route table has return prefix → correct next hop 3) Security policy allows it (SG/NSG, NACL, host firewall) 4) No overlapping CIDRs across peering/transit paths 5) If using NAT/appliances, confirm symmetric routing expectations

Why inter-subnet traffic may fail even when instances are healthy

Failure mode 1: missing or incorrect routes (especially with peering, VPN, transit)

If Instance A tries to reach 10.20.0.0/16 but the route table only has a default route to the internet gateway (or no route at all for 10.20.0.0/16), packets won’t reach the destination network. In cloud terms, “no route” often means the packet is dropped by the routing plane.

• Practical steps: Verify the source subnet’s route table contains the destination CIDR pointing to the correct target (peering, transit hub, VPN, etc.). Verify the destination side has a return route (or that the path is inherently bidirectional).

Failure mode 2: policy blocks traffic (stateful vs stateless)

Even with perfect routes, security controls can block traffic:

• Security groups/NSGs (often stateful): You might allow inbound TCP/443 to a web server but forget to allow inbound from the specific source subnet or security group.
• Network ACLs (often stateless): You might allow inbound TCP/443 but forget ephemeral return ports, causing connections to fail in ways that look like routing problems.
• Host firewall: OS-level rules can block ICMP or application ports, making network tests misleading.

Failure mode 3: asymmetric routing introduced by virtual appliances

When you insert a firewall/NVA (network virtual appliance) and use custom routes to steer traffic through it, return traffic might take a different path (for example, destination subnet still uses the default “local” route back). Many appliances and security policies expect symmetric flows; asymmetry can cause drops.

• Practical steps: Ensure both directions are steered through the same appliance when required, and confirm the appliance has routes back to all involved prefixes.

Failure mode 4: overlapping CIDRs across connected networks

Peering and transit designs often break when two connected networks use the same IP range (for example, two VPCs both using 10.0.0.0/16). Even if instances are healthy, routing decisions become ambiguous or are rejected by the platform.

• Practical steps: Check for CIDR overlap across VPC/VNet, peering connections, and on-prem networks; renumber or use NAT where appropriate.

Quick troubleshooting flow: same-subnet vs cross-subnet

• If same subnet: Focus on security policy and host firewall first; then verify IP/mask and that both are truly in the same subnet.
• If different subnets: Verify route tables (both sides), then security policy (both sides), then appliance symmetry/overlap issues.