CCNA-Level Networking for Cloud and Web Hosting: The Essentials You Actually Use

New course

13 pages

All courses > Information Technology > Web Servers and Cloud Computing ::

ARP and Layer 2 Resolution: Why the Gateway Matters

Capítulo 4

Estimated reading time: 8 minutes

+ Exercise

ARP as the Local “Who Has This IP?” Mechanism

On an Ethernet LAN (including most virtualized cloud networks that emulate Ethernet), devices deliver frames using MAC addresses, not IP addresses. IP is still used to decide where you want to send traffic, but before an IP packet can be placed into an Ethernet frame, the sender must know the destination MAC address for the next hop on the local network.

Address Resolution Protocol (ARP) is the mechanism that maps an IPv4 address to a MAC address on the local Layer 2 segment. ARP only works within the local broadcast domain; it does not cross routers.

ARP Request/Reply Flow (What Actually Happens on the Wire)

When the destination IP is on the same LAN

If a server wants to send an IP packet to another IP on the same Layer 2 network, it needs the MAC address of that destination host.

Step 1: Check ARP cache (local table of IP-to-MAC mappings). If there is a valid entry, the server can immediately encapsulate the IP packet into an Ethernet frame addressed to that MAC.
Step 2: If missing, send an ARP Request as a broadcast Ethernet frame (destination MAC ff:ff:ff:ff:ff:ff). The ARP payload asks: “Who has IP X.X.X.X? Tell Y.Y.Y.Y.”
Step 3: The owner of that IP replies with an ARP Reply (typically unicast back to the requester) containing its MAC address.
Step 4: Cache the mapping and send the original IP packet in an Ethernet frame to the learned MAC.

Key properties that matter operationally

ARP Request is broadcast on the local segment; every host sees it, but only the host with that IP should answer.
ARP Reply is usually unicast to the requester, reducing noise.
ARP is “trusting” by default: many systems will update their cache when they see ARP traffic, which is why bad ARP information (misconfigurations or conflicts) can cause real outages.

ARP Cache Behavior (Why Problems Can Be Intermittent)

To avoid broadcasting for every packet, hosts keep an ARP cache (sometimes called the neighbor table). Each entry has a lifetime; after a timeout it is removed or marked stale and must be refreshed.

Common cache states and behaviors

Reachable/Valid: the mapping is considered good; traffic flows normally.
Stale: the mapping exists but hasn’t been confirmed recently; the host may still use it while attempting to revalidate in the background.
Incomplete: the host has initiated ARP resolution but hasn’t received a reply yet; packets to that IP may queue briefly or fail.

Hosting symptoms caused by ARP cache issues

Intermittent connectivity: a stale or incorrect ARP entry can work “sometimes” (until refreshed) and fail at other times, producing confusing patterns like occasional timeouts to a database, load balancer, or gateway.
Sudden break after a change: moving an IP to a new VM/NIC (or replacing a firewall) can leave peers with old ARP entries pointing to the previous MAC, causing traffic to go to the wrong place until caches expire or are refreshed.
Duplicate IP address: two devices answering ARP for the same IP leads to ARP cache “flapping” (the mapping alternates between two MACs). This can look like random packet loss, random SSH drops, or “it works from some hosts but not others.”

Practical step-by-step: checking and clearing ARP/neighbor entries

Use these when you suspect wrong MAC resolution or duplicate IP behavior.

Continue in our app.

You can listen to the audiobook with the screen off, receive a free certificate for this course, and also have access to 5,000 other free online courses.

Or continue reading below...

Download the app

Linux (iproute2): view neighbor table:
```
ip neigh show
```
show a specific IP:
```
ip neigh show 192.0.2.1
```
delete an entry (forces re-ARP):
```
sudo ip neigh del 192.0.2.1 dev eth0
```
Linux (legacy):
```
arp -n
```
Windows:
```
arp -a
```
delete one entry:
```
arp -d 192.0.2.1
```
delete all:
```
arp -d *
```
macOS: view:
```
arp -a
```
delete one:
```
sudo arp -d 192.0.2.1
```

Clearing an entry is not a fix by itself; it is a diagnostic move to force fresh resolution and confirm whether the mapping was wrong or stale.

When the Destination Is Remote: ARP for the Default Gateway MAC

ARP does not resolve the MAC of a remote host on the internet. Instead, the host resolves the MAC of its next hop on the local network—almost always the default gateway.

The decision process is:

If the destination IP is local, ARP for the destination host’s MAC.
If the destination IP is remote, send the packet to the default gateway’s MAC (the router will forward it).

Why the gateway matters for hosting

Wrong default gateway: the server may be able to talk to local neighbors but cannot reach anything off-subnet (DNS resolvers, package repos, external APIs). This is one of the most common “server can’t reach the internet” causes.
Gateway IP correct but ARP points to the wrong MAC: can happen after gateway replacement, VRRP/HSRP failover, or duplicate IPs. The server sends frames to a MAC that is no longer the router, so traffic disappears.
Duplicate gateway IP: two devices claim the gateway IP; ARP flaps; outbound connectivity becomes intermittent for many servers at once.

Packet-Flow Narrative: Server Querying an External DNS Resolver

Scenario: a web server needs to resolve a hostname and sends a DNS query to an external resolver (for example, 8.8.8.8). The server is on a LAN with default gateway 192.0.2.1. The server’s IP is 192.0.2.10.

Step-by-step flow (what must happen before routing can occur)

Step 1: Application triggers DNS. The server prepares a UDP packet (source ephemeral port, destination port 53) with destination IP 8.8.8.8.
Step 2: Host decides next hop. Because 8.8.8.8 is not on the local network, the host chooses the default gateway 192.0.2.1 as the next hop.
Step 3: ARP for the gateway (if needed). The server checks its ARP cache for 192.0.2.1. If missing or invalid, it broadcasts an ARP Request: “Who has 192.0.2.1? Tell 192.0.2.10.”
Step 4: Gateway replies with its MAC. The router (or virtual gateway) responds with an ARP Reply containing its MAC address (for example, 00:11:22:33:44:55). The server caches 192.0.2.1 → 00:11:22:33:44:55.
Step 5: Server sends the DNS packet to the gateway’s MAC. The Ethernet frame destination MAC is the gateway’s MAC, but the IP destination remains 8.8.8.8. This is the key point: Layer 2 destination is the gateway, Layer 3 destination is the remote resolver.
Step 6: Router forwards. The gateway receives the frame, decapsulates to the IP packet, routes it onward, and the response eventually returns. On the way back, the gateway uses ARP on the server’s LAN (if needed) to deliver the response to 192.0.2.10’s MAC.

What failure looks like at each step

Failure at Step 3 (no ARP reply): the server cannot learn the gateway MAC. You’ll see ARP requests repeating and no replies. Result: no outbound connectivity; DNS times out; pings to remote IPs fail.
Failure at Step 2 (wrong gateway configured): the server ARPs for the wrong gateway IP (one that doesn’t exist on the LAN). Same symptom: repeated ARP requests, no replies.
Failure due to duplicate IP: ARP replies come from different MACs over time. Connectivity becomes intermittent; some packets go to the wrong device.

Diagnosing Real Hosting Incidents with ARP and Captures

Checklist: “Server can’t reach anything” (focus on gateway + ARP)

Verify the configured default gateway (don’t assume). On Linux:
```
ip route show
```
Look for a default route (0.0.0.0/0) via the expected gateway.
Ping the gateway IP. If pinging the gateway fails, you likely have a local L2/L3 issue (VLAN/port group, security group, wrong IP, wrong mask, or ARP failure).
Check ARP/neighbor entry for the gateway. On Linux:
```
ip neigh show 192.0.2.1
```
If it’s missing/incomplete/failed, ARP resolution is the blocker.
Force re-resolution by deleting the neighbor entry and retrying ping/DNS:
```
sudo ip neigh del 192.0.2.1 dev eth0
```

Detecting duplicate IPs and ARP flapping

Duplicate IPs are common during migrations, templated VM deployments, or manual “quick fixes.” ARP makes duplicates visible because multiple devices may answer for the same IP.

Symptom: ARP entry for an IP changes MAC repeatedly.
On Linux: watch for changes while you generate traffic:
```
watch -n 1 'ip neigh show | grep 192.0.2.1'
```
Packet capture proof: capture ARP and look for multiple replies claiming the same IP.

Packet capture: confirming ARP behavior

Captures are the fastest way to confirm whether ARP requests are leaving and whether replies are coming back.

Linux tcpdump: capture ARP on an interface:
```
sudo tcpdump -ni eth0 arp
```

Capture ARP plus DNS to see the dependency:

sudo tcpdump -ni eth0 'arp or (udp port 53)'

Interpretation tips:

If you see ARP Requests for the gateway IP with no ARP Replies, the server cannot learn the next-hop MAC (gateway down, wrong VLAN/segment, filtering, wrong gateway IP, or L2 isolation).
If you see ARP Replies for the same IP from different source MAC addresses, suspect a duplicate IP or a misconfigured HA gateway setup.
If ARP looks healthy but DNS still fails, you’ve proven Layer 2 resolution to the gateway works; the issue is likely beyond ARP (routing, firewall, NAT, resolver reachability).

Now answer the exercise about the content:

A server needs to send a DNS query to an external IP (off-subnet). What MAC address will it use as the Ethernet destination after ARP completes?

You are right! Congratulations, now go to the next page

You missed! Try again.

For off-subnet traffic, the host ARPs for the next hop on the local LAN (usually the default gateway) and sends the Ethernet frame to the gateway’s MAC. The IP destination remains the remote resolver.