Skip to content

TCP Troubleshooting: TIME_WAIT / RST / SYN Half-Open / Retransmission

TCP failures are almost never about TCP itself — they're about something on one side of the socket doing something unusual, and TCP faithfully reflecting it. The failure you observe depends entirely on which side is doing the unusual thing. That's why the first step is always to figure out which side is doing what, not to tune anything.

Before You Touch Anything: 3 Commands That Cut 80% of Cases

Every TCP investigation starts the same way. These three commands take under a minute and eliminate most of the noise.

Command 1: Overall TCP Summary

ss -s

What you're looking for: - TCP: 500 (established 400, closing 5, timewait 80, orphaned 0) — healthy baseline for a small server. - timewait in the low thousands on a reverse proxy is normal. In the hundreds of thousands is not. - closing > 50 suggests something is closing sockets improperly. - timewait_orphaned > 0 means sockets were closed without a peer.

Command 2: State Distribution

ss -tn | awk '{print $1}' | sort | uniq -c | sort -rn

What each state means when abnormal:

State Normal on a small server What abnormal means
ESTAB Proportional to active users Leak, or a real load spike
TIME-WAIT < 500 unless it's a proxy Churn from a reverse proxy or client
CLOSE-WAIT < 50 Local app not closing sockets — always a bug
SYN-SENT < 100 Outbound connect hangs — network path issue
SYN-RECV < 50 Accept queue pressure or scan
FIN-WAIT-* Small numbers Normal close in progress

The rule of thumb: CLOSE-WAIT > 100 is a bug on the local side, SYN-SENT > 100 is a bug on the network path, TIME-WAIT > 10k is churn.

Command 3: Kernel TCP Counters

netstat -s | grep -iE 'retrans|reset|listen|overflow'

What to watch: - TCPRetransFail growing → packets dropped on the path - ListenOverflows growing → accept queue full, real SYN drops - ListenDrops growing → SYN cookies in use or backlog exceeded

If all three are flat, TCP is not the problem and you should look elsewhere.

Scenario 1: TIME-WAIT Flood (35% of production cases)

Symptom match: ss -s shows thousands of timewait. ss -tn state time-wait | wc -l returns 5000+. CPU spikes correlate with connection close rate.

Root cause: A server-side application closes connections first (typically a web server or reverse proxy after an HTTP response). Every close enters TIME-WAIT for 60 seconds (2× MSL). Under high request rates this exhausts the ephemeral port range.

Diagnosis:

# Count TIME-WAIT sockets
ss -tn state time-wait | wc -l

# Who's generating them? Group by peer
ss -tn state time-wait | awk '{print $5}' | sort | uniq -c | sort -rn | head

# Are we running out of ephemeral ports?
ss -s | grep -i 'timewait'
cat /proc/sys/net/ipv4/ip_local_port_range

Fix — server-side only:

# /etc/sysctl.d/90-tcp.conf
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.ip_local_port_range = 1024 65535

sysctl --system

Fix — reduce churn at the source (better than tuning):

If the TIME-WAIT comes from Nginx acting as a client to upstreams, enable upstream keepalive (see Nginx 502/504 troubleshooting for the full config):

upstream backend {
    server 127.0.0.1:8080;
    keepalive 32;
}
location / {
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_pass http://backend;
}

Keepalive is the correct fix. tcp_tw_reuse is the emergency valve.

Verification: TIME-WAIT count stays below 500 for 10 minutes under load, ss -s shows stable timewait number.

Pitfall: Never enable tcp_tw_reuse on a client-side connection path (i.e., a workload that opens many outbound connections). It was designed for the server side and can cause correctness bugs under NAT on the client side. If in doubt, use keepalive instead.

Scenario 2: RST Storm (25% of production cases)

Symptom match: Connections randomly dropping with ECONNRESET in app logs. tcpdump shows RST packets. Some workloads (e.g., long HTTP/1.1 connections to a load balancer) fail intermittently.

Root cause: One side of the TCP connection is closed while the other side is still sending data. The closed side responds with RST. In production this is almost always one of:

  1. Load balancer idle timeout shorter than app idle time — most common. The LB closed, the app sends on the next tick, and gets an RST.
  2. App socket idle timeout shorter than peer's — symmetric to above.
  3. Application process crash — kernel sends RST on behalf of the dead process.
  4. Firewall with short idle timeout — middlebox is silently pruning idle TCP sessions.

Diagnosis:

# Live RST capture
tcpdump -nn -i any 'tcp[tcpflags] & tcp-rst != 0'

# Count RST rate over time
watch -n1 'netstat -s | grep -i reset'

# For a specific connection pair
tcpdump -nn -i any host 10.0.0.5 and port 8080 and 'tcp[tcpflags] & tcp-rst != 0'

Fix — socket-level keepalive on both ends:

export TCP_KEEPIDLE=60    # Start probing after 60s idle
export TCP_KEEPINTVL=10   # Probe every 10s
export TCP_KEEPCNT=3      # Give up after 3 failed probes (60+30 = 90s total)

Or via code — Python:

import socket
s = socket.create_connection(('backend:8080',))
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 60)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 10)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 3)

Fix — align LB timeout with app timeout:

AWS ALB default idle timeout is 60s. If your upstream app sends data every 90s, you'll get RST storms. Either lower the LB idle timeout below your app's send interval, or send TCP keepalives.

Verification: RST rate from netstat -s becomes flat over 5 minutes under steady load.

Scenario 3: SYN Half-Open (20% of production cases)

Symptom match: ss -tn state syn-recv | wc -l returns thousands. netstat -s | grep -i listen shows ListenOverflows growing. App can accept connections but new client connections time out.

Root cause: The server's SYN receive queue is full. Incoming SYNs are dropped, and the client retransmits. Two sub-cases:

  1. SYN flood attack — many SYNs from spoofed sources. Kernel's SYN cookies (net.ipv4.tcp_syncookies, on by default) usually absorb this.
  2. Legitimate burst — real clients sending real SYNs, but the accept queue is too small.

Diagnosis:

# Are we seeing listen overflows?
netstat -s | grep -i listen

# SYN-RECV count
ss -tn state syn-recv | wc -l

# Backlog vs current accept queue
ss -lnt | awk '$4 ~ /LISTEN/' | awk '{print $2, $3}'
# Recv-Q = current accept queue length, Send-Q = backlog limit

Fix — raise backlog and queues:

# /etc/sysctl.d/90-tcp.conf
net.core.somaxconn = 8192
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_tw_buckets = 2000000

The app must also use a large listen() backlog:

int server_fd = socket(AF_INET, SOCK_STREAM, 0);
bind(server_fd, &addr, sizeof(addr));
listen(server_fd, 8192);  // Match somaxconn, not the default 128

Note: listen() backlog is capped by somaxconn. Raising one without the other does nothing.

Fix — SYN flood protection:

If SYN-RECV is dominated by one IP, it's a scan or flood. Enable SYN cookies (default), consider tcp_synack_retries:

net.ipv4.tcp_synack_retries = 2  # Down from 5; faster give-up

Verification: ListenOverflows stops growing, SYN-RECV count stabilizes under burst.

Scenario 4: Retransmission Spike (15% of production cases)

Symptom match: Slow requests, timeouts, but no RSTs. netstat -s | grep -i retrans shows TCPRetransFail climbing.

Root cause: Packets dropped on the path — either wire-level loss (cable, switch, Wi-Fi) or congestion-induced drops. TCP's retransmission algorithm kicks in, but if the loss rate exceeds a threshold the effective throughput collapses.

Diagnosis:

# Cumulative retransmission count
netstat -s | grep -i retrans

# Per-connection retransmission (modern kernels, ss >= 2.19)
ss -ti 10.0.0.5:8080

# Check link errors
ethtool -S eth0 | grep -iE 'drop|error|crc'

# Is it one direction or bidirectional?
tcpdump -nn -i any host 10.0.0.5
# Look for duplicate seq numbers

Fix — kernel tuning helps slightly, doesn't fix the root cause:

# More aggressive retransmission
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_rmem = 4096 87380 16777216

The kernel tuning above doesn't fix packet loss. If retransmissions are real, find and fix the lossy segment:

# Where's the loss happening? Probe hop by hop
mtr -rwc 100 10.0.0.5
# Look for loss at a specific hop

# Or use IP fragmentation detection
ping -M do -s 1472 host  # MTU probing

If mtr shows loss at hop 3 but not hop 2 or 4, that switch or router is the problem — file a ticket with your network team.

Verification: nstat -az | grep -i retrans shows flat count over 5 minutes. mtr loss stabilizes at baseline.

Closing the Loop: TCP Cheat Sheet

20 ss / netstat / sysctl One-Liners

# Command What it tells you
1 ss -s Overall TCP summary (states, timewait, orphaned)
2 ss -tn All TCP connections with state
3 ss -tn state time-wait Only TIME-WAIT sockets
4 ss -tn state close-wait Local app not closing — always a bug
5 ss -tn state syn-sent Outbound connects that haven't completed
6 ss -tn state syn-recv Inbound half-open connections
7 ss -lnt Listening sockets with Recv-Q / Send-Q
8 ss -tnp TCP connections with owning process
9 ss -tni TCP connections with detailed info (retrans, cwnd)
10 ss -tn dst 10.0.0.5 Connections to a specific host
11 ss -tn sport = :8080 Connections on a specific port
12 ss -tn \| awk '{print $1}' \| sort \| uniq -c State distribution
13 ss -tn state time-wait \| wc -l TIME-WAIT count
14 netstat -s \| grep -iE 'retrans\|reset\|listen\|overflow' Kernel TCP counters
15 nstat -az \| grep -i retrans Per-nstat retrans counters (delta-friendly)
16 cat /proc/net/tcp Raw TCP table
17 sysctl net.ipv4.tcp_tw_reuse Current TIME-WAIT reuse setting
18 sysctl net.core.somaxconn Current accept queue cap
19 ethtool -S eth0 \| grep -iE 'drop\|error\|crc' NIC error counters
20 mtr -rwc 100 host Loss by hop

6 tcpdump / sysctl / Config Snippets

1. tcpdump: only RST packets

tcpdump -nn -i any 'tcp[tcpflags] & tcp-rst != 0'

2. tcpdump: SYN without ACK (handshake attempts)

tcpdump -nn -i any 'tcp[tcpflags] & tcp-syn != 0 && tcp[tcpflags] & tcp-ack == 0'

3. tcpdump: zero window (receiver backpressure)

tcpdump -nn -i any 'tcp and (tcp[12] == 0)'

4. sysctl: production server baseline

# /etc/sysctl.d/90-tcp.conf
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.ip_local_port_range = 1024 65535
net.core.somaxconn = 8192
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2

5. Socket-level keepalive (Python)

s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 60)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 10)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 3)

6. listen() backlog must match somaxconn

listen(server_fd, 8192);  // not 128

When You're Still Stuck

If you've covered all four scenarios and TCP still looks wrong, drop to the packet layer:

  1. Packet capture: tcpdump -nn -i any -w /tmp/conn.pcap for the duration of the issue. Open in Wireshark (see Wireshark basics for filters).
  2. Look for 3 signatures: (a) unacknowledged data → retransmissions, (b) RST in one direction → app closed early, (c) duplicate ACKs → loss on path.
  3. Compare to a known-good path: same workload, different IP, same port. If one works and the other doesn't, it's the path.

502 vs 504 — Where the Boundary Is

If you came here from Nginx 502/504 issues: Nginx reports what it saw on its side of the socket. Your Nginx may report upstream timed out while the real problem is a retransmission storm on the app side. Or Nginx may report connection refused while the real problem is the upstream's accept queue is full. Nginx error.log tells you the symptom; ss and netstat tell you the state; tcpdump tells you the packet-level truth. Read them in that order.

Authoritative References

Written for ops.jiunotes.top. Original work based on production incident experience. Not copied from any third-party blog. CC BY-NC-SA 4.0.