A virtual machine's security group allows inbound TCP 443 from the load balancer and has no other inbound rules. Outbound rules allow everything. A health check on port 8080 from the load balancer fails. Which explanation fits?
Show the full answer Hide the answer
The mechanism
A security group is stateful: traffic allowed in one direction is automatically allowed back. The group tracks connections, so a permitted inbound request on 443 produces a permitted outbound response, and a permitted outbound request produces a permitted inbound response, regardless of the rules in the other direction.
So nothing about return traffic is in play here. The health check is a new inbound connection to port 8080, and the group permits new inbound connections only on 443. It is blocked on arrival - and blocked by a group that drops rather than rejects, which is why the symptom is a timeout rather than a connection refused.
Fix: add an inbound rule for 8080 from the load balancer's security group. Referencing the group rather than an address range is the idiomatic form, because the load balancer's addresses change and the group reference follows them.
Why the other options fail
- "Return traffic needs an inbound rule for the ephemeral port range." This is the correct answer for a stateless access control list - a network ACL - and getting the two confused is the single most common cause of this class of confusion. A stateless list evaluates every packet independently, so a reply arriving on an ephemeral port in the 32768–60999 range needs its own explicit allow. Security groups do not work this way.
- "Outbound rules must name the load balancer explicitly." Outbound is already allow-all, and in any case the health check's reply is permitted by connection state rather than by an outbound rule.
- "Stateful groups require a rule in both directions." This states the opposite of what stateful means. It is a widely held belief and it produces configurations with twice the rules and no additional safety.
The distinction to keep
| Security group (stateful) | Network ACL (stateless) | |
|---|---|---|
| Return traffic | automatic | needs an explicit rule, usually the ephemeral range |
| Evaluation | per connection | per packet |
| Rules | allow only | allow and deny |
| Attached to | the instance's interface | the subnet |
When both are in use, a connection must satisfy the ACL in both directions and the group in one. A flow that works for short-lived connections and fails for long ones is usually a stateless-ACL or connection-tracking timeout issue, not a group rule.
The design trade-off is worth stating, because teams reach for both layers by default. Stateless lists cost operational complexity in exchange for a deny rule and a subnet-wide blast radius, which is occasionally exactly what a compliance control asks for and is otherwise two sets of rules to keep consistent. In production the common failure is a half-remembered ephemeral-port rule copied from an ACL into a group, where it does nothing but reassure. Prefer groups referencing other groups as the only layer, and add a subnet list only when you need an explicit deny - for example blocking one address range across 20 subnets at once, which groups cannot express.
When this is not the wrong answer in disguise
If port 8080 is in the inbound rules and the check still fails, the group has been eliminated and the next candidates are the subnet's ACL, the host firewall inside the instance, the application not listening on the expected interface (binding to localhost rather than 0.0.0.0 is extremely common), and a health check path returning a non-2xx status. Check in that order and stop guessing at rules: a packet capture on the instance settles whether traffic arrives at all, which splits the problem in half in one step.