DoS, DDoS
Introduction
DoS or Denial of Service attacks effectively make service unavailable for legitimate use. DoS come in two flavors - DoS and DDoS (Distributed DoS). DoS occurs from a single server and DDoS is a coordinated attack by many servers simultaneously. DoS may originate externally or be non-maliciously self-inflicted due to accidental bugs or mis-configuration. Some examples of how big and simple they are to accomplish.
Reasons for DoS
There are many possible reasons for being the target of a DoS attack. The reasons below were derived from research found here and here.
- Self-inflicted - bug or misconfiguration (non-malicious)
- Random target or Prestige - challenge, bragging rights for hacker or group
- Politics - anti-capitalism, anti-american
- Training ground - experiment, training for other attacks
- Punishment for undesired action - disgruntled customer, employee, vendor, partner, etc.
- Anti-competitive business practice
- Ideological conflicts or Principle Driven - disagrees with religious, LGBT, etc. policy
- Blackmail or extortion - via threat of DoS for financial gain or to elicit change, i.e., wants action taken against a particular employee for his actions or beliefs, or statements
- Cause SLA violations to get out of contract - Phil Lombardi
Classes and Types of DoS
This list is intended to be representative and not exhaustive. AWS 2014 presentation.
According to AWS presentation:
- 78% of attacks at infrastructure (Volume/Protocol), avg. < 1 Gb/sec, 50% < 500 Mb/s, easy to launch rent botnets by the hour
- 22% of attacks at application layer
- More recently seeing multi-vector attacks
- Hit and runs 91% less than an hour, exhausting for admin staff often used as smokescreens for other hacking like password attacks
- Most attacks at port 8443 (https)
Volume
Saturate the bandwidth, measured in bits/sec
- UDP Flood - Amplification Attack (high response to request ratio) NTP, SSDP, DNS, Chargen, SNMP
- ICMP (Ping) Flood
Protocol
Consumes server resources, measured in Packets/sec
- SYN floods - ties up all available connections
- Ping of Death - malformed excessively large packets that overrun buffers
- Slowloris and derivatives - ties up connections and server resources slowly sending or receiving data
- HTTP Flood - resource exhaustion, looks like real traffic, can be hard to distinguish valid traffic malicious traffic
Application
Consume resources and crash the services, measured in Requests/sec
Prevention/Mitigation
AWS Topology Recommendation, comprehensive
- ELB blocks/discards UDP flood attacks
- ELB discards syn floods
- CloudFront spreads out attacks and absorbs them
- CloudFront discards syn floods
- HAProxy with Nginx + NAXSI can be utilized as WAF
Standard Approaches
From AWS presentation:
- Minimize surface
- Know normal traffic - typical and atypical (when are typical spikes), types and levels
- Be ready to scale
- Safeguard expensive non-scalable resources
- Plan in advance, difficult while under attack
Recommendations (not from AWS presentation):
- Only expose services through AWS ELB
- Change SSH from standard port - attacks are more frequently attempted against standard ports
- Only allow SSH traffic from specific networks (expected origination) to prevent/filter attacks from the cloud
- Enable x-forwaded-for logging
- group, sum, and update appropriate ip filtering
AWS Recommended Services
Route53
Provides:
- Worldwide distributed DNS
- Anycast routing requests to most locally available resources
- Health checks to validate resources are available and healthy
- Provides A-records and cnames to provide resource abstraction to make it easier to quickly make topology changes
- Infima Shuffle-sharding handled by a RubberTree - don't fully understand this, but it sounds interesting
CloudFront
Provides more than just CDN:
- CDN (static content caching)
- Massively diffuse to absorb flash traffic and deflect many DDoS attacks, i.e., UDP / amplification attacks, ICMP, syn flood, ping-of-death,etc.
- SSL termination ($600/month per cert)
- Spreads load around the globe
- Reuses TCP connections (pooled keep-alive) - reduces load on back-end servers
Elastic Load Balancer (ELB)
Provides:
- Cross zone load balancing - balances auto-scaling quickly (off by default - turn it on)
- Elastic scaling
- Deflects many DDoS attacks, i.e., UDP / amplification attacks, ICMP, syn flood, ping-of-death,etc.
- SSL termination
- Make sure health checks of backup-end are long enough to allow startup or instances will fail and be terminated
Slow to warm up and not as diffuse as CloudFront.
EC2 Auto-Scaling
Provides:
- Automatic increasing and decreasing of resources to process load
- Can scale up at one rate, down at another rate, scale up quickly, down slowly
- Can scaled based on various metrics:
- concurrent connections
- connection rate change
- response time change
Don't recommend CPU load
During increased load, choice to use larger instances or more instances or both
Multi-Region, Multi-Zone
Provides:
- 11 regions each with multiple availability zones
- More EC2 scalability
- More network bandwidth
- Quicker scaling/reactivity
CloudWatch
Provides:
- Monitoring
- EC2 instances
- Auto-scaling groups
- ELB
- Route53 healthchecks
- Custom application metrics
- Monitor and store logs
- Threshold Alarms
- Log Event Alarms
- Provides metrics graphs and statistics
Standard metrics:
- CPU utilization
- Data xfer
- Disk utilization
- of requests/failures(400s)
- of connections
- duration of connections
Web Application Firewalls (WAF)
Web Application Firewalls (WAF) provide:
- OWASP top 10 detection/protection
- Rate limiting
- Whitelist/Blacklist
- Native auto-scaling
- Learning engine
Open Source WAF
- HAProxy - configurable to detect and prevent excessive requests from individual IPs and automatically drop slow leaking connections, see this and this and thisfor example configurations
- Slowloris and derivatives both send and receive
- Excessive host traffic
- too many connections opened
- new connection rate too high
- http request rate too high
- bandwith usage too high
- Can tarpit to slow down agents
- Nginx + NAXIS- validation and filtering, see slide presentationon using Nginx
- Checks URLs, request parameters, file attachments, body, headers
- Excessive host traffic, some overlap with HAProxy
- Has standard rule sets
- Learning mode
- white list for exceptions
COTS WAF
Amazon has 15 options in Amazon Marketplace
Sample Providers:
Tools
DoS Deflate monitors with netstat and updates iptables when excessive connections from an ip address
Profile Traffic
Benefits from knowing typical and atypical traffic profile:
- Allows spotting abnormalities
- Useful for setting alarms and thresholds
- Forensic data for law enforcement
Case Study
From Amazon partner CrownPeak, see prersentation
CrownPeak successfully defended sustained large scale DDoS attack for customer in April 2013.
Statistics from the peak of the event:
- 39 hour sustained attack
- 100K agents from around the world
- 60% from US
- 100% legitimate URL requests, could not filter them
- 86 million concurrent requests per second
- 20 Gb/sec sustained
Peak auto-scaled environment:
- 17 - C3.8xlarge EC2 instances running HAProxy
- 34 - m3.large web servers
- scaled up 2-4 instances at a time (180 sec interval)
- maintained 1-3 second response time for legitimate traffic
Total cost: $1500
Recommendations
Below are recommendations for a hybid or frugal approach. The information was derived from AWS documentation, best practices and various presentations. Injecting the WAF sandwich after the attack started is similar to what occurred in the CloudPeak event presented in the Amazon presentation.
- Build and test a comprehensive system utilizing Route53, CloudFront, auto-scaled ELB + Web Application Firewall sandwich comprised of either or both HAProxy and.or Nginx + Naxis in front of the application servers.
- Once tested, route traffic with Route53 alias through ELB to application server. Need to use low DNS TTLs (1-10 minutes) so changes will propagate quickly enough.
- Monitor the system and understand normal traffic patterns and set thresholds and alerts. If an attack is detected stand-up auto-scaled ELB + WAF sandwich and update the Route53 alias to route traffic through the ELB + WAF sandwich.
- By not running the ELB + WAF sandwich during most of the time when not being attacked the extra server and traffic costs are saved.
- The risk is that there may be a brief outage due to overloaded infrastructure at the start of an attack until the routes are changed and WAF is added.
Other specific recommendations:
- Create a comprehensive test plan
- Create comprehensive metrics, thresholds, and alerts
- Create comprehensive documentation and playbook
- Train/practice with real traffic load simulating an attack
- Script all necessary changes
- Train/practice periodically with real EC2 configuration, instances, and scripts
- Periodically review new attack vectors, update plans, training, testing, scripts, and documentation
If practical, utilize the following low-cost open source services:
- Nginx + NAXIS
- HAProxy
Comments