Several Government of Canada servers were recently knocked off the Internet as described in this Globalnews.ca story. Targets included canada.ca, sencanada.ca, justice.gc.ca, and csis.gc.ca. The story captions a video with a reference to government servers “being hacked”, but that implies an intrusion. It wasn’t an intrusion, it was a denial-of-service attack. A flood of traffic made them unavailable, as described elsewhere in the story.
The classic “triad” of information security is CIA — Confidentiality, Integrity, and Availability. Encryption is a protective technology for confidentiality. Choose good ciphers, use them correctly, and manage your keys carefully, and an exploit becomes difficult enough and unlikely enough that you don’t have to worry about the attacker breaking the encryption.
Hash functions are detective technology for data integrity and also for authentication of users and hosts. For data, you realize that it has been modified and should no longer be trusted. For authentication, you realize that the purported user or host identity is not what it claims to be. In both cases you don’t know exactly what is going on, but you know that something is being attempted and you block or ignore the data or request.
Unfortunately, we don’t have any math-based defenses for availability, so we can’t put numbers on the risks. We can’t make an attack less likely to happen, and we can’t even put meaningful numbers on the risk of an attack.
Distributed Denial-of-Service or DDOS attacks are an unexpectedly heavy load, they’re just a more extreme version of a network service suddenly becoming much more popular than expected. With the addition of malicious intent, of course.
It is best practice to be resilient under heavy load, malicious or not, so what can we do to help our servers hold up to unexpected traffic levels? Learning Tree’s Linux optimization and troubleshooting course looks at this problem from the viewpoint of providing service under heavy legitimate loads, but the same techniques help when it’s malicious DDOS.
Here is a version of the classic TCP state transition diagram. A FIN-FIN-ACK 3-way handshake shuts down a connection. The problem is that the client may simply ignore the rules after getting the requested data, leaving the connection hanging open with the server politely trying to shut it down cleanly. Meanwhile, limited resources including file handles and buffer memory are tied up for what the server is only very slowly realizing is a defunct connection. Older versions of Microsoft Internet Explorer have been notorious for this faulty client behavior.
Remember that TCP/IP dates from around 1980, with some assumptions and default timers and counters based on the computers and Internet of that era. The default in the current Linux kernel is to keep waiting for 60 seconds when the client has simply disappeared during the connection shutdown. You can verify that with the following command. All TCP and UDP parameters are in
/proc/sys/net/ipv4, they apply to both IPv4 and IPv6.
You can experiment with tuning by using
echo to put a new value into that “file”, which is really a kernel parameter:
echo 10 > /proc/sys/net/ipv4/tcp_fin_timeout
As we show you in Learning Tree’s Linux server administration course, once you decide on an appropriate setting you should put that in a file named
/etc/sysctl.d/*.conf so it will be automatically applied at each boot.
This is just one simple example of making a system more resilient under heavy legitimate load. But…
Antiquated attacks like ping floods and the Smurf and Fraggle attacks make for useful examples when you are explaining simple DOS attacks for the first time, but the current reality is very different. Modern DDOS uses DNS amplification and NTP amplification, and commonly result in attack traffic of tens to hundreds of gigabits per second. These attacks are often run by DDOS-for-hire operators calling themselves “booters”. For example, see this CloudFlare description of an attack of almost 400 Gbps against one of their customers.
DDOS attacks saturate the network between your servers and the Internet, there’s nothing you can do yourself. Multiple providers might help, but not if they all use the same upstream provider.
Good net citizenship is important, make sure that your public-facing DNS and NTP servers are appropriately configured so they cannot be used as DDOS weapons. Contract with a major provider to better withstand DDOS attacks, if appropriate. And prepare and respond as the Canadian government did — they reminded people that even though the web sites were unavailable, the government was still operating, the offices were open, and the phones still worked.