Did You Know the /var/log Folder in Linux is Telling a Story?


Every action you take when you work on your computer or device is recorded in its logs. These logs tell a story to those who want to read it, and come in very handy in case a system is attacked.

Sumerian writings and Egyptian hieroglyphs have one thing in common — both use pictorial signs to represent events, thoughts, or abstract ideas. The Sumerian language is said to be the oldest written language in existence, first attested about 3100 BCE in Southern Mesopotamia (currently Iraq). Hieroglyphs were the writing system used by ancient Egyptians. They date back to a similar time period. But what is so special about them? Well, they represent the time when man decided to keep written records for all the events taking place. Records prior to that were kept verbally, passed on from generation to generation in the form of songs or stories. However, note that keeping a record of events has always been important in the history of mankind.

Now, let us go to the history of record-keeping in computers and computing.

In the 1960s, MIT (Massachusetts Institute of Technology), AT&T Bell Labs and GE (General Electric) jointly developed an experimental time-sharing operating system called Multics for the GE-645 mainframe. This was the predecessor to the UNIX operating system.

The year 1966 marked the development of the US Advanced Research Projects Agency Network (ARPANET), the predecessor of the modern internet. The aim was to connect various mainframes together for fault-resistant communications. Then a revolution took place on October 29, 1969, when the first message was sent over the ARPANET from the University of California Los Angeles (UCLA) to Stanford Research Institute (SRI).

A UCLA student, Charley Kline, tried to log in to the SRI SDS Sigma 7 mainframe. He had to type the command ‘LOGIN’ on the teletype writer (TTY) to log in. However, the SRI host crashed just after ‘LO’ was received; so the first ever message to be sent over the internet was ‘LO’!
After the initial crash, proper host to host communication was established between UCLA and SRI the same day itself. This was logged in the ARPANET IMP log (Figure 1).

Figure 1: ARPANET IMP log (Source: Lo and Behold, Reveries of the Connected World, Magnolia Pictures)
Figure 1: ARPANET IMP log (Source: Lo and Behold, Reveries of the Connected World, Magnolia Pictures)

The initials CSK in Figure 1 stand for Charles S. Kline. This marked the beginning of the modern-day internet, which used a packet switching network. The first ever node of the internet is still there at UCLA, standing the test of time.

The ARPANET was decommissioned in 1990, giving rise to the modern TCP/IP standards. As we can see, every written record has a story to tell.

Let us go back to another well-known story of 13th century BCE — the Trojan War. The Greek army failed to enter the city of Troy because of its high walls. So, they left a wooden horse outside the city and pretended to leave. The people of Troy brought the horse inside, and the Greek soldier hiding in it opened the city’s gates. This led to the downfall of Troy in spite of the high walls protecting it. This was recorded and written much later. Trojan Horse is the computer malware that gets its name from this historic event.

Coming to today’s world, everything is digitalized. And all digital activities leave a footprint in the form of logs. These logs tell a very vivid story, and it is very important to be able to read the story.

For example, we have a set of logs for just after a computer’s boot sequence, as shown in Figure 2.

Figure 2: Computer boot logs
Figure 2: Computer boot logs

The computer is secured with the McAfee antivirus and security system. So, malware may not be able to affect it. However, every piece of log has a story to tell. Let us see this story from an attacker’s point of view.

Figure 3: Linux logs
Figure 3: Linux logs

As an attacker, we can see that the DHCP client completed the handshake at 07:07:59. This means the computer is successfully online from 07:07:59. The WLAN-AutoConfig action to configure the Wi-Fi took place at 07:08:00. But the McAfee Service Controller started at 07:08:03. This means the attacker has a three second window to attack the system when it boots up.

Most of the world’s servers run on Linux based operating systems and the logs are usually stored in the /var/log folder.

For example, if you want to view who requested sudo privileges to your system, you can view the auth.log file in this folder.

All the logs placed side by side generate a story that tells who and which applications did exactly what on the computer without any ambiguity.

Figure 4: Logs stored in auth.log
Figure 4: Logs stored in auth.log

The logs of a Linux machine act in exactly the same way as a log register you would use for noting down every small step you take. It’s similar to a photo album that records how a child grows up gradually.

Logs generated by intrusion detection and prevention systems (IDPS) are specifically important for maintaining the security of systems. For example, Figure 5 shows an example of a server protected by the IDPS Suricata, which is under a possible SSH brute force attack.

The IDPS logs are important to understand how an attacker might be exploiting the flaws in the network to gain access to hosts. The vulnerabilities need to be patched as and when they are spotted.

There are various automated tools for visualising the logs, one of them being the ELK (Elasticsearch, Logstash, Kibana) stack. It can form easy-to-read dashboards to visualise, analyse and find anomalies in the logs.

Figure 5: Suricata logs
Figure 5: Suricata logs

Some attackers may try to modify and erase the logs, but those operations also get stored. It is possible to make the story difficult to read but it is never possible to erase it.

Logs, in the end, are very useful when it comes to telling the story of the machine, especially from the perspective of cyber security. To a lay person using the computer, logs may seem irrelevant. We do not see many people opening the Event Viewer in Windows or the log folder in Linux. But then again, when it comes to an incident response, the first step is always to analyse the logs. Because the logs always have a story to tell to those who can read it.


Please enter your comment!
Please enter your name here