IPsec VPN Penetration Testing with BackTrack Tools

IPsec VPN Penetration Testing

Penetration testing

This article outlines the value of penetration-testing VPN gateways for known vulnerabilities and also shows you how to prevent a breach into the internal network.

IPsec is the most commonly used technology for both gateway-to-gateway (LAN-to-LAN) and host to gateway (remote access) enterprise VPN solutions. It offers complete data protection for tunnelled traffic, with confidentiality, integrity, data origin authentication and anti-replay services. IPsec uses a lot of components to achieve high-level security. The major protocols that IPsec uses are:

  • ESP (Encapsulation Security Payload): ESP can provide data confidentiality and integrity, but cannot protect the IP header. The IP protocol number of ESP is 50.
  • AH (Authentication Header): AH can provide the integrity service to the data packet, but cannot offer confidentiality to data packets like ESP. The IP protocol number of AH is 51.
  • IKE (Internet Key Exchange): IKE provides support for the negotiation of parameters between end points or VPN peers and thus establishes, maintains and terminates security associations (SA). The SA termination can be based on time (seconds) or transfer (kilobytes) rate. Actually, IKE is a type of ISAKMP (Internet Security Association Key Management Protocol) implementation, which is a framework for authentication and key exchange. IKE establishes the security association (SA) between two endpoints through a three-phase process.
    • IKE Phase 1: IKE Phase 1 sets up a secure channel between two IPsec endpoints by the negotiation of parameters like the encryption algorithm, integrity algorithm, authentication type, key distribution mechanism, life time, etc. IKE Phase 1 can either use the main mode or aggressive mode to establish the bidirectional security association. Main mode negotiates SA through three pairs of messages, while aggressive mode offers faster operations through the exchange of three messages.
    • IKE Phase 2: IKE Phase 2 is used for data protection. The VPN peers negotiate the IPsec parameters needed for data security with ESP and AH. Finally, a unidirectional SA is built between pairs with a special mode known as Quick Mode. The establishment of the Phase 2 security association can use an entirely different algorithm from what’s used by Phase 1 — the Diffie Hellman Algorithm — for more security. This concept is known as Perfect Forward Secrecy (PFS).
    • IKE Phase 1.5: IKE Phase 1.5 or the Extended Authentication Phase is an optional phase and is commonly used in remote access VPN solutions. IKE Phase 1.5 will enhance security by adding end-user-level authentication.

Commercial VPN gateways from different manufacturers like Cisco, Checkpoint, Juniper, Microsoft, etc., are readily available. Some of those vendors offer both hardware- and software-based solutions for IPsec implementations. Quite a few robust open source solutions like Openswan, StrongVPN, etc., can also be used for IPsec implementations.

Does your IPsec VPN solution offer complete protection?

VPN penetration testing will help the organisation to baseline (identify the loopholes that exist in the present implementation and modify the configuration accordingly to protect itself from known problems) its current VPN security posture, identify threats and weaknesses, and implement a new security policy that will mitigate risks.

Setting up the test lab for VPN pen-testing

GNS3 is a great tool for simulating Cisco devices (and other vendor devices like Juniper too). There are many tutorials on the Internet for IPsec remote access and site-to-VPN configurations using GNS3. One such tutorial is available here.

By using a PC with a Core 2 Duo or higher processor with 2 GB RAM, you can completely simulate the test lab. Distros like Ubuntu now have GNS3 in their repository. Command-line installation in Ubuntu is as simple as sudo apt-get install gns3. A complete coverage of the GNS3 setup is beyond the scope of this article. So those who are new to GNS3, consult the project documentation.

For this test lab, you need to simulate a router with IPsec support with two interfaces. Do not forget to add the BackTrack PC to the simulated Internet region of your test lab setup (external interface of router), as shown in the sample topology (Figure 1). All tests will be performed from the BackTrack PC.

Sample topology
Figure 1: Sample topology

The same test lab setup can also be arranged with other solutions like Checkpoint SPLAT (Secure Platform or SPLAT is a software-based gateway solution from Checkpoint Software), Microsoft Server 2003 or 2008 (configured as an IPsec VPN gateway), etc.

Penetration testing an IPsec VPN

Penetration testing an IPsec VPN includes several phases like:

  1. Scanning or identifying the VPN gateway.
  2. Fingerprinting the VPN gateway for guessing implementation.
  3. PSK mode assessment and PSK sniffing.
  4. Offline PSK cracking.
  5. Checking for default user accounts.
  6. Testing the VPN gateway for vendor specific vulnerabilities.

Scanning or identifying the VPN gateway

To determine the presence of an IPsec VPN gateway, the penetration tester needs to port-scan the target. Most IPsec implementations will be ISAKMP-based. ISAKMP is an application layer key-exchange protocol that provides mechanisms to establish, negotiate, modify and delete Security Associations. ISAKMP uses UDP port 500, so a direct UDP port-scan on the suspected VPN gateway may give you the results. You can use Nmap or Ike-scan for this.

Scanning with Nmap

A direct port-scan on the VPN gateway with this powerful open source scanner provides supplemental information on the presence of the VPN gateway. Nmap can later be used at the fingerprinting phase for version or OS identification.

root@bt:~# nmap -sU -p 500
Starting Nmap 5.51 (http://nmap.org) at 2011-11-26 10:56 IST
Nmap scan report for
Host is up (0.00036s latency).
500/udp open  isakmp
MAC Address: 00:1B:D5:54:4D:E4 (Cisco Systems)

Nmap done: 1 IP address (1 host up) scanned in 0.17 seconds

The options used were -sU for UDP scan, and -p to only scan the specified port. The scan output shows the ISAKMP port (UDP port 500) open.


Ike-scan is a simple but powerful command-line tool that is used to find and fingerprint VPN gateways. It sends specially crafted IKE packets to target gateways and enlists any IKE responses that are received. By default, Ike-scan works in main mode, and sends a packet to the gateway with an ISAKMP header and a single proposal with eight transforms inside it.

Each transform contains a number of attributes like DES or 3DES as the encryption algorithm, SHA or MD5 as the integrity algorithm, a pre-shared key as the authentication type, Diffie-Hellman 1 or 2 as the key distribution algorithm and 28800 seconds as the lifetime.

Initial IPsec VPN discovery with Ike-scan is as shown below:

root@bt:~# ike-scan -M
Starting ike-scan 1.9 with 1 hosts (http://www.nta-monitor.com/tools/ike-scan/)    Main Mode Handshake returned
    SA=(Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800)
    VID=4048b7d56ebce88525e7de7f00d6c2d3c0000000 (IKE Fragmentation)

Ending ike-scan 1.9: 1 hosts scanned in 0.015 seconds (65.58 hosts/sec). 1 returned handshake; 0 returned notify

The -M shows each payload in a line, so that the output will be neat and easy to understand. The output can be any of the following:

  • 0 returned handshake; 0 returned notify: This means the target is not an IPsec gateway.
  • 1 returned handshake; 0 returned notify: This means the target is configured for IPsec and is willing to perform IKE negotiation, and either one or more of the transforms you proposed are acceptable.
  • 0 returned handshake; 1 returned notify: VPN gateways respond with a notify message when none of the transforms are acceptable (though some gateways do not, in which case further analysis and a revised proposal should be tried).

In the example shown, the VPN gateway replies with one returned handshake and the acceptable transform set has these parameters:

Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800

Custom transform sets can be tried against the target with the 
"--trans" switch:


……where 1=Encryption Algorithm, 2=Hash Algorithm, 3=Authentication Method, 4=Group Description, and 5=Group Type.

Kindly refer to RFC 2409 Appendix A for a complete understanding of transform set values. There are a number of other tools like ipsectrace, ipsecscan, etc., available for IPsec scanning, but undoubtedly Ike-scan is one of the best and a frequently updated tool.

Vulnerability assessment tools like Nessus, Nexpose, etc, can be used to identify the vulnerabilities of VPN implementations. A full security audit on the target gateway with such types of tools will generate a detailed report with all identified problems and the mitigation steps available.

Fingerprinting the VPN gateway for guessing implementation

Vendor identification and software detection of the gateway is achieved in the fingerprinting phase. To proceed with fingerprinting, you need to get a handshake message from the gateway, containing the acceptable transform set details. As the default IKE doesn’t offer reliability for transmitted packets, VPN gateway vendors will use their own back-off algorithm to deal with the “lost in transit” traffic.

The attacker sends an initial IKE proposal to the VPN gateway with an acceptable transform set. The attacker doesn’t reply and carefully analyses the server response messages for some time. (The default time Ike-scan waits for back-off fingerprinting is 60 seconds.) By analysing the time difference between the received messages from the server and the matching response pattern, the pen tester can successfully fingerprint the VPN gateway vendor.

Some VPN servers will use the optional Vendor ID (VID) payload with IKE to carry some proprietary extensions. This will really make fingerprinting easy for the attacker. Most of the time, VID is a hashed text string. Ike-scan can use the --vendor switch to add the VID payload to outbound packets. The received VID payload can be displayed by Ike-scan directly, as shown below:

root@bt:~# ike-scan -M --showbackoff
Starting ike-scan 1.9 with 1 hosts (http://www.nta-monitor.com/tools/ike-scan/)    Main Mode Handshake returned
    SA=(Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800)
    VID=4048b7d56ebce88525e7de7f00d6c2d3c0000000 (IKE Fragmentation)

IKE Backoff Patterns:

IP Address       No.  Recv time            Delta Time    1    1322286031.744904    0.000000    2    1322286039.745081    8.000177    3    1322286047.745989    8.000908    4    1322286055.746972    8.000983    Implementation guess: Cisco VPN Concentrator

Ending ike-scan 1.9: 1 hosts scanned in 84.080 seconds (0.01 hosts/sec). 1 returned handshake; 0 returned notify

Note that the Ike-scan fingerprinting can be done without even using the --trans option, but adding it will make the process faster.

So you have been successful in fingerprinting the vendor of the VPN gateway; in this case, it is a Cisco VPN Server like ASA or PIX.

PSK mode assessment and PSK sniffing

The aggressive mode of IPsec does not use a key distribution algorithm like Diffie-Hellman to protect the authentication data exchange. This makes it possible for the attacker to capture the authentication data. A server that works with aggressive mode will send the authentication hash in clear-text mode, which can be captured and cracked offline by tools like ike-crack. In the following example, the penetration tester sniffs the PSK hash and saves it into a file for offline cracking:

root@bt:~# ike-scan --pskcrack --aggressive --id=peer > psk.txt

Ike-probe or Ike-scan can be used to capture authentication data, as the following example shows:

root@bt:~# ike-scan --pskcrack --aggressive --id=peer

Starting ike-scan 1.9 with 1 hosts (http://www.nta-monitor.com/tools/ike-scan/)    Aggressive Mode Handshake returned HDR=(CKY-R=7eb59f437bbc5445) SA=(Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800) KeyExchange(128 bytes) Nonce(20 bytes) ID(Type=ID_IPV4_ADDR, Value= Hash(20 bytes) VID=12f5f28c457168a9702d9fe274cc0100 (Cisco Unity) VID=09002689dfd6b712 (XAUTH) VID=4048b7d56ebce88525e7de7f00d6c2d3c0000000 (IKE Fragmentation) VID=1f07f70eaa6514d3b0fa96542a500100 (Cisco VPN Concentrator)

IKE PSK parameters (g_xr:g_xi:cky_r:cky_i:sai_b:idir_b:ni_b:nr_b:hash_r):
Ending ike-scan 1.9: 1 hosts scanned in 0.018 seconds (55.19 hosts/sec). 1 returned handshake; 0 returned notify

Offline PSK cracking

Before cracking the captured hashed authentication string offline, edit the output file to include only the hash value. (It should only include 9 colon-separated values.) The offline cracking in Backtrack is done with psk-crack, which supports the dictionary, brute-force and hybrid mode cracking.

There are a number of other tools like Cain and Abel available for the offline PSK hash cracking. The following example shows the dictionary mode of psk-crack:

root@bt:~# psk-crack -d /usr/local/share/ike-scan/psk-crack-dictionary psk.txt

Starting psk-crack [ike-scan 1.9] (http://www.nta-monitor.com/tools/ike-scan/)
Running in dictionary cracking mode
key "ADMIN" matches SHA1 hash c1dc52bbb88d4b434c1050a6e77e923f03afbc82
Ending psk-crack: 136 iterations in 0.001 seconds (153153.15 iterations/sec)

So the VPN gateway is configured with a simple pre-shared key ADMIN!

Checking for default user accounts

Most VPN solutions have the end-user-level authentication Xauth (Phase 1.5 of IKE) or Extended Authentication enabled by default. So with the psk-crack output alone, it will not be possible to get into the internal network. After the initial peer authentication, Xauth is required before the VPN gateway grants access.

Xauth login credentials can be captured by using fiked, a command-line tool that impersonates the VPN gateway’s IKE responder and sniffs the authentication data by intercepting the IKE traffic. You need to redirect IKE traffic to fiked for sniffing, which can be done with the help of ARP spoofing.

Given below is a simple example of fiked. The -g switch specifies the IP address of the gateway, captured data is written to a file with the -l switch, -d is used to run it in daemon mode, and -k is for “group id: shared key” representation:

root@bt:~# fiked -g -k testgroup:secretkey -l output.txt -d

In some cases, the VPN gateway will have default user accounts, which the pen-tester can use for Xauth. If not, extensive social engineering or information gathering will do the trick. You may use a proper IPsec VPN client like the Cisco EasyVPN client for the final verification.

Testing the VPN gateway for vendor specific vulnerabilities

There are hundreds of known IPsec/IKE vulnerabilities. Exploitation of these can cause disruption of VPN gateway services, so testing the gateway for these is very important. The following three websites are very useful; they list the known vulnerabilities of different VPN solutions:

  1. National Vulnerability Database
  2. Secunia
  3. SecurityFocus

Sophisticated vulnerability assessment tools like MetaSploit Framework Pro, Qualys, Core Impact, etc., can be used to test the VPN gateway against known vulnerabilities, along with custom-created exploit scripts. Security patches, OS upgradation or additional configuration may be needed to mitigate these threats, as guided by the vendor.

By compromising a VPN gateway server, the attacker can gain access to valuable internal resources, so organisations need to make sure that VPN gateways are hardened against these threats. Vulnerability assessment and penetration-testing of the VPN gateway, along with periodic reviews of configured security policies, can help organisations to tighten up overall security.

Feature image courtesy: Walter Logeman. Reused under the terms of CC-BY-NC-ND 2.0 License.


  1. Admin , please share the contact details of the author , we are looking for the services of an experienced penetration tester like this.

  2. can u just elaborate this part “edit the output file to include only the hash value. (It should only include 9 colon-separated values.)”…in the psk offline cracking…any link to do this..or any method…i m facing extreme difficulty in this part…thnks in advance…


Please enter your comment!
Please enter your name here