HowTo IPSec tunnel and VPN – Diffie-Hellman

Overview of Diffie-Hellman

Diffie-Hellman is an asymmetric key algorithm used for public key cryptography. As well as IPSec it is also used for SSL, SSH, PGP and other PKI systems.

The Diffie-Hellman algorithm was created to address the issue of secure encrypted keys from being attacked over the internet when in transmission, though using the Diffie-Hellman algorithm in distributing symmetric keys securely over the internet.

The process works by two peers generating a private and a public key. Peer A would send it’s public key to peer B and peer B would send it’s public key to peer A. Peer A would then use the public key sent from peer B and it’s own private key to generate a symmetric key using the Diffie-Hellman algorithm. Peer B would also take the same process as peer A and in turn produce the exact same symmetric key as peer A, though enabling them to communicate securely over the in-secure internet. Both peers can now encrypt, transmit and decrypt data using their symmetric keys.

However some concerns were found later within the Diffie-Hellman algorithm such as Man-in-the-middle attacks as there is no authentication in place before keys are exchanged. How would peer B know that it is about to exchange keys with peer A? It could easily be a hacker spoofing peer A’s identity. This led to the more advanced public key cryptography in RSA. However using authentication methods such as pre-shared keys and digital certificates to authenticate VPN gateways have overcome this issue. So using Diffie-Hellman along side authentication algorithms is a secure and approved solution. Diffie-Hellman is based on calculating discrete logarithms in a finite field.

Diffie-Hellman public key cryptography is used by all major VPN gateway’s today, supporting Diffie-Hellman groups 1,2 and 5. DH group 1 consists of a 768 bit key, group 2 consists of 1024 bit key and group 5 comes with 1536 bit key. Group 5 is the strongest and most secure.

Diffie-Hellman just does key exchange and does not do data encryption, digital signatures or any authentication.

As well as Diffie-Hellman, some other asymmetric encryption algorithms are RSA, ECC, El Gamal, DSA, LUC and Knapsack.

IPsec Host-to-Host Configuration


IPsec can be configured to connect one desktop or workstation to another by way of a host-to-host connection. This type of connection uses the network to which each host is connected to create the secure tunnel to each other. The requirements of a host-to-host connection are minimal, as is the configuration of IPsec on each host. The hosts need only a dedicated connection to a carrier network (such as the Internet) and Red Hat Enterprise Linux to create the IPsec connection.
The first step in creating a connection is to gather system and network information from each workstation. For a host-to-host connection, you need the following information:
  • The IP address for both hosts
  • A unique name to identify the IPsec connection and distinguish it from other devices or connections (for example, ipsec0)
  • A fixed encryption key or one automatically generated by racoon
  • A pre-shared authentication key that is used to initiate the connection and exchange encryption keys during the session
For example, suppose Workstation A and Workstation B want to connect to each other through an IPsec tunnel. They want to connect using a pre-shared key with the value of foobarbaz and the users agree to let racoonautomatically generate and share an authentication key between each host. Both host users decide to name their connections ipsec0.
The following is the ifcfg file for Workstation A for a host-to-host IPsec connection with Workstation B (the unique name to identify the connection in this example is ipsec0, so the resulting file is named /etc/sysconfig/network-scripts/ifcfg-ipsec0):
Workstation A would replace X.X.X.X with the IP address of Workstation B, while Workstation B replaces X.X.X.Xwith the IP address of Workstation A. The connection is set to initiate upon boot-up (ONBOOT=yes) and uses the pre-shared key method of authentication (IKE_METHOD=PSK).
The following is the content of the pre-shared key file (called /etc/sysconfig/network-scripts/keys-ipsec0) that both workstations need to authenticate each other. The contents of this file should be identical on both workstations and only the root user should be able to read or write this file.


To change the keys-ipsec0 file so that only the root user can read or edit the file, perform the following command after creating the file:
 chmod 600 /etc/sysconfig/network-scripts/keys-ipsec0 
To change the authentication key at any time, edit the keys-ipsec0 file on both workstations. Both keys must be identical for proper connectivity.
The next example shows the specific configuration for the phase 1 connection to the remote host. The file is named X.X.X.X.conf (X.X.X.X is replaced with the IP address of the remote IPsec router). Note that this file is automatically generated once the IPsec tunnel is activated and should not be edited directly.
remote X.X.X.X { exchange_mode aggressive, main; my_identifier address; proposal { encryption_algorithm 3des; hash_algorithm sha1; authentication_method pre_shared_key; dh_group 2 ; } }
The default phase 1 configuration file created when an IPsec connection is initialized contains the following statements used by the Red Hat Enterprise Linux implementation of IPsec:
remote X.X.X.X
Specifies that the subsequent stanzas of this configuration file applies only to the remote node identified by the X.X.X.X IP address.
exchange_mode aggressive
The default configuration for IPsec on Red Hat Enterprise Linux uses an aggressive authentication mode, which lowers the connection overhead while allowing configuration of several IPsec connections with multiple hosts.
my_identifier address
Defines the identification method to be used when authenticating nodes. Red Hat Enterprise Linux uses IP addresses to identify nodes.
encryption_algorithm 3des
Defines the encryption cipher used during authentication. By default, Triple Data Encryption Standard (3DES) is used.
hash_algorithm sha1;
Specifies the hash algorithm used during phase 1 negotiation between nodes. By default, Secure Hash Algorithm version 1 is used.
authentication_method pre_shared_key
Defines the authentication method used during node negotiation. Red Hat Enterprise Linux by default uses pre-shared keys for authentication.
dh_group 2
Specifies the Diffie-Hellman group number for establishing dynamically-generated session keys. By default, the 1024-bit group is used.
The /etc/racoon/racoon.conf files should be identical on all IPsec nodes except for the include "/etc/racoon/X.X.X.X.conf" statement. This statement (and the file it references) is generated when the IPsec tunnel is activated. For Workstation A, the X.X.X.X in the include statement is Workstation B’s IP address. The opposite is true of Workstation B. The following shows a typical racoon.conf file when IPsec connection is activated.
# Racoon IKE daemon configuration file.
# See 'man racoon.conf' for a description of the format and entries.

path include "/etc/racoon";
path pre_shared_key "/etc/racoon/psk.txt";
path certificate "/etc/racoon/certs";

sainfo anonymous
	pfs_group 2;
	lifetime time 1 hour ;
	encryption_algorithm 3des, blowfish 448, rijndael ;
	authentication_algorithm hmac_sha1, hmac_md5 ;
	compression_algorithm deflate ;
include "/etc/racoon/X.X.X.X.conf"
This default racoon.conf file includes defined paths for IPsec configuration, pre-shared key files, and certificates. The fields in sainfo anonymous describe the phase 2 SA between the IPsec nodes — the nature of the IPsec connection (including the supported encryption algorithms used) and the method of exchanging keys. The following list defines the fields of phase 2:
sainfo anonymous
Denotes that SA can anonymously initialize with any peer insofar as the IPsec credentials match.
pfs_group 2
Defines the Diffie-Hellman key exchange protocol, which determines the method in which the IPsec nodes establish a mutual temporary session key for the second phase of IPsec connectivity. By default, the Red Hat Enterprise Linux implementation of IPsec uses group 2 (or modp1024) of the Diffie-Hellman cryptographic key exchange groups. Group 2 uses a 1024-bit modular exponentiation that prevents attackers from decrypting previous IPsec transmissions even if a private key is compromised.
lifetime time 1 hour
This parameter specifies the life cycle of an SA and can be quantified either by time or by bytes of data. The Red Hat Enterprise Linux implementation of IPsec specifies a one hour lifetime.
encryption_algorithm 3des, blowfish 448, rijndael
Specifies the supported encryption ciphers for phase 2. Red Hat Enterprise Linux supports 3DES, 448-bit Blowfish, and Rijndael (the cipher used in the Advanced Encryption Standard, or AES).
authentication_algorithm hmac_sha1, hmac_md5
Lists the supported hash algorithms for authentication. Supported modes are sha1 and md5 hashed message authentication codes (HMAC).
compression_algorithm deflate
Defines the Deflate compression algorithm for IP Payload Compression (IPCOMP) support, which allows for potentially faster transmission of IP datagrams over slow connections.
To start the connection, either reboot the workstation or execute the following command as root on each host:
/sbin/ifup ipsec0
To test the IPsec connection, run the tcpdump utility to view the network packets being transfered between the hosts (or networks) and verify that they are encrypted via IPsec. The packet should include an AH header and should be shown as ESP packets. ESP means it is encrypted. For example:
17:13:20.617872 > \
	    AH(spi=0x0aaa749f,seq=0x335): ESP(spi=0x0ec0441e,seq=0x335) (DF)

Set up an L2TP/IPsec VPN server on Linux

In this tutorial, we’ll set up a VPN server using Openswan on Debian Linux. To do this, we’ll be using the Layer 2 Tunnelling Protocol (L2TP) in conjunction with IPsec, commonly referred to as an ‘L2TP/IPsec’ (pronounced “L2TP over IPsec”) VPN.
For more information, see the L2TP/IPsec standard (RFC 3193).

Note: If you were looking for our tutorial on how to build an L2TP/IPsec VPN on a Windows Server, you can find it here.


Step 1: Initial setup

You’ll need to have set up a Cloud Server running Linux. The steps in this tutorial assume that you are using Debian Linux, but should be similar for other versions of Linux or BSDs if you have a preference. We recommend running all the commands below as root, or using sudo.

If you are looking to use the VPN to connect to several servers within ElasticHosts, make sure that the others are connected to the VPN server by a VLAN as described in our tutorial on VLANs.

If you don’t intend to connect to other machines within your ElasticHosts account (for example, if you want to use the VPN for increased privacy while browsing), you won’t need the second server. Finally, if you’re using a firewall such as iptables or the built-in ElasticHosts firewall, you’ll need to make sure that UDP traffic is allowed to port 500 (IKE) and port 4500 (for IPsec Nat traversal). For the purposes of this tutorial, we will give our VPN server an address of on the VLAN, and connect a second server over the VLAN at If you are not using a VLAN, you can add an internal address to your server’s eth0 device as follows:

$ nano /etc/network/interfaces
#These lines, or something similar to them, should already exist
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

#We need to add the lines below this one
auto eth0:1
iface eth0:1 inet static

Next, we need to enable IP forwarding. Using your text editor of choice, open your server’s sysctl configuration file:

$ nano /etc/sysctl.conf

This allows you to change several operating system parameters within Linux. We’re going to add three lines: the first enables IP forwarding, and is essential. The other two lines disable ICMP redirects: this is not essential but is highly recommended unless you believe they are specifically required.

net.ipv4.ip_forward = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0

Tell sysctl to re-read the configuration file to start using the new parameters.

$ sysctl -p /etc/sysctl.conf

Finally, we must install the packages we will be using – the Openswan IPsec VPN, and xl2tpd, the Layer 2 Tunneling Protocol Daemon.

$ apt-get update
$ apt-get install openswan xl2tpd

When installing Openswan, you will be asked whether you want to create an X.509 certificate. This tutorial will cover preshared key (PSK) authentication, so you can select No here: if you change your mind at any time you can reach this prompt again by running the command dpkg-reconfigure openswan.

  • Important: While PSK authentication is secure enough for most uses, this may leave servers vulnerable to ‘Man in the Middle’ (MitM) attacks, potentially allowing a malicious server to masquerade as the VPN gateway. While this is only possible if the attacker is in possession of the PSK, authentication with X.509 certificates or with RSA keypairs makes this type of attack significantly more difficult. If you are planning to allow VPN clients to use the server’s internet connection, you may also wish to install the iptables-persistent package, as this will come in useful later.
$ apt-get install iptables-persistent

Step 2: IPsec configuration

First, we’ll configure the IPsec part of our network: this will provide a secure channel for our L2TP-tunnelled data. We’ll start by editing the main Openswan configuration file as follows.

$ nano /etc/ipsec.conf
config setup
conn L2TP-PSK

[lns default]
ip range =
local ip =
refuse chap = yes
refuse pap = yes
require authentication = yes
ppp debug = yes
pppoptfile = /etc/ppp/options.xl2tpd
length bit = yes
  • local ip – This is the IP of your VLAN interface. As mentioned above, we are using in this example.
  • ip range – A range of IPs from which internal addresses will be assigned to remote clients. This should be on the same subnet as this machine, but should not conflict with addresses in use on the VLAN. Here, we specify, allowing for plenty of servers within the VLAN and over 200 potential remote users.
  • We have specified refuse chap and refuse pap. When we specify PPP options we will choose to require the use of MS-CHAPv2, for better compatibility with Windows clients.

Now the PPP options.

$ nano /etc/ppp/options.xl2tpd
asyncmap 0
  • ms-dns – Some clients will request DNS settings from the server when connected over the VPN. Here, we are simply using public DNS provided by Google and Level3.
  • proxyarp – Proxy ARP is required to rewrite the source hardware address and prevent traffic being dropped for coming from an invalid location.

Finally, the CHAP secret for PPP authentication.

$ nano /etc/ppp/chap-secrets
# Secrets for authentication using CHAP
# client        server                           secret                       IP addresses
*               elastichosts-vpn-server          a-secure-chap-secret         *

Again, you can specify several lines following this format, in order to provide different secrets for different users.

  • client – This allows you to set a username to go with the CHAP secret. Here, we use the wildcard * to allow any username.
  • server – The name of the server. We specified this in /etc/xl2tpd/xl2tpd.conf above.
  • secret – The CHAP secret itself. Again, this is a password and should follow good practice for password strength.
  • IP addresses – An IP address or range of addresses that will be used to allocate an address to any user who connects using this CHAP secret. This should be a subset of the range specified above: here we have used * once again, meaning the server should allocate addresses from the whole range.

This nearly completes our tutorial – if you would like to make this server’s internet connection available for remote clients to use a proxy, you must follow one more step. If you are simply aiming to use your IPsec VPN to connect remote users to servers within an ElasticHosts VLAN, you can skip to the final step: starting Openswan.

Step 4 (Optional): Enable NAT for remote clients

In order for remote users to connect to the public Internet through this VPN server, the gateway must be configured to perform Network Address Translation (NAT) for remote clients. To do that, we will use Linux’s built-in iptables firewall: the following command tells it to perform NAT for the IP range

$ iptables -t nat -A POSTROUTING -o eth0 -s -j MASQUERADE

You may recall that we installed the package iptables-persistent during the first step. This provides a simple method of ensuring that the firewall rules we create will be loaded when the server boots. To save the firewall rules, run:

$ iptables-save > /etc/iptables/rules.v4

Remote users can now set a default route through this server in order to securely access the internet through the VPN.

Step 5: Start Openswan

All we need to do now is start Openswan and xl2tpd. On Debian, we can do this as follows:

$ /etc/init.d/ipsec start
$ /etc/init.d/xl2tpd start

To make Openswan and xl2tpd start automatically on boot, simply run the following two commands.

$ update-rc.d ipsec defaults
$ update-rc.d xl2tpd defaults

You’re finished!

That’s it – your L2TP Debian server is now ready to be used.

Take a look at our Windows or Linux client tutorials for more information on connecting a client to the VPN.



Ron Jagannathan has written 54 articles

Ronan is a Caffeine dependent life-form from Planet Earth who wants to be a Jedi Knight of cloud computing. A man of mystery and power, whose power is exceeded only by his mystery. Quantum Physicist, TransHumanist, Systems Architect, Unix Administrator, Artificial Intelligence, Machine Learning and DIY Gadget enthusiast. Believes that the Universe has a high probability of being a simulation.
But he's real and hopefully some of his readers are too.
email: ph: 202 355 5205
My Famous Quotes:
“In a Unix Universe, God is known by a four letter word called root. To err is really foul requires you to be root.. err.. god.” ― Ron Jagannathan


“Quotes found on the Internet are not always accurate.” ― Abraham Lincoln

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>