Linux Box as Router

In continuation of my previous posts where i’m create a distributed cloud infrastructure i need to connect VM’s on multiple Host Machines

Pre knowledge of routing (IPTables) and networking is required for information below. Technically this is what containers like docker or software routers do internally when they need to connect 2 different network’s


Lets assume

HostMachine1 has VM’s on network
HostMachine2 has VM’s on network

Now our Gateway1 and Gateway2 have 2 Network Interfaces/NIC cards.

Gateway1 and Gateway2 are connected by switch hence on same network as well as their respective VM networks as they have 2 NIC cards connected.

Let assume Gateway1 has IP
Let assume Gateway2 has IP

Both Gateway1 and Gateway2 can connect to each other as they are directly connected.

My Current Configuration on Gateway1 which is our target router and i’ve below Network Interfaces on that

enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:9d:08:3f brd ff:ff:ff:ff:ff:ff
    inet brd scope global enp0s9
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe9d:83f/64 scope link
       valid_lft forever preferred_lft forever
 enp0s10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:7b:12:89 brd ff:ff:ff:ff:ff:ff
    inet brd scope global enp0s10
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe7b:1289/64 scope link 

Now we need to add routing table configurations on either gateway1 or gateway2 to forward packets from one network to another.

We can do this 2 ways.

Either by creating network bridge

# Install brige utils that gives us easy commands to creatr bridge network
apt-get install bridge-utils
yum install bridge-utils

# Create new Bridge
brctl addbr br0

# Enable Spanning Tree Support if you need
brctl stp br0 on

# Make sure you get your Devices down before creating bridge and explicity assign to make sure they loose ip
ifconfig enp0s9 down
ifconfig enp0s10 down

# Add them to our newly created bridge network
brctl addif br0 enp0s9
brctl addif br0 enp0s10

# Finally get all interfaces up.
ifconfig enp0s9 up
ifconfig enp0s10 up
ifconfig br0 up


by modifying routing table. I’m explaining second concept here

Enable forwarding in the kernel:

echo 1 >> /proc/sys/net/ipv4/ip_forward

To set this value on boot uncomment this line in/etc/sysctl.conf


Now i need to route traffic from one Interface to another using routing tables
Below are statements that can do that

# Always accept loopback traffic
iptables -A INPUT -i lo -j ACCEPT

# We allow traffic from the HostMachine1 side
iptables -A INPUT -i enp0s9  -j ACCEPT

# We allow traffic from the HostMachine2 side
iptables -A INPUT -i enp0s10  -j ACCEPT

#                         ROUTING

# enp0s9 is HostMachine1 Network
# enp0s10 is HostMachine2 Network

# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Masquerade.
iptables -t nat -A POSTROUTING -o enp0s10  -j MASQUERADE

# fowarding
iptables -A FORWARD -i enp0s9 -o enp0s10  -m state --state RELATED,ESTABLISHED -j ACCEPT

# Allow outgoing connections from the HostMachine1 side.
iptables -A FORWARD -i enp0s10 -o enp0s9  -j ACCEPT

# Repeat same steps on reverse route
iptables -t nat -A POSTROUTING -o enp0s9   -j MASQUERADE

iptables -A FORWARD -i enp0s10 -o enp0s9  -m state --state RELATED,ESTABLISHED -j ACCEPT

iptables -A FORWARD -i enp0s10 -o enp0s9  -j ACCEPT

Finally on HostMachine1 route all traffic on subnet to Gateway1

I use Mac as one my host machine hence below command

sudo route -n add -net

If you are using any *nix systems command would be

ip route add via dev

Linux Networking Fundamentals Part 2

This is in continuation of previous article. I’m going to start from scratch.

I’m going to build

  1. Two datacenters with name DC1 & DC2 by creating 2 different Vagrant VM networks
  2. Two Rack’s per Datacenter say DC1-RC1, DC1-RC2  and DC2-RC1,DC2-RC2
  3. Each Rack is connected by a Gateway
  4. Each Datacenter is connected by a Router
  5. Finally openvpn to connect both datacenter’s


All the hardware node and device cooking is mostly done via shell scripts and ruby and vagrant coding.

I’m assuming who ever is interested to go over this first understand basics of networking, Ruby, ShellScripting and Vagrant and Docker Environments.

Before moving ahead i need a simple utility to generate IP address range for given CIDR

Wrote a basic code in ruby that generates that.

# Generate IP's in given Range
# IpList = Nodemanager.convert_ip_range('', '')

module Nodemanager

	# Generates range of ips from start to end. Assumption is that i'm only using IPv4 address
  def convertIPrange first, last
    first, last = [first, last].map{|s| s.split(".").inject(0){|i, s| i = 256 * i + s.to_i}}
    (first..last).map do |q|
      a = []
      (q, r = q.divmod(256)) && a.unshift(r) until

Now i need to load all dependencies in by Berksfile. Berksfile is like a dependency manger for chef (Provisioning tool)

It can be compared with Maven/Gradle(Java), Nuget(Dotnet),Composer (PHP), Bundler (Ruby) , Grunt/Gulp (NodeJS)

name             'basedatacenter'
maintainer       'Ashwin Rayaprolu'
maintainer_email ''
license          'All rights reserved'
description      'Installs/Configures Distributed Workplace'
long_description 'Installs/Configures Distributed Workplace'
version          '1.0.0'

depends 'apt', '~> 2.9'
depends 'firewall', '~> 2.4'
depends 'apache2', '~> 3.2.2'
depends 'mysql', '~> 8.0'  
depends 'mysql2_chef_gem', '~> 1.0'
depends 'database', '~> 5.1'  
depends 'java', '~> 1.42.0'
depends 'users', '~> 3.0.0'
depends 'tarball'

Before moving ahead i want to list my base environment.
I have 2 host machines. One on CentOS 7 and other one on CentOS 6

[ashwin@localhost distributed-workplace]$ uname -r
[ashwin@localhost distributed-workplace]$ vboxmanage --version
[ashwin@localhost distributed-workplace]$berks --version
[ashwin@localhost distributed-workplace]$ vagrant --version
Vagrant 1.8.5
[ashwin@localhost distributed-workplace]$ ruby --version
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]
[ashwin@localhost distributed-workplace]$ vagrant plugin list
vagrant-berkshelf (5.0.0)
vagrant-hostmanager (1.8.5)
vagrant-omnibus (1.5.0)
vagrant-share (1.1.5, system)

Now let me write a basic Vagrant file to start my VM’s

# -*- mode: ruby -*-
# vi: set ft=ruby :

require './modules/Nodemanager.rb'

include Nodemanager

@IPAddressNodeHash = {|h,k| h[k] = }
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!

Vagrant.require_version '&gt;= 1.5.0'

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  # Create Share for us to Share some files
  config.vm.synced_folder &quot;share/&quot;, &quot;/usr/devenv/share/&quot;, disabled: false
  # Disable Default Vagrant Share
  config.vm.synced_folder &quot;.&quot;, &quot;/vagrant&quot;, disabled: true

  # Setup resource requirements
  config.vm.provider &quot;virtualbox&quot; do |v|
    v.memory = 2048
    v.cpus = 2

  # vagrant plugin install vagrant-hostmanager
  config.hostmanager.enabled = false
  config.hostmanager.manage_host = false
  config.hostmanager.manage_guest = true
  config.hostmanager.ignore_private_ip = false
  config.hostmanager.include_offline = true

  # NOTE: You will need to install the vagrant-omnibus plugin:
  #   $ vagrant plugin install vagrant-omnibus
  if Vagrant.has_plugin?(&quot;vagrant-omnibus&quot;)
    config.omnibus.chef_version = '12.13.37'
  end = 'bento/ubuntu-16.04' :private_network, type: 'dhcp'
  config.berkshelf.enabled = true

  # Assumes that the Vagrantfile is in the root of our
  # Chef repository.
  root_dir = File.dirname(File.expand_path(__FILE__))

  # Assumes that the node definitions are in the nodes
  # subfolder
  nodetypes = Dir[File.join(root_dir,'nodes','*.json')]

  ipindex = 0
  # Iterate over each of the JSON files
  nodetypes.each do |file|
    puts &quot;parsing #{file}&quot;
        node_json = JSON.parse(

        # Only process the node if it has a vagrant section
          @IPAddressNodeHash[node_json[&quot;vagrant&quot;][&quot;name&quot;]] = Nodemanager.convertIPrange(node_json[&quot;vagrant&quot;][&quot;start_ip&quot;], node_json[&quot;vagrant&quot;][&quot;end_ip&quot;])

          1.upto(node_json[&quot;NumberOfNodes&quot;]) do |nodeIndex| 

            ipindex = ipindex + 1

            # Allow us to remove certain items from the run_list if we're
            # using vagrant. Useful for things like networking configuration
            # which may not apply.
            if exclusions = node_json[&quot;vagrant&quot;][&quot;exclusions&quot;]
              exclusions.each do |exclusion|
                if node_json[&quot;run_list&quot;].delete(exclusion)
                  puts &quot;removed #{exclusion} from the run list&quot;

            vagrant_name = node_json[&quot;vagrant&quot;][&quot;name&quot;] + &quot;-#{nodeIndex}&quot;
            is_public = node_json[&quot;vagrant&quot;][&quot;is_public&quot;]
            #vagrant_ip = node_json[&quot;vagrant&quot;][&quot;ip&quot;]
            vagrant_ip = @IPAddressNodeHash[node_json[&quot;vagrant&quot;][&quot;name&quot;]][nodeIndex-1]
            config.vm.define vagrant_name, autostart: true  do |vagrant|

              vagrant.vm.hostname = vagrant_name
              puts  &quot;Working with host #{vagrant_name} with IP : #{vagrant_ip}&quot; 

              # Only use private networking if we specified an
              # IP. Otherwise fallback to DHCP
              # IP/28 is CIDR
              if vagrant_ip
       :private_network, ip: vagrant_ip,  :netmask =&gt; &quot;;

              if is_public
       &quot;public_network&quot;, type: &quot;dhcp&quot;, bridge: &quot;em1&quot;

              # hostmanager provisioner
              config.vm.provision :hostmanager

              vagrant.vm.provision :chef_solo do |chef|
                chef.data_bags_path = &quot;data_bags&quot;
                chef.json = node_json

            end  # End of VM Config

          end # End of node interation on count
        end  #End of vagrant found
      end # End of each node type file


Finally run vagrant up . Sample output attached below. I’m creating 2 VM’s for 2 Racks and 1 VM for Gateway. There are now 3 VM’s up and running. 2 VM’s represent our 2 virtual racks and third a gateway. If you notice all of them are running on private ip network which is inaccessible from external world except our gateway node. Our gateway node has 2 different ethernet devices 1 connecting private network and other connecting host network. I’ve marked specific lines that define the kind of network that gets created.

# Only use private networking if we specified an
              # IP. Otherwise fallback to DHCP
              # IP/28 is CIDR
              if vagrant_ip
       :private_network, ip: vagrant_ip,  :netmask =&gt; &quot;;

              if is_public
       &quot;public_network&quot;, type: &quot;dhcp&quot;, bridge: &quot;em1&quot;

Sample output on Vagrant up



I define node configuration in a json file so as to make it more simple. Attached is sample node type json for both Gateway node and Rack Node
Below is definition for Rack. I tried to add as much comments as possible to explain each field

If you observer below node definition’s i’ve give Node Name prefix in the config file and also from and to range for IP’s in config file. Apart from that i define the kind of recipe that need to loaded by chef for this specific node type.

  "authorization": {
    "sudo": {
      // the deploy user specifically gets sudo rights
      // if you're using vagrant it's worth adding "vagrant"
      // to this array
      // The password for the dpeloy user is set in data_bags/users/deploy.json
      // and should be generated using:
      // openssl passwd -1 "plaintextpassword"
      "users": ["deploy", "vagrant"]
  // See for more on this
  "vagrant" : {
    "exclusions" : [],
    "name" : "dc1-rc",
    "ip" : "",
  "mysql": {
      "server_root_password": "rootpass",
      "server_debian_password": "debpass",
      "server_repl_password": "replpass"

Below is node definition for Gateway.

  "authorization": {
    "sudo": {
      // the deploy user specifically gets sudo rights
      // if you're using vagrant it's worth adding "vagrant"
      // to this array
      // The password for the dpeloy user is set in data_bags/users/deploy.json
      // and should be generated using:
      // openssl passwd -1 "plaintextpassword"
      "users": ["deploy", "vagrant"]
  // See for more on this
  "vagrant" : {
    "exclusions" : [],
    "name" : "dc1-gw",
    "ip" : "",
  "mysql": {
      "server_root_password": "rootpass",
      "server_debian_password": "debpass",
      "server_repl_password": "replpass"


Before moving on to next step i need to install 5 nodes on each rack. Which is taken care by docker. Docker is a containerization tool that mimic’s VM but very light weight. We are using docker containers to mimic realworld nodes

apt-get install -y curl &&
apt-get install  -y  apt-transport-https ca-certificates &&
apt-key adv --keyserver hkp:// --recv-keys 58118E89F3A912897C070ADBF76221572C52609D &&
touch /etc/apt/sources.list.d/docker.list &&
echo "deb ubuntu-xenial main" >> /etc/apt/sources.list.d/docker.list  &&
apt-get update &&
apt-get purge lxc-docker &&
apt-get install -y linux-image-extra-$(uname -r) linux-image-extra-virtual &&
apt-get update &&
apt-get install -y docker-engine &&
curl -L`uname -s`-`uname -m` > /usr/local/bin/docker-machine && 
chmod +x /usr/local/bin/docker-machine &&
curl -L`uname -s`-`uname -m` > /usr/local/bin/docker-compose &&
chmod +x /usr/local/bin/docker-compose &&
sudo usermod -aG docker docker

Once docker is setup on all racks  we need to install all nodes. Below is base version of docker file that i use
My next step is to setup containers on each of the rack so that we can replicate multiple datacenter’s and multiple rack scenarios

I’m going to create 5 containers on each rack and each one of the container will again be using Ubuntu Xenial as base OS. I’m going to install oracle 7 jdk on all of them.

My usecase for distributed architecture is based on HDFS, Cassandra setup hence i need to install java first . Below script is run by vagrant/chef to install docker on each of the rack.

FROM ubuntu:16.04
MAINTAINER Ashwin Rayaprolu

RUN apt-get update
RUN apt-get dist-upgrade -y

RUN DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
RUN DEBIAN_FRONTEND=noninteractive apt-get -y install python-software-properties
RUN DEBIAN_FRONTEND=noninteractive apt-get -y install software-properties-common
RUN DEBIAN_FRONTEND=noninteractive apt-get -y install byobu curl git htop man unzip vim wget

# Install Java.
  echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \
  add-apt-repository -y ppa:webupd8team/java && \
  apt-get update && \
  apt-get install -y oracle-java7-installer && \
  rm -rf /var/lib/apt/lists/* && \
  rm -rf /var/cache/oracle-jdk7-installer
# Install InetUtils for Ping/traceroute/ifconfig
RUN apt-get update
# For Ifconfig and other commands
RUN apt-get install -y net-tools
# For ping command
RUN apt-get install -y iputils-ping 
# For Traceroute
RUN apt-get install -y inetutils-traceroute

# Define working directory.

# Define commonly used JAVA_HOME variable
ENV JAVA_HOME /usr/lib/jvm/java-7-oracle

# Define default command.
CMD ["bash"]


Docker has a very elegant way of creating network’s. As our
Rack Network is on 192.168.1.*
We want
Node Network on

We have multiple options to create network in docker. I would like to go with bridge networking. Will discuss on those specific topic later. For now assuming we are using bridge network below is code to create network and attach to some container

We need to make sure we have different range of network on each rack and each datacenter so that we don’t overlap IP’s between different rack’s and datacenter’s

# Below command will create a network in our desired range (dc1-rack1)
#  to
docker network create -d bridge \
  --subnet= \
  --gateway= \
  --ip-range= \

# Below command will create a network in our desired range (dc1-rack2)
# From  to
docker network create -d bridge \
  --subnet= \
  --gateway= \
  --ip-range= \

# Below command will create a network in our desired range (dc2-rack1)
#  to
docker network create -d bridge \
  --subnet= \
  --gateway= \
  --ip-range= \

# Below command will create a network in our desired range (dc2-rack2)
#  to
docker network create -d bridge \
  --subnet= \
  --gateway= \
  --ip-range= \
# -d option to run in background  -t option to get a duplicate tty
docker run -itd multinode_node1

# Connect the newly created network on each node to the node name.
docker network connect my-multihost-network docker_node_name


I would write code to automate all the above tasks in subsequent articles. I’m going to use docker-compose to build individual nodes in each rack.

Very basic code would look like this

version: ‘2’
build: node1/
image: node2/
image: node3/

You can checkout First version of code from


Linux Networking Fundamentals Part 1

I’ve been working on setting up couple of host and guest machines at my home/office for last couple of years. I wanted to replicate real world scenario where we have multiple hosts on multiple subnet and still interacting with each other. While working on this i  found that lot of people are either not aware of basic concepts or confused because of abundant information available online. I want to simplify networking in a short concise manner all in single post.

I would also show application layers along with network layers to make it more clear. I see that these 2 department don’t have any common knowledge in common everywhere in industry. I would love my developers to understand network connectivity while designing distributed application or even basic non distributed applications.

There are 5 major components of that any company or network admin has to look at

  1. Gateway
  2. Firewall
  3. Switch
  4. Router
  5. Hub

and 2 basic concepts

  1. Nic Cards/ Network Interface Cards
  2. Subnets.


Gateway : This is the target applicance/device/machine to which source computer sends all its packets for any networking. Assuming there is only one network interface card in source computer. (Note: A machine can have any number of interface cards as hardware supports). If we use gateway we mostly need to add static routes to all machines

Firewall:  This generally sits between cloud/internet and private network which can control flow of data packets from inside/outside interfaces. Example: Cisco ASA Applicance.  Without a firewall every network is susceptible to brute force attacks and heavy load on systems. (This is funny link which shows how everyone around the world are trying to attach each other digitally

Hub: This is a device to which all networking devices connect to. This is good for lab testing but not efficient way of connecting devices.Major disadvantage it has is that it broadcasts information sent from one device over common channel that is visible to all other devices connected to that hub. This also creates unnecessary bandwidth issues as it floods information to everyone irrespective of who it is destined to.

Switch: This is an efficient way of creating a minimal network. Switches have multiple sockets to connect different available in network. Switches separate communication channels between all devices which makes it very efficient in connecting devices.

Router:  As the name suggests it routes traffic between different networks. This generally sits coupled with gateway and knows where to send packets to. It stores all information in routing table and decides best possible route for given packet. Router automatically takes care of routing removing need for nodes to add static routes

Below diagram explain a very high level connection scenario’s


If you observe in above diagram i’ve one network with subnet connected to one switch and other network connected to different switch.

Both of them are connected via a  router/gateway

And gateway is connected to internet via Firewall.

This explain a minimalistic design of any company.

We can have multiple layers of switches and router sitting on top of each other in tree fashion but basic concepts remain the same.

Now coming to NIC Card/Network Interface Cards

NIC: This is located in each and every networking device and enable the device to get connected via ethernet. These days we can create Virtual NIC cards on any Operating system which behave same as physical NIC. There are other concepts of NIC called Nic Teaming or Bonding which improve bandwidth and performance of Network IO apart from providing failover connectivity. I would explain all these concepts in details in subsequent series.

Subnet:  Subnets are single most important concept that allow us to segregate different kind of networks. Originally when networks originated in 80’s there was concept of Class Based Networking

Example: for

Class A:   Network portion is just 192 and rest  168.1.1 defines the node address

Class B:  192.168 defines network portion and rest 1.1 is node address

Class C: 192.168.1 defines network and remaining 1 represents node address

This approach was good till early 90’s but then issue of ip address exhaustion started.

Then IETF introduced a concept of CIDR (Classless Internet Domain Routing). With introduction of this approach now IP Address is never complete without Network Mask. Example doesn’t represent a node unless we define netmask. So now ip address have to defined as  where 24 represents 24 bits which is prefix or network portion of above address.

With advent of Virtual Machines/Containerization of networks and Operating system like linux we can replicate all above devices with software code.

I would start explain each concept in detail and how to configure and build multiple computers and network routes in subsequent series.

Technologies i’m going to use and topic coverage in next couple of months. I’m going to write each one fresh using latest version so that my code is not outdated.

  1. Vagrant ( To manage Virtual machines)
  2. Oracle VirtualBox ( Virtual machine provider . We can also use HyperV or VMWare)
  3. Docker (Lightweight replica of VM which can provide similar features)
  4. CentOS/Ubuntu ( I’m going to use both flavors of linux in different devices)
  5. Linux Routing Table for basic
  6. Quagga/Zebra to create powerful router
  7. Iptables used as simple firewall
  8. OpenVPN to setup VPN connections.
  9. Chef/Berkshelf (One of the most versatile OS provisioner )
  10. Shell Scripting ( This is obvious)
  11. Apache/PHP/Mysql For a demo website
  12. Cassandra as demo database to show how data gets across different racks in out virtual data centers (VM’s on different networks).


I’ve already have one base version written which i’m going to reuse in bits and pieces.

Part 2 of the series is at