VMware NSX how to firewall between IP’s and issues

The first thing everyone does with NSX is try to create firewall rules between IP addresses.  I consider this a mistake because the DFW can key off a lot better markers than IP addresses.   Either way at some point you will want to use IP addresses in your rules.  This post will describe how to setup firewall rules between IP addresses.



I have two Linux machines each on their own subnet:

Linux1 – – network

Linux3 – – network

Routing is setup between the hosts so they can connect to each other.  I would like to block all traffic except ssh between these subnets.   We are going to assume that both of these networks exist in NSX.

NSX Setup:

First we have to set up an IP set in NSX Manager.  This is suprisingly a set of IP addresses.

  • Login to the vSphere web client
  • Click networking and security
  • Select your NSX Manager and expand it
  • Select Manage -> grouping objects
  • On the lower pane select IP Sets
  • Press the green plus button to add a new set
  • Setup each set as shown below:



Tale of multiple cities:

Here is where NSX gets interesting you have multiple ways to block access.  First a little understanding of firewall constructs in NSX:

  • Security Groups – these are groups of machines / constructs they can include IP sets, MAC sets, dynamic name based wild card information.  They can contain whole datacenters or a single virtual machine.  It can be very dynamic with boolean conditions.
  • Security Policies – These are groups of firewall rules and introspection services.  These are policies that are applied to security groups.  Each of the firewall policies assume that they are assigned to one or more security groups.   So your source or destination needs to be the policies assigned security group.  The opposite side (source/destination) needs to either be a security group or any.

Remember we want the following rules:

  • SSH between and should be allowed bi-directional
  • Everything else between them should be blocked

Within these constructs there are a number of possible options for the firewalls:

  • Option 1 – rules in this order
    • Firewall rule allowing ssh between source: assigned policy group and destination:
    • Firewall rule allowing ssh between source: and destination: assigned policy group
    • Firewall rule blocking any between source: assigned policy group and destination:
    • Firewall rule blocking any between source: and destination: assigned policy group
    • Assign the security policy to
  • Option 2 – Security Groups
    • Firewall rule allowing ssh between source: assigned policy group and destination: assigned policy group
    • Firewall rule blocking any between source: assigned policy group and destination: assigned policy group
    • Assign the security policy to and
  • Option 3 – Two rules
    • Rule 1
    • Firewall rule allowing ssh between source: Assigned Policy group and destination:
    • Firewall rule blocking any between source: Assigned Policy group and destination:
    • Assign Policy to
    • Rule 2
    • Firewall rule allowing ssh between source: Assigned Policy group and destination:
    • Firewall rule blocking any between source: Assigned Policy group and destination:
    • Assign Policy to

First question anyone will ask is why would I not use option 2?  It’s smaller and easier to read.  It does accomplish the same goal.   It does lack granularity in design.  What if you had a third subnet and you only wanted it to access  Option 1 would easily be able to do this, while option 2 would mistakenly open up   This is the heart of firewall design.  Layer rules to create granularity.    I am not a master of the firewall but I do have a few suggestions:

  • Outbound firewall rules sound great but right away will kill you in complexity
  • Protect the end points… apply rules to the destination (think apply rules to the web server instead of every PC)  If you need to apply source rules do it on the destination
  • Use naming conventions that describe the purpose of the rule  Allow-SSH-Into-Production
  • Consider using a DROP all on your default rule and then applying only allow rules in security groups
  • Rules that are part of the default and not created in service composer don’t show up in the GUI so don’t use them beyond the default DROP apply everything as a security policy


Let’s do Option 1

  • Return to networking and security and select service composer
  • Select security groups and create a security group for each IP Set



  • Repeat for the other subnet
  • Click on security policies
  • Create a new policy as shown belowCapture






  • Now that you have it build your just need to apply it to a security group
  • Click on the text of your Security Policy
  • Select Manage -> Security Groups
  • Click edit and add

Now your rules should work.  You can test with ping and SSH.   Using the same dialog’s you can create option 2 or 3.   The same rules you use for firewalls on physical entities need to apply to DFW.   You need to think before you create or you will be in firewall spawl.

Deep Dive: How does NSX Distributed Firewall work

This is a continuation of my posts on NSX features you can find other posts on the Deep Dive page.   My favorite feature of VMware NSX is the Distributed firewall.   It provides some long over due security features.  At one time I worked in an environment where we wanted to ensure that every type of traffic was filtered with a firewall.   This was an attempt to increase security.  We wanted to ensure that there was no east <-> west traffic between hosts; so everyone was in its own subnet.  Each virtual machine was deployed inside a /27 subnet alone.   Every communication required a trip to the firewall which was also serving as a router.

LunchThis model worked but made us very firewall centric.  Everything required multiple firewall changes.  Basic provisioning took weeks because of the constant need for more firewall changes.   In addition we wanted secondary controls so each host ran their own host based firewall as well.   This model caused a few major design constraints: you had to buy larger firewalls to handle all the routing and you had to take your firewall guys to lunch all the time to avoid mega rage.

Enter the distributed firewall

The distributed firewall applies firewall rules at the virtual machine kernel and network interface right above the guest OS.  This has a few advantages:

  • No one on the OS can change firewall rules
  • Only traffic that should be on the network is on the network everything else gets blocked before leaving the virtual machine (Think mega cost savings, and less garbage traffic)
  • You can inspect each packet before it gets to the network and take action (lots of third-party plugins will be able to do this)
  • You can scale out your firewalls capacity by adding more hosts in a modular fashion that matched your server growth

The firewall has a api for third-party solutions like virus scanners or IDS.   This allows them to be part of the data stream in real-time.

Components of Distributed firewall (DFW)

The DFW has a management plane, control plane and data plane which should be familiar to network admins.

  • Management Plane – is Done via vCenter plugin or API access to the NSX manager – This allows you to use any vCenter object as the source or destination (Datacenter, VM name, vNic etc..) It also allows you to define IP ranges for more traditional firewalls between IP’s
  • Control Plane – is done by the NSX manager it takes changes from vCenter and stores them in a central database and then pushes the rules down to each ESXi host.  (Database is /etc/vmware/vsfwd/vsipfw_ruleset.dat on each ESXi host)
  • Data Plane – ESXi hosts are the data plane doing the actual work of the firewall.  All firewall functions take place in kernel modules on the ESXi hosts.  Remember that enforcement is done locally and at the destination reducing the traffic on the wire.

Each vNIC get its own instance of DFW put into place and managed by a set of daemons called vsfwd.

How does it work?

Each firewall rule is created and applied via the NSX manager GUI or API.   When published it pushes all rules down to each ESXi host.  They create a file on disk which holds the all the firewall rules.   The ESXi host applies rules to the instance of DFW when a change in vCenter (remember management plane – like a new vNic vlan change etc..) happens the firewall rules are re-consulted.  IP-based rules require VMware tools to identify the IP address / addresses of the server.

How about vMotion?

Since the rules are applied to the virtual container they are moved with the host when vMotion is used, no effect.

How about HA events?

Rules are loaded off disk and applied to virtual machines.

What about if NSX Manager is not available?

Rules are loaded off disk. New systems will get the rule set that apply to them, for example if my new server is called Web-Machine12 and I have rules that are applied to all vm’s named Web-* then it will get them from disk.  This entourages the use of naming standards.

How about if I create a new virtual machines and it does not have any rules?

At the bottom is a default rule (some vote for allow all other deny all, I vote deny all) so you machine will have deny all.

Group and Policies

DFW has the concept of Security Groups (yep like it sounds) groups of similar systems, these can be hard-coded to specific entities or dynamic using regular expresses on any vCenter entity.   They also have security policies these are groups of like-minded rules to be processes in order.   So you define the scope of the rules in the Security Groups and define what is done in Security policies.  It can be a one to many reference on both sides.  (A security group can have many policies or a policy can have may groups) providing the ability to layer rules.

How do I track my firewall drops / accepts?

This is the first thing your firewall guys are going to ask for…  And I don’t like the answer right now.  They are logged to the ESXi hosts syslog.   So you need to centralize your host logs and do some searches to gather the firewalls into one place.   If you search your host based logs for “vsip_pkt” (In 6.1 they changed this to dfwpktlogs:) you will find the firewall drops / accepts.


Deep Dive: How does NSX Distributed routing work

As a continuation of my previous host How does NSX virtual switch work I am now writing about how the routing works.   I should thank Ron Flax and Elver Sena for walking through this process with me.   Ron is learning about what I mean by knowledge transfer and being very patient with it.   This post will detail how routing works in NSX.  The more I learn about NSX the more it makes sense.  It is really a great product.  My post are 100% about VMware NSX not multi-hypervisor NSX.

How does routing work anyways

Must like my last post I think it’s important to understand how routing in a physical environment works.   Simply put when you want to go from one subnet to another subnet (layer 2 segment) you have to use a router.   The router receives a packet and has two choices (some routers have more but this is generic):

  • I know that destination network lives down this path and send it
  • I don’t know that destination network and forward it out my default gateway

IP packets continue to forward upward until a router knows how to deliver the IP packet.  It all sounds pretty simple.

Standardized Exterior Gateway protocols

It would be simple if someone placed a IP subnet at a location and never changed it.  We could all learn a static route to the location and never have it change.   Think of the internet like a freeway.  Every so often we need to do road construction this may cause your journey to take longer using an alternate route but you will still get there.  I am afraid that the world of IT is constant change.  So protocols were created to dynamically update router of these changes standardized exterior gateway protocols were born. (BGP, OSPF etc..) I will not go into these protocols because I have a limited understanding and because it’s not relevant for the topic today (It will be relevant later).   It’s important to understand that routes can change and there is an orderly way of updating (think DNS for routing.. sorta).


A key component of routing is the internet protocol.  This is a unique address that we all use every single day.   There are public ip address and internal IP addresses.  NSX can use either and has solutions for bridging both.  This article will use two subnets and   The /24 after the IP address denotes the size of the range in cidr notation.  For this article it’s enough to denote that these ranges are on different layer 2 segments and normally cannot talk to each other without a router.


We are going to place these two networks on different VXLAN backed network interfaces (VNI’s) as shown below:


If you are struggling with the term VNI just replace with VLAN and it’s about the same thing.  (Read more about the differences in my last post).   In this diagram we see that each virtual machine is connected to its own layer 2 segment and will not be able to talk to each other or anything else.  We could deploy another host into VNI 5000 with the address and they would be able to talk using NSX switching but no crossing from VNI 5000 to VNI 5001 will be allowed.  In a physical environment a router would be required to allow this communication, in NSX this is also true:


Both of the networks shown would set their default gateway to be the router.  Notice the use of distributed router.  In a physical environment this would be a single router or a cluster.  NSX uses a distributed router.  It’s capability scales up as you scale up your environment, each time you add a server you get more routing capacity.

Where does the distributed router live?

This was a challenge for me when I first started working with NSX, I thought everything was a virtual machine.   The NSX vSwitch is really just a code extension of the dVS.  This is also true of the router.  the hypervisor kernel does the router with mininal physical overhead.  This provides an optimal path for data, if the data is on the same machine communication never leaves the machine (much like switching in normal vss).  The data plane for the router lives on the dVS.    There are a number of components to consider:

  • Distributed router – code that lives on each ESXi host as part of the dVS that handles routing.
  • NSX Routing Control VM – this virtual machine that controls aspects of routing (such as BGP peering)  it is in the control plane not data plane (in order words it is not required to do routing) (Design Tip: You can make it highly available by clicking the HA button at anytime, this will create another vm with a anti-affinity rule)
  • NSX Control cluster – This is the control cluster mentioned in my last post.   It syncs configuration between the management and control plane elements to the data plane.

How does NSX routing work?

Here is the really neat part.  A routers job to deliver IP packets.  It is not concerned if the packets should be delivered it just fings IP’s to their destination.   So let’s go through a basic routing situation in NSX.  Assume that Windows virtual machine wants to talk to Linux virtual machine.



The process is like this:

  1. The L3 Local router becomes aware of each virtual machine as it talks out and updates the control cluster with arp entry including VNI and ESXi Node
  2. The control cluster updates all members of the same transport zone so everyone knows the arp entries
  3. Windows virtual machine wants to visit the website on Linux so it arps
  4. ESXi1’s DLR (Distributed Logical Router) returns its own mac address
  5. Windows sends a packet to ESXi1’s LDR
  6. Local LDR knows that Linux is on VNI 5001 so it routes the packet to the local VNI 5001 on ESXi1
  7. Switch on ESXi1 knows that Linux lives on ESXi2 so it sends the packet to VTEP1
  8. VTEP1 sends the packet to VTEP2
  9. VTEP2 drops the packet into VNI 5001 and Linux gets the message

It really makes sense if you think about it.  It works just like any router or switch you have mostly ever used.  You just have to get used to the distributed nature.   The greatest strength of NSX is the ability to handle everything locally.   If Linux was on the same ESXi host then the packet would never leave ESXi1 to get to Linux.

What is the MAC address and IP address of the DLR?

Here is where the fun begins.   It is the same on each host:


Yep it’s not a typo each router is seen as the default gateway for each VNI.  Since the layer 2 networking is done over VXLAN (via VTEP) each local router can have the same IP address and mac address.  The kernel code knows to route it locally and it all works.   This does present one problem: External access.

External Access

In order for your network to be accessible via external networks the DLR has to present the default gateway outside, but if each instance has the same IP / Mac who responds to requests to route traffic?   Once instance gets elected as the designated instance (DI) and answers all questions.   If a message needs to be sent to another ESXi host than the one running the DI then it routes like above.   It’s a simple but great process that works.

Network Isolation

What if your designated instance becomes isolated?  There is an internal heartbeat that if not responded to will cause a new DI election to happen.   What about if networking fails on my ESXi host?  Well then every other instance will continue to communicate with everyone else, packets destined for the failed host will fail.

Failure of the control cluster

What about if the control cluster fails?   Well since all the routing is distributed and held locally everything will continue to operate.  Any new elements in the virtual world may fail but everything else will be good.  It’s a good idea to ensure that you have enough control clusters and redundancy as they are a critical component of the control plane.

Deep Dive: How does the NSX vSwitch Work

Edit: Thank to Ron Flax, Todd Craw for helping me correct some errors.

I have been blessed of late to be involved in some VMware NSX deployments and I am really excited about the technology.   I am by no means a master of NSX but I will post about my understand as a method to spread information and assist with my personal learning.   In this post I will be covering only the switch capabilities of NSX.


Traditional Switches

The key element of a layer 2 ethernet switch is the MAC address.  This is a unique (perhaps)  identifier on a network card.  Each network adapter should have a unique address.   A traditional physical switch learns the mac addresses connected on each port when the network device first tries to communicate.  For example:


When you power on Windows Physical server the physical switch learns that MAC 00:00:00:00:01:01 is connected to port 1.  Any messaged destined for 00:00:00:00:01:01 should be sent to port 1.   This allows the switch to create logical connections between ports and limit the amount of wasted traffic.   This entry in the switches MAC table (sometimes called a cam table) stays present for 5 minutes (user configurable)  and is refreshed whenever the server uses it’s network card.   The Linux server on port two is discovered exactly the same way via physically talking on the port, the table is updated for port 2.   If Windows wants to talk to linux their communication never leaves the switch as long as they are in the same subnet.   If the MAC address is unknown by the switch it will broadcast the request out all ports with hopes that something will respond.

Address Resolution Protocol (ARP)

ARP is a protocol used to resolve IP addressed to their MAC addresses.  It is critical to understand that ARP does not return the MAC address of the final destination it only returns the mac address of the next hop unless the final destination is on the same subnet.  This is because ethernet is only concerned with next hop via mac not end destination.


You can follow the communication with ARP’s between each layer of the diagram the key component is that if the IP is not local then it returns its own MAC and forwards it out the default gateway.

Traditional Virtual Switches

In order to understand NSX vSwitch it is critical that you understand how the traditional virtual switch works.  In a traditional virtual switch (VSS and dVS) the switch learns the mac addresses of virtual machines when they are powered on.  As soon as a virtual machine is assigned a switch port it becomes hard-coded in the MAC table for that virtual switch.   Anything that is local to that switch in the same vlan or segment will be delivered locally.    Otherwise the virtual switch just forwards the message out it’s uplink and allows the physical switches to resolve the connection.

NSX Virtual Switch

The NSX virtual switch includes additional functionality from the traditional virtual switch.  The key feature is the ability to use VXLAN to span layer 2 segments between hosts without the use of multiple streched VLAN’s.   VXLAN also allows strech layer 2 to distant datacenters and up to 16 million segements vs the current limit of 4096 vlans.  There are some common components that need to be understood:

  • VTEP (VXLAN Tunnel End Point)  – this is a ESXi virtual adapter that has its own vlan and ip address including gateway.  This interface must be set for 1600 MTU and all physical switches/routers that handle this traffic must allow at least 1600 MTU.
  • NSX virtual switch (also called logical switch) – This is a software kernel based construct that does the heavy lifting. This is deployed to a dVS switch and works as extensions to the dVS.
  • NSX Manager – This is the management plane for NSX, it acts as a central point for communication, scripting and control.  It is only required when making changes as part of the management plane
  • NSX Control cluster – This is a series of virtual machines that are clustered via software.  Each node (should be a odd number and at least three)  contains all required information and load is distributed between all three.  (Best Practice: Do a DRS rule to keep these on separate hosts, future releases may do this for you)
  • VNI – Virtual network interface – this is an identifier used by VXLAN to separate networks (think vlan tag) they start at 5000 and go to 16,000,000.  It easiest for people to think vlan tags when working with VNI’s.

With all the terminology out-of-the-way it’s time to get down to the path.   The NSX Virtual switch includes one key component the ability to switch packets between nodes or clusters without having the layer 2 streched between the clusters.  For my networking friends this means reduction in spanning tree issues.

So let me lay it out below:


We have a three node NSX control cluster that has been deployed.  We have two ESXi hosts running dVS’s with the NSX Virtual switch.  VXLAN has been enabled and a virtual network VNI:5000 has been created.   The VTEP’s have been configured.   We have created two virtual machine as shown in green.  Neither has been connected to the VNI network yet.


Time to learn our first MAC:

  • We connect the Windows server to VNI:5000 as shown below
  • The MAC table on our local switch is updated (Learns) then passes it’s learned information to the control cluster
  • The control cluster passes it to all members of the logical switch (there are three methods to pass the information which I will cover in another post unicast, multicast and hybrid)



This syncing of the MAC table ensures that each member of VNI knows how to handle switching creating a distributed switch (like a switch stack that has multiple switches that act as one).

When we power on the linux server the same method is used:

  • We connect the Windows server to VNI:5000 as shown below
  • The MAC table on our local switch is updated (Learns) then passes it’s learned information to the control cluster
  • The control cluster passes it to all members of the logical switch (there are three methods to pass the information which I will cover in another post unicast, multicast and hybrid)


Now we have a ARP table available on each switch that works great.   Let’s follow the flow of communication: Assume the following.   Windows server wants to open a web page on Linux server on port 80:

  • User on Windows server brings up internet explorer and types in
  • Windows server sends out a arp entry for
  • ESXi1 ‘s virtual switch returns the MAC address 00:00:00:00:02:02
  • Windows server sends out a IP packet with the MAC address of 00:00:00:00:02:02
  • ESXi’s virtual switch forwards the packet out VTEP1 by encapsulating it destined for the IP of VTEP2
  • VTEP2 opens the packet and removes the VTEP encapsulation and forwards the packet to ESXi2 virtual switch on VNI:5000
  • The switch on ESXi2 sends the packet to the virtual port that the linux servers network card is connected on.


This is how a NSX virtual switch handles switching.  At first you may say this makes no sense at all… wouldn’t a VLAN just be easier.   There are a number of benefits this brings:

  • Limits your Spanning tree to potentially top of rack switches if architected correctly
  • Allows you to expand past the 4096 VLAN limit
  • Opens the door for other NSX services (which I will post about in the future.)


As I mentioned this is just my understanding I do not have inside knowledge if I have made a mistake let me know, I’ll test then correct it.

Central Ohio VMware Lunch and Learn

I have been toying with the idea of starting a community series of lunch and learn sessions to assist people in learning about VMware technology.   I am happy to announce that the first session will be Sep. 25th at Noon at:


OARnet – Bale Conference Room

1224 Kinnear Road

Columbus, OH 43212


It was very kind of my previous employer to be willing to host us for these sessions.   I am excited to announce that VMware education has also provided some certification discount codes for me to pass out.   The format will be a bit loose.  I will be focusing on VCP content but it will be open to discussion.   I want it to be a forum.   I have also invited others to present in the future and hope to make it a monthly occurrence.   The topic for this month with be vSphere networking.  It will be a great refresher course for anyone looking to study for the VCP-NV.    The event it 100% open to the public and we have seating for about 60 people.  There is standing room for about 40 more.   Bring your lunch and join us.  Feel free to contact me via comments or twitter if you have questions or would like to present a future topic.  The one request I have is this is a technical conversation not a sales pitch.  I want it to be a discussion between technical people.


Looking forward to seeing you there.

How do I explain virtualization to my Mother

IMG_20140730_174059As I have progressed in my career it’s been increasing hard to explain my job to people both inside and outside IT.    There used to be a time when people in IT understood what I did… at this point most people really don’t understand what I do or why.   I have given up explaining it to people I just say I work with computers.    Two years ago while at VMworld the crew from VMware TV stopped me on the street and asked how do you explain virtualization to your mother.   They totally stumped me.   I am lucky my mother has some technology in her life.   She recently got a nook and has discovered she can get books without leaving the house.  For a woman in her 70’s she is about as technically savvy as I can expect.    My religious studies have taught me that analogies can be a great way to teach.   So I present my analogy to explain virtualization.

The Apartment building

Imagine with me that I have just bought a 30,000 square foot housing space.   As the owner I could rent out this space to a single four person family.   They would be very happy and have more space than they could ever use.   It does present some critical problems.  The family would have to be very rich in order to pay for my whole building.   There is no way they could possibly use all the space so there would be lots of wasted space.   In my case if the one family moved out I would have a huge expense that I would have to shoulder until I found another rich family who wanted 30,000 square feet.   I have other issues unless I was very handy I would have to hire someone to fix and repair the apartment when things broke.  This is an expense that is wasted when no one is living in the apartment.   The cost for heating, cooling and powering the apartment would be a huge expense that I would pass on to my single family.  At this point the power bill alone might force the family to move out, once again leaving me to shoulder the whole bill.   In reality running a 30,000 square foot apartment building with a single tenant is a huge risk.  In some neighborhoods it’s totally possible to rent out a space like this to a single family and make a huge profit, either because money is no object or they have some requirement that offsets the costs (like a home office).

The subdivided apartment

I prefer investments with less risk.    After some examination I have discovered that in the neighborhood there is a demand for one, two and three bedroom apartments.   Each type of apartment has some common components: bathroom, a living room and a kitchen.   I create three standard configurations and start to subdivide my building into separate living spaces.  Some of my living space is lost to overhead like hallways and doors.  There are some shared area which represent a space saver for example stairs, elevators and laundry rooms.   Making some area’s shared reduces the lost space to overhead.    I may even consider putting in a pool on the roof to increase the price of my apartments individual rent and increase my profit.   Each of the apartments have their own plumbing with sinks, toilets and showers.   Once these shared components leave your individual space they join the building plumbing and water and utilize shared resources.   It’s important that I take into account the total amount of possible shared utilization at the same time to avoid loss of individual services.   After all if everyone flushes their toilet at 5:00 PM I cannot have the pipes get stuck.   I have to be careful that the individual actions of a single tenant cannot create a failure for all other tenants.   This is one of the key reasons why each apartment has their own water heater, we never want the actions of a single bad neighbor to affect everyone else’s experience.

What does the apartment have to do with virtualization

Virtualization is very much like the apartment.   I have a large computer.  Most of the time it’s 30,000 square feet is about 2% utilized.   If I engineered the correct solution I could utilize the other 98% of wasted space.   Much like humans my applications don’t like to live in the same space.   Virtualization creates separate apartments for each service, these virtual apartments have some shared components and some individual components.   For example I may have shared network connections, power, even portions of memory (hallways) and shared storage (laundry room)  while I have my own water heater (reservation/allocation of resources).  I may have a flash cache on my server (pool on the room) to improve the amenities and encourage higher rent.    All of this is done in a fashion to protect the security of individual families and homes (hypervisor security).   Virtualization has to take into account peak usage to avoid having the pipes filled with you know what at 5:00 PM.   Much like my apartment I need to hire systems administrators to provide care and feeding to my virtualization, the more apartments I deploy the better my cost savings in theory (Yes I know there is diminishing returns when I need more workers)

What does virtualization not have to do with an apartment building

Virtualization brings a few key differences to the table over my apartment building.   It is very costly for me to reconfigure my available space into larger to smaller apartments to fulfill demand, virtualization can do this on demand.   If my apartment burns to the ground due to faulty wiring my families cannot be moved within minutes to another apartment nearby with their furniture and home goods intact.  Virtualization can do that.

Key elements

  • Virtualization is like an apartment building created to make efficient use of large wasted space
  • Virtualization has overhead due to shared components but the overhead uses what would be wasted space so it’s a net gain in most situations
  • Virtualization has limits on shared components and should be sized correctly (no full pipes at 5:00PM)
  • Virtualization is better than single homes in almost every way except one: It is still a shared resource and bad neighbors can still make it unlivable

Death of the sysadmin and birth of….


I started my career as a sysadmin..  I didn’t want to spend all day sitting in a chair writing applications.  I wanted to touch the hardware.   I am a firm believer that every sysadmin is a control freak to some respect.  They love how the machine obeys them.   They enjoy telling users no you cannot and figuring out ways limit access.  The essence of every good sysadmin is the innate need for improvement.   In the early days of my career I was exposed to systems administrators who had hundreds of shell scripts they had everything automated.   As the years past these older sysadmins seem to be replaced with younger admins who had been raised in an easy world.   They were used to clicking next to install applications and things that just work… (hence the appeal of the iPhone).   I am all for simple and easy gadgets and bringing computers to every old persons life.   The relative ease of the solutions have made life a little too easy for us.


Then this crazy thing happened…. the cloud.   Amazon brought the easy button to server deployments.   Some embraced the ease of the solution, others liked the agility.   What ever your motivation for using AWS they have changed IT again.  Everywhere I go business units want to know why it takes so long to deploy a server.   They want to know how to create their own AWS cloud.   People every where have been deploying operating systems and getting IT done without systems admins..

The Auto Industry

When the auto industry first started Henry Ford and his engineers would assemble a car from scratch.  Everyone working on the car understand each component and how it worked.  They understood the flow of assembly to make the car.   Each of the assembly guys could build a car from scratch or design a car.   As time past demand increased for the product and Ford had to increase his agility to create cars.   He hired workers to build cars, assigned them specific roles with rote tasks.   These workers would do the same task over and over again.  This provided a few advantages: first they got good at the task and they did not need to know how to build a whole car.   It also introduced some challenges: if they missed their task due to human error failures were introduced.   Eventually humans workers were replaced with robotics.  This reduced the errors and increased the cost.  It also allowed Ford to build a lot more cars.  Not all the jobs went away they just changed.  Workers were replaced with robotics and automation engineers.   The people working on the cars had no idea how to build cars they just kept the robotics working.   Every other car manufacturer followed suite to compete.   Auto manufacturing plants became huge, downtime cost millions of dollars.   Massive amounts of money are spent to ensure the plants keep running.

What lessons can we learn from Auto Industry

  • Having highly skilled humans build the cars worked great
  • Having architects design cars then hand off work instructions to workers introduced a lot of errors
  • Having automation reduces errors and requires workers with a new skill set
  • It is not required that the people keeping the automation running have an understanding of the product, they just need to understand the automation
  • The cost of automation will force a centralization of building cars
  • As manufacturing became centralized downtime became a critical issue

What does this have to do with sysadmins?

Thanks for bearing with me this far.  If your still reading and wondering why I wrote this article let me explain.  I want to suggest that the world is changing for systems administrators and as control freaks they don’t really like it.   AWS has created a golden standard we can deploy the system in minutes why can’t you?  If you have not faced this question you will soon.   Every shop wants to have AWS.

Every shop wants to have AWS but do they need it?

AWS has a very specific business model.  Deploy base templates for customers then step away and collect cash.   It’s a great model.  Functionality of the virtual product beyond being powered on is 100% your problem.   AWS ensures uptime of power and networking.   You still have to do a lot of work to make that server deployed in minutes usable.   AWS has saved you the time of procurement of hardware and working with silo’ed team to get a server in place, but you don’t have a money-making machine until you install your product.  What is it about AWS that you really need?  I suggest it is not agility instead it’s less hassle.  AWS provides you freedom from people who seem to create never-ending road blocks while doing their job.  Yes, I am looking at you security team.  Yes, I am looking at you server deployment team.  Yes,  I am looking at you…

Why is IT so hard

IT is hard because it’s never the same.  In my career I have rarely seen the same request twice.   If your business is netflix and you have three types of servers then automation makes sense.   Most IT shops are not netflix’s every single business unit wants to drive IT choices and so we get a spaghetti mess of IT.   IT is hard because the business unit wants to drive technical choices instead of business requirements.   Many years ago we had a business unit demand that their new workflow be built in Sharepoint, forget the fact that we were a linux shop with no sharepoint.   So the lesson is:

  • Business unit’s stop messing with IT.  Bring your needs well-defined to IT and let us implement it.  Trust us to do our job it’s why we cost so much.

Why do Menu’s exist

Restaurants have menu’s for the following reasons:

  •  To limit customers options – they could not possibility have all ingredients
  • To help customers make choices –  if left to their own customers would become confused by the options and leave
  • Create standard workflows and realize cost savings
  • Give the customers illusion of choice

Why doesn’t IT have a Menu…. here it comes the ITIL service catalog.   Most service catalogs are too technical and don’t represent what the customer really needs.  What the customer needs is a service which is normally a lot more complex than a single server.  They have a project.


Yep that word again project… it’s so important we have a certification and role who manages it.    Business unit’s rarely want one more netflix streaming server.. they expect IT to handle that if needed.  They want to create a whole new business and that requires a project.   Our menu really needs to be a project menu not a server menu.   We need to stop offering the business unit separate components of our offering or they will keep getting into our business.  We need to provide the business unit the correct choices that keep them away from dictating technology.

Death of a sysadmin … birth of a ..process engineer

So now that I have ranted for too long what is the future of systems administration.   I think we need to become process engineers.   Very few people are going to understand the whole product.  More will administrate from a automation console rather than logging into a server.   How do we re-tool for this change? I have a few suggestions:

  • Learn to examine process.  Do something manually first.  Document the process in extreme detail, use a process diagram. Critically look at your process diagram.  Do you see how many manual processes you have?   How can you automate them.
  • Help customers standardize, learn the language stop jumping to technical solutions with your customers.  Focus on their needs and requirements allow the technology to be a black box.
  • Develop standard methods for documenting and ingesting new projects… create a documented process and follow it.
  • Automate everything you can, develop solution with the automation mind set.  How would I do this if I had to deploy 100 servers instead of two.
  • Ask your self does this process, technology or choice scale up?   If I had to increase the amount of these by 1,000 would this process work.

Well thanks for reading my rant.  Let me know where I am wrong.