Deep Dive: Standard Virtual Switch vs Distributed Virtual Switch

Posted on by Joseph Griffiths

Let the wars begin.  This article will discuss the current state of affairs between virtual switches in ESXi.   I have not included any third party switches because I believe them to becoming quickly not part of the picture with NSX.

 

Whats the deal with these virtual switches?

Virtual switches are a lot like ethernet layer 2 switches.  They have a lot of the same common features.  Both switch types feature the following configurable items:

  • Uplinks – connections from the virtual switch to the outside world – physical network cards
  • Port Groups – groups of virtual ports with similar configuration

In addition both switch types support:

  • Layer 2 traffic handling
  • VLAN segmentation
  • 801.1 Q tagging
  • nic teaming
  • Outbound traffic shaping

So the first question everyone ask’s is if two virtual machines are in the same vlan and on the same server does their communication leave the server?

No… communication between the two vm’s on the same ESXi host can communicate without leaving the switch.

 

Port Groups what are they?

Much like the name suggests port groups are groups of ports..  They can be best described as a number of virtual ports (think physical port 1-10) that are configured the same.  Port groups can have a defined number of ports and expanded at will (like a 24 port switch or 48 port switch)  There are two generic types of port groups:

  • Virtual machine
  • VMkernel

Virtual machine port groups is for guest virtual machines.  VMkernel port groups are for ESXi management functions and storage. The follow are valid uses for VMkernel ports

  • Management Traffic
  • Fault Tolerance Traffic
  • IP based storage
  •  vMotion traffic

You can have one or many port groups for VMkernel but each requires a valid IP address that can reach other VMkernel ports in the cluster.

At time of writing (5.5) the follow maximum’s apply

  • Total switch ports per host: 4096
  • Maximum active ports: 1016
  • Port groups per standard switch:512
  • Port groups per distributed switch: 6500
  • VSS port groups per host: 1000

So as you can see vDS scales a lot higher.

Standard Virtual Switch

The standard switch has one real advantage.  It does not require enterprise plus licensing to use.  It has a lot less features and some draw backs including:

  • No configuration sync – you have to create all port groups exactly the same on each host or lots of things will fail (even upper case vs lower case will cause it to fail)

Where do standard switches make sense?  Small shops with a single port group they make a lot of sense.  If you need to host 10 virtual machine on the same subnet then standard switches will work fine.

Advice

  • Use scripts to deploy switches and keep them in sync to avoid manual errors
  • Always try vMotions between all hosts before after each change to ensure nothing is broken
  • Don’t go complex on your networking design – it will not pay off

Distributed Virtual Switch

Well the distributed virtual switch is a different animal.  It is configured by vCenter and deployed to each ESXi host.  The configuration is in sync.  It has the following additional features:

  • Inbound Traffic Shaping – Throttle incoming traffic to the switch – useful to slow down traffic to a bad neighbor
  • VM network port block – Block the port
  • Private VLAN’s – This feature requires switches that support PVLAN so you can create VLAN’s inbetween vlans
  • Load – Based teaming – Best possible load balancing (another article on this topic later)
  • Network vMotion – Because the dVS is owned by vCenter traffic stats and information can move between hosts when a virtual machine moves… on a standard switch that information is lost with a vMotion
  • Per port policy – dVS allows you to define policy at the port level instead of port group level
  • Link Layer Discoery Protocol – LLDP enables virtual to physical port discovery (your network admins can see info on your virtual switches and you can see network port info – great for troubleshooting and documentation)
  • User defined network i/o control – you can shape outgoing traffic to help avoid starvation
  • Netflow – dVS can output netflow traffic
  • Port Mirroring – ports can be configured to mirror for diagnostic and security purposes

As you can see there are a lot of features on the vDS with two draw backs:

  • Requires enterprise plus licensing
  • Require vCenter to make any changes

The last draw back has provided a number of hybrid solutions over the years.  At this point VMware has created a work around with the empherial port group type and the network recovery features of the console.

Advice in using:

  • Backup your switch with PowerCli (a number of good scripts out there)
  • Don’t go crazy just because you can if you don’t need the feature don’t use it
  • Test your vCenter to confirm you can recover from a failure

So get to the point which one should I use?

Well to take the VCDX model here are the elements of design:

Availability

  • VSS – deployed and defined on each ESXi host no external requirements + for availability
  • dVS – deployed and defined by vCenter and requires it to provision new ports/ port groups – for availability

Manageability

  • VSS – pain to manage in most environments and does not scale with lots of port groups or complex solutions – for manageability
  • dVS – Central management can be deployed to multiple hosts or clusters at the same datacenter + for manageability

Performance

  • VSS – performance is fine no effect on quality
  • dVS – performance is fine no effect on quality other than it can scale up a lot larger

Recoverability

  • VSS – is deployed to each host and stored on each host… if you loose it you have to rebuild from scratch and manually add vm’s to the new switch – for recoverability
  • dVS – is deployed from vCenter and you always have it as long as you have vCenter.  If you loose vCenter you have to start from scratch and cannot add new hosts.  (don’t remove your vCenter it’s a very bad idea) + as long as you have a way to never loose your vCenter (does not exist yet)

Security

  • VSS – Offers basic security features not much more
  • dVS – Wider range of security features + for security

 

End Result:

dVS is better is most ways but costs more money.   If you want to use dVS it might be best to host vCenter on another cluster or ensure it’s availability.

 

 

Deep Dive: vSphere Network Link Failure Settings

Posted on by Joseph Griffiths

In this series of posts I will tackling different topics around vSphere and attempting to explain what they mean and how to use them.  This article will discuss the link fail over detection methods.

 

Link Fail over detection

Link fail over detection is a critical component in any infrastructure this is the method ESXi used to determine if a link has failed and should not be used for traffic.   ESXi provides two options:

  • Link Status
  • Beacon Probing

 

Link Status

Link status is just as it sounds.  The link is either up or down.  This method can detect switch failure or cable failure on the next hop. For example if switch A were to loose power ESXi move move all possible traffic from NIC1 and NIC2 to NIC3 and 4.

Drawing1

 

Link status does have some drawbacks:

  • It cannot detect mis-configuration on the switches or upstream.
  • It cannot detect upstream failures (for example the router attached to each switch)

For these reasons it is critical that you implement some type of link state tracking on your network gear.  A common setup is to configure ports to shutdown when their uplink ports fail.   This type of link state tracking is a function of the switch gear and it’s critical that it be configured all the way to the ESXi ports so ESXi see’s a link failure.   It still cannot overcome the misconfiguration.  This is really bad in situations where MTU is misconfigured upstream.   For this reason VMware implemented a Network health check and can help identify MTU mismatches and VLAN issues.  I would 100% recommend turning it on.  It’s a free health check that can save you hours.

Beacon probing

Beacon probing is a really simple process.  It requires a odd number of network devices.  Each network card sends out a broadcast message.  As each nic receives the other network cards broadcast it knows it is not isolated from the others and assumes good link state.   This process has a number of advantages:

  • Can detect upstream failures
  • Can detect some misconfigurations

It does have a downside

  • Requires at least three network cards for a quorum (2 would vote each other out)
  • Can lead to false positives

I would like to explain the false positives.  There are a number of situations where it would be possible for broadcast message to not reach the destination during these times all links determined as isolated would be shutdown.   You could put your host into a isolation event very quickly all at once.

 

Link State Tracking Choice

This one is 100% up to you.  If you only have two or less network cards use link state.   If you have three or more then you might want to use beacon probing.  Either way test every possible failure scenario for possible issues before depoying in production.

 

Notify Switch of failure

Should you notify the switch of a failure?  I would think this is a good idea.  Without going into a discussion of arps.  This setting chooses to send out gratuitous arp messages after a fail over event.   These messages allow switches to quickly update their arp tables.  Without these updates messages destined for moved virtual machines may take up to five minutes before they get the message.   This is unlikely but possible in complex network configurations.   My vote is always yes… I cannot think of a downside but suggestion one if you know it.

 

Failback:

This setting allows traffic to be moved back to a link after a link state failure is set to yes.  If set to no you have to manually move the traffic flow back.    There are two schools of thought on this matter.  Failback yes creates a automated fail back when outages occur.  Less work is good.  But it’s possible that a link starts flapping and traffic keeps moving back and forth all night between working and failed links… causing availability problems in your environment.  It’s really up to your requirements but I suggest that if you use failback:NO enable a vSphere alarm to let you know so you can re-add the link after the failure is resolved.

 

Radically simple storage design with VMware

Posted on by Joseph Griffiths

Storage is my bread and butter.  I cut my teeth on fine storage arrays from EMC.  Since then I have moved on to many different vendors and I have learned one truth: storage can be hard or simple.   VMware can make storage easy.   I am very excited about SDS (software defined storage)  I personally love VSAN and Nutanix they are the commercial solution to something google figured out long ago.   Storage is simple but storage arrays are hard.   VMware has been making great strides to simply storage but I find lots of people are afraid to use them.   They prefer to stick non-flexable arrays and provisioning methods.   Please don’t get me wrong these designs are required for some solutions.  Some transnational processing requires insane IOP’s or low latency.  This design is for the rest of you.

Design Overview:

You have a VMware cluster with highly available shared storage.   You have a mixed VMware cluster running lots of different applications.  Some of your virtual machines have lots of drives spread all over your storage luns.   Some of your virtual machines have 2TB drives attached so you have standardized on 4TB lun’s for all VMFS datastores.  All of your luns are thin provisioned.  You need to provide a solution that is easy to manage but avoids lun’s running out of disk space in the middle of the night.  You are also concerned about performance you would love an automated way to move virtual machines if I/O on a lun is a problem.

Assumptions:

The following assumptions have been made:

  • You have enterprise plus licensing
  • You are running 5.5 and all VMFS luns are at least 5.XX format native
  • You do not have an array that provides auto tiering
  • You do not need to take into account path selection in the process or physical array

 

Storage:

VMware’s Storage cluster provides for all the requirements and needs.  By using all storage in a storage cluster management of storage becomes easy.  Just group storage together based on IO metrics (do not mix 15,000 disks with 7,200 k disks)  into a pool or datastore cluster.  Enable storage DRS and your life just got a lot easier.  Enable automated storage DRS for ease of management.   This will help you place new virtual machine and move virtual machines off luns that are above a certain threshold (80%) by default.   Now you just need to enable IO latency moves.  This will move virtual machines to other datastores if the latency on the datastore passes a threshold (default 10ms) for a specific duration.   I have used storage DRS just like this with over 2,000,000 successful storage moves without a single outage.

 

Abstract storage -> Pool -> Automate

 

All are provided by this design.

PowerCLI Change vSphere Alarms to send emails

Posted on by Joseph Griffiths

vSphere 5 comes with a slew of really great alarms prebuilt by VMWare.   There is only one problem by default they all just alarm in the GUI.  I really want to avoid having someone login to find out status.  One organization I worked for had a ticketing system that accepted input as emails.  It was not the cleanest but it worked.  So here are some generic alarms that I recommend switching to email notifications.

 

“Datastore usage on disk”
“Health status changed Alarm”
“Health status monitoring”
“Insufficient vSphere HA failover resources”
“VMKernel NIC not configured correctly”
“vSphere HA failover in progress”
“vSphere HA host status”

 

Yes you can manually change them to send emails in the GUI… but it’s a lot quicker to use PowerCLI.  Here are the commands:

 

Get-AlarmDefinition "Datastore usage on disk" | New-AlarmAction -Email -To “your_email(Replace this parenthesis with the @ sign)domain.com” -Subject "Datastore usage on disk" 
Get-AlarmDefinition "Health status changed Alarm" | New-AlarmAction -Email -To “your_email(Replace this parenthesis with the @ sign)domain.com” -Subject "Health status changed Alarm" 
Get-AlarmDefinition "Health status monitoring" | New-AlarmAction -Email -To “your_email(Replace this parenthesis with the @ sign)domain.com” -Subject "Health status monitoring" 
Get-AlarmDefinition "Insufficient vSphere HA failover resources" | New-AlarmAction -Email -To “your_email(Replace this parenthesis with the @ sign)domain.com” -Subject "Insufficient vSphere HA failover resources"
Get-AlarmDefinition "VMKernel NIC not configured correctly" | New-AlarmAction -Email -To “your_email(Replace this parenthesis with the @ sign)domain.com” -Subject "VMKernel NIC not configured correctly"
Get-AlarmDefinition "vSphere HA failover in progress" | New-AlarmAction -Email -To “your_email(Replace this parenthesis with the @ sign)domain.com” -Subject "vSphere HA failover in progress"
Get-AlarmDefinition "vSphere HA host status" | New-AlarmAction -Email -To “your_email(Replace this parenthesis with the @ sign)domain.com” -Subject "vSphere HA host status"

 

 

VMware Certified Design Expert (VCDX) Journey

Posted on by Joseph Griffiths

bestSome of my normal readers will have noticed that I have been missing from the blog of late.  My family has been noticing me missing in life for the last six months.  This is all due to my goal to become a VMware certified design expert.  It’s a certification created by VMware to identify individuals who are IT architects.  It’s a design focused certification instead of a technical one.   It requires that you create a datacenter or multi-datacenter design and submit it.  Once the submission is accepted you have a opportunity to defend it before a group of current VCDX’s for a chance at becoming a VCDX.   On Thursday exactly one day after defending my design I learned that I passed the defence.  I am now VCDX #143.  You can read about the VCDX program here.  The number gives you an idea of how few people hold the certification.  You can see the full list here.   I would like to share some things about my journey in order to help and encourage others to join me.  If you don’t want to read my story skip to the bottom I have some bullet point tips to help you.

 

A year ago I had an experience that had a profound effect on my career path.  My ememployer kindly sent me to VMworld.  At the time I was the primary VMware systems administrator as well as being the lead for all systems administration.  As I attended the conference my eye were opened to all the future possibilities.  It was the year of automation and cloud computing 2012.  During the sessions I was able to interact with a lot of the virtualization personalities and the attendees.  I found a group of kindred spirits and an excitement and passion for IT that I had lost in the last few years.  I came home excited my parents game me a ride home from the airport and mention multiple times how excited I was..  For those who know me excited is not a term normally associated with me.  At this point I was a VMware Certified Professional (VCP) 5 that my employer had encourage me to obtain.   I came back with the desire to learn more about VMware products.

I have always learned more while on a project. I have to have a deadline or life will get in the way. So I set a goal along with my employers support (they were awesome) that I would become a VMware Certified Advanced professional (VCAP) by the next VMworld. Turns out the goal was way to far out.. and I did nothing for a long time. A few months before VMworld 2013 I really started hitting the VCAP-DCA (DCA Data Center Administration) content. I was determined to install and play with every VMware features which as it turns out is a great way to prepare for this exam. I had also been working with my employer to install and setup vCloud director in order to become a service provider of IaaS. This allowed me to get a  VCP-Cloud certification. When it was posted that exams would be 75% off at VMworld I now had a concrete goal to achieve. I took both the VCAP-DCA and the IaaS test during the general sessions at VMworld 2013.. I managed to pass both. Now realizing that certifications provide the solid ground to force me to learn I looked at the certification track for my next learning path. VCAP-DCD (Data Center Design) seemed like the most logical next step. After VMworld they offered a coupon for 50% off certifications if done within two months. I had my goal and bought a test. I read and studied a lot for this test. I have done a lot of architecting but in education we didn’t always use the same terms. I found the VCAP-DCD book from VMware press the greatest help.  The night before the test I came down with a 103 fever.. but unwilling to waste my money and Pearson’s inflexible reschedule policy in my way I took the test. It went by quickly and I was faced with a score of 297. Passing is a 300. So exactly two weeks later without a fever I passed the test. Now I was faced with the VCDX-DV certification.  I bloged about my experiences with both exams in other posts.

 

VCDX-DCV the Journey

I started on my design in Dec 2013. This was after securing this goal with my wife, family and employer. Through this process my employer has been very supportive which is critical. If you are an employer wondering if your employee with benefit from this certification, the answer is a simple yes. It’s cheaper than a week of training at $1200 and trust me I learned more about vSphere and architecture during the design than I have in the last three years combined.

The design
I started with a fictional design.. based in VSAN. I was going for the wow factor. I spend most of the month playing with VSAN and reading every word about VSAN. In January as I struggled with the fictional design around business requirements I gave it up. I went with a real world design. If you have not read it somewhere else read it here… don’t do 100% fictional. Do something you know. VCDX is not about the wow factor it’s about meeting a customers requirements and constrains in your design. It’s not about the perfect design. It’s about meeting the customers needs. You may have to add some portion of fiction to your design to meet the VCDX requirements (included in the blueprint – read it over and over it’s the greatest clue to what they want)
I did have to add some fictional elements in order to show mastery of all the elements of design but I kept them limited. Remember to include business requirements: SLA’s, RTO, RPO and define them in detail. Once I had an idea around my design, I needed to figure out the format. I again struggled with the format to use. My day to day job did not use anything as formal as was requested by the design submission document. I used several publicly available documents and started to pattern a document but I still struggled to find the correct format.

Solutions Enablement toolkit

I read in multiple places that a lot of the people who submitted used the solutions enablement toolkit from VMware as a template. This tookit is only available to partners. This posed a problem. I worked for a partner on a enterprise licensing agreement but they did not have the required partner level to access the kit. In order to get the partner level I had to pass a whole bunch of pre-sales and post sales certifications. So after two days of certifications and online learning (thank goodness the tests are free to partners) I was able to access the SET for vSphere. This document really helped me and partners really have a leg up with these documents. I did not use them as templates beyond understanding which each document type should contain.  If you are not with a partner then I recommending using this document as a template.  With a lot more detail.  Remember that everything should align with the elements of design.  (RAMPS, Recoverability, Availability, Manageability, Performance, Security)  Make sure you justify each design choice against these elements and understand the effect of each design choice positive and negative against these elements.

Adjusted Expectations

So fast forward we are now at Lat February. My goal was to get VCDX-DCV at VMworld 2014 but they didn’t have any defenses scheduled for VMworld. I have a choice between Cambridge MA in July or late fall in Palo Alto. Since I had some question if my employer would pay for the trip to defend I went for Cambridge. This gave me three months to complete the design by May 9th.   I feel it’s critical to have balance in your life.  I did not have any all night design documents writing sessions.  I did however spend about one hour after work and two hours after my kids went to bed each night for three months.  I took three days off work and it really did occupy a lot of my time and thoughts.  I took the 8th off work and submitted my design by 5 PM.   They were kind enough to let me know they got my documents.  At this point I had invested about 600-700 hours in the process.  Most of this time was spend figuring out the format I wanted to use for my documents.   I changed my mind a lot and reformed the document multiple times.    The real document only took about 100 hours plus some proof reading.

Perfect Document

After providing my perfect document (about 600 pages in total) I went on vacation with my family.  A carefully crafted beach visit to Lake Michigan to end on the day I would hear back if I get to defend.   I was happy to hear that I was invited to defend and terrified that I now had to prepare to be judged.  I started reviewing my design document to create slides… and there were errors… things I had missed and new problems I discovered.  Which goes to show you no design is perfect and it’s ok to have errors just let them know you changed something and why … and of course what effect that change had to the elements of design.

Slides and presentation

Putting together a slide deck I stuck to the recommendations in the VCDX book.   I did not do anything fancy.  I practices my presentation a few times but not a ton… I mostly studied VMware books (HA and DRS deep dive etc..) and my design.

Defense

I am not allowed to talk a huge amount about the defense.  What I can tell you is I really enjoyed spending time with the other VCDX’s explaining my design.   It was just like VMworld all over again… I had a great time.   They asked great questions and helped me to think critically about my design choices.  I spend two years as a missionary going door to door which perhaps prepared me for this experience.  After eight hours every day for two years of having people reject your message you get used to rejection and different opinions.   It was great practice for this experience and life as a whole.    My panelists were awesome and helped me a lot.

Ok … Ok enough about me here is the tips

  • Get lots of practice talking about VMware in public speaking situations (Your local VMUG is awesome for this)  My local VMUG allowed me to present  three times in the last year… it was great experience.
  • Get lots of practice public speaking… in your church community or something somewhere…. practice and push yourself out of that comfortable skin
  • Study to learn… figure out how your learn… if it’s writing after you learn start a blog… if it’s teaching others start a lunch and learn session at your work… Figure out your learning style and start using it on VMware products
  • Set deadlines and don’t miss them… also don’t give up with failure.  The only real way to fail is to not try or stop trying.  You can do it… your a smart person
  • Reach out to people in twitter… (yes I said the evil word.)  #VCDX find local resources that are studying and join a VCDX group
  • Focus on the business side of it all… I bet your great at technical but it’s time to understand the business… they are why we exist.  Find out what the business wants and why.  Learn the business lingo and terms.
  • Don’t make a design choice just because…. know why you make the choice and the consequences then stick by your design.   Don’t let the first stiff wind blow you down.
  • Use a white board and pictures to explain your thoughts…  a picture is really worth a million words.  Learn to use Visio or some application like it…  Draw pictures to explain everything.    Trust me use pictures…
  • Conflicts are your friend.  Every design has conflicts and risks… if your does not it’s not real.  Conflicts just show critical thinking.
  • Use RAMPS like crazy.   Think about it, drink it, and live it…
  • VCDX is about becoming not achieving – It’s not a one time event it’s a life long journey.   My journey to VCDX did not really begin in January and it did not end in July it continues.  Life should be about becoming something..  How can you become a better architect.
  • The journey to VCDX has taught me so much more about virtualization technology… networking… storage.. disaster recovery and people skills find your weak spot and learn about it
  • Getting your design accepted is a huge deal don’t treat that lightly may people don’t try or ever get that far.
  • Read the VCDX book and blueprint and live it… don’t think you know better just follow them… it’s like being in college again… figure out what the professor wants and do it… don’t worry if you think you know a better way.
  • View the recorded VCDX boot camps and attend a boot camp if you can.. they both help a lot.
  • If your ready to submit a design you already know a lot about VMware products… go do what you know be confident.  I am a VCDX now and I don’t know everything.. There are lots of holes in my knowledge…
  • Do a real design with real requirements and constraints… don’t go for WOW factor… just make a good design.
  • Know what else you could have done… and the effect of that choice.  If you had six more nic’s how would you use them?
  • Mock Design sessions… get on twitter and locate a mock design and defense group they are happening all around and will really help you prepare.
  • Push yourself.. if you don’t enjoy public speaking… do it.  If you don’t enjoy speaking up to defend your design choices do it… push yourself
  • Do not sacrifice your family, health, employment etc… to the certification.  Balance is hard in life but worth it.   When you die you don’t want you tombstone to say divorced VCDX.  No amount of success in life can replace failure in your home.
  • Enough with my preaching… I hope it helps.  Contact me via twitter or here anytime if you have questions.

 

Whats Next for me?

Well a VCDX-Cloud would be a good choice for me since I am doing a lot of cloud now… but in reality I think I might focus on a CCNA… because I am a little weak on networking and I want to round up my experiences, plus it will justify buying some cool looking cisco switches.  (Yes that means you are going to see Cisco posts soon.)

 

Network IO Control failing to shape traffic with multi-nic vMotion

Posted on by Joseph Griffiths

NIOCI have always used network IO Control to shape my traffic on a virtual switch (Enterprise Plus required).   It does a great job of balancing traffic when contention comes into play.  Unfortunately it cannot shape traffic as it comes into the virtual switch.  It can only shape traffic going out.   A new friend (@VMPrime)  pointed this out to me at the perfect time.   He was knowledgeable and encouraging an all around great guy.   He pointed out that NIOC only have effect on the machine during traffic flows exiting the machine.  When traffic goes to another machine NIOC has no effect.  I remember reading about this but the terminology was a little fuzzy from a VMware perspective.  Joe provided a simple scenario when that lack of control could be a problem.  Take into account the following scenario.  Assume that we have a two host cluster each running two 10GB nics.  We have a vlan for management, virtual machines and we have setup multi-nic vMotion as shown in the diagram 1 below.   We have NIOC setup with shares to protect each traffic type during contention.  Assume that the network utilization of host A is 2GB.  While the network utilization of host b is 15GB.Assuming that host B has capacity for all of host A virtual workloads I put host A into maintenance mode host A now utilizes up to 18GB of network to transfer the running state of virtual machines to host B.  Host A’s NIOC kicks in preserving 2GB for virtual machines and allocated 18GB to vMotion to Host B.  We are now shoving 18GB into Host B who’s virtual machine need 15GB’s.  Now both sides are contending for space and we might have availability issues on our host in addition the vMotion might fail.

How do we solve this issue?

This is exactly why we have Network Limits. Unlike CPU and memory Limits NIOC limits can really help with this exact issue. Putting a limit on vMotion of for example 2.5 GB per link would create a scenario when it could never use more than 5GB per host. Will this still have an effect? Maybe it’s a cost benefit anaylisis. You have to weigh your options and you might have to adjust you limit lower.

 

Drawing1

Apache and F5′s

Posted on by Joseph Griffiths

Everyone who uses any type of reverse proxy runs into this issue.  The original user IP address is lost when the proxy is used.  F5 and most proxy vendors have implements an additional header that contains the original source ip for usage known as the X-Forwarded-For Header.   You have to train your apache logs to look at this or your logs show the wrong IP.   This does present a problem that if someone goes directly the webserver the logs don’t show their ip address.  I ran across this little trick to display the correct IP either way just place this in your apache configuration replacing other log configuration.

 

httpd.conf in LogFormat section

LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”” proxy

SetEnvIf X-Forwarded-For “^.*\..*\..*\..*” forwarded

Then in your virtual host or host entry use the following

CustomLog “logs/ssl_access_log” combined env=!forwarded

CustomLog “logs/ssl_access_log” proxy env=forwarded

 

Enjoy!