Tag Archive for How-Tos

How To: DELL DSET Report Tool Live CD and Linux VLAN Config

Here is a quick post on generating support logs for DELL cases if you are running VMware ESX(i) on any of the DELL server hardware. I had a CPU alert appear in my vSphere Hardware status and raised a support ticket with DELL. Previously I’ve had to wrestle with the config/setup of the DSET tool on ESX(i) and even had it cause a boot up failures due to a comparability bug.

The Dell tech send me the link below which is a CENTOS LiveCD which can be downloaded and booted up on the server in question.

http://linux.dell.com/files/openmanage-contributions/omsa-70-live/

Once downloaded and attached via the iDRAC Virtual Media Manager you will automatically go through to the desktop where you can double click on the DSET Tool Icon. Let it do it’s thing and gather all the relevant info which is then packaged into a zip file under \tmp\data\

dell_dset_1

dell_dset_2

Ok, so now that you have the file…how do you get it off the LiveCD instance? The answer would be simple if you had interfaces configured with DHCP, but the majority of these servers are configured with NICs on VLAN enabled ports which are not easily switched over or able to be reconfigured without going through change management etc etc.

The Network Configuration GUI in CENTOS doesn’t have the ability to configure VLAN tagging on the interfaces so you need to jump into the shell and manually configure the network settings as shown below.

Create a new config file for eth0 and configure it as shown below…key here is to take note of the MAC Address, no include and IP or Subnet details and I disabled IPv6.

#vi ifcfg-eth0
DEVICE=eth0
HWADDR=xx:xx:xx:xx:xx:xx
ONBOOT=yes
BOOTPROTO=none
TYPE=Ethernet
IPV6INIT=no
IPV6_AUTOCONF=no

Once saved, copy that file and save to ifcfg-eth0.x where is is the VLAN you want the interface to communicate in. This time you are adding relevent IP info along with specifying the device name as eth0.x and VLAN=yes which obviously enabled the VLAN tag config.

#cp ifcfg-eth0 ifcfg-eth0.x
#vi ifcfg-eth0.x
DEVICE=eth0.x
HWADDR=xx:xx:xx:xx:xx:xx
ONBOOT=yes
HOTPLUG=no
BOOTPROTO=none
TYPE=Ethernet
VLAN=yes
IPADDR=192.168.0.10
NETMASK=255.255.255.0
GATEWAY=192.168.0.1

Fire up the new interfaces and restart network and you have a VLAN enabled connection that you can now grab the DSET zip file off and send to DELL for analysis.

#/sbin/ifup eth0
#/sbin/ifup eth0.x
#/sbin/service network restart

As a side note, being the good VMware fanboy that I am, I used my Octopus Beta service to upload the file and make it available via the Octopus URL for sharing…because getting access to the Horizon Suite BETA is currently near on impossible :)

How-To: Citrix NetScaler GeoIP Restrictions

I had a request from a Hosting Client this week to look at options around blocking malicious users from causing trouble on a local Auction site. As the site was only for Australian and New Zealand users we needed to come up with a solution to block the whole world except AU and NZ visitors. Obviously I know there are mechanisms in existence that have annoyed me in the past while trying to source overseas content and getting the message telling you that you can’t access this site in your region.

I’ve never personally had to act on a request like this, and thought about options relating to some sort of code based filtering or filtering at the gateway level. I’ve known that in real terms I haven’t even scratched the surface of what our Citrix NetScaler VPX’s can do, and with that I searched for some guidelines on getting up GeoIP Responder rules at the Load Balancer’s Virtual Server level. Not being able to find anything definitive end to end, here are the steps I took to achieve the end result.

Citirx NetScaler ArticleHow to Block Access to a Site by Country using a Location Database

First step is to enable the Responder Feature is it’s not already enabled. Citrix suggest you disable any feature not in use to save on system resources.

> enable feature RESPONDER

In order for the NetScaler to work our what location a visitor is coming from it needs to reference a GeoIP database. MaxMind offer a free database from here: These are updated on the first Tuesday of everymonth, so a little upkeep is required moving forward. There are IPv4/6 versions as well as an extended database City version which lets you get very granular in terms of allowing city access. For this exercise we will use the GeoIPCountryWhois CSV database.

Jump into the shell of the NetScaler and create a new directory. Note that if you have a HA setup, you need to do this on each NetScaler node.

172.1.1.1) Done
> shell
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 The Regents of the University of California. All rights reserved.
root@NSLB01# cd /var/
root@NSLB01# mkdir geoip

Use SCP to upload the CSV database to that location just created on the NetScaler and then run the following command to import the location parameters. Once done you can query the location database to ensure you have  imported the CSV line items.

> add locationfile /var/geoip/GeoIPCountryWhois.csv -format GeoIP-Country
 Done
> show locationparameter
Static Proximity
----------------
Database mode: File
Flushing: Idle; Loading: Idle
Context: geographic
Qualifier 1 label: Continent
Qualifier 2 label: Country
Qualifier 3 label: Region
Qualifier 4 label: City
Qualifier 5 label: ISP
Qualifier 6 label: Organization
Location file (format: geoip-country):
 /var/geoip/GeoIPCountryWhois.csv
Lines: 170195 Warnings: 0 Errors: 0
Current static entries: 170195 Current custom entries: 0
 Done

Now that you have the GeoIP location locked and loaded, you can created the Responder Policy. I had a little trouble trying to work out how to structure the rule to work correctly limiting visitors to only .AU and .NZ. I’ll be honest here and admit that trial and error was the winner here, but eventually I came up with the following that works.

> add responder policy GeoAusNZOnly "CLIENT.IP.SRC.MATCHES_LOCATION("*.AU.*.*.*.*").NOT && CLIENT.IP.SRC.MATCHES_LOCATION("*.NZ.*.*.*.*").NOT" RESET

Reading through the policy it’s easy enough to see what’s going on…this page references the Location Database General Information and formats, however it’s confusing at best..my advice is for Country Based GeoIP use the above as a template and simply change the country codes to suit.

Back to the GUI of the NetScaler and under Load Balancing settings of the Virtual Server(s) in question, open the Virtual Server for editing and go to the Policies Tab -> Click on the Responder sub tab and right click to Insert Policy and the end result will be similar to what’s shown below.

I was able to use Twitter contacts with servers in global locations to test out the rule which was behaving exactly as expected. If you go back to the Policy menu item under Responder and check the Responder Policies you will be able to see if the rule is active and how many hits the rule has triggered.

The default action of the policy is to DROP or RESET the connection. You do have the option of creating a custom REDIRECT rule that will allow you to make the end user a little nicer in terms of presenting the user with a HTML page letting them know the page is restricted ..with the DROP and REST the browser simply shows a page not found or connection reset. I’ll update this post once i’ve created the REDIRECT rule.

Update: Turns out that if you apply the above rule it’s not that great for Google Analytics and the bots that hit your site. If you want to get the GoogleBot User Agent through the rule, create a rule similar to below

> add responder policy GeoAusNZGoogleOnly "CLIENT.IP.SRC.MATCHES_LOCATION("*.AU.*.*.*.*").NOT && CLIENT.IP.SRC.MATCHES_LOCATION("*.NZ.*.*.*.*").NOT && HTTP.REQ.HEADER("User-Agent").SET_TEXT_MODE(IGNORECASE).REGEX_MATCH(re/Googlebot/).NOT" RESET
Looking at that additional condition  you are looking in the HTTP Request Header, ignoring case and matching Googlebot.

How To: Setting Up Nagios User for ESXi

Unless my Google skills are seriously on the decline I wasn’t able to find a definitive post on correctly setting up a nagios user to facilitate esx_checks for monitoring systems such as OpsView and similar Nagios based systems.

For our ESXi monitoring we have chosen to use the OS – vSphere Service Checks. Post here on how to get the OpsView side going. Having a look at the checks in OpsView you see the following checks are used

Our current ESXi 4.1 hosts where working a treat, and this being my first time adding new hosts to the monitoring set, I was confronted with the following errors via Nagstamon:

So, to get the reporting up the following actions need to be taken on the ESXi host.

  • Log into the host directly with the VI Client
  • Go to the Local Users & Groups Tab
  • Add/Modify the nagios user and set the password
  • Add the user to the users group and click ok
  • On the Permissions tab and Add a new permission for the nagios user as Read-Only.

If you want you can run the commands to test access directly from the cli of the OpsView system.

root@opsview-appliance:/home/conf# ./check_esx -H x.x.x.x -u nagios -p xxxxx -l runtime -s health -w 0 -c 5
CHECK_ESX CRITICAL - Error: Permission to perform this operation was denied.
root@opsview-appliance:/home/conf# ./check_esx -H x.x.x.x -u nagios -p xxxxx -l runtime -s health -w 0 -c 5
CHECK_ESX OK - All 236 health checks are Green | Alerts=0;;

Alerts are gone and we are good to move onto the next job…finally!

How-To: vCenter 5.1 SSO Adding AD Identity Source


The SSO Component of vCenter 5.1 throws a couple of spanners in the works with regards to a straight forward upgrade of an existing vCenter install. While not overly complicated in terms of understanding what and how the SSO Service fits into the 5.1 puzzle, I found that it did add a couple of additional configuration steps that where not expected during and after the upgrade process. There are a heap of resources out there already on the end to end install of the SSO…be it a Simple Server install or a multi-server HA set-up, but your best bet is to catch up on the official VMware Documentation here.

EDIT: @VMwareKB vSphere Blog SSO Help Page http://t.co/9y20Kk22

In my environment we already employed AD authentication by way of Group Membership that dictated access to the vSphere Datacenters and Clusters. This was well established and working without too much hassle. My first attempt at the vCenter 5.1 upgrade yielded mixed results with the SSO, but lesson learnt was that I made the mistake of being too eager to jump into the upgrade without RTFM!

What I am now calling an exercise in executing a roll-back plan came about because I didn’t bother to understand how the SSO component affects an existing set-up and also from not paying attention during the install. In truth, I thought the first upgrade failed to install SSO correctly as I was getting errors when trying to login and the Web Client wasn’t able to connect to the SSO service. Couple of points here is that I rushed through the “Error 29155 Identity source discovery error” which is referenced by KB 2034374 and I attempted to “fix” the service by messing with the SSO Service Log-on user configuration. In the end I got impatient and rolled back the vCenter SnapShot I had taken before upgrade and started again. (Side note: that actually worked ok even after 5.1 agents where deployed to hosts managed by the vCenter…after rolling back the snap the 5.0 agent’s where re-redeployed without hassle)

So, once 5.1 had been installed and all components have been upgraded, you need to add your AD LDAP profile as an Identity Source via the vCenter Web Client. Without this, your existing AD credentials will not be honoured.

Log into the vCenter Web Client with the credentials provided during setup:

Click through Administration -> Sign-On and Discovery -> Configuration and click on the green + Button in the centre window pane.

Collect all your relevant AD LDAP information and complete the set-up as shown below.

If all the settings are correct you will get a positive Test Connection response.

Now that you have your Identity Source configured you can add the new source to the default domains by clicking Add to default domains in the top bar and bump the new source to the top of the list in the bottom pane. This allows you to not have to enter the NETBIOS name of the domain during login. eg DOMAN\username vs username.

Final thing to check now is to ensure that your previous Permissions based on AD groups are still in place relative to the vCenter, Datacenters, Clusters etc. As shown below, from this point forward you can configure access as you would have previously…the only change now is you have the option of selecting the Domain to reference.

What this means, is in theory you could pull in external/client LDAP Identity Sources to use as authentication mechanisms on your vCenter…not sure it’s totally useful for vCenter’s, but can see this being extremely useful for management and automation layers like vCloud and vCOPs or even vCO.

Load Balancer Internal IP’s Appearing in IIS/Apache Logs: Quick Fix

If you are NAT’ing public to private addresses with a load balancer in between your web server and your Gateway/FireWall device you might come across the situation where the IIS/Apache logs report the IP of the Load Balancer, when what you really want, is the client IP.

203.123.4.100 - - [29/Jun/2012:00:12:43 +0800] "GET /test.html HTTP/1.1" 404 261 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
172.20.0.16 - - [29/Jun/2012:00:12:45 +0800] "GET /test.html HTTP/1.1" 404 261 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"

It’s obvious that the biggest issue with this is that any Log Parser/Analytic’s you do against the site will all be relative to the IP of the load balancer. All useful client and geographical information is lost.

Most Load Balancer’s get around this by inserting a Header into the packet that relates to Client IP. In most cases that I have seen, both Juniper and NetScalers the Header is set to rlnclientipaddr.

What needs to be done at the web server configuration level to help pick up on and translate the header info so it can be used to translate the correct client IP into the log files. There are obviously different way to achieve this in Apache, compared to IIS and Apache has a much simply solution than IIS.

Apache:

In your apache.conf go to the LogFormat sections and modify the default format as shown below (Replace the Red text with the green text) and restart the Apache Service.

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat “%h %l %u %t \"%r\" %>s %b \"%{Referer}i\” \"%{User-Agent}i\""combined 
LogFormat "%{rlnclientipaddr}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent


IIS
:

The IIS 5/6/7/8 solution is a little more involved, but still just as efficient and not overly complicated at the end of the day…in fact for me the hardest part was actually chasing up the DLL’s linked below. It must be noted that while this has worked perfectly for me against both a Juniper DX and NetScaler VPX load balancer I would suggest testing the solution before putting it into production. Reason being is that the ISAPI filters are specifically sourced for the Juniper DX series, but in my testing I found that they worked for the NetScalers as well. Sourcing the x64 DLL’s was a mission, so in this I am saving you a great deal of time by provided the files below.

rllog-ISAPI

Download and extract those files into your Windows root. Go to the Features View -> ISAPI Filters and Click on Add. Enter in the Name and Executable Location and click ok. Note that it’s handy to add both 32 and 64 bit version to a 64bit IIS Web Server just in case you are dealing with legacy Application that are required to run in 32bit mode. Adding the ISAPI Filter at the root config of the Web Server so it propagates down to all existing sites and any newly created sites.