Monthly Archives: September 2015

VSAN for Service Providers

Since VMworld in San Francisco, VMware have been on a tear backing up all the VSAN related announcements at the show by starting to push a stronger message around the improvements in the latest VSAN release. Cormac Hogan and Rawlinson Rivera have published articles while Duncan Epping has also released a number of articles around VSAN since VMworld including this one on VSAN Licensing and what’s included as part of the different Enterprise Packages…There has also been an official post on the vCloud Team Blog on use cases for VSAN Storage Policies in a vCloud Director environment.

Last year I wrote a couple of posts around the time VSAN pricing was being released and also on the specifics of the vCloud Air Network program bundles that allow VSAN to be consumed via the VSPP. At the time there was no All Flash Array option and the pricing through the VSPP was certainly competitive when you compared it to a per socket price.

As a platform, VSAN is maturing as an option for hyper converged deployments and VMware Service Providers are starting to deploy it not only for their Management Clusters, but also main compute and resource clusters. The wording and messaging from VMware has shifted significantly from the first 5.5 VSAN release where they mainly talked about Test/Dev and VDI workloads to now talk about mission critical workloads with 6.x.

While doing research into our new Management Clusters that will use VSAN on top of the new Dell FX2 PowerEdge Converged Platform I was looking into the costs on the vCAN and how it stacks up next to per socket pricing…Shown below are the different Product Bundles included in the vCAN Program…each one contains a different combination of VMware Products which you get access to depending on the bundle of choice (details here).
While I can’t exactly disclose what level we get at Zettagrid (or other providers for that matter) due to the commercial nature of the programs its safe to assume that service providers at scale can drill that US $1 per point price point down by up to 50%…while some could actually pay more.

When you start to look at the cost of running a storage platform for IaaS you start to get an appreciation for the cost per month on the vCAN program that running VSAN offers. At a small to medium scale VSAN via the vCAN Program stacks up…mainly because the program is structured to make the Points Per Month value cheaper the more volume you transact against the program. So an SP consuming large amounts of vRAM will have a lower entry point for VSAN.

Looking at the larger picture, below is an an example software (only) cost of 10 Hosts (64GB RAM) with 100TB of Storage with an expected utilization of 80% assuming 2 hosts are reserved for HA.

VSAN 80TB Allocated vRAM 410GB (205GB Reserved) Per Month
 $6,400 $1,433  $ 7,833

If we scale that to 20 hosts with 128GB and 200TB of Storage with an expected utilization of 80% assuming 4 hosts are reserved for HA.

VSAN 160TB Allocated vRAM 1.6TB (820GB Reserved) Per Month
 $ 12,800 $5,734  $ 18,534.40

You start to see that the cost per month starts to get somewhat questionable when comparing OpEx vs CapEx costs of a traditional SAN purchase outside of the vCAN Program. As an example you should be able to source a traditional SAN under finance with roughly the same usable storage as whats in the second example for about US $4000-6000 per month on a finance plan over 36 months.

Personally I believe the cost per allocated GB is a little on the high side at scale and it could start to become cost prohibitive for Service Providers when comparing to traditional storage pricing models or even some of the latest pricing for newer scale out platforms on the market…and that’s not even thinking about the additional cost of the AFA Add-On.

So, for me VMware need to look at slightly tweaking the vCAN cost model for VSAN to either allow some form of tiering (ie 0-500GB .08, 500-1000GB .05, 1TB-5TB .02 and so on) and/or change over the metering from allocated GB to consumed GB which allows Service Providers to take advantage of over provisioning and only pay for whats actually being consumed in the VSAN Cluster.

If VMware can push those changes through it will make VSAN even more attractive to vCloud Air Network Partners and have VSAN move from mainly Management Cluster consideration to full blown production IaaS use.


First Look: PernixData Architect (Beta)

PernixData had a number of announcements at Virtualization Field Day back in June including details around their v3 release of FVP as well as the announcement of their Freedom Edition. The other announcement was around Architect which is PernixData’s play at deep storage Analytics and Intelligence. With the advantage of being plugged right into the ESXi kernel the FVP Architect Host Extensions allows the software to represent (at a deeper level) statistics around VM performance, IO Profile and working set.

I’ve installed the beta into my FVP Freedom Parent lab to get a feel for what Architect can do and what level of detail it could provide. As mentioned I choose to upgrade my existing 3.0 Manager to the 3.1 binaries you get access to as part of the beta and I have it running side by side with my Freedom Edition.

You can see above that there is a slightly different installer with the FVP Architect Logo and the upgrade process was no different to upgrading between minor or major releases of FVP. After the upgrade you need to go to the PernixData Management Console and activate the Architect license which is set for a 60 day trial.

One thing to mention before moving on is that the Architect platform started to populate data even with the FVP 3.0 Host Extensions installed…meaning that I didn’t have to upgrade to the 3.1 beta bits…not sure what I’m missing out on short term buy not going up to 3.1 but it appears to be doing it’s job ok.

Working through the various Dashboards of Architect you start to get a feel as to what PernixData are trying to achieve with this software…there are the usual performance metrics we are used to seeing (Throughput, Latency, IOPS and Acceleration Rates) but right from the start you see additional bits of information that up until now, have been hard to capture and represent…IO Block Size breakdown distribution and percentage, working sets of VMs and Hosts and then (after an 4-8 hour wait) recommendations are given against VMs as to weather they are best suited for read or write back acceleration.

A few things that I was surprised with from looking through the data was that my NestedESXi Labs where extremely write intensive and the average block size we a tiny 4-8K. Overall I am pretty impressed with Architect and for someone that loves analytics and metrics I find myself being engrossed by all the data that’s being presented and intrigued by what I am finding out about this particular platform.

The beta program for their Architect software is open and you can register below for the 3.1 Beta of PernixData Manager and the FVP Host Extensions.

Below is an overview video taken from the PernixData site:

First Look: Dell PowerEdge FX2 Converged Platform

For the last six months or so I’ve been on the look out for server and storage hardware to satisfy the requirement for new Management Clusters across our Zettagrid vCloud Zones… After a fairly exhaustive discovery and research stage the Dell PowerEdge FX2 dropped at the right time to make the newly updated converged architecture hardware platform a standout choice for a HCI based solution.

I plan on doing a couple of posts on the specifics of the hardware chosen as part of the build that will end up as a VMware VSAN configuration but for the moment there is a little more info on the FX2 PowerEdge (below) as well as a Virtual Unboxing video that goes through the initial familiarization with the CMC and then walks through the FC430 System and Storage Configuration as well as what the new BIOS menu looks like:


Below are some specs from the Dell site going through the compute and storage hardware…as you saw in the video above we went for the 1/4 Blade FC430’s with two FD322 Storage Sleds.

Server blocks at the heart of the FX converged architecture are powered by the latest Intel® Xeon® processors. They include:

  • FC430: 2-socket, quarter-width 1U high-density server block with optional InfiniBand configuration
  • FC630: 2-socket, half-width 1U workhorse server block ideal for a wide variety of business applications
  • FC830: Powerful 4-socket, full-width 1U server block for mid-size and enterprise data centers
  • FM120x4: Half-width 1U sled housing up to four separate Intel® Atom® powered single-socket microservers offers up to 16 microservers per 2U.

The FC430 features:

  • Two multi-core Intel® Xeon® E5-2600 v3 processors or one multi-core Intel® Xeon® E5-1600 v3 processor (up to 224 cores per FX2)
  • Up to 8 memory DIMMs (up to 64 DIMMs per FX2)
  • Two 1.8″ SATA SSDs or one 1.8″ SATA SSD (w/front IB Mezzanine port)
  • Dual-port 10Gb LOM
  • Access to one PCIe expansion slot in the FX2 chassis

The FD332 provides massive direct attached storage (DAS) capacity in easily scalable, modular half-width, 1U blocks. Each block can house up to 16 direct-attached small form factor (SFF) storage devices. Combined with FX servers, the FD332 drives highly flexible, scale out computing solutions and is an excellent option for dense VSAN environments using optimized ratios of HDD/SSD storage (including all flash) .

  • Up to 16 SFF 2.5″ SSDs/HDDs , both SATA and SAS
  • Up to three FD332 blocks per chassis (with one FC630 for processing). Other storage options include one or two blocks with different combinations of server blocks
  • 12Gbps SAS 3.0 and 6Gbps SATA 3.0
  • PowerEdge RAID Controller (PERC9), single or dual controllers, RAID or HBA modes, or mix and match modes with dual controllers

My first impressions are that this is a very very sexy bit of kit! I am looking forward to getting it up and firing and putting it to use as the basis for a solid Management Cluster platform. 

NSX Bytes: Migrating NSX Edges Between Clusters….The Right Way

Had a situation pop up today where an NSX Edge needed to be moved from it’s current location to another location due to an initial VM placement issue. The VM was being moved within the bounds of the NSX Transport Zone so no issues with it being out of scope… At first the VM was vMotioned from one Cluster to another via the Web Client…however when the VM was brought back online the NSX Edge Status in the Web Client showed the error below:

While the VIX_E_TIMEOUT group of error messages are common for vShield and NSX Managers the VM was up and actually passing traffic ok…though no config could be applied in this state.

Looking through the specific Edge System Events under the Monitor Tab you see:

First thing I tried was a Force Sync which only served to reboot the Edge. Looking under the Manage Tab under NSX Edges and the NSX Edge Appliances I saw that the Edge details was Deployed and reporting on the correct Host and Datastore it was vMotioned to. From here I attempted a Re-Deploy operation, but that only moved it back to it’s original location fixing the error and making it manageable again.

I went back to the Manage Tab under NSX Edge Appliances and this time I changed the location of the Edge Cluster and Datastore directly from this menu.

This triggered another Edge Redeployment, but this time it was deployed to the desired location and the Edge was manageable again from the NSX Manager’s perspective. Lesson learnt is to not vMotion NSX Edges but take advantage of the fact that the Edges are effectivly stateless VMs that are highly transportable and have their configuration stored centrally for easy redeployment and recovery.

vSphere 5.5 Update 3 Released: Features and Top Fixes

vSphere 5.5 Update 3 was released earlier today and there are a bunch of bug fixes and feature improvements in this update release for both vCenter and ESXi. For most Service Providers updating to vSphere 6.0 is still a while away so it’s good to have continued support and improvement for the 5.5 platform. I’ve scanned through the release notes and picked out what I consider some of the more important bug fixes and resolved issues as they pertain to my deployments of vSphere.

Note: Still appears that there is no resolution to the vMotion errors I reported on earlier in the year or the bugs around the mClock Scheduler and IOPS Limiter on NFS.

ESXi 5.5 Update 3:

  • Status of some disks might be displayed as UNCONFIGURED GOOD instead of ONLINEStatus of some disks on an ESXi 5.5 host might be displayed as UNCONFIGURED GOOD instead of ONLINE. This issue occurs for LSI controller using the LSI CIM provider.
  • Cloning CBT-enabled virtual machine templates from ESXi hosts might failAttempt to clone CBT-enabled virtual machines templates simultaneously from two different ESXi 5.5 hosts might fail. An error message similar to the following is displayed:Failed to open VM_template.vmdk': Could not open/create change tracking file (2108).
  • ESXi hosts with the virtual machines having e1000 or e1000e vNIC driver might fail with a purple screenESXi hosts with the virtual machines having e1000 or e1000e vNIC driver might fail with a purple screen when you enable TCP segmentation Offload (TSO). Error messages similar to the following might be written to the log files:cpu7:nnnnnn)Code start: 0xnnnnnnnnnnnn VMK uptime: 9:21:12:17.991 cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x65b stack: 0xnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x18ab stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0xa2 stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0xae stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x488 stack: 0xnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x60 stack: 0xnnnnnnnnnnnnnnn cpu7:nnnnnn)0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn][email protected]#nover+0x185 stack: 0xnnnnnnnnnnnn
  • Attempts to reboot Windows 8 and Windows 2012 server on ESXi host virtual machines might failAfter you reboot, the Windows 8 and Windows 2012 Server virtual machines might become unresponsive when the Microsoft Windows boot splash screen appears. For more information refer, Knowledge Base article 2092807.
  • Attempts to install or upgrade VMware Tools on a Solaris 10 Update 3 virtual machine might fail
    Attempts to install or upgrade VMware Tools on a Solaris 10 Update 3 virtual machine might fail with the following error message:Detected X version 6.9
    Could not read /usr/lib/vmware-tools/configurator/XOrg/7.0/ Execution aborted.This issue occurs if the script copies the file, which should not be used in Xorg 6.9.

In going through the remaining Known Issues you come across a lot of Flash Read Cache related problems…maybe VMware should call it a day with this feature…not sure if anyone has the balls to actually use it in production…be interested to hear? There are also a lot of VSAN issues still being reported as known with workarounds in place…all the more reason to start a VSAN journey with vSphere 6.0.

For a look at what’s new and for the release notes in full…click on the links below:

VMware ESXi™ 5.5 Update 3 | 16 SEP 2015 | Build 3029944

vCenter Server 5.5 Update 3 | 16 SEP 2015 | Build 3000241

#vBrownBag TechTalk – NSX…An Unexpected Journey

While at VMworld a couple of weeks ago I presented a short talk around my journey working with NSX-v and how it has shifted (pivoted) the direction of what I consider to be important in my day to day role. The unexpected part of the journey dragged me kicking and screaming into the world of APIs and dare I say…Devops.

And while I don’t consider myself a DevOp (far from it)…I find myself more and more getting sucked into that world and with that I am trying adjust how I consume IT. In any case if you have a spare 10 minutes have a listen about how NSX kickstarted my interest and got me looking more under the covers of the server platforms and services we sometimes take for granted. Before this change I was comfortable accepting a UI as the only way to interact and consume services…are you?

For those interested the full schedule is here, along with direct links to the YouTube Channel with all the talks.

Altaro VMBackup – The Maltese Connection

Those that know me know that I am a very proud Australian of Maltese decent…so when I was emailed a while back to see if I would review the beta for Altaro’s new version of their Hyper-V Backup solution I was drawn by the surnames of the Altaro founders. After a quick Google I discovered that this Backup Software company was founded and based out of Malta which instantly sparked my interest…I’ve known for a while that Malta has a very strong tech industry and the Island has become known for producing strong programmers and developers.

That aside, I was also interested in their 6.0 product which introduces VMware support to extend what is now a well rounded backup alternative for the small to mid market at a very competitive price point. There are a lot of Backup software products out there in the market but there are not a lot that keep things simple…and from my time trying out Altaro in the lab it seems that they have hit that mark while still offering enough of a feature set to keep more advanced user expectations in check.

As mentioned, version 6 introduces VMware support and now allows backup and recovery of both Hyper-V and ESXi VMs to a local of offsite location. It provides a central Management Console for all backups and doesn’t rely on an external database for it’s configuration. One thing I would like to see added to the product is the ability to choose an offsite repository thats not another Altaro Backup Server…ie direct to S3 or a Swift Compatible endpoint, though this can be achieved by pointing the backup target at a iSCSI/NFS mount that could be an AWS Gateway Appliance or a Twinstrata Appliance.

Other features include:

Versions of VMware supported:

  • VMware vCenter 5.0, 5.1, 5.5 and 6.0
  • VMware ESXi 5.0, 5.1, 5.5 and 6.0

Versions of Hyper-V supported:

  • Windows 2008/2012 R2
  • Windows Hyper-V Server 2008/2012 R2

Below is a ten minute video that goes through the simple next next setup and shows you how easy it is to connect to a vCenter and start backing up VMs.

Altaro offers a couple of versions at different price points and also has a free edition that backs up 2 VMs with a four week retention.

  •     Standard Edition : Most functionality with protection of up to five VMs (~$370USD)
  •     Unlimited Edition : All functionality for an unlimited number of VMs and supports clusters (~$560USD)
  •     Free Edition: Protects up to two VMs with four weeks retention

For small to medium setups with 2-5 Hosts its an attractive alternative to the existing smaller players in the market…though you would still be looking at the larger vendors for bigger virutalized footprints it’s great to see news guys coming into the market. Well done to David, Conrad and rest of the Altaro Team out of Malta…great to see tech of this standard coming out of my Island home!

PernixData FVP Freedom: Setup and First Look

At the Virtualization Field Day back in June PernixData Announced they would be releasing a free version of their FVP software…FVP Freedom. Overnight Freedom was released and as I have been eager to put it to use in my Lab for a while now I’ve gone through the install steps and put together a guide/first look.

As a quick reminder FVP Freedom gives you the following:

  • Acceleration for any virtualized application
  • Up to 128 GB of read acceleration (Using only RAM as Cache – DFTM)
  • Community support

Required Bits for Installation:

  • FVP Management Server (v3.0)
    • 4 vCPU and 8GB vRAM
    • Windows 2008/R2 2012/R2
    • MSSQL 2008/12/14
    • .NET 3.5 SP1 or Later
  • FVP Host Extension for vSphere 5.1, 5.5 or 6.0 (v3.0.0.0)

FVP Management Server and Console:

For my NestedESXi Labs I run five hosts that are a mix of ESXi 5.5 and 6.0 Clusters managed by the 6.0 vCenter Server Appliance. The FVP Management Server is hooked up to the Active Directory for authentication and I am running MSSQL locally on the server to keep the required services local.

Once you have the pre-requisites installed launch the FVP Manager Installer ( I choose Custom Setup )  and work through the install options including configuring the vCenter details and authentication, database and FVP Manager IP and Port settings.

Note: There is still this password issue when special characters are used.

At this point, before installing the Manager you are told that the FVP Freedom Edition is being installed and you can activate using the license page.

If, like me you have installed MSSQL on the same server you may get the following warning.

My suggestion is to cap MSSQL to allow for the 4GB free memory the FVP Manager is asking for…the Management VM has 12GB of vRAM and I’ve limited MSSQL to use 6GB max. Once the installation is complete you can now access the new standalone FVP Management Console. If you choose the default options you can browse to the Manager Console via:


Once logged in using the credentials supplied during the setup process you will see a familiar layout to what was previously in the VI Client and will show up in the Web Client.


License Install:

To install your FVP license (you should have received the license in a seperate email from PernixData after registering for FVP Freedom) click on the Licensing Tab and select the vCenter Server from the list and Click on the Enter FVP License Key.

At this point the Online Activation should take place and you should see the FVP License Type change to Freedom Edition. Clicking on the License details in the lower pane will show you just want PernixData are giving away with this edition.

FVP Host Extension and Host Resource Configuration:

Installing the Host Extensions is a straight forward process and I won’t go into too much detail as there are other blog posts around that cover the subject. In a nutshell you can use Update Manager to push the bundle out or use the following esxcli command on the hosts directly.

esxcli software vib install-d /vmfs/volumes/ds01/

For the rest of the FVP Cluster Configuration I’ve created a quick video:

vCenter Web Client:

With the new Stand Alone Management Console the vCenter Web Client is used as a pointer. As shown below the PernixData Logo now shows up in the Home Page and clicks through to a summary screen where you see the FVP Cluster details and get some basic insights into how the cluster is performing.

Enjoy the Freedom!

Released: PernixData FVP 3.0 – New Console + vSphere 6.0 Support

Last week at VMworld I was lucky enough to spend some quality time with the PernixData team and got some great insights into their future products including FVP, Architect and FVP Freedom. Today PernixData has released into GA FVP 3.0 which includes some significant new features and supportability for all those who are on vSphere 6.0. There are also some performance enhancements, support for offsite metadata analytics of if you choose to allow PernixPlus as well as an improved license activation experience through the new HTML5 Portal.

Support for vSphere 6.0

FVP 3.0 introduces support for vSphere 6.0.  Maintain your world class performance as you transition over to vSphere 6.0.  FVP 3.0 also supports mixed environments of vCenter 6.0 with hosts running ESXi 5.1 or newer as you phase in vSphere 6.0 across your data center.  With the support of vSphere 6.0, FVP 3.0 will no longer be supporting vSphere 5.0. 

New HTML5 based User Interface

FVP 3.0 offers a completely new user experience.  Administer FVP using a modern, fast, and flexible interface with a web browser while moving past the inherent limitations of the vSphere Web Client.  This new interface can be used on its own, or easily launched from the vSphere Web Client.  FVP is known for its elegant presentation of performance metrics, and version 3.0 improves on this experience even more with enhancements to the performance data presented.

FVP is an amazingly simple and elegant solution for those wanting to get more out of their existing backend storage or looking to add new levels of performance via SSD or PCI Based Flash…FVP 3.0 software, along with documentation, will be available on the PernixData support portal at  

As a sidenote, with this release PernixData have stopped official support for FVP 1.0.

vExpert VMworld Reception and Program Thoughts

Last week at VMworld we had the annual vExpert Reception…this year the party was held at the very swanky Julia Morgan Ballroom in the Merchants Exchange building. From looking around the room I counted at least a couple hundred vExperts which is a fantastic turnout given the spread of the 1300 vExperts around the world.

The agenda was split into four presentations including Cory Romero with an update on the program and also some futures on where VMware want to take the program…we also had updated from VMware’s Digital Marketing lead, NSX Certification Program Manger Chris McCain, a VCDX Program Update (did you know all VCDX’s are automatically vExperts now) and the night was capped off with a very personable presentation from Pat Gelsinger.

Pat didn’t talk vision as he did in the Tuesday Keynote…he kept the talk very casual and talked about his experiences around getting excited about hearing about the first vMotion while working at Intel and the potential use cases around that. For him, it’s all about the technology and he expressed in no uncertain terms that as people who work in and around the IT world…

We all get turned on by cool tech!

And while that got a truthfully embarrassing reaction from the crowd it speaks volumes as to a big part of why programs like the vExpert Program exist. No one should be part of a program like this if they don’t get off on technology. Passion translates to action…and VMware (and other vendors that run similar programs) understands that there is huge power in bringing together like minded people to help push and champion their products.

Interestingly, after I posted this to Twitter during the presentations I had a number of replies from Michael Stump (sidenote: Michael has a great Blog here) echoing his previous thoughts that the vExpert program is a marketing tool for VMware which he couldn’t partake in anymore…and while I respect his choice I completely disagree with his rational. There is no confusion in my head (and in others I have spoken to or that have commented) that being part of these vendor programs means we need to “whore” ourselves off a little…for the most we graciously receive the free gifts from vExpert Vendor Partners and we troll the Solutions Exchange looking for our free battery packs, speakers and apparel.

We should not feel guilty about receiving these gifts but we must also understand that they do come with a small price…However the fact that we are passionate and all get off on technology means that in accepting these gifts we are doing exactly what the program is designed for…that is to help promote VMware and their partners…and in turn great tech!

Now, if vExperts choose to take the gifts and run that’s totally up to the individual…but this program gives back as much as you put in…the true value of the community isn’t the gifts and awesomely good food at the reception party…it’s the community aspect that it breeds. For me I have made many new friends and contacts in the industry that I wouldn’t have thought possible without the that I have been able to extend my technical knowledge and it has also served as a distribution point for the content I create and in that I believe that I can give back to the vExpert Community.

Embrace the program as it continues to grow! It won’t be around forever!