Category Archives: VMware

9.5 Update 3 Officially Compatible with VMware Cloud on AWS

At VMworld 2017 Veeam was announced as one of only two foundation Data Protection partners for VMware Cloud on AWS. This functionality was dependant on the release of Veeam Backup & Replication 9.5 Update 3 that contained the enhancements for it to interoperate with VMware Cloud on AWS locked down vCenter.

This week 9.5 Update has been listed on the VMware Compatibility Guide (VCG) for Data Protection.

In terms of what you now get in Update 3, there is little noticeable difference in the process to configure and run backup or replication jobs from within Veeam Backup & Replication. The VMware Cloud on AWS resources are treated as just another cluster so most actions and features of the core platform work as if the cloud based cluster was local or otherwise.

There were a few limitations that VMware have placed on the solution which means that our NFS based features such as Instant VM Recovery, Virtual Labs or Surebackups won’t work at this stage. HotAdd mode is the only supported backup transport mode (which isn’t a bad thing as it’s my preferred transport mode) which talks to a new VDDK library that is part of the VMC platform.

With that the following features work out of the box:

  • Backup with In Guest Processing
  • Restores to original or new locations
  • Backup Copy Jobs
  • Replication
  • Cloud Connect Backup
  • Windows File Level Recovery
  • Veeam Explorers

I’m really excited where VMware takes VMware Cloud on AWS and I see a lot of opportunities for the platform to be used as an availability resource. Over the next couple of months I’m hoping to be able to dive a little more into how Veeam can offer both backup and replication solutions for VMware Cloud on AWS.

Resources:

https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsanps&details=1&partner=594&releases=282&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc

Homelab : Supermicro 5028D-TNT4 One Year On

It’s been just over a year since I unboxed my Supermicro 5028D-TNT4 and setup my new homelab. A year is a long time in computing so I though I would write up some thoughts on how the server has performed for me and give some feedback on what’s worked and what hasn’t worked with having the Supermicro system as my homelab machine.

As a refresher, this is what I purchased…

I decided to go for the 8 core CPU mainly because I knew that my physical to virtual CPU ratio wasn’t going to exceed the processing power that it had to offer and as mentioned I went straight to 128GB of RAM to ensure I could squeeze a couple of NestedESXi instances on the host.

https://www.supermicro.com/products/system/midtower/5028/sys-5028d-tn4t.cfm

  • Intel® Xeon® processor D-1540, Single socket FCBGA 1667; 8-Core, 45W
  • 128GB ECC RDIMM DDR4 2400MHz Samsung UDIMM in 4 sockets
  • 4x 3.5 Hot-swap drive bays; 2x 2.5 fixed drive bays
  • Dual 10GbE LAN and Intel® i350-AM2 dual port GbE LAN
  • 1x PCI-E 3.0 x16 (LP), 1x M.2 PCI-E 3.0 x4, M Key 2242/2280
  • 250W Flex ATX Multi-output Bronze Power Supply

In addition to what comes with the Super Server bundle I purchased 2x Samsung EVO 850 512GB SSDs for initial primary storage and also got the SanDisk Ultra Fit CZ43 16GB USB 3.0 Flash Drive to install ESXi onto as well as a 128GB Flash Drive for extra storage.

One Year On:

The system has been rock solid however I haven’t been able to squeeze the two NestedESXi instances that I wanted to initially. 128GB of RAM just isn’t enough to handle a full suit of systems. As you can see below, I am running three NestedESXi hosts with NSX, vCloud Director and Veeam. The supporting systems for the NestedESXi lab make up the majority of the resource consumption but I also run the parent VCSA and domain controller on the server leaving me with not a lot of breathing room RAM wise.

In fact I have to keep a couple of servers offline at any one point to keep the RAM resources in check.

What I Wanted:

For me, my requirements where simple; I needed a server that was powerful enough to run at least two NestedESXi lab stacks, which meant 128GB of RAM and enough CPU cores to handle approx. twenty to thirty VMs. At the same time I needed to not not blow the budget and spend thousands upon thousands, lastly I needed to make sure that the power bill was not going to spiral out of control…as a supplementary requirement, I didn’t want a noisy beast in my home office. I also wasn’t concerned with any external networking gear as everything would be self contained in the NestedESXi virtual switching layer.

One Year On:

As mentioned above to get to two NestedESXi lab stacks I would have needed to double the amount of RAM from 128GB to 256GB however the one stack that I am running covers most of my needs and I have been able to work within the NestedESXi platform to do most of my day to day tinkering and testing. The CPU hasn’t been an issue and i’ve even started using some of the spare capacity to mine cryptocurrency…something that I had no intention of doing one year earlier.

In terms of the power consumption the Xeon-D processor is amazing and I have not noticed any change in my power bill over the last 12 months…for me this is where the 5028D-TNT4 really shines and because of the low power consumption the noise is next to nothing. In fact as I type this out I can hear the portable room fan only…the Micro Tower it’s self is unnoticeable.

From a networking point of view I have survived this far without the need for external switching while still being able to take advantage of the vSphere private VLANs to accomodate my routing needs within the system.

Conclusion:

Looking at the WiredZone pages the SuperMicro systems haven’t really changed much in the past 12 months and prices seem to have stayed the same, however the price of RAM is still higher than when I purchased the system. For me you can’t beat the value and relative bang for buck of the Xeon-D processors. My only real issues where with not having a “management cluster” meaning that I had to take down all the systems to perform upgrades on the VCSA and hosts. To get around that I might consider purchasing a smaller NUC to run the core management VMs which would free up 16GB of RAM on the SuperMicro meaning I could squeeze a little more into the system.

All in all I still highly recommend this system for homelab use as it’s not only proven to be efficient, quiet and powerful…but also extremely reliable.

NSX Bytes – What’s new in NSX-T 2.1

In Feburary of this year VMware released NSX-T 2.0 and with it came a variety of updates that looked to continue to push of NSX-T beyond that of NSX-v while catching up in some areas where the NSX-v was ahead. The NSBU has big plans for NSX beyond vSphere and during the NSX vExpert session we saw how the future of networking is all in software…having just come back from AWS re:Invent I tend to agree with this statement as organisations look to extend networks beyond traditional on-premises or cloud locations.

NSX-T’s main drivers relate to new data centre and cloud architectures with more hetrogeneality driving a different set of requirements to that of vSphere that focuses around multi-domain environments leading to a multi-hypervisor NSX platform. NSX-T is highly extensible and will address more endpoint heterogeneity in future releases including containers, public clouds and other hypervisors. As you can see before the existing use cases for NSX-T are mainly focused around devops, micro-segmentation and multi-tenant infrastructure.

Layer 3 accessibility across all types of platforms.

What’s new in NSX-T 2.1:

Today at Pivotal SpringOne, VMware is launching version 2.1 of NSX-T and with it comes a networking stack underpinning Pivotal Container Services, direct integration with Pivotal Cloud Foundry and significant enhancements to load balancing capabilities for OpenStack Neutron and Kubernetes ingress. These load balancers can be virtual or bare metal. There is also native networking and security for containers and Pivotal operations manager integration.

NSX-T Native Load Balancer:
NSX-T has two levels of routers as shown above…then ones that connect to the physical world and the ones which are labeled T1 in the diagram above. Load balancing will be active on the T1 routers and have the following features:

  • Algorithms – Round Robin, Weighted Round Robin, Least Connections and Source IP Hash
  • Protocols – TCP, UDP, HTTP, HTTPS with passthrough, SSL Offload and End to end SSL
  • Health Checks – ICMP, TCP, UDP, HTTP, HTTPS
  • Persistance – Source IP, Cookie
  • Translation – SNAT, SNAT Automap and No SNAT

As well as the above it will have L7 manipulation as will as OpenStack and Kubernetes ingress. Like NSX-v these edges can be deployed in various sizes depending on the workload.

Pivotal Cloud Foundry and NSX-T:

For those that may not know, PCF is a cloud native platform for deploying and operating modern applications and in that NSX-T providers the networking to support those modern application. This is achieved via the Network Container Plugin. Cloud Foundry NSX-T topology include a separate network topology per orginization with every organization getting one T1 router. Logical switches are then attached per space. High performance north/south routing uses NSX routing infrastructure, including dynamic routing to the physical network.

For east/west traffic that happens container to container with every container having distributed firewall rules applied on it’s interface. There is also a number of visibility and troubleshooting counters attached to every container. NSX also controls the IP management by supplying subnets from IP blocks to namespaces and individual IPs and MACs to containers.

Log Insight Content Pack:

As part of this release there is also a new Log Insight NSX-T Content Pack that builds on the new visibility and troubleshooting enhancements mentioned above and allows Log Insight to monitor a lot of the container infrastructure with NSX.

Conclusion:

When it comes to the NSX-T 2.1 feature capabilities, the load balancing is a case of bringing NSX-T up to speed to where NSX-v is, however the thing to think about is that how those capabilities will or could be used beyond vSphere environments…that is the big picture to consider here around the future of NSX and it can be seen with the deeper integration into Pivotal Cloud Foundry.

AWS re:Invent – Expectations from a VM Hugger…

Today is the first day offical day of AWS re:Invent 2017 and things are kicking off with the global partner summit. Today also is my first day of AWS re:Invent and I am looking forward to experiencing a different type of big IT conference with all previous experiences being at VMworld or the old Microsoft Tech Eds. Just buy looking at the agenda, schedule and content catalog I can already tell re:Invent is a very very different type of IT conference.

As you may or may not know I started this blog as Hosting is Life! and the first half of my career was spent around hosting applications and web services…in that I gravitated towards looking at AWS solutions to help compliment the hosting platforms I looked after and I was actively using a few AWS services in 2011 and 2012 and attended a couple of AWS courses. After joining Zettagrid my use of AWS decreased and it wasn’t until Veeam announced supportability for AWS storage as part of our v10 announcements that I decided to get back into the swing of things.

Subsequently we announced Veeam Availability for AWS which leverages EBS snapshots to perform agentless backups of AWS instances and more recently we where announced as a launch partner for VMware Cloud on AWS data availability solutions. For me, the fact that VMware have jumped into bed with AWS has obviously raised AWS’s profile in the VMware community and it’s certainly being seen as the cool thing to know (or claim to know) within the ecosystem.

Veeam isn’t the only backup vendor looking to leverage what AWS has to offer by way of extending availability into the hyper-scale cloud and every leading vendor is rushing to claim features that offload backups to AWS cloud storage as well as offering services to protect native AWS workloads…as with IT Pros this is also the in thing!

Apart from backup and availability, my sessions are focused on storage, compute, scalability and scale as well as some sessions on home automation with Alexa and alike. This years re:Invent is 100% a learning experience and I am looking forward to attending a lot of sessions and taking a lot of notes. I might even come out taking the whole serverless thing a little more seriously!

Moving away from the tech the AWS world is one that I am currently removed from…unlike the VMware ecosystem and VMworld I wouldn’t know 95% of the people delivering sessions and I certainly don’t know much about the AWS community. While I can’t fix that by just being here this week, I can certainly use this week as a launching pad to get myself more entrenched with the technology, the ecosystem and the community.

Looking forward to the week and please reach out if you are around.

Released: NSX-v 6.3.5 and New Features and Fixes

Last week VMware released NSX-v 6.3.5 (Build 7119875) that contains a few new features and addresses a number of bug fixes from previous releases. Going through the release notes there are a lot of known issues that should be known and there are more than a few that apply to service providers…specifically there are a lot around Logical and Edge Routing functions. The other interesting point to highlight about this release is that this is apparently the same build that runs on VMware on AWS instances as mentioned by Ray Budavari.

The new features in this build are:

  • For vCenter 6.5 and later, Guest Introspection VM’s, on deployment, will be named Guest Introspection (XX.XX.XX.XX), where XX.XX.XX.XX is the IPv4 address of the host on which the GI machine resides. This occurs during the initial deployment of GI.
  • Guest Introspection service VM will now ignore network events sent by guest VMs unless Identify Firewall or Endpoint Monitoring is enabled
  • You can also modify the threshold for CPU and memory usage system events with this API: PUT /api/2.0/endpointsecurity/usvmstats/usvmhealththresholds
  • Serviceability enhancements to L2 VPN including
    • Changing and/or enabling logging on the fly, without a process restart
    • Enhanced logging
    • Tunnel state and statistics
    • CLI enhancements
    • Events for tunnel status changes
  • Forwarded syslog messages now include additional details previously only visible on the vSphere Web Client
  • Host prep now has troubleshooting enhancements, including additional information for “not ready” errors

That last new feature above is seen below…you can see the EAM Status message just below the NSX Manager IP which is a nice touch given the issues that can happen if EAM is down.

If you click on the Not Ready Installation Status you now get a more detailed report of what could be wrong and suggestions of how to resolve.

Important Fixes :

  • VMs migrated from 6.0.x can cause host PSOD When upgrading a cluster from 6.0.x to 6.2.3-6.2.8 or 6.3.x, the VM state exported can be corrupted and cause the receiving host to PSOD
  • “Upgrade Available” link not shown if cluster has an alarm. Users are not be able to push the new service spec to EAM because the link is missing and the service will not be upgraded
  • NSX Manager crashes with high NSX Manager CPU NSX Manager has an OOM (out of memory) error and continuously restarts
  • NSX Controller memory increases with hardware VTEP configuration causing high CPU usage A controller process memory increase is seen with hardware VTEP configurations running for few days. The memory increase causes high CPU usage that lasts for some time (minutes) while the controller recovers the memory. During this time the data path is affected
  • Translated IPs are not getting added to vNIC filters which is causing Distributed Firewall to drop traffic When new VMs are deployed, the vNIC filters do not get updated with the right set of IPs causing Distributed Firewall to block the traffic.

Those with the correct entitlements can download NSX-v 6.3.5 here.

References:

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/rn/releasenotes_nsx_vsphere_635.html

Awarded vExpert Cloud – A New vExpert Sub Program

Last week Corey Romero announced the inaugural members of the vExpert Cloud sub-program. This is the third vExpert sub-program following the vSAN and NSX programs announced last year. There are 135 initial vExpert Cloud members who have been awarded the title. As it so happens I am now a member of all three which reflects on the focus I’ve had and still have around VMware’s cloud, storage and networking products leading up to and after my move to Veeam last year.

Even with my move, that hasn’t stopped me working around these VMware vertices as Veeam works closely with VMware to offer supportability and integration with vCloud Director as well as being certified with vSAN for data protection. And more recently as it pertains specifically to the vExpert Cloud program, we are going to be supporting vCloud
Director in v10 of Backup & Replication for Cloud Connect Replication and also at VMworld 2017 we where announced as a launch partner for data protection for VMware Cloud on AWS.

For those wondering what does it take to be a part of the vExpert Cloud program:

We are looking for vExperts who are evangelizing VMware Cloud and delivering on the principles of the multi-cloud world being the new normal. Specificity we are looking for community activities which follow the same format as the vExpert program (blogs, books, videos, public speaking, VMUG Leadership, conference sessions speaking and so on).

And in terms of the focus of the vExpert Cloud program:

The program is focused on VMware Cloud influencer activities, VMware, AWS and other cloud environments and use of the products and services in way that delivers the VMware Cloud reality of consistency across multi-cloud environments.

Again, thank you to Corey and team for the award and I look forward to continuing to spread the community messaging around Cloud, NSX and vSAN.

Released: vCloud Director 8.10 and 8.20 Point Updates

Last week VMware snuck out two point releases for vCloud Director 8.10 and 8.20. For those still running those versions you now have 8.10.1.1 (Build 6878548) and for 8.20 there 8.20.0.2 (Build 6875354) available for download. These are both patch upgrades and resolve a number of bugs, some of which appear to be mirrored in both versions.

Scanning the Release Notes, below are some of the more notable fixes:

8.10

  • Resource limit change for a vCloud Edge Gateway Resolves an issue where the memory limit for a compact and full-4 Edge Gateway was insufficient. Memory was increased from 512MB to 2048MB
  • Performing hardware changes to a VM fails Resolves an issue where performing hardware changes to a VM in vCloud Director fails with an error message:
  • Degraded performance due to insufficient memory Resolves an issue that could lead to an insufficient memory reservation of the NSX Edge VMs, which might cause poor performance.
  • Catalog synchronization failure Resolves an issue where synchronization of a remote catalog item fails with an out of memory, causing the vCloud Director cell to crash.

8.20

  • Incorrect status update for VMs storage profile or disk-level storage Resolves an issue that could cause a VM storage profile or disk-level storage profile to be updated incorrectly when the VM is included in a recompose operation. This fix ensures that PvdcComputeGuaranteeValidator runs even when the deployment fails in Pay-As-You-Go allocation model. With this fix, the undeploy workflow ignores the VM deployment state if the undeploy operation is called with a force=true flag.
  • Failure to move virtual machines between shared datastores Resolves a storage issue where moving a virtual machine from one shared datastore to another fails.
  • Failure to revert VM snapshots Resolves an issue that could cause reverting to a virtual machine snapshot to fail
  • Failure to allocate an external IP address and a gateway IP address Resolves several issues in managing the allocation of external IP a gateway IP addresses during VM boot and runtime when the NAT service is enabled and IP Translation is set manually.
  • Failure to delete Organization VDC Resolves an issue that could cause various operations to fail.

So a small point release for good to see the team continuing to improve the platform for those not yet able to upgrade to the latest 9.0 release. If you have the entitlements, head to the MyVMware site to download the builds.

References:

http://pubs.vmware.com/Release_Notes/en/vcd/81011/rel_notes_vcloud_director_8-10-1-1.html

http://pubs.vmware.com/Release_Notes/en/vcd/82002/rel_notes_vcloud_director_8-20-0-2.html

vCloud Director 9.0: Digging into the new Standalone VM Feature

vCloud Director 9.0 was released late last month and brought with it a number of big new features and enhancements. If you are interested in a overview of what’s new, head here to my launch post. Getting back to this post I wanted to focus on what I think is a significant change to the way in which workloads are thought about in vCD…the Standalone VM.

Standalone Virtual machines can be instantiated and viewed along with virtual machines as part of a vApp container. A filter button creates a list based on Virtual machines, virtual applications or both.

The vApp container construct in vCloud Director carries divided opinion from both services providers and customers of vCD with one side liking the fact that VMs could be grouped into logical vApps and treated as a like group or VMs such as an Exchange Cluster. While others wanted the ability to deploy standalone VMs that where more like VM instances you find in public clouds. Historically from a programatic point of view the creation of a VM within a vApp had it’s challenges in a chicken and egg type of scenario where by the composition and recomposiontion of the VM within the vApp required a specific order. This was improved from 8.0 with enhancements to vApp functionality, including the ability to reconfigure virtual machines within a vApp, and network connectivity and virtual machine capability during vApp instantiation.

Standalone Virtual Machines:

In vCloud Director 9.0 you can now create and configure individual Virtual Machines form the new HTML5 Tenant UI. Under the compute menu you now have a Virtual Machines and vApps tab. From here you can view either standalone VMs, VMs in a vApp or both. This is also where you can create a new VM. Note that you can’t create new vApps from the new UI just yet…that still needs to be done in the Flash based UI.

You now have the ability to choose from three pre-canned instance sizes which come with default resources depending on the type of VM selected. However you can still customize the VM as shown below.

When provisioned the VM is available from the new tenant UI with all the normal operations possible. The biggest difference here is that you don’t need to worry about the vApp state and that it’s independent from any other VMs. As a side note as it’s not 100% obvious, to view the console of the VM click on the icon top right of the Virtual Machine box.

Standalone VMs in vCenter and Flash UI:

Taking a look under the covers of the HTML5 UI the standalone VMs are represented slightly differently in vCenter. in Previous versions each VM was created with the VM name plus a UUID…when a standalone VM is created the VM name is just that…the VM name.

However what is interesting is when you look in the Flash UI you will see that in fact the standalone VM is still contained within a vCD vAPP construct.

So in effect, that HTML5 UI is presenting the VM as standalone, but in actual fact there is still a one to one relationship with a vApp under the covers. Taking a look back in vCenter under the folder view it’s more representative of what you see in the Flash UI.

Standalone VMs via the API:

Querying the API shows that the Standalone VMs are indeed composed within a traditional vCD vApp.

References:

https://docs.vmware.com/en/vCloud-Director/9.0/rn/rel_notes_vcloud_director_90.html

Released: NSX-v 6.3.4 and Upgrade Notes and Fixes

Last week VMware released NSX-v 6.3.4 (Build 6845891) that contains no specific new features but addresses a couple of bug fixes from previous releases. Going through the release notes there are a lot of known issues that should be known and there are more than a few that apply to service providers…specifically there are a lot around NSX Edge functions. The other interesting point to highlight about this release is that for those on NSX-v 6.3.3 there is are a couple of scripts to run against the API before upgrading to ensure all controllers are upgradable.

As mentioned, before upgrading the release notes stage that for those on NSX-v 6.3.3 they follow this VMwareKB. In a nutshell there is a bug in 6.3.3 where the NSX Controllers are reported as disconnected in the Web Client as shown below.

To fix that situation you need to execute a couple of API calls that POSTs a script to the NSX Manager as documented in the VMwareKB. This needs to be done as the NSX Manager Admin user as I found this didn’t work with an NSX Domain User or an SSO Administrator Account with NSX Org admin level permissions.

Once the second script has been run you should see a similar output to what’s shown above and have all NSX Controllers ready in a connected state which allows you to prepare for the upgrade. Once done, you can go through the normal NSX upgrade steps which will get you to the latest build.

Important Fixes :

  • Fixed Issue 1970527: ARP fails to resolve for VMs when Logical Distributed Router ARP table crosses 5K limit
  • Fixed Issue 1961105: Hardware VTEP connection goes down upon controller rebootA BufferOverFlow exception is seen when certain hardware VTEP configurations are pushed from the NSX Manager to the NSX Controller. This overflow issue prevents the NSX Controller from getting a complete hardware gateway configuration. Fixed in 6.3.4.
  • Fixed Issue 1955855: Controller API could fail due to cleanup of API server reference filesUpon cleanup of required files, workflows such as traceflow and central CLI will fail. If external events disrupt the persistent TCP connections between NSX Manager and controller, NSX Manager will lose the ability to make API connections to controllers, and the UI will display the controllers as disconnected. There is no datapath impact.

Those with the correct entitlements can download NSX-v 6.3.4 here.

References:

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/rn/releasenotes_nsx_vsphere_634.html

https://kb.vmware.com/kb/2151719

 

Enabling, Configuring and Viewing Metrics in vCloud Director 9.0

Last week I released a post on configuring Cassandra for vCloud Director 9.0 metrics. As a refresher, one of the cool features released in vCloud Director SP 5.6.x was the ability to expose VM metrics that service providers could expose to their clients via a set of API calls. With the release of vCloud Director 9.0, the metrics can now be viewed from the new HTML5 tenant UI, meaning that all service providers should be able to offer this to their customers.

With the Cassandra configuration out of the way, the next step is to use the Cell Management Tool to tell the vCD cells to push the VM Metric data. Before this, if you log into the HTML5 UI you will notice no menu for Monitoring…this only gets enabled once the metrics have have been enabled by the tool.

The command has changed from previous versions in line with removing the dependancy on the KairosDB and we are now calling a cassandra argument that has the following options:

Those familiar with the previous command to configure the metrics will see a lot more options that specify the Cassandra nodes, the original command to configure the schema, the username and password to connect to the Cassandra database with and the ttl for the data, meaning that if you wanted you could keep more than two weeks of data.

If you tail the Cassandra system.log while the process is happening you will see a bunch of tables being created and populated with the initial data.

With the done, if you go into the new HTML5 Tenant UI and go to the Virtual Machine view you should now see a Monitoring Chart drop down in the menu in the main window. From here you can choose any of the available metrics across a half hour, hour, day and week timescale.

API Calls to Retrieve Current and Historical Metrics:

If you still want to go old school the following API Calls are used to gather current and historical VM metrics for vCD VMs. The Machine ID required used the VM GUID as seen in vCenter. The ID can be sourced from the VM Name. The vCD Machine ID shown below in the brackets is what you are after.



« Older Entries