Monthly Archives: October 2017

Awarded vExpert Cloud – A New vExpert Sub Program

Last week Corey Romero announced the inaugural members of the vExpert Cloud sub-program. This is the third vExpert sub-program following the vSAN and NSX programs announced last year. There are 135 initial vExpert Cloud members who have been awarded the title. As it so happens I am now a member of all three which reflects on the focus I’ve had and still have around VMware’s cloud, storage and networking products leading up to and after my move to Veeam last year.

Even with my move, that hasn’t stopped me working around these VMware vertices as Veeam works closely with VMware to offer supportability and integration with vCloud Director as well as being certified with vSAN for data protection. And more recently as it pertains specifically to the vExpert Cloud program, we are going to be supporting vCloud
Director in v10 of Backup & Replication for Cloud Connect Replication and also at VMworld 2017 we where announced as a launch partner for data protection for VMware Cloud on AWS.

For those wondering what does it take to be a part of the vExpert Cloud program:

We are looking for vExperts who are evangelizing VMware Cloud and delivering on the principles of the multi-cloud world being the new normal. Specificity we are looking for community activities which follow the same format as the vExpert program (blogs, books, videos, public speaking, VMUG Leadership, conference sessions speaking and so on).

And in terms of the focus of the vExpert Cloud program:

The program is focused on VMware Cloud influencer activities, VMware, AWS and other cloud environments and use of the products and services in way that delivers the VMware Cloud reality of consistency across multi-cloud environments.

Again, thank you to Corey and team for the award and I look forward to continuing to spread the community messaging around Cloud, NSX and vSAN.

What I’ve Learnt from 12 Months Working From Home

This week marks one year since I started at Veeam and it feels like that twelve months has flown by. Before I started here at Veeam I had only worked for local companies here in Perth, though the last two had national presence which meant some travel interstate and occasionally overseas for events like VMworld. Prior to this role I was office bound however this role, being part of a Global Team meant that I had to work remotely from home. Something that I thought would be a walk in park…however the reality of working from home is far from that.

There is a growing norm (especially in IT) where location doesn’t matter and working remotely is embraced. For the employer they win by getting the person they want…and for the employee the boundaries of locality are lessoned meaning that more opportunities can be pursued. In my case, living in one of the most isolated cities in the world I was aware of other vendor roles where people worked from home so knew if the right role popped up, I had a chance to remain in Perth…travel a lot…and work from home.

The role that I’m in has me traveling roughly once every three days, however that has come in waves and I’ll have periods of travel, followed by periods at home meaning I could have times where i’m working from home for weeks on end. While this isn’t a definitive guide as such to working from home, I wanted to jot down some experiences and lessons learnt from my last 12 months, because the adjustment was tougher than expected.

If you want some generic advice there are lots of articles out there that list the Top Working from Home Tips, but below are my key takeaways from my experiences.

Getting into a Routine:

This is the obvious one, however it’s actually hard to achieve unless you really put your mind to it. Over the first two to three months I was finding myself still stuck int the old routine of getting up and effectively going to work. I sat in front of the computer from 8am to 5pm, had dinner, played with the kids and had family time. Problem was my team was spread across the globe and I was then working from 9pm to 12-1am so my screen time was significant. I wasn’t burning out, but came to the realisation that because of the working from home and the fact that timezones meant nothing I had to stop thinking as a 9-5 worker.

This involved setting a routine that was achievable. When home I now get up, have breakfast with the family (when possible) and then get ready to go into the study. For me, having a shower first thing is still optional and while that might disgust some people out there, I tend to wash up during my first break of the day. That break is usually around 11am after dealing with emails and when the east coast of the US starts to go to bed.

One of the things that I try and do during the middle part of the day is get out the backyard and shoot some hoops…basketball is a great game to play by yourself. Once I have lunch I usually get back on the computer for a couple of hours and then head out to the gym for a workout. Once I get back home the kids are back from school and generally its time for dinner and I try to do some family time where possible.

After family time I then do the nightshift when most of EMEA is well into their day and the US is starting to wake up. From 9pm till 12am (or later) I can work efficiently and tend to get a lot fo work done during this period. It’s also when most of the timezones I deal with are awake at the same time, so interaction with workmates is at it’s peak.

Getting over the Feeling on Loneliness:

Those that know me know that I am a pretty social guy…I love a good chat and enjoy interacting with people in the office. Those that have worked with me, also know that I like to muck around a little bit and have a laugh during the day. All in all I enjoy peoples company, so probably the biggest adjustment to working from home was the fact that I did feel lonely to begin with. It was good to hear that people I have mentioned that to that also work from home had felt that as well…good in that I wasn’t alone in this.

The key that i’ve found to combating that sense of isolation is to ensure that I am not home bound 24 hours a day, five days a week. The thing that solved this for me was developing a routine where I get out of the house to go to the gym to be around other humans…and while I am not exactly having conversations with people at the gym I’m at least physically around people which seemed to help.

In addition to that, social media platforms like Slack, WhatsApp and Skype are critical social interactions and while they can be sometimes distracting…they are critical to making sure that I feel connected with the outside world which in turn helps beat the isolation.

Having a Proper Home Office Setup:

The last thing is around having a decent home office setup. I know a lot of people that work from home but work at the dinner table or on the kitchen bench. This isn’t conducive to being able to work constantly or efficiently. I made sure that there was a decent study when looking for a house and I’m lucky enough to have a good one at the moment. It’s isolated from the main living area of the house and setup in such a way that it closely replicates an proper office.

Apart from all the right technology one of the biggest things for me is keeping this space tidy and organised. It’s important to maintain a high standard even though no one else gets to see the setup. Apart from that the other thing I’ve learnt is to make it as desirable as possible to be around…because I spend all day there I want to feel like I want to be there. Apart from the job being rewarding, for me it’s important to have a sense of pride in your work space so even if working from home it’s an important point to consider.

Wrap Up:

One thing to finish up on is that support comes in many flavours while working from home. I’m lucky that I have a great boss and a great team that I work with…they help tremendously in making the working from home thing work. Without a great team and support structure it would be indeed be a lonely gig.

All in all, after a period of adjustment I’ve settled into a decent routine while keeping myself sane during the periods when I am working from home. Ultimately what I learnt during the first twelve months of working from home is that you have to be disciplined. With the discipline to stick to my routines and get into a rhythm day in and day out it’s become easier and more natural. That said, I still miss the office atmosphere however there is some sacrifice that needs to be taken in order to work in a role that is ultimately very rewarding.

And like actually being at an office…the key is to minimize distractions!

Released: vCloud Director 8.10 and 8.20 Point Updates

Last week VMware snuck out two point releases for vCloud Director 8.10 and 8.20. For those still running those versions you now have 8.10.1.1 (Build 6878548) and for 8.20 there 8.20.0.2 (Build 6875354) available for download. These are both patch upgrades and resolve a number of bugs, some of which appear to be mirrored in both versions.

Scanning the Release Notes, below are some of the more notable fixes:

8.10

  • Resource limit change for a vCloud Edge Gateway Resolves an issue where the memory limit for a compact and full-4 Edge Gateway was insufficient. Memory was increased from 512MB to 2048MB
  • Performing hardware changes to a VM fails Resolves an issue where performing hardware changes to a VM in vCloud Director fails with an error message:
  • Degraded performance due to insufficient memory Resolves an issue that could lead to an insufficient memory reservation of the NSX Edge VMs, which might cause poor performance.
  • Catalog synchronization failure Resolves an issue where synchronization of a remote catalog item fails with an out of memory, causing the vCloud Director cell to crash.

8.20

  • Incorrect status update for VMs storage profile or disk-level storage Resolves an issue that could cause a VM storage profile or disk-level storage profile to be updated incorrectly when the VM is included in a recompose operation. This fix ensures that PvdcComputeGuaranteeValidator runs even when the deployment fails in Pay-As-You-Go allocation model. With this fix, the undeploy workflow ignores the VM deployment state if the undeploy operation is called with a force=true flag.
  • Failure to move virtual machines between shared datastores Resolves a storage issue where moving a virtual machine from one shared datastore to another fails.
  • Failure to revert VM snapshots Resolves an issue that could cause reverting to a virtual machine snapshot to fail
  • Failure to allocate an external IP address and a gateway IP address Resolves several issues in managing the allocation of external IP a gateway IP addresses during VM boot and runtime when the NAT service is enabled and IP Translation is set manually.
  • Failure to delete Organization VDC Resolves an issue that could cause various operations to fail.

So a small point release for good to see the team continuing to improve the platform for those not yet able to upgrade to the latest 9.0 release. If you have the entitlements, head to the MyVMware site to download the builds.

References:

http://pubs.vmware.com/Release_Notes/en/vcd/81011/rel_notes_vcloud_director_8-10-1-1.html

http://pubs.vmware.com/Release_Notes/en/vcd/82002/rel_notes_vcloud_director_8-20-0-2.html

vCloud Director 9.0: Digging into the new Standalone VM Feature

vCloud Director 9.0 was released late last month and brought with it a number of big new features and enhancements. If you are interested in a overview of what’s new, head here to my launch post. Getting back to this post I wanted to focus on what I think is a significant change to the way in which workloads are thought about in vCD…the Standalone VM.

Standalone Virtual machines can be instantiated and viewed along with virtual machines as part of a vApp container. A filter button creates a list based on Virtual machines, virtual applications or both.

The vApp container construct in vCloud Director carries divided opinion from both services providers and customers of vCD with one side liking the fact that VMs could be grouped into logical vApps and treated as a like group or VMs such as an Exchange Cluster. While others wanted the ability to deploy standalone VMs that where more like VM instances you find in public clouds. Historically from a programatic point of view the creation of a VM within a vApp had it’s challenges in a chicken and egg type of scenario where by the composition and recomposiontion of the VM within the vApp required a specific order. This was improved from 8.0 with enhancements to vApp functionality, including the ability to reconfigure virtual machines within a vApp, and network connectivity and virtual machine capability during vApp instantiation.

Standalone Virtual Machines:

In vCloud Director 9.0 you can now create and configure individual Virtual Machines form the new HTML5 Tenant UI. Under the compute menu you now have a Virtual Machines and vApps tab. From here you can view either standalone VMs, VMs in a vApp or both. This is also where you can create a new VM. Note that you can’t create new vApps from the new UI just yet…that still needs to be done in the Flash based UI.

You now have the ability to choose from three pre-canned instance sizes which come with default resources depending on the type of VM selected. However you can still customize the VM as shown below.

When provisioned the VM is available from the new tenant UI with all the normal operations possible. The biggest difference here is that you don’t need to worry about the vApp state and that it’s independent from any other VMs. As a side note as it’s not 100% obvious, to view the console of the VM click on the icon top right of the Virtual Machine box.

Standalone VMs in vCenter and Flash UI:

Taking a look under the covers of the HTML5 UI the standalone VMs are represented slightly differently in vCenter. in Previous versions each VM was created with the VM name plus a UUID…when a standalone VM is created the VM name is just that…the VM name.

However what is interesting is when you look in the Flash UI you will see that in fact the standalone VM is still contained within a vCD vAPP construct.

So in effect, that HTML5 UI is presenting the VM as standalone, but in actual fact there is still a one to one relationship with a vApp under the covers. Taking a look back in vCenter under the folder view it’s more representative of what you see in the Flash UI.

Standalone VMs via the API:

Querying the API shows that the Standalone VMs are indeed composed within a traditional vCD vApp.

References:

https://docs.vmware.com/en/vCloud-Director/9.0/rn/rel_notes_vcloud_director_90.html

Released: NSX-v 6.3.4 and Upgrade Notes and Fixes

Last week VMware released NSX-v 6.3.4 (Build 6845891) that contains no specific new features but addresses a couple of bug fixes from previous releases. Going through the release notes there are a lot of known issues that should be known and there are more than a few that apply to service providers…specifically there are a lot around NSX Edge functions. The other interesting point to highlight about this release is that for those on NSX-v 6.3.3 there is are a couple of scripts to run against the API before upgrading to ensure all controllers are upgradable.

As mentioned, before upgrading the release notes stage that for those on NSX-v 6.3.3 they follow this VMwareKB. In a nutshell there is a bug in 6.3.3 where the NSX Controllers are reported as disconnected in the Web Client as shown below.

To fix that situation you need to execute a couple of API calls that POSTs a script to the NSX Manager as documented in the VMwareKB. This needs to be done as the NSX Manager Admin user as I found this didn’t work with an NSX Domain User or an SSO Administrator Account with NSX Org admin level permissions.

Once the second script has been run you should see a similar output to what’s shown above and have all NSX Controllers ready in a connected state which allows you to prepare for the upgrade. Once done, you can go through the normal NSX upgrade steps which will get you to the latest build.

Important Fixes :

  • Fixed Issue 1970527: ARP fails to resolve for VMs when Logical Distributed Router ARP table crosses 5K limit
  • Fixed Issue 1961105: Hardware VTEP connection goes down upon controller rebootA BufferOverFlow exception is seen when certain hardware VTEP configurations are pushed from the NSX Manager to the NSX Controller. This overflow issue prevents the NSX Controller from getting a complete hardware gateway configuration. Fixed in 6.3.4.
  • Fixed Issue 1955855: Controller API could fail due to cleanup of API server reference filesUpon cleanup of required files, workflows such as traceflow and central CLI will fail. If external events disrupt the persistent TCP connections between NSX Manager and controller, NSX Manager will lose the ability to make API connections to controllers, and the UI will display the controllers as disconnected. There is no datapath impact.

Those with the correct entitlements can download NSX-v 6.3.4 here.

References:

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.3/rn/releasenotes_nsx_vsphere_634.html

https://kb.vmware.com/kb/2151719

 

Enabling, Configuring and Viewing Metrics in vCloud Director 9.0

Last week I released a post on configuring Cassandra for vCloud Director 9.0 metrics. As a refresher, one of the cool features released in vCloud Director SP 5.6.x was the ability to expose VM metrics that service providers could expose to their clients via a set of API calls. With the release of vCloud Director 9.0, the metrics can now be viewed from the new HTML5 tenant UI, meaning that all service providers should be able to offer this to their customers.

With the Cassandra configuration out of the way, the next step is to use the Cell Management Tool to tell the vCD cells to push the VM Metric data. Before this, if you log into the HTML5 UI you will notice no menu for Monitoring…this only gets enabled once the metrics have have been enabled by the tool.

The command has changed from previous versions in line with removing the dependancy on the KairosDB and we are now calling a cassandra argument that has the following options:

Those familiar with the previous command to configure the metrics will see a lot more options that specify the Cassandra nodes, the original command to configure the schema, the username and password to connect to the Cassandra database with and the ttl for the data, meaning that if you wanted you could keep more than two weeks of data.

If you tail the Cassandra system.log while the process is happening you will see a bunch of tables being created and populated with the initial data.

With the done, if you go into the new HTML5 Tenant UI and go to the Virtual Machine view you should now see a Monitoring Chart drop down in the menu in the main window. From here you can choose any of the available metrics across a half hour, hour, day and week timescale.

API Calls to Retrieve Current and Historical Metrics:

If you still want to go old school the following API Calls are used to gather current and historical VM metrics for vCD VMs. The Machine ID required used the VM GUID as seen in vCenter. The ID can be sourced from the VM Name. The vCD Machine ID shown below in the brackets is what you are after.



Veeam Vault #9: Backup for Office 365 1.5 GA, Azure Stack and Vanguard Roundup

Welcome to another Veeam Vault! This is the ninth edition and given the last edition was focused around VMware and VMworld I thought just for a change, the focus for this edition will be Microsoft. Reason for that is over the past couple of weeks we have had some significant announcements around Azure Stack and the GA release of Backup for Office 365 1.5. I’ll cover both of those announcements, share some Veeam employee automation work that shows off the power of our new APIs and see what the Veeam Vanguard’s have been blogging about in the last month or so.

Backup for Office 365 1.5 GA:

The early part of my career was dedicated to Exchange Server however I drifted away from that as I made the switch to server virtualization and cloud computing. The old Exchange admin in my is still there however and it’s for that reason that I’m excited about the GA of our Backup for Office 365 product which is now at version 1.5. This release caters specifically for service providers adding scalability and automation enhancements as well as extended support for on-premises and hybrid Exchange setups.

New features and enhancements:

  • Distributed, scalable architecture: Enhanced scalability in distributed environments with several Remote Offices/Branch Offices and in service providers infrastructures
  • Backup proxies: take the workload off the management server, providing flexible throttling policy settings for performance optimization.
  • Support for multiple repositories: Streamlines data backup and restore processes.
  • Support for backup and restore of on-premises and hybrid Exchange organizations: Allows a variety of configurations and usage scenarios and implement those that meet your particular needs.
  • Increased performance: Restore operations allows for up to 5 times faster restores than in v1.0.
  • Restore of multiple datastore mailboxes using Veeam Explorer for Microsoft Exchange: simplifies workflow and minimizes workload for restore operators, as well as 1-Click restore of a mailbox to the original location.
  • RESTful API and PowerShell cmdlets: Helpful for automation of routine tasks and integration into existing or new portals.
  • UI Enhancements: Including main window, wizards, dialogs, and other elements, facilitating administration of the solution.
Examples of the Power of the Veeam APIs:

One of the features of Backup for Office 365 was the addition of a power set of RESTful APIs and PowerShell commandlets that are aimed are service providers automating the setup and management of their offerings around the product. A couple of our employees have written example interfaces for the Backup for Office 365 product and it shows that any service provider with some in house programming skill set can build customer portals that enhances their offerings and increases efficiency through automation.

Special welcome to Niels who this week joined our team. Great to have you on board!

Microsoft Azure Stack Support:

Last week at Microsoft Ignite, we announce our supportability for Azure Stack. This is based around our Windows Agent, Cloud Connect and Availability Console products that combine together to off an availability solution

Key benefits of Veeam’s support for the Azure Stack include:

  • Multi-tenancyVeeam Cloud Connect isolates backup copies for each tenant ensuring security and compliance; 
  • Multiple recovery options: Veeam Backup & Replication supports both granular item level recovery through Veeam Explorers for Microsoft Exchange, SQL Server, Microsoft SharePoint, Microsoft Active Directory and for Oracle, as well as full file level restores for tenant files that were deleted or corrupted;
  • Reporting & Billing: Veeam Availability Console supports real-time monitoring and chargeback on tenant usage, allow either Hosting providers or Enterprise organizations to easily manage and bill their tenants for Availability usage.

Veeam Vanguard Blog Post Roundup:

References:

https://helpcenter.veeam.com/docs/vbo365/guide/vbo_what’s_new_in_v1_5.html?ver=15

Configuring Cassandra for vCloud Director 9.0 Metrics

One of the cool features released in vCloud Director SP 5.6.x was the ability to expose VM metrics that service providers could expose to their clients via a set of API calls. Some service providers took advantage of this and where able to offer basic VM metrics to their tenants through customer written portals. Zettagrid was one of those service providers and while I was at Zettagrid, I worked with the developers to get VM metrics out to our customers.

Part of the backend configuration to enable the vCloud Director cells to export the metric data was to stand up a Cassandra/KairosDB cluster. This wasn’t a straight forward exercise but after a bit of tinkering due to a lack of documentation, most service providers where able to have the backend in place to support the metrics.

With the release of vCloud Director 9.0, the requirement to have KairosDB managed by Apache has been removed and metrics can now be accessed natively in Cassandra using the cell management tool. Even cooler is that the metrics can now be viewed from the new HTML5 tenant UI, meaning that all service providers should be able to offer this to their customers.

Cassandra is an open source database that you can use to provide the backing store for a scalable, high-performance solution for collecting time series data like virtual machine metrics. If you want vCloud Director to support retrieval of historic metrics from virtual machines, you must install and configure a Cassandra cluster and use the cell-management-tool to connect the cluster to vCloud Director. Retrieval of current metrics does not require optional database software.

The vCloud Director online docs have a small install guide but it’s not very detailed. It basically says to install and configure the Cassandra cluster with four nodes, two of which are seed nodes, enabling encryption and user authentication with Java Native Access installed. Not overly descriptive. I’ve created an script below that installs and configures a basic single node Cassandra cluster that will suffice for most labs/testing environments.

Setting up Cassandra on Ubuntu 16.04 LTS:

I’ve forked an existing bash script on Github and added modifications that goes through the installation and configuration of Cassandra 2.2.6 (as per the vCD 9.0 release notes) on a single node, enabling authentication while disabling encryption in order to keep things simple.

This will obviously work on any distro that supports apt-get. Once configured you can view the Cassandra status by using the nodetool status command as shown below.

The manual steps for the Cassandra installation are below…note that they don’t include the configuration file changes required to enable authentication and set the seeds.

From here you are ready to configure vCD to push the metrics to the Cassandra database. I’ll cover that in a seperate post.

References:

https://docs.vmware.com/en/vCloud-Director/9.0/com.vmware.vcloud.install.doc/GUID-E5B8EE30-5C99-4609-B92A-B7FAEC1035CE.html

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/vcloud/vmware-vcloud-director-whats-new-9-0-white-paper.pdf

vCloud Director 9.0: Manual Quick fix for VXLAN Network Pool Error

vCloud Director 9.0, released last week has a bunch of new enhancements and a lot of those are focused around it’s integration with NSX. Tom Fojta has a what’s new page on the go with a lot of the new features being explained. One of his first posts just after the GA was around the new feature of being able to manually create VXLAN backed Network Pools.

VXLAN Network Pool is recommended to be used as it scales the best. Until version 9, vCloud Director would create new VXLAN Network Pool automatically for each Provider VDC backed by NSX Transport Zone (again created automatically) scoped to cluster that belong to the particular Provider VDC. This would create multiple VXLAN network pools and potentially confusion which to use for a particular Org VDC.

In vCloud Director 9.0 we now have the option of creating a VXLAN backed network pool manually instead of one being created at the time of a setting up a Provider vDC. In many of my environments for one reason or another the automatic creation of VXLAN network pool together with NSX would fail. In fact my current NextedESXi SliemaLabs vCD instance shows the following error:

There is a similar but less serious error that can be fixed by changing the replication mode from within the NSX Web Client as detailed here by Luca, however like my lab I’ve know a few people to run into the more serious error as shown above. You can’t delete the pool and a repair operation will continue to error out. Now in vCD 9.0 we can create a new VXLAN Network Pool form the Transport Zones created in NSX.

Once that’s been done you will have the newly created VXLAN Network Pool that’s truly more global and tied to best practice for NSX Transport Zones and one that can be used with the desired replication mode. The old one will remain, but you can now configure Org vDCs to consume the VXLAN backed network pool over the traditional VLAN backed pool.

References:

vCloud Director 9: What’s New

vCloud Director 9: Create VXLAN Network Pool