Monthly Archives: May 2017

Quick Fix: VCSA 503 Service Unavailable Error

I’ve just had to fix one of my VCSA’s again from the infamous 503 Service Unavailable error that seems to be fairly common with the VCSA even though it’s was claimed to be fixed in vCenter version 6.5d. I’ve had this error pop up fairly regularly since deploying my homelab’s vCenter Server Appliance as a version 6.5 GA instance and for the most part I’ve refrained from rebooting the VCSA just in case the error pops up upon reboot and have even kept a snapshot against the VM just in case I needed to revert to it on the high change that it would error out.

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000559b1531ef80] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

After doing a Google search for any permanent solutions to the issue, I came across a couple of posts referencing USB passthrough devices that could trigger the error which was plausible given I was using an external USB Hard Drive. IP changes seem to also be a trigger for the error though in my case, it wasn’t the cause. There is a good Reddit thread here that talks about duplicate keys…again related to USB passthrough. It also links externally to some other solutions that where not relevant to my VCSA.

Solution:

As referenced in this VMware communities forum post, to fix the issue I had to first find out if I did have a duplicate key error in the VCSA logs. To do that I dropped into the VCSA shell and went into /var/logs and did a search for any file containing device_key + already exists. As shown in the image above this returned a number of entries confirming that I had duplicate keys and that it was causing the issue.

The VMware vCenter Server Appliance vpxd 6.5 logs are located in the /var/log/vmware/vmware-vpx folder

What was required next was to delete the duplicate embedded PostGres database table entries. To connect to the embedded postgres database you need to run the following command from the VCSA shell:

To remove the duplicate key I ran the following command and rebooted the appliance, noting that the id and device_key will vary.

Once everything rebooted all the services started up and I had a functional vCenter again which was a relief given I was about five minutes away from a restore or a complete rebuild…and ain’t nobody got time for that!

vCenter (VCSA) 6.5 broken after restart from vmware

Reference:

https://communities.vmware.com/thread/556490

 

Released: vCloud Director SP 8.20.0.1

Last week VMware released an update for vCloud Director SP (Build 5439762) and while the small version increment suggest a small release, it actually contains a couple of important new features and introduces support for VVOL datastores which hopefully will get vCAN Service Providers looking at VVOLs a little more as well as a new Cell Management Tool command to help the debugging of the auto import feature. There are also a number of resolved issues that are worth reading through the release notes for.

  • Support for VVol (Virtual Volume) datastores
    These datastores were introduced in vSphere 6.0 and can now be selected for use by vCloud Director. See VMware Knowledge Base article 2113013 for more information about VVols.
  • New Cell Management Tool subcommand debug-auto-import 
    You can use this command to get more information about why an adopted VM was not imported. See the debug-auto-import command help for information about command options.

Also, just as a reminder that if you are a vCAN Service Provider currently running vCloud director or looking to run it, the vCloud Director Team has a VMLive session in June that will provide a sneak peek at vCloud Director.Next roadmap. Looking forward to seeing what goodies the vCD Product Team are going to announce in regards to vCD enhancements.

Those with the the correct entitlements can download the build here.

References:

http://pubs.vmware.com/Release_Notes/en/vcd/8-20/rel_notes_vcloud_director_8-20-0-1.html

VeeamON 2017 Wrap

VeeamON 2017 has come and gone and even though I left New Orleans on Friday afternoon, I just arrived back home…54 hours of travel, transit and delays has meant that my VeeamOFF continued longer than most! What an amazing week it was though for Veeam, our partners and our customers…The announcements that we made over the course of the event have been extremely well received and it’s clear to me that the Availability Platform vision that we first talked about last year is in full execution mode.

The TPM team executed brilliantly and along with the core team and the other 300 Veeam employee’s that where in New Orleans it was great to see all the hard work pay off. The Technical Evangelist’s main stage live demo’s all went off (if not for some dodgy HDMI) without a hitch and we all felt privileged to be able to demo some of the key announcements. On a personal note, It was a career highlight to be able to present to approximately 2000 people and be part of a brand new product launch for Veeam with Veeam PN.

From a networking point of view it was great to meet so many new people and put faces to Twitter handles. It was also great to see the strong Veeam Vanguard representation at the event and even though I couldn’t party with the group like previous years, it looked like they got a lot out of week, both from a Veeam technical point of view and without doubt on the social front…I was living vicariously through them as they where partying hard in New Orleans.

VeeamON Key Announcements:

Availability Suit 10

  • Built-in Management for Veeam Agent for Linux and Veeam Agent for Microsoft Windows
  • Scale-Out Backup Repository — Archive Tier
  • NAS Backup Support for SMB and NFS Shares
  • Veeam CDP (Continuous Data Protection)
  • Primary Storage Integrations — Universal Storage Integration API
  • DRaaS Enhancements (for service providers)
  • Additional enterprise scalability enhancements

For me, the above list shows our ongoing commitment to the Enterprise but more importantly for me working on enhancing our platform so that our Veeam Cloud and Service Providers can continue to leverage our technology to create and offer cloud based Disaster Recovery and Backup services.

Product Announcements and Releases:

I have been lucky enough to work as the TPM lead on Veeam PN and I was extremely excited to be able to demo it for the first time to the world. I’ve written a blog post here that goes into some more detail around Veeam PN and if you want to view the main stage demo I’ve linked to the video in the last section…I start the demo at the 29th minute mark if you want to skip through.

vCloud Director Cloud Connect Enhancements:

As mentioned above we have enhanced core capabilities in v10 when it comes to Cloud Connect Replication and Cloud Connect Backup. Obviously, the announcement that we will be supporting vCloud Director is significant and one that a lot of our Cloud and Service Providers are extremely happy with. It just makes the DRaaS experience that much more complete and when you add that to the CDP features in the core platform which will allow for sub minute RPO’s for replica’s it firmly places Cloud Connect as the market leader in Replication as a Service technologies.

We also announced backup to tape features for Cloud Connect Backup which will allow Cloud and Service Providers to offload long term backup files to cheaper storage. Note that this isn’t limited to tape if used in conjunction with a Virtual Tape Library. Hopefully our VCSP’s can create revenue generating service offerings around this feature as well.

VCSP Council Meeting:

On Thursday, our R&D leads met with a select group of our top Cloud and Service Provider partners over a three hour lunch meeting which could have gone all day if time permitted. It was great to be on the other side of the fence for the first time and hear all the great feedback, advice and suggestions from the group. It’s encouraging to hear about how Veeam Backup & Replication had become the central platform for IaaS, Cloud Replication an Backup offerings and with the v10 enhancements I expect that to be even more the case moving forward.

Main Stage Recordings:

Wednesday and Thursday morning both saw main stage general sessions where we announced our new products and features along with keynotes from Sanjay Poonen and Mark Russinovich as well as co-CEO Peter McKay and co-founder Ratmir Timashev. They are worth a look and I’ve posted links to the video recordings below. Note that they are unedited and contain all change overs and wait times.

https://www.veeam.com/veeamon/live

Press Releases:

Veeam is now in the Network Game! Introducing Veeam Powered Network.

Today at VeeamON 2017 we announced the Release Candidate of Veeam PN (Veeam Powered Network) which together with our existing feature, Direct Restore to Microsoft Azure creates a new solution called Veeam Disaster Recovery for Microsoft Azure. At the heart of this new solution is Veeam PN which extends an on-premises network to one that’s in Azure enhancing our availability capabilities around disaster recovery.

Veeam PN allows administrators to create, configure and connect site-to-site or point-to-site VPN tunnels easily through an intuitive and simple UI all within a couple of clicks. There are two components to Veeam PN, that being a Hub Appliance that’s deployable from the Azure Marketplace and a Site Gateway that’s downloadable from the veeam.com website and deployable on-premises from an OVA meaning it can be installed onto

Veeam PN for Microsoft Azure (Veeam Powered Network) is a free solution designed to simplify and automate the setup of a disaster recovery (DR) site in Microsoft Azure using lightweight software-defined networking (SDN).

  • Provides seamless and secure networking between on-premises and Azure-based IT resources
  • Delivers easy-to-use and fully automated site-to-site network connectivity between any site

Veeam PN is designed for both SMB and Enterprise customers, as well as service providers.

From my point of view this is a great example of how Veeam is no longer a backup company but a company that’s focused on availability. Networking is still the most complex part of executing a successful disaster recovery plan and with Veeam PN easily extending on-premises networks to DR networks as well as providing connectivity from remote sites back to DR networks via site-to-site connectivity while also providing access for remote endpoints the ability to connect into the HUB appliance and be connected to networking configured via a point-to-site connection.

Look out for more information from myself on Veeam PN as we get closer to GA.

Veeam DRaaS v10 Enhancements: vCloud Director Support!

Today at VeeamON 2017 we announced two very important enhancements to our DRaaS capabilities around Cloud Connect Replication and Tape Backup for our Veeam Cloud and Service Provider partners that help customer minimize the cost and reduce recovery times during a disaster. The press release can be found here, however as you could imagine I wanted to talk a little bit about the vCloud Director support.

A lot of service providers have been asking us to support vCloud Director in Veeam Cloud Connect Replication and I’m very happy to write that today we announced that v10 of Backup & Replication will have support for replica’s to be replicated and brought up into at service providers vCloud Director environment.

This is a significant enhancement to Cloud Connect replication end even with it being somewhat of a no brainer I am still sure it will make many VCSP people happy. With vCloud Director support in v10 tenants can now replace existing hardware plans with vCloud Director Virtual Datacenter resources. A tenant can either leverage an existing virtual datacenter or have the service provider create a dedicated one for the purpose of replication.

While Cloud Connect Replication was a strong product already with industry leading networking and ease of use, the flexibility that can be harnessed by tenants (and service providers) through the vCD platform means that there is even more control when a failover takes place. Look out for more information on our vCD integration as the v10 release gets closer…again for me, this is huge and bring’s together two of the best platforms for cloud based services even closer!

Veeam Vault #6: Pre VeeamON 2017 Edition! New Logo, Update 2, VAW and Vanguard Roundup

Time flies quickly when you’re having fun! VeeamON 2017 kicks off in New Orleans kicks off in just a few days and to say that it’s been a hectic period for the Technical Product Marketing Team and anyone at Veeam involved with VeeamON would be an understatement. All the hard work being done behind the scenes should result in a brilliantly executed event and there is going to be a lot on offer in terms of content, product announcements, learning and networking opportunities during the event. I would encourage everyone going to make sure you attend all three (one partner only) General Sessions to hear about how Veeam will continue to innovate and deliver around our Availability vision.

In this Veeam Vault I am going to round up some of the blogging content around VeeamON 2017, briefly talk about Backup & Replication 9.5 Update, the RTM of the Agent for Microsoft Windows and finish with a Veeam Vanguard Blog Post roundup since the last Veeam Vault edition.

For those attending VeeamON next week, see you there!

New Logo:

For those that hadn’t notice Veeam has rebranded and produced a new logo. Most, if not all of our public facing sites have been updated to reflect the new branding and even though I now have to throw out a number of relatively new polo shirts, I am a big fan of the new logo.

Update 2 and Agent for Windows:

A couple of weeks ago we released RTM builds of Backup & Replication 9.5 Update 2 as well as Veeam Agent for Microsoft Windows. GA isn’t far away so if you are a Veeam Cloud and Service Provider and haven’t upgraded to the RTM build yet you are probably behind the eight ball in terms of being in a position to support VAW for when it does GA. Veeam Agent for Microsoft Windows represents a massive opportunity for our VCSPs to tap into a market that was previously not accessible…this being, physical severs and workstations, workstation endpoints and more significantly cloud based Windows instances…all of which can now be backed up to Cloud Connect Repositories.

VeeamOn 2017 Blog Roundup and Mobile App:

A number of my team mates have written veeam.com blog posts about what to expect at VeeamOn this year and they are well worth a read. I, myself wrote a post last week where I listed my top sessions for 2017. Check out the posts below to get even more info on happenings at VeeamON.

Speaking of sessions and session registration, if you haven’t downloaded the VeeamON app for IOS or Android, scan the QR codes below or search for VeeamON in the App Stores:

There is also an online version of the app which can be found here.

Veeam Vanguard Blog Post Roundup:

Quick Thought: VMUG is now part of DTUC

I awoke this morning to the news that an announcement was made at DELL|EMC World that VMUG had been rolled into a the recently formed Dell Technologies User Community (DTUC – doesn’t quiet roll off the tongue now does it?) …I also awoke to a lot of VMware community backlash on Twitter not only in response to the news but also in the way in which it was not communicated to the existing local VMUG leadership and steering committee members.

From the reaction i’ve seen, most people are fairly ticked off with the fact that almost everybody found out about this through public channels…mainly Twitter. It’s worth watching the video below to get an overview of the changes from the VMUG President and CEO as it does go some way to clarifying the what’s what of the announcement.

Just to clarify, VMUG is not changing it’s name to DTUC.

https://dtusercommunity.com

My Take:

I think everybody knew that VMUG was in trouble from an organisational standpoint with a lot of changes during the first few months of 2017 and some interesting moves around the removing of Nutanix staff from leadership role. So this news isn’t a total surprise however for me, the one key ingredient that VMUG offered is now well and truly in danger of being wiped away…and that is it’s relative independence.

The VMUG community was born out of the technology ecosystem that grew around VMware’s success in the virtualization market and it meant that all of VMware’s technology and alliance partners where given a seat at the table in terms of event sponsorship and presentations. It was a place equally where smaller startup’s could come and talk about their new technology solutions and where the more established vendors could talk around why there where still cool and relevant.

Now, with DELL|EMC plus VMware product portfolio my fear is that finding sponsors will become even more of a challenge as it has been worldwide for the last 12 to 18 months. This is an interesting move but again, but not a surprising one given what I’ve seen with my involvement in VMUG over the past two years. It’s not all doom and gloom though as I feel the VMUG UserCons are still brilliant events as was the case with the recent ones held in Sydney and Melbourne.

Time will tell how this plays out, but there is one thing I believe the wider VMware community doesn’t want to see drop off or disappear…and that is the community it’s self!

VMware Flings: Top 5 – 2017 Edition

VMware has had their Lab Flings program going for a number of years now and in 2015 I wrote this post listing out my Top 5 Flings. Since then there have been some awesome Flings released and I thought it was a good time to update my Top 5 Flings to reflect the continued awesomeness generated within the VMware Labs. Since my last post there have also been a number of flings that have found their way into product releases:

Flings are apps and tools built by our engineers that are intended to be played with and explored.

There are 128 (up from 57 from August of 2015) Flings listed on the site though some have been depreciated. They range across most of VMware’s Product stack…most of them have been created out of some requirement or function that was/is lacking in the current toolset for their respective products. Most of them solve usability issues or look to resolve performance bottlenecks and look to optimize product experience.

Fling Number 5 – Storage Profile Updater

This Fling is a simple tool that enables the migration of vCloud Director virtual machines and templates from the default any storage profile to a specific storage profile. The tool can be run from the command-line with the help of a configuration file, and it allows you to change storage profiles in a batch style of processing.

For those that upgraded vCloud Director from 1.5 to 5.x you would know about the Any profile issue…this fling allows you to migrate all VMs from that default storage policy to any new one you might have configured in your Provider vDC.

Fling Number 4 – Cross vCenter VM Mobility – CLI

Cross vCenter VM Mobility – CLI is a command line interface (CLI) tool that can be used to migrate or clone a VM from one host to another host managed by a linked or isolated vCenter (VC) instance. It has been built using vSphere Java-based SDK APIs.

Currently, as of vSphere 6.0, the vSphere HTML5 Web Client allows users to perform Cross-VC operations like migration and cloning if two VCs are linked. If VCs are not linked, users cannot view the infrastructure across multiple VCs and thus, cannot utilize this functionality through UI. This Fling provides a way for users to access this vSphere feature through simple CLI commands. It also supports cross-cluster placement and shared storage vMotion between two VCs.

Cross vCenter migrations is probably one of the most underrated features VMware has released and has been present since vSphere 6.0. Originally exposed via the API’s William Lam blogged about a wrapper he wrote to use the functionality and this Fling sits beside that as possible tools to perform the cross vCenter actions.

Fling Number 3 – Embedded Host Client

The ESXi Embedded Host Client is a native HTML and JavaScript application and is served directly from your ESXi host! It should perform much better than any of the existing solutions

This Fling was a revelation when it was first released and adds a very usable and functional HTML5 web interface from which to manage your ESXi hosts. It’s now productized and packaged into ESXi 5.5, 6.0 and 6.5 and there is continues on going development of the tool along with bug fixes and features that can be installed via the vib on the Fling site.

Fling Number 2 – VMware Tools for Nested ESXi

This VIB package provides a VMware Tools service (vmtoolsd) for running inside a nested ESXi virtual machine. The following capabilities are exposed through VMware Tools:

Provides guest OS information of the nested ESXi Hypervisor (eg. IP address, configured hostname, etc.).
Allows the nested ESXi VM to be cleanly shut down or restarted when performing power operations with the vSphere Web/C# Client or vSphere APIs.
Executes scripts that help automate ESXi guest OS operations when the guest’s power state changes.
Supports the Guest Operations API (formally known as the VIX API).

The release of this Fling was met with a lot of thankyou’s from those who had battled with NestedESXi Hosts not having VMTools available. If anything, the ability to cleanly shutdown or restart the ESXi Guest was welcomed. With the release of ESXi 6.0 (and subsequent 6.5 release) the Tools are included in the OS by default…but for those running 5.x Nested Hosts its a must have.

Fling Number 1 – ESXi Mac Learning dvFilter v2.0

MAC learning functionality solves performance problems for use cases like nested ESX.  This ESX extension adds functionality to ESX to support MAC-learning on vswitch ports. For most ESX use cases, MAC learning is not required as ESX knows exactly which MAC address will be used by a VM. However, for applications like running nested ESX, i.e. ESX as a guest-VM on ESX, the situation is different. As an ESX VM may emit packets for a multitude of different MAC addresses, it currently requires the vswitch port to be put in “promiscuous mode”. That however will lead to too many packets delivered into the ESX VM, as it leads to all packets on the vswitch being seen by all ESX VMs. When running several ESX VMs, this can lead to very significant CPU overhead and noticeable degradation in network throughput. Combining MAC learning with “promiscuous mode” solves this problem. The MAC learning functionality is delivered as a high speed VMkernel extension that can be enabled on a per-port basis. It works on legacy standard switches as well as Virtual Distributed Switches

This Fling is close to my heart as I learnt at VMworld 2014 that it was born out of a blog post I did on Promiscuous Mode that triggered William Lam to approach Christian Dickmann with the issues and look for a way to solve the issue. As you can see from my followup post it works as designed and is the single must have Fling for those who run Nested ESXi labs. It was recently upgraded to version 2.0 to support ESXi 6.5.

As of last week there a new ESXi Learnswitch Fling was released which builds upon (but can’t be used with) the MAC Learning fling.

ESXi Learnswitch is a complete implementation of MAC Learning and Filtering and is designed as a wrapper around the host virtual switch. It supports learning multiple source MAC addresses on virtual network interface cards (vNIC) and filters packets from egressing the wrong port based on destination MAC lookup. This substantially improves overall network throughput and system performance for nested ESX and container use cases.

To learn more, read ESXi Learnswitch – Enhancement to the ESXi MAC Learn DvFilter.

For a full list of the Flings available for download, head to this link

https://labs.vmware.com/flings/?utf8=%E2%9C%93&order=date+DESC

VCSP Important Notice: 9.5 Update 2 RTM Is Out…With Remote Access and more!

Last Friday, Veeam made available to our VCSP partners the RTM of Update 2 for Backup & Replication 9.5 (Build 9.5.0.1038). Update 2 for Backup & Replication is what we term a breaking update, meaning that if a Cloud Connect client upgrades from the 9.5 GA (9.5.0.711) or Update 1 (9.5.0.823) this will break backup or replication functionality. With that in mind the RTM has been made available for our VCSP partners to ensure it is installed and tested before being pushed out to production before the GA release. Veeam Backup & Replication releases from 8.0 (build 8.0.0.817) can write backups to a cloud repository on 9.5 Update 2, and any release from 9.0 (build 9.0.0.902) can write replicas to a cloud host on 9.5 Update 2.

Update 2 is a fairly significant update and contains a number of enhancements and bug fixes with a lot of those enhancements aimed at improving the scalability of the Backup & Replication platform that VCSPs can take advantage of. The biggest and most anticipated update is for the support of the upcoming Veeam Agent for Microsoft Windows, but we also pushed out an awesome feature for our VCSPs to help manage their client or tenant Cloud Connect jobs or in fact manage their remote Veeam Backup & Replication instances with Remote Access.

VCSP and Cloud Connect Features and Enhancements:

Below are the main VCSP related features and enhancements included in Update 2. For a full list, head to the Veeam Cloud and Service Provider channels of the Veeam forum. For VCSPs with more than 300 connections there is a post regarding tweaks for improving performance at scale.

Core

  • Veeam Backup Remote Access: Full remote management of Veeam Backup & Replication installations at customer sites with a Veeam backup console or Microsoft RDP client over the existing Cloud Connect tunnel.
  • Veeam Agent for Microsoft Windows integration: Full support for Veeam Agent for Microsoft Windows 2.0, including direct backup to a cloud repository backed by Veeam Cloud Connect. This includes the introduction of separate workstation and server counters for Cloud Connect Backup
  • Usage reporting enhancements: Usage reports are now saved in two formats, CSV and PDF.

Cloud Connect

  • Major scalability improvements: In light of support for direct agent backups, numerous optimizations to the Cloud Connect framework on both the service provider and the tenant sides have been introduced to significantly increase its scalability (including number of concurrent tenant jobs per Cloud Connect server) and reliability large at scale. The release was tested with 1000 concurrent agent backups per single Cloud Connect instance. Reaching this number may require applying additional tweaks, please refer to this post for more information.
  • Improved reporting and monitoring: Better visibility from the service provider side into Cloud Connect tenant jobs, including current job activity and statistics such as current rate and amount of data sent and received.

Cloud Connect Replication

  • Improved failback performance: Failback to the original location can now optionally leverage VMware changed block tracking information, instead of scanning the entire VM disks to determine the difference, improving failback performance up to 20 times.
  • Hardware plan enhancements: Added the ability to increase a number of networks assigned to the Hardware Plan after it has been created.

Important Note for Veeam One Users:

As mentioned in the VCSP forum digest this week, there is a patch available for Veeam One which enables compatibility with Veeam Backup & Replication 9.5 Update 2. This hotfix must be deployed to any Veeam ONE server that is monitoring Cloud Connect services…it doesn’t need to be installed on instances not monitoring Cloud Connect services.

Conclusion:

Once again, this is an important update to apply for VCSPs running Cloud Connect services in preparation for the GA release which will happen in about two weeks. Once released I’ll link to the VeeamKB for a detailed look at the fixes but for the moment, if you have the ability to download the update do so and have it applied to your instances.