Tag Archives: vCloud SP

Veeam 9.5 – What’s in it for Service Providers

Since Backup and Replication 7 Veeam have continued to develop new features in enhancements to support their Cloud and Service Provider community. This started with vCloud Director support…continued with Cloud Connect Backup in v8 and more recently with Cloud Connect Replication in the v9 release. Veeam Backup and Replication 9.5 was released a couple of weeks ago and with it came a bunch of new features and enhancements that VCSPs can take advantage of.

In my initial v9.5 What’s New post I covered off core features and enhancements and without question some of those I talked about will help VCSPs gain enhanced scalability and efficiency in their current availability offerings based on Veeam, weather that be general IaaS and VM backups or if they are offering Cloud Connect Backup and/or Replication. From that first post, I have listed in short some of those features and enhancements that VCSPs will benefit from:

  • Advanced Data Fetcher and Infrastructure Cache
  • Database Optimizations
  • Improved Instance VM recovery
  • Parallel Processing of Full VM Restores
  • Advanced ReFS Support
  • Parallel Processing of Per VM Backups
  • Proxy Affinity
  • Scale Out Repository Temporary Expansion
  • PowerShell and RESTful API Enhancements

Moving on from the core enhancements, without question my favourite new Cloud and Service Provider feature in 9.5 is the introductions of the vCloud Director Self Service Tenant Portal. I’ve blogged about that in preview here, and without going into too much detail in this post, Veeam saw the need to increase efficiency for VCSPs by empowering their vCloud Director tenants with a self service backup and restore portal based on Veeam Backup Enterprise Manager.

vCloud Director Self Service Tenant Portal:

  • Native vCloud Authentication and Integrated Access: Allows tenants to continue to use their existing vCloud Director credentials for the self service portal and restricts them from being able to backup and restore only the VMs belonging to their respective organizations.
  • Enhanced Backup Control: Self service backup allows tenants to maintain better control over their backups by controlling existing jobs and creating new ones. Job setup is simplified so tenants only need to select VMs to perform backups of, as well as select essential parameters such as guest credentials, retention and notifications. Tenants are blocked from accessing advanced settings, such as repository or backup mode selection, which are managed by service providers through job templates
  • Enhanced Restore: Self service restore allows tenants to perform a wide variety of restore options for VMs that Veeam Backup Enterprise Manager currently provides including application items, guest files, full VM and full vApp restores. These restores can now be easily performed in just a few clicks by tenants themselves, improving RTOs and reducing costs for service providers

Just as a reminder this feature required Enterprise Plus to access the vCloud Director self service portal and I will be writing a more in depth blog post on this over the next few weeks. With regards to Cloud Connect, 9.5 has added improvements for both the Cloud Service Providers and their tenants with the biggest enhancement again adding to the scalability of the service by adding parallel processing for both Backup and Replication jobs. We have also extended support for key v9 features such as Per-VM backup chains, Scale Out Backup Repositories and ReFS.

Cloud Connect – Service Provider:

  • Per-VM backup file chains: Cloud repositories can now be backed by backup repositories with the per-VM backup file chains setting enabled to improve scalability and better support for deduplicating storage appliances.
  • Scale-out Backup Repository: Cloud repositories can now be backed by scale-out backup repositories to simplify backup storage management and save costs for service providers.
  • Advanced ReFS Integration: Cloud repositories backed by backup repositories meeting the requirements for Advanced ReFS Integration fully support the corresponding functionality.
  • Improved Diagnostic Logging: Multiple improvements were made to specific areas of tenant job logging where it was possible to do so without exposing a tenant’s confidential information

 Cloud Connect – Tenant:

  • Parallel Processing: Tenants can now backup or replicate multiple VMs and disks in parallel, based on the concurrent task limit set by the service provider, thus improving job performance on fast links. Please note that parallel processing applies to direct transport mode only meaning that WAN accelerators are not supported.
  • Replication From Backup In A Cloud Repository. Tenants can now perform replication from a backup residing in a cloud repository, enabling a new DR option without generating additional network traffic or impacting production VMs.
  • Security enhancements: Veeam Cloud Connect service will now attempt to use more secure TLS 1.2 and TLS 1.1 authentication algorithms when establishing a connection to the service provider. Failover to SSL 3.0 has been disabled for all Veeam Cloud Connect components.
  • Configuration Backup To Cloud Repository: Added support for performing configuration backups to a cloud repository, except for those backed by a scale out repository.

Overall another great update for the VCSPs and their tenants alike and again, if you have Veeam 9 running do yourself a favour and go through the required change controls to upgrade to v9.5…your backups will thank you! 🙂

References:

https://www.veeam.com/veeam_backup_9_5_whats_new_en_wn.pdf

Released: vCloud Director SP 8.10.1 Important Upgrade for Zerto Clients

This week VMware released vCloud Director SP 8.10.1 Build 4655197. This is the sister build for vCD SP 8.0.2 and like that release, while there a a number of minor bug fixes in this release there is one important fix that will make service providers who offer replication services built upon Zerto happy, as it resolves a bug that had stopped many service providers upgrading from vCD SP 5.6.x…however unlike the release notes in 8.0.2 it doesn’t mention the specific fix in the notes. By all acounts the hot-fix that was released prior to this offical build is in this build…if you still have issues after this build please let VMware know through GSS.

 Apart from the bug fixes, there is one new feature in this build and that is something that will be welcomed by a lot of vCD users and that is Enhanced Boot Options.

  • New boot customization options added to delay the boot time and to enter into the BIOS setup screen. You can use the vCloud Director Web console or the vCloud API to set Boot Delay and EnterBIOS mode options.

There is also official support for NSX-v 6.2.4 and that’s now covered by all the latest vCD SP versions as you can see below.

As usual I’ve gone through the Resolved Issues list and highlighted the ones I feel are most relevant…the ones in red are issues we had seen my old employers vCloud Zones and Zettagrid Labs.

  • Deployment of vApp template in My Cloud with Hardware Modification fails with null UI Error
    Attempts to deploy vApp in My Cloud from the vApp template with hardware modificat
  • After vCloud Director upgrade, the vCloud Director version does not change in vCenter Solutions Manager
    After successful upgrade of the vCloud Director from version 8.0.1 to 8.10.0, the vCloud Director version in vCenter Solutions Manager does not update and remains 8.0.1.
  • Uploading ISO media file does not consume quota that is set after the storage policy is configured to organization vDC
    When you configure the storage policy to organization virtual datacenter (vDC) and set a quota limit, the quota is not consumed while uploading the ISO media file.
  • vCloud Director database upgrade takes long time to complete when the audit_event table contains millions of records
    Database upgrade of vCloud Director from versions 5.5.x, 5.6.x to versions 8.0, 8.0.x, 8.10 might take up to 8 hours time to complete if the audit_event table contains millions of records. This issue is resolved in vCloud Director 8.10.1. The database upgrade might now take up to 20 minutes.
  • VMware vCloud Director (vmware-vcd) services do not start automatically upon a reboot
    The VMware vCloud Director (vmware-vcd) services do not start automatically after a reboot because of an issue in the systemd-219-19.el7 module of Red Hat Enterprise Linux 7.2 that includes the upgrade to Red Hat Enterprise Linux 7.3.

This will more than likely be the last build of the current 8.0 and 8.10 releases with a closed BETA of the next vCD SP currently underway. This next major release of vCD SP promised to deliver on new UI enhancements (HTML5) and deep NSX-v integration.

References:

http://pubs.vmware.com/Release_Notes/en/vcd/8-10/rel_notes_vcloud_director_8-10-1.html

VMworld Europe 2016: vCloud Air Network Out in Force!

VMworld Europe is a little over a week away and while I won’t be attending the even in Barcelona in looking through the session catalog for Partner Exchange and VMworld proper the refocus on the vCloud Air Network that was announced last year at VMworld 2015 is being put well into action. Though since VMworld in the US I have been a little more skeptical given the Cloud Foundations and Cross Cloud Platform announcements…however I am sure that VMware is focused on ensuring the success of it’s vCAN Service Providers as IaaS continues to grow in this new hybrid world.

With another successful VMworld US and VMware Partner Exchange in the books, the vCloud Air Network team is headed to Barcelona, Spain for VMworld Europe 2016 – held at the Fira Gran Via from 17-20 October.

If you are attending Partner Exchange there are a bunch of great sessions that should be on your list for the Sunday ranging from an overview of the vCloud Architecture Toolkit, sessions on Hybrid Cloud with vCloud Director and sessions around storage and networking specific for vCAN Service Providers. I’ve listed down my top picks below pulled from the vCloud Teams recent blog and added links to them for easy searchability in the VMworld Session Catalog.

  • PAR3700 – Building and Enabling a Hybrid Cloud with vCloud Director – a Perspective for Service Providers
  • RTM3702 – Route to Market Session – vCloud Air Network Service Provider Partners
  • PAR3831 – Hybrid Cloud Networking & Security
  • PAR3708 Hyper-Converged, Software-Defined Storage: A New Profitable Revenue Path for Cloud Service Providers
  • PAR3714 – Winning SDDC and Hybrid Cloud Deals Against Competition
  • PAR3728 – Hybrid Cloud Migration Strategies

Looking through the breakout sessions and quick talks at VMworld proper there is still a lot of content relating to vCloud Air…however this seems to be more DR focused and there are a lot of sessions around leveraging NSX for Hybrid Cloud and enhanced vMotion capabilities in vCenter.

  • HBC7602 – Build True Hybrid Clouds: See How Service Providers Can Use NSX to Extend Customers On-prem Data Centers
  • HBC8474 – Making it Easy to Orchestrate and Automate Your Hybrid Cloud Environment
  • HBC8799 – How OVH, vCloud Air Network Service Provider, is Using NSX to Easily Onboard Your Workloads to the Cloud
  • HBC7700 – Disaster Recovery in the Cloud with VMware Availability
  • HBC9171 – Intercontinental vMotion with Purpose

There is also a decent looking self paced HoL looking at building a vCD based IaaS Platform.

  • SPL-1787-USE-1 – VMware vCloud Director for Service Providers – Building IaaS Platform

Apart from what I have listed above there will be a bunch of vCAN talent hovering around the conference so make sure you make an effort to connect, network and share vCAN experiences. We are effectively all in this together and if the vCAN grows stronger…we all grow stronger.

#LongLivevCD

VCA-CLI for vCloud Director: New Networking Features

There is a lot of talk going around how IT Pros can more efficiently operate and consume Cloud Based Services…AWS has lead the way in offering a rich set of APIs for it’s clients to use to help build out cloud applications and infrastructure and there are a ton of programming libraries and platforms that have seen the rise of the DevOps movement…And while AWS has lead the way, other Public Clouds such as Azure (with PowerShell Packs) and Google have also built self service capability through APIs.

vCloud Director has always had a rich set of APIs (API Online Doco Here) and as I blogged about last year Paco Gomez has been developing a tool called VCA-CLI which is based on pyvcloud which is a Python SDK for vCloud Director and vCloud Air. This is an alternative to Web Based creation and management of vCloud Director vDCs and vApps. Being Python based you have the option of running it pretty much on any OS you like…the posts below show you how to install and configure VCA on a Mac OS X OS and Windows and how to connect up to a vCloud Director based Cloud Org.

Initial releases of VCA-CLI didn’t have the capability to configure the Firewall settings of a vDC Edge Gateway, but since the release of version 16, Firewall rule management has been added. In the below example, I connect up to my vCD Org in Zettagrid, gather some information about my vDC, deploy a SexiLog VM template, set the Syslog setting on the Gateway and then configure a new NAT and Firewall rules to open up port 8080 to the SexiLog Web interface.

And the end result:

Again, this highlights the power of the vCloud Director API and what can be done with the pyvcloud Python SDK. Once perfected the set of commands above can be used to deploy vApps and configure networking in seconds instead of having to work through the vCloud Director UI…and that’s a win win!

References:

https://pypi.python.org/pypi/vca-cli

https://github.com/vmware/vca-cli

http://www.sexilog.fr/

 

NSX Bytes: NSX-v 6.2.4 Released …Important Upgrade!

NSX-v 6.2.4 was released the week before VMworld US so might have gotten somewhat lost in the VMworld noise…For those that where fortunate enough to not upgrade to or deploy a greenfield 6.2.3 site you can now safely do so without the nasty bugs that existed in the 6.2.3 build. In a nutshell this new build delivers all the significant features and enhancements announced in 6.2.3 without the dFW or Edge Gateway bugs that forced the build being pulled from distribution a few weeks back.

In terms of how and when to upgrade from previous versions the following table gives a great overview of the pathways required to get to 6.2.4.

The take away from the table above is that if possible you need to get onto NSX-v 6.2.4 as soon as possible and with good reason:

  • VMware NSX 6.2.4 provides critical bug fixes identified in NSX 6.2.3, and 6.2.4 delivers a security patch for CVE-2016-2079 which is a critical input validation vulnerability for sites that uses NSX SSL VPN.
  • For customers who use SSL VPN, VMware strongly recommends a review of CVE-2016-2079 and an upgrade to NSX 6.2.4.
  • For customers who have installed NSX 6.2.3 or 6.2.3a, VMware recommends installing NSX 6.2.4 to address critical bug fixes.

Prior to this release if you had upgraded to NSX-v 6.1.7 you where stuck and not able to upgrade to 6.2.3. The Upgrade matrix is now reporting that you can upgrade 6.1.7 to 6.2.4 as shown below.

I was able to validate this in my lab going from 6.1.7 to 6.2.4 without any issues.

NSX-v 6.1.4 is also fully supported by vCloud Director SP 8.0.1 and 8.10

References:

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.4/releasenotes_nsx_vsphere_624.html

http://www.theregister.co.uk/2016/07/22/please_dont_upgrade_nsx_just_now_says_vmware/

Released: NSX 6.2.3 – Packed Full Of New Features!

Last week VMware released NSX-v 6.2.3 Build 3979471 and it’s anything but your standard point release. Running through the list off the release notes this could have easily been a major dot release. In good news for vCloud Air Network Service Providers there have been some major enhancements to the Edge Services Gateways which adds availability and protocol enhancements as well as added general stability through bug fixes and security updates.

There has also been additional management and monitoring built into the Web Client and other UI enhancements. The new licensing features as previously discussed in this post have come into effect as of this build so you will now see the license type and number of licenses used for VXLAN and DFW in the Web Client under NSX Managers -> Summary

As this is a big release I am going to filter through the release notes and pick the best features and fixes as it pertains to Service Providers and highlight the ones that I feel improve the ability to SPs to deliver strong networking services based on NSX-v as part of their service offerings.

Web Client Additions:

As mentioned above there have been a few UI enhancements in the 6.2.3 release including a new NSX Dashboard (shown below) that provides visibility into the overall health of NSX components in one view, Traceflow Enhancement for Network Introspection Services and the Firewall rules UI now displays configured IP protocols and TCP/UDP port numbers associated with services.

Going through the upgrade from previous NSX versions I noticed a few other UI additions. Once the Controllers are upgraded you can now see Disk Latency of each controller disk. The Controllers are extremely disk sensitive so it’s good to see this worked into the UI.

In addition to that new installations of NSX 6.2.3 will deploy NSX Controllers with updated disk partitions to provide extra cluster resiliency. Previously log overflow on the controller disk might impact controller stability. If you upgrade to NSX 6.2.3 the Controller will retain their original disk layout.

I also noticed a Channel Health option in the Host Preparation Tab that shows the status of the NSX Host agents and there are some other UI additions letting you modify the UUID of the NSX Instance and modify the VXLAN Port which can be done under Logical Network Preperation -> VXLAN Transport.

NSX Edge Service Gateway Changes:

As mentioned there have been a number of enhancements to the NSX ESGs which have further added to the maturity of the Edge appliance and makes it even more attractive for use with vCloud Director offering Hybrid Networking solutions…or just as a web frontend for key internet services. IS-IS has also been removed as a routing protocol option under dynamic routing as support has been pulled. TLS 1.0 has been depreciated and there have been some Cipher support changes for the IPSec, SSLVPN and L2VPN.

  • New Edge DHCP Options: DHCP Option 121 supports static route option, which is used for DHCP server to publish static routes to DHCP client; DHCP Options 66, 67, 150 supports DHCP options for PXE Boot; and DHCP Option 26 supports configuration of DHCP client network interface MTU by DHCP server.
  • Increase in DHCP Pool, static binding limits: The following are the new limit numbers for various form factors: Compact: 2048; Large: 4096; Quad large: 4096; and X-large: 8192.
  • Edge Firewall adds SYN flood protection: Avoid service disruptions by enabling SYN flood protection for transit traffic. Feature is disabled by default, use the NSX REST API to enable it.
  • NSX Edge — Resource Reservation: Reserves CPU/Memory for NSX Edge during creation. Admin user can modify the CPU/Memory settings after NSX Edge deployment using REST API to configure VM appliances.
  • Change in NSX Edge Upgrade Behavior: Replacement NSX Edge VMs are deployed before upgrade or redeploy. The host must have sufficient resources for four NSX Edge VMs during the upgrade or redeploy of an Edge HA pair. Default value for TCP connection timeout is changed to 21600 seconds from the previous value of 3600 seconds.
  • Flexible SNAT / DNAT rule creation: vnicId no longer needed as an input parameter; removed requirement that the DNAT address must be the address of an NSX Edge VNIC.
  • Maximum number of NAT rules: For NSX Edge versions prior to 6.2, a user could configure 2048 SNAT and 2048 DNAT rules separately, giving a total limit of 4096 rules. Since NSX Edge version 6.2 onwards, a limit is enforced for the maximum allowed NAT rules, based on the NSX Edge appliance size: 1024 SNAT and 1024 DNAT rules for a total limit of 2048 rules for COMPACT edge. 2048 SNAT and 2048 DNAT for a total limit of 4096 rules for LARGE edge and QUADLARGE edge. 4096 SNAT and 4096 DNAT rules for a total limit of 8192 rules for XLARGE edge.
  • Logging is now enabled by default for SSL VPN and L2 VPN. The default log level is notice.
  • NSX Edge technical support logs have been enhanced to report memory consumption per process.

Other Key Features and Additions:

  • NSX Hardware Layer 2 Gateway Integration: expands physical connectivity options by integrating 3rd-party hardware gateway switches into the NSX logical network
  • New VXLAN Port 4789 in NSX 6.2.3 and later: Before version 6.2.3, the default VXLAN UDP port number was 8472. See the NSX Upgrade Guide for details.
  • Firewall — Granular Rule Filtering: simplifies troubleshooting by providing granular rule filters in UI, based on Source, Destination, Action, Enabled/Disabled, Logging, Name, Comments, Rule ID, Tag, Service, Protocol.
  • Guest Introspection — Windows 10 support
  • SSL VPN ClientMac OS El Capitan support
  • Service Composer — Performance Improvements: enables faster startup/reboot of NSX Manager by optimizing synchronization between security policy and firewall service, and disabling auto-save of firewall drafts by default
  • VMware vRealize Log Insight 3.3.2 for NSX provides intelligent log analytics for NSX, This version accepts NSX Standard/Advanced/Enterprise edition license keys issued for NSX 6.2.2+

Upgrade Notes – RTFM:

In the release notes there is a detailed section on the upgrade and interoprability of this version of NSX with other key VMware components. It’s important that it’s read so as to not have a poor experience during the upgrade.

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.3/releasenotes_nsx_vsphere_623.html#upgradenotes

Resolved Issues:

There are a large number of Resolved Issues which can be found on the release notes…below are the ones that relating to Service Providers running Edge Services Gateways.

  • Extended HA failover times for Edge Services Gateway (ESG) or DLR with Edge VM when using only static routes
  • NAT does not translate IP addresses when NSX Edge firewall is disabled
  • vCenter 6.0 restart/reboot may result in duplicate VTEPs on VXLAN prepared ESX hosts
  • After upgrading the NSX Edge from 6.1.x to 6.2.x, the NSX Manager vsm.log shows “INVALID DHCP CONFIG”
  • Unexpected TCP interruption on TCP sessions during Edge High Availability (HA) failover in NSX 6.2.x

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.3/releasenotes_nsx_vsphere_623.html#resolvedissues

NSX Design Guide v3:

https://communities.vmware.com/servlet/JiveServlet/previewBody/27683-102-8-41631/NSX%20Reference%20Design%20Version%203.0.pdf

Overall a huge release for NSX-v. If you have the right entitlements you can login to MyVMware and download the binaries.

References:

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.3/releasenotes_nsx_vsphere_623.html

Gotcha: vCloud Stops Logging After 5.5.x to 5.6.x SP Upgrade

This is something to look out for if you are upgrading from vCloud Director 5.5.x to 5.6.x SP and if you have altered the syslog settings of your cells….This little gotcha nearly caused a couple of our production vCloud Director Cells to fail due to insufficient storage on the their file systems. Prior to our alerting warning us about an low disk space alert on the cells I couldn’t remember any issues with disk space and vCD Guest Systems since I first started with vCD in 2011…as is normally the case storage space is generally consumed by logs that don’t rotate or grow unexpectedly…so I went looking in the vCD Logs directory under /opt/vmware/vcloud-director/logs And sure enough I found that the jmx.log had ballooned to 17GB and had not rotated since the date of the upgrade to 5.6.4

After getting a support ticket raised it was found that the vCD container debug logs where not even rolling…in fact there was no debug log activity on the cells (except for the cell.log) since the upgrade.

Looking at the log4j.properties file under /opt/vmware/vcloud-director/etc it looked like everything was in order and that the logs should be rotating after the file hit 20MB however it was obvious that this wasn’t happening.

Working with VMware Support we came across this section of the Install and Upgrade guide where it talks about

(Optional) Update logging properties.

After an upgrade, new logging properties are written to the file /opt/vmware/vcloud-director/etc/log4j.properties.rpmnew

Option

Action

If you did not change existing logging properties

Copy this file to /opt/vmware/vcloud-director/etc/log4j.properties.

If you changed logging properties

Merge /opt/vmware/vcloud-director/etc/log4j.properties.rpmnew file with the existing /opt/vmware/vcloud-director/etc/log4j.properties. Merging these files preserves your changes.

Sure enough our log4j.properties file had been altered to add some custom logging formats and targets and in each of the directories there was the log4j.properties.rpmnew file. That is to say there are new logging properties that come with vCD 5.6.x SP and if you had altered the file prior to upgrade those additions meant the logging broke. As suggested above all that needed to be done was to merge both files, or append the customer entries to the bottom of the .rpmnew version and save that as log4j.properties and restart the cells. Once I did this the debug logs started to flow again and the jmx.log file was rotated allowing me to shrink/delete the file consuming the storage.

 

References:

http://pubs.vmware.com/vcd-56/index.jsp#com.vmware.vcloud.install.doc_56/GUID-CEF834DA-1FF5-4819-9D24-88DE6F005C78.html?resultof=%2522%2572%2570%256d%256e%2565%2577%2522%2520