Tag Archives: PowerShell

Mapping vCloud Director Backup Jobs to Self Service Portal Tenants

Since version 7 of Backup & Replication, Veeam has lead the way in regard to the protection of workloads running in vCloud Director. With version 7 Veeam first released deep integration into vCD that talked directly to the vCD APIs to facilitate the backup and recovery of vCD workloads and their constructs. More recently in version 9.5, the vCD Self Service Portal was released which also taps into vCD for tenant authentication.

The portal leverages Enterprise Manager and allows service providers to grant their tenants self-service backup for their vCD workloads. More recently we have seen some VCSPs integrate the portal into the new vCD UI via the extensibility plugin which is a great example of the power that Veeam has with vCD today while we wait for deeper, native integration.

It’s possible that some providers don’t even know that this portal exists let alone the value it offers. I’ve covered the basics of the portal here…but in this post, I am going to quickly mention an extension to a project I released last year for the vCD Self Service Portal, that automatically enables a tenant, creates a default backup jobs based on policies, tie backup copy jobs to default job for longer retention and finally import the jobs into the vCD Self Service Portal ready for use.

Standalone Map and Unmap PowerShell Script:

From the above project, the job import part has been expanded to include its own standalone PowerShell script that can also be used to map or unmap existing vCD Veeam Backup jobs to a a tenant to manage from the vCD Self Service Portal. This is done using the Set-VBRvCloudOrganizationJobMapping commandlet.

As shown below, this tenant has already configured a number of jobs in the Portal.

There was another historical job that was created outside of the portal directly from the Veeam console. Seen below as TEST IMPORT.

To map the job, run the PowerShell script is with the -map parameter. A list of all existing vCloud Director Backup jobs will be listed. Once the corresponding number has been entered the commandlet within the script will be run and the job mapped to the tenant linked to the job.

Once that has been run, the tenant now has that job listed in the vCD Self Service Portal.

There is a little bit of error checking built into the script, to that it exits nicely on an exception as shown below.

Finally, if you want to unmap a job from the vCD Self Service portal, run the PowerShell script with the -unmap parameter. Conclusion:

Like most things I work on and then publish for general consumption, I had a request to wrap some logic around the Set-VBRvCloudOrganizationJobMapping commandlet from a service provider partner. The script can be taken and improved, but as-is, provides an easy way to retrieve all vCloud Jobs belonging to a Veeam Server, select the desired job and then have it mapped to a tenant using the vCD Self Service Portal.

References:

https://github.com/anthonyspiteri/powershell/blob/master/vCD-Create-SelfServiceTenantandPolicyJobs/vCD_job.ps1

https://helpcenter.veeam.com/docs/backup/powershell/set-vbrvcloudorganizationjobmapping.html?ver=95u4

Orchestration of NSX by Terraform for Cloud Connect Replication with vCloud Director

That is probably the longest title i’ve ever had on this blog, however I wanted to highlight everything that is contained in this solution. Everything above works together to get the job done. The job in this case, is to configure an NSX Edge automatically using the vCloud Director Terraform provider to allow network connectivity for VMs that have been replicated into a vCloud Director tenant organization with Cloud Connect Replication.

With the release of Update 4 for Veeam Backup & Replication we enhanced Cloud Connect Replication to finally replicate into a Service Providers vCloud Director platform. In doing this we enabled tenants to take advantage of the advanced networking features of the NSX Edge Services Gateway. The only caveat to this was that unlike the existing Hardware Plan mechanism, where tenants where able to configure basic networking on the Network Extension Appliance (NEA), the configuration of the NSX Edge had to be done directly through the vCloud Director Tenant UI.

The Scenario:

When VMs are replicated into a vCD organisation with Cloud Connect Replication the expectation in a full failover is that if a disaster happened on-premises, workloads would be powered on in the service provider cloud and work exactly as if they where still on-premises. Access to services needs to be configured through the edge gateway. The edge gateway is then connected to the replica VMs via the vOrg Network in vCD.

In this example, we have a LAMP based web server that is publishing a WordPress site over HTTP and HTTPs.

The VM is being replicated to a Veeam Cloud Service Provider vCloud Director backed Cloud Connect Replication service.

During a disaster event at the on-premises end, we want to enact a failover of the replica living at in the vCloud Director Virtual Datacenter.

The VM replica will be fired up and the NSX Edge (the Network Extension Appliance pictured is used for partial failovers) associated to the vDC will allow the HTTP and HTTPS to be accessed from the outside world. The internal IP and Subnet of the VM is as it was on-premises. Cloud Connect Replication handles the mapping of the networks as part of the replication job.

Even during the early development days of this feature I was thinking about how this process could be automated somehow. With our previous Cloud Connect Replication networking, we would use the NEA as the edge device and allow basic configuration through the Failover Plan from the Backup & Replication console. That functionality still exists in Update 4, but only for non vCD backed replication.

The obvious way would be to tap into the vCloud Director APIs and configure the Edge directly. Taking that further, we could wrap that up in PowerShell and invoke the APIs from PowerShell, which would allow a simpler way to pass through variables and deal with payloads. However with the power that exists with the Terraform vCloud Director provider, it became a no brainer to leverage this to get the job done.

Configuring NSX Edge with Terraform:

In my previous post around Infrastructure as Code vs APIs I went through a specific example where I configured an NSX Edge using Terraform. I’m not going to go over that again, but what I have done is published that Terraform plan with all the code to GitHub.

The GitHub Project can be found here.

The end result after running the Terraform Plan is:

  • Allowed HTTP, HTTPS, SSH and ICMP access to a VM in a vDC
    • Defined as a variable as the External IP
    • Defined as a variable as the Internal IP
    • Defined as a variable as the vOrg Subnet
  • Configure DNAT rules to allow HTTP, HTTPS and SSH
  • Configure SNAT rule to allow outbound from the vOrg subnet

The variables that align with the VM and vORG network are defined in the terraform.tfvars file and need to be modified to match the on-premises network configuration. The variables are defined in the variables.tf file.

To add additional VMs and/or vOrg networks you will need to define additional variables in both files and add additional entires under the firewall_rules.tf and nat_fules.tf. I will look at ways to make this more elegant using Terraform arrays/lists and programatic constructs in future.

Creating PowerShell for Execution:

The Terraform plan can obviously be run standalone and the NSX Edge configuration can be actioned at any time, but the idea here is to take advantage of the script functionality that exists with Veeam backup and replication jobs and have the Terraform plan run upon completion of the Cloud Connect Replication job every time it is run.

To achieve this we need to create a PowerShell script:

GitHub – configure_vCD_VCCR_NSX_Edge.ps1

The PowerShell script initializes Terraform and downloads the Provider, ensures there is an upgrade in the future and then executes the Terraform plan. Remembering that that variables will change within the Terraform Plan its self, meaning these scripts remain unchanged.

Adding Post Script to Cloud Connect Replication Job:

The final step is to configure the PowerShell script to execute once the Cloud Connect Replication job has been run. This is done via a post script settings that can be found in Job Settings -> Advanced -> Scripts. Drop down to selected ps1 files and choose the location of the script.

That’s all that is required to have the PowerShell script executed once the replication job completes.

End Result:

Once the replication component of the job is complete, the post job script will be executed by the job.

This triggers the PowerShell, which runs the Terraform plan. It will check the existing state of the NSX Edge configuration and work out what configuration needs to be added. From the vCD Tenant UI, you should see the recent tasks list modifications to the NSX Edge Gateway by the user configured to access the vCD APIs via the Provider.

Taking a look at the NSX Edge Firewall and NAT configuration you should see that it has been configured as specified in the Terraform plan.

Which will match the current state of the Terraform plan

Conclusion:

At the end of the day, what we have done is achieved the orchestration of Veeam Cloud Connect Replication together with vCloud Director and NSX… facilitated by Terraform. This is something that Service Providers offering Cloud Connect Replication can provide to their clients as a way for them to define, control and manage the configuration of the NSX edge networking for their replicated infrastructure so that there is access to key services during a DR event.

While there might seem like a lot happening, this is a great example of leveraging Infrastructure as Code to automated as otherwise manual task. Once the Terraform is understood and the variables applied, the configuration of the NSX Edge will be consistent and in a desired state with the config checked and applied on every run of the replication job. The configuration will not fall out of line with what is required during a full failover and will ensure that services are available if a disaster occurs.

References:

https://github.com/anthonyspiteri/automation/tree/master/vccr_vcd_configure_nsx_edge

Automated Configuration of Backup & Replication with PowerShell

As part of the Veeam Automation and Orchestration for vSphere project myself and Michael Cade worked on for VMworld 2018, we combined a number of seperate projects to showcase an end to end PowerShell script that called a number of individual modules. Split into three parts, we had a Chef/Terraform module that deployed a server with Veeam Backup & Replication installed. A Terraform module that deployed and configured an AWS VPC to host a Linux Repository with a Veeam PN Sitegateway. And finally a Powershell module that configured the Veeam server with a number of configuration items ready for first use.

The goal of the project was to release a PowerShell script that fully deployed and configured a Veeam platform on vSphere with backup repositories, vCenter server and default policy based jobs automatically configured and ready for use. This could then be adapted for customer installs, used on SDDC platforms such as VMware Cloud on AWS, or for POCs or lab use.

While we are close to releasing the final code on GitHub for the project, I thought I would branch out the last section of the code and release it separately. As I was creating this script, it became apparent to me that it would be useful for others to use as is or as an example from which to simplify manual and repetitive tasks that go along with configuring Backup & Replication after installation.

Script Overview:

The PowerShell script (found here on GitHub) performs a number of configuration actions against any Veeam Backup & Replication Server as per the included functions.

All of the variables are configured in a config.json file meaning nothing is required to be modified in the main PowerShell script. There are a number of parameters that can be called to trigger or exclude certain functions.

There are some pre-requisites that need to be in place before the script can be executed…most importantly the PowerShell needs to be executed on a system where the Backup & Replication Console is installed to allow access to the Veeam PowerShell Snap-in. From there you just need a new Veeam Backup & Replication server and a vCenter server plus their login credentials. If you want to add a Cloud Connect Provider offering Cloud Connect Backup or/and Replication you enter in all the details in the config.json file as well. Finally, if you want to add a Linux Repository you will need the details of that plus have it configured for key based authentication.

You can combine any of the parameters listed above. An example is shown above where -ClearVBRConfig has been used to reverse the -RunVBRConfigure parameter that was executed first to do an end to end configure. For Cloud Connect Replication, if you want to configure and deploy an NEA there is a specific parameter for that. If you didn’t want to configure Cloud Connect or the Linux Repository the parameters can be used individually, or together. If those two parameters are used, the Default Backup Repository will be used for the jobs that are created.

Automating Policy Based Backup Jobs:

Part of the automation that we where keen to include was the automatic creation of default backup jobs based on vSphere Tags. The idea was to have everything in place to ensure that once the script had been run, VMs could be backed up dependant on them being added to vSphere Tags. Once done the backup jobs would protect those VMs based on the policies set in the config.json.

The corresponding jobs are all using the vSphere Tags. From here the jobs don’t need to be modified when VMs are added…VMs assigned those Tags will be included in the job.

Conclusion:

Once the script has been run you are left with a fully configured Backup & Replication server that’s connected to vCenter and if desired (by default) has local and Cloud Connect repositories added with a set of default policy based jobs ready to go using vSphere Tags.

There are a number of improvements that I want to implement and I am looking out for Contributors on GitHub to help develop this further. At its base it is functional…but not perfect. However it highlights the power of the automation that is possible with Veeam’s PowerShell Snap-In and PowerCLI. One of the use-cases for this was for repeatable deployments of Veeam Backup & Replication into POCs or labs and for those looking to standup those environments, this is a perfect companion.

Look out for the full Veeam SDDC Deploy Toolkit being released to GitHub shortly.

References:

https://github.com/anthonyspiteri/powershell/tree/master/BR-Configure-Veeam

Creating Policy Based Backup Jobs for vCloud Director Self Service Portal with Tenant Creation

For a long time Veeam has lead the way in regard to the protection of workloads running in vCloud Director. Veeam first released deep integration into vCD back in version 7 of Backup & Replication that talked directly to the vCD APIs to facilitate the backup and recovery of vCD workloads and their constructs. More recently in version 9.5, the vCD Self Service Portal was released which also taps into vCD for tenant authentication.

This portal leverages Enterprise Manager and allows service providers to grant their tenants self-service management of their vCD workloads. It’s possible that some providers don’t even know that this portal exists let alone the value it offers. I’ve covered the basics of the portal here…but in this post, I want to talk about how to use the Veeam APIs and PowerShell SnapIn to automatically enable a tenant, create a default backup jobs based on policies, tie backup copy jobs to default job for longer retention and finally import the jobs into the vCD Self Service Portal ready for use.

Having worked with a service provider recently, they requested to have previously defined service definitions for tenant backups ported to Veeam and the vCD Self Service Portal. Part of this requirement was to have tenants apply backup policies to their VMs…this included short term retention and longer term GFS based backup.

One of the current caveats with the Veeam vCD Self Service Portal is that backup copy jobs are not configurable via the web based portal. The reason for this is that It’s our belief that service providers should be in control of longer term restore operations, however some providers and their tenants still request this feature.

Translated to a working solution, the PowerShell script combines a previously released set of code by Markus Kraus that uses the Enterprise Manager API to setup a new tenant in the vCD Self Service portal and a set of new functions that create default backup and backup copy jobs for vCD and then imports them into the portal ready for use. The variables are controlled by a JSON file making the script portable for Veeam Cloud and Service Providers to use as a base and build upon.

The end result is that when a tenant first logs into the vCD Self Service Portal they have jobs, dictated by the desired polices ready for use. The backup jobs are set to disabled without a schedule set. The scope of the default jobs is the tenant’s Virtual Datacenter. If there is a corresponding backup copy job, this is tied to the backup job and is ready to do its thing.

From here, the tenant can choose which policy that want to apply to their workloads and edit the desired job, change or leave the scope and add a schedule. The job name in the Backup and Replication console is modified to indicate which policy the tenant selected.

Again, if the tenant chooses a policy that requires longer term retention, the corresponding backup copy job is enabled in the Backup & Replication console…though not managed by the tenant.

Self service recovery is possible by the tenant for through the portal as per usual, including full VM recovery, file and application item level recovery. For recovery of the longer term workloads and/or items, this is done by the Service Provider.

This is a great example of the power of the Veeam API and PowerShell SnapIn providing a solution to offer more than what is out of the box and enhance the offering around the backup of vCloud Director workloads with Veeam’s integration. Feel free to use as is, or modify and integrate into your service offerings.

GitHub Page: https://github.com/anthonyspiteri/powershell/tree/master/vCD-Create-SelfServiceTenantandPolicyJobs

OVFTool: vCloud Director OVA Upload PowerShell Script

Earlier this year I put together a quick and nasty PowerShell Script that exports a vApp from vCloud Director using the OVFTool …for those that don’t know the OVFTool is a command line tool that has a powerful set of functions to import/export VMs and vApps from vCenter, ESXi and vCloud Director weather it be from a vCloud Air or a vCloud Air Network Provider.

You can Download and install the tool from here:

This week I needed to upload an Virtual Machine that was in OVA format and for those that have worked with vCloud Director you would know that the OVA format is not supported using the upload functionality in the current web interface. With that I thought it was a good time to round out the export using OVTTool post with an import using OVFTool post. Again, doing some research I found a bunch of posts relating to importing OVAs into vCloud Director and after working through the Admin Guide and some examples I was ready to build out a basic import command and start work on the PowerShell Script. On Windows you can run the tool from CMD but I would suggest using PowerShell/CLI as in the example below I go through building a variable.

What Info is Required:

  • vCloud URL
  • vCloud Username and Password
  • Org Name
  • vDC Name
  • vApp Name
  • Catalog Name
  • Path to OVA

Command Line Example:

Below is a basic example of how to construct the vCloud String and use it as a variable to execute the tool.

PowerShell Script:

Again, I’ve taken it a step further to make it easier for people to import OVAs into vCloud Director and put together another, slightly improved PowerShell Script that I have coded in to work with my old companies vCloud Zones…though this can be easily modified to use any vCloud Air Network vCD endpoint.

The output of the script can be seen below:

It’s a very basic script that gathers all the required components that make up the vCloud Source Connection String and then exports the OVA into the vCD vApp. I’ve even done a little more PowerShell improvements around password security and added a little colour.

Save the code snippet as a .ps1 into the OFVTool Windows Folder and execute the script from the same location. If there are any errors with the inputs provided the OVFTool will fail with an error, but apart from that it’s a very simple straight forward way to import OVAs into any vCloud Director enabled endpoint.

Additional Reading:

http://www.virtuallyghetto.com/tag/ovftool

http://www.vmwarebits.com/content/import-and-export-virtual-machines-command-line-vmwares-ovf-tool 

OVFTool: vCloud Director vApp Export PowerShell Script

Last week I had a requirement to look at how to allow customers to export VM’s and vApps from our vCloud Director Zones without using the UI. I’ve known about the OVFTool for a while but never really had the need to use it in anger…for those that don’t know the OVFTool is a command line tool that has a powerful set of functions to import/export VMs and vApps from vCenter, ESXi and vCloud Director weather it be from a vCloud Air or vCloud Air Network Provider.

You can Download and install the tool from here: https://my.vmware.com/group/vmware/details?downloadGroup=OVFTOOL410&productId=491

Upon doing some research I found a bunch of posts relating to importing OVFs into vCloud Director, vCloud Air or vCenter’s but not a lot around the export side of things…after working through the Admin Guide and some examples I was ready to build out a basic export command and start work on the PowerShell Script. On Windows you can run the tool from CMD but I would suggest using PowerShell/CLI as in the example below I go through building a variable.

What Info is Required:

  • vCloud URL
  • vCloud Username and Password
  • Org Name
  • vDC Name
  • vApp Name

Note: The VM/vApp needs to be offline for the export process to take place.

Command Line Example:

Below is a basic example of how to construct the vCloud String and use it as a variable to execute the tool.

PowerShell Script:

Wanting to take it a step further to make it easier for our customers to download their vApps I put together a quick and nasty PowerShell Script that can be used for all ZettaGrid Zones. The output of the script can be seen below:

It’s a very basic script that acts to break down the required components that make up the vCloud Source Connection String and then saves the OVF to the same folder where the OVFTool is installed.

Save the code snippet as a .ps1 into the OFVTool Windows Folder and execute the script from the same location. If there are any errors with the inputs provided the OVFTool will fail with an error, but apart from that it’s a very simple straight forward way to export and download VMs and vApps from any vCloud Director enabled endpoint.

Behind the Scenes:

I thought it would be interesting to see what happens behind the scenes on the vCloud Director Cells when the OVFTool is brought in do it’s magic….When the OVFTool Authenticates against vCloud the following entries are seen in the cell.log of the active vCenter Proxy Cell.

When the Enable Download task is executed the cell begins to copy the vApp to the Cell Transfer directory which is the staging area vCloud Director uses for all VM/vApp related copy/move/import/export functions…During this copy the OVFTool displays the Waiting for Server Task status. If you where able to view the contents of the director created in the transfer location you would see the vmdk growing in size as shown below:

If you check into the vCloud Director UI and browse to the vApp you will see that the vApp is Busy with a status of Enabling Download.

Once the copy has finished the OFVTool starts the download and once that is complete (or there is an error) the files in the vCloud Director Transfer area are deleted. In my testing I haven’t witnessed any continuation or pick off where it last failed mechanism.

Feel free to take the script and do with it what you will…it can be pretty easily modified to connect to any vCloud Air Network Partner or vCloud Air its self.

Additional Reading:

http://www.virtuallyghetto.com/tag/ovftool

http://www.vmwarebits.com/content/import-and-export-virtual-machines-command-line-vmwares-ovf-tool 

PowerCli IOPS Metrics: vCloud Org and VPS Reporting

We have recently been working through a product where knowing and reporting on VM Max Read/Write IOPS was critical. We needed a way to be able to provide reporting on our clients VPSs and vCloud Organisation VMs.

vCOPs is a seriously great monitoring and analytics tool, but it has got a flaw in it’s reporting in that you can’t search, export or manipulate metrics relating to VM IOPS in a useful way. VeeamOne gives you a Top 10 list of IOPS, CloudPhysics has a great card showing DataStore/VM performance…but again, not exportable or granular enough for what we needed.

If you search on Google for IOPS Reporting you will find a number of guys who have created excellent PowerCLI Scripts. Problem I found was that most worked in some cases, but not for what we required. One particular post I came across this Post on the VMware Community Forums gave a quick and dirty script to gather IOPS stats for all VMs. This lead me to the Alpacapowered Blog. So initial credit for the following goes to MKguy…I merely hacked around it to provide us with additional functionality.

Before You Start:

Depending on your Logging Level in vCenter (I have run this against vCenter 5.1 with PowerCLI 5.5) you may not be collecting the stats required to get Read/Write IOPS. To check this run the following in PowerCLI connected to your vCenter

If you don’t get the output it means your logging level is set to a lower level than is required. Read through this Post to have vCenter logging the required metrics on a granular level. Once thats been done, give vCenter about 30 minutes to collect its 5 minute samples. If you ever want to check individually how many samples you have for a particular VM you can run the following command. It will also show you the Min/Max Count plus the average.

The Script:

I’ve created two versions of the script (one for Single VMs and on for vCloud Org VMs) and as you can see below, I added in a couple niceties to make this more user friendly and easy to trigger for our internal support staff. Idea is that anyone with the right access to vCenter can double-click on the .ps1 script, and with the right details produce a report for either a single VM or a vCloud Organisation.

Script Notes:

Line 1: Adds the PowerCLI Snap-In to be able to call ESXi Commandlets from PowerShell on click of the .ps1

Line 3: Without notes from MKguy, i’m assuming this is telling us to use the last 30 days of stats if they exist.

Line 7: I discovered the -menu flag for Connect-VIServer which lists a 10 list of your most recently connects vCenter or ESXi servers…from there you enter a number to connect (ease of use for helpdesk)

Line 16: Does uses the Get-Folder command to allow us to get all the VMs in a vCloud Org…you can obviously enter in your own preferred search flags here.

Lines 17-22 are the ones I picked up form the Community post which basically takes the command we used above to check for samples metrics and feeds it into a read/write variable which is then displayed in a series of columns as shown below.

Script Output:

Executing the .ps1 will open a PowerShell window, Ask you to enter in the vCenter/Host and finally the VM name or vCloud Org Description. If you have a folder with a number of VMs, the script can take a little time going through the math and spit out the values.

From there you can do a select and copy to export the values out for manipulation…I haven’t done a csv export option due to time constraints, however if anyone want to add that to the end of the script, please do and let me know 🙂

Hope this script is useful for some!