Tag Archives: APIs

Infrastructure as Code vs RESTful APIs … A Working Example with Terraform and vCloud Director

Last week I wrote an opinion piece on Infrastructure as Code vs RESTful APIs. In a nutshell, I talked about how leveraging IaC instead of trying to code against APIs directly can be more palatable for IT professionals as it acts as a middle man interpreter between yourself and the infrastructure endpoints. IaC can be considered a black box that does the complicated lifting for you without having to deal with APIs directly.

As a follow up to that post I wanted to show an example about the differences between using direct APIs verses using an IaC tool like Terraform. Not surprisingly the example below features vCloud Director…but I think it speaks volumes to the message I was trying to get across in the introduction post.

The vCloud Director Terraform Provider was recently upgraded with the release of vCloud Director 9.7 and now sits at version 2.1 itself.

The Terraform Provider has been developed using Python and GO. It uses Client-Server model inside the hood where the client has been written using GO and server has been written using Python language. The core reason to use two different languages is to make a bridge between Terraform and Pyvcloud API. Pyvcloud is the SDK developed by VMware and provides an medium to talk to vCloud Director. Terraform uses GO to communicate where Pyvcloud has been written in Python3.

The above explanation as to how this provider is doing its thing highlights my previous points around any IaC tools. The abstraction of the infrastructure endpoint is easy to see… and in the below examples you will see it’s benefit for those who have not got the inclination to hit the APIs directly.

The assumption for both examples is that we are starting without any configured Firewall or NAT rules for the NSX Edge Services Gateway. Both methods are connecting as tenant’s of the vCD infrastructure and authenticating with Organisation level access.

The end result will be:

  • Allow HTTP, HTTPS and ICMP access to a VM living in a vDC
    • External IP is 82.221.98.109
    • Internal IP of VM is 172.17.0.240
    • VM Subnet is 172.17.0.0/24
  • Configure DNAT rules to allow HTTP and HTTPS
  • Configure SNAT rule to allow outbound from the VM subnet
Configuring Firewall and NAT Rules with RESTful API:

Firstly, to understand what vCD API operations need to be hit, we need to be familiar with the API Documentation. This will cover initial authentication as either a SYSTEM or Organizational admin and then what calls need to be made to get information relating to the current configuration and schema. Further to this, we need to also be familiar with the NSX API for vCD Documentation which covers how to interact with the network specific API operations possible from the vCD API endpoint.

We are going to be using Postman to execute against the vCD API. Postman is great because you can save your call history and reuse them at a later date. You can also save variable into Workspaces and also insert specific code to assist with things like authentication.

First step is to authenticate against the API and get a session authorization key that will allow you to feed that key back into subsequent requests. This authorization key will only last you a finite amount of time and will need to be regenerated.

Because we are using a decent RESTful API Client like Postman, there is a better way to programatically authenticate using a bearer access token as described in Tom Fojta’s post here when talking to the vCD API.

Once that is done we are authenticated as a vCD Organizational Admin and we can now query the NSX Edge Services Gateway (ESG) Settings for Firewall and NAT rules. I’ll walk through configuring a NAT rule for the purpose of the example, but the same method will be used to configure the Firewall as well.

A summary of the NAT requests can be seen below and found here.

Below we are querying the existing NAT rules using a GET request against the NSX ESG. What we are returned is an empty config in XML.

What needs to be done is to turn that request into a POST and craft an XML payload into the Body of the request so that we can configure the NAT rules as desired.

Redoing the GET request will now show that the NAT rules have been created.

And will be shown in the vCD Tenant UI

From here we can update, append, reset or delete the NAT rules as per the API documentation. Each one of those actions will require a new call to the API and the same process followed as above.

Configuring Firewall and NAT Rules with Terraform:

For a primer on the vCloud Director Terraform Provider, read this post and also head over to Luca’s post on Terraform with vCD. As with the RESTful API example above, I will use Terraform IaC to configure the same Tenant NSX Edge Gateway’s Firewall and NAT rules. What will become clear using Terraform for this is that it is a lot more efficient and elegant that going at it directly against the APIs.

Initially we needs to setup the required configuration items in order for the Terraform Provider to talk to the vCD API endpoint. To do this we need to setup a number of Terraform files that declare the variables required to connect to the vCD Organization and then configure the terraform.tfvars file that contains the specific variables.

We also create a provider .tf file to specifically call out the required Terraform provider and set the main variables.

We contain all this in a single folder (seen in the left pane above) for organization and portability…These folders can be called as Terraform Modules if desired in more complex, reusable plans.

We then create two more .tf files for the Firewall and NAT rules. The format is dictated by the Provider pages which gives examples. We can make things more portable by incorporating some of the variables we declared elsewhere in the code as shown below for the Edge Gateway name and Destination IP address.

Once the initial configuration work is done, all that’s required in order to apply the configuration is to initialize the Terraform Provider, make sure that the Terraform Plan is as expected… and then apply the plan against the Tenant’s Organization.

As the video shows… in less than a minute we have the NSX Firewall and NAT rules configured. More importantly, we now have a desired state which can be modified at any time by simple additions or subtractions to the Terraform code.

Wrapping it up:

From looking at both examples, it’s clear that both methods of configuration do the trick and it really depends on what sort of IT Professional you are in terms of which method is more suited to your day to day. For those that are working as automation engineers, working with APIs directly and/or integrating them into provisioning engines or applications is going to be your preferred method. For those that want to be able to deploy, configure and manager their own infrastructure in a more consumable way, using a Terraform provider is probably a better way

The great thing about Terraform in my eyes is the fact that you have declared the state that you want configured and once that has been actioned, you can easily check that state and modify it by changing the configuration items in the .tf files and reapplying the plan. For me it’s a much more efficient way to programatically configure vCD than doing the same configuration directly against the API.

Ohhh… and don’t forget… you are still allowed to use the UI as well… there is no shame in that!

Creating Policy Based Backup Jobs for vCloud Director Self Service Portal with Tenant Creation

For a long time Veeam has lead the way in regard to the protection of workloads running in vCloud Director. Veeam first released deep integration into vCD back in version 7 of Backup & Replication that talked directly to the vCD APIs to facilitate the backup and recovery of vCD workloads and their constructs. More recently in version 9.5, the vCD Self Service Portal was released which also taps into vCD for tenant authentication.

This portal leverages Enterprise Manager and allows service providers to grant their tenants self-service management of their vCD workloads. It’s possible that some providers don’t even know that this portal exists let alone the value it offers. I’ve covered the basics of the portal here…but in this post, I want to talk about how to use the Veeam APIs and PowerShell SnapIn to automatically enable a tenant, create a default backup jobs based on policies, tie backup copy jobs to default job for longer retention and finally import the jobs into the vCD Self Service Portal ready for use.

Having worked with a service provider recently, they requested to have previously defined service definitions for tenant backups ported to Veeam and the vCD Self Service Portal. Part of this requirement was to have tenants apply backup policies to their VMs…this included short term retention and longer term GFS based backup.

One of the current caveats with the Veeam vCD Self Service Portal is that backup copy jobs are not configurable via the web based portal. The reason for this is that It’s our belief that service providers should be in control of longer term restore operations, however some providers and their tenants still request this feature.

Translated to a working solution, the PowerShell script combines a previously released set of code by Markus Kraus that uses the Enterprise Manager API to setup a new tenant in the vCD Self Service portal and a set of new functions that create default backup and backup copy jobs for vCD and then imports them into the portal ready for use. The variables are controlled by a JSON file making the script portable for Veeam Cloud and Service Providers to use as a base and build upon.

The end result is that when a tenant first logs into the vCD Self Service Portal they have jobs, dictated by the desired polices ready for use. The backup jobs are set to disabled without a schedule set. The scope of the default jobs is the tenant’s Virtual Datacenter. If there is a corresponding backup copy job, this is tied to the backup job and is ready to do its thing.

From here, the tenant can choose which policy that want to apply to their workloads and edit the desired job, change or leave the scope and add a schedule. The job name in the Backup and Replication console is modified to indicate which policy the tenant selected.

Again, if the tenant chooses a policy that requires longer term retention, the corresponding backup copy job is enabled in the Backup & Replication console…though not managed by the tenant.

Self service recovery is possible by the tenant for through the portal as per usual, including full VM recovery, file and application item level recovery. For recovery of the longer term workloads and/or items, this is done by the Service Provider.

This is a great example of the power of the Veeam API and PowerShell SnapIn providing a solution to offer more than what is out of the box and enhance the offering around the backup of vCloud Director workloads with Veeam’s integration. Feel free to use as is, or modify and integrate into your service offerings.

GitHub Page: https://github.com/anthonyspiteri/powershell/tree/master/vCD-Create-SelfServiceTenantandPolicyJobs

VCA-CLI for vCloud Director: New Networking Features

There is a lot of talk going around how IT Pros can more efficiently operate and consume Cloud Based Services…AWS has lead the way in offering a rich set of APIs for it’s clients to use to help build out cloud applications and infrastructure and there are a ton of programming libraries and platforms that have seen the rise of the DevOps movement…And while AWS has lead the way, other Public Clouds such as Azure (with PowerShell Packs) and Google have also built self service capability through APIs.

vCloud Director has always had a rich set of APIs (API Online Doco Here) and as I blogged about last year Paco Gomez has been developing a tool called VCA-CLI which is based on pyvcloud which is a Python SDK for vCloud Director and vCloud Air. This is an alternative to Web Based creation and management of vCloud Director vDCs and vApps. Being Python based you have the option of running it pretty much on any OS you like…the posts below show you how to install and configure VCA on a Mac OS X OS and Windows and how to connect up to a vCloud Director based Cloud Org.

Initial releases of VCA-CLI didn’t have the capability to configure the Firewall settings of a vDC Edge Gateway, but since the release of version 16, Firewall rule management has been added. In the below example, I connect up to my vCD Org in Zettagrid, gather some information about my vDC, deploy a SexiLog VM template, set the Syslog setting on the Gateway and then configure a new NAT and Firewall rules to open up port 8080 to the SexiLog Web interface.

And the end result:

Again, this highlights the power of the vCloud Director API and what can be done with the pyvcloud Python SDK. Once perfected the set of commands above can be used to deploy vApps and configure networking in seconds instead of having to work through the vCloud Director UI…and that’s a win win!

References:

https://pypi.python.org/pypi/vca-cli

https://github.com/vmware/vca-cli

http://www.sexilog.fr/

 

#vBrownBag TechTalk – NSX…An Unexpected Journey

While at VMworld a couple of weeks ago I presented a short talk around my journey working with NSX-v and how it has shifted (pivoted) the direction of what I consider to be important in my day to day role. The unexpected part of the journey dragged me kicking and screaming into the world of APIs and dare I say…Devops.

And while I don’t consider myself a DevOp (far from it)…I find myself more and more getting sucked into that world and with that I am trying adjust how I consume IT. In any case if you have a spare 10 minutes have a listen about how NSX kickstarted my interest and got me looking more under the covers of the server platforms and services we sometimes take for granted. Before this change I was comfortable accepting a UI as the only way to interact and consume services…are you?

For those interested the full schedule is here, along with direct links to the YouTube Channel with all the talks.

http://professionalvmware.com/2015/08/vbrownbag-techtalks-schedule-vmworld-usa-2015/

NSX vCloud Retrofit: vShield Edges Become Unmanageable – API Fix

If you are familiar with vCloud Director Edge Gateway Services you might have come across situations where Edges become unmanageable and you see the following options greyed out in the vCD UI. All Edge services remain functional, however no changes can be made.

In environments where NSX has been retrofitted with an in place upgrade over vCNS you may hit a bug with the NSX Manager to do with how it interprets vShield Edge Statuses. Basically vCloud Director talks to the NSX Manager to grab the status of deployed Edge Gateways. A situation can occur where the NSX Manager sends back the incorrect status resulting in the Edge becoming unmanageable from vCloud Director as shown above.

A Re-Deploy will work in resetting the status and making the Edge manageable again, however this will result in downtime for the services sitting behind the effected Edge device. During the course of an SR I raised with the NSX and vCloud VMware Engineering Teams a fix was created that uses the NSX Manager APIs to POST a new status that makes vCloud Director pick up the edge as manageable without having to Re-Deploy.

First step is to find out which Edges might be effected by this condition…apart from going through each Edge in the vCD UI I suggest looking at this post from @fojta in which he creates a PowerCLI Script to grab the current statuses of all edges. In addition to the Edge Name you will also need the EDGE-ID

Get the EDGE-ID by going into the vSphere Web Client’s Networking and Security Tab -> NSX Edges and Search to the Edge Name that matches the vCD UI

Using your favourite Rest Client, take the EDGE-ID and replace the identifier in the following API Call to get more details of the Edge.

https://NSX-MANAGER-IP/api/4.0/edges/edge-xxx

Next take the EDGE-ID and NAME (checking the DATACENTERID) from the response above and modify the Payload below increment the ID Number as you go along

Executing the following API POST

https://NSX-MANAGER-IP/api/2.0/systemevent 

You should see a 201 Created Status Returned after the POST…Refresh the list in vCloud Director and the edge should be manageable. Repeat the process for any effected Edges.

Thanks to the NSX and vCloud Engineering team for working through an elegant solution that means zero impact on client services…As I am discovering, there are lots of cool things that can be done with APIs!

Further Reading:

This blog series extends my NSX Bytes Blog Posts to include a more detailed look at how to deploy NSX 6.1.x into an existing vCloud Director Environment. Initially we will be working with vCD 5.5.x which is the non SP Fork of vCD, but as soon as an upgrade path for 5.5.2 -> 5.6.x is released I’ll be including the NSX related improvements in that release.

NSX Bytes: API Response Times and User Accounts

If there is one thing that working with NSX has done is drag me kicking and screaming into the world of APIs. Apart from being on the VCIX-NV Blueprint, in my current role it was becoming increasingly important to understand how our developers consume and manipulate the automation of our products and services…just being able to do a manual step by step with a little PowerCLI wasn’t enough any more.

NSX was born out of APIs…for those that took a look at the first NVP examples before the release of NSX-v it was all API driven…with that, learning how to consume NSX via it’s APIs is paramount for anyone serious about automating NSX-v platform services.

The work I’ve been doing in getting NSX Productized resulted in a situation where I needed to look for efficiencies in response times for API requests. As has been the case since I first used NSX-v (6.05) I had noticed that depending on what account you log in as to the vCenter Web Client seemed to dictate the speed of the Networking and Security Menu. This had been put down to the Web Client making sure the logged in account has access to all the NSX parts…The NSX Service Account was snappier than AD Accounts…with the SSO Admin Account snappier again.

Using the Chrome Advanced Rest Client I wanted to see if I could get consistent results pointing out which account was best to use to squeeze out quicker response times.

  • NSX Enterprise Admin (AD Account)
  • NSX Service Account (AD Account)
  • SSO Admin (Local Account vCenter)

After a bit of testing with different accounts authenticating again the call to get all NSX Edges under the NSX Manager (https://NSX-manger-ip/api/4.0/edges/) I got some strange results…I can’t 100% say with confidence that the SSO Admin account was quicker, but in most circumstances there was a 10 second difference between the SSO Admin and AD Account…

To get the above XML response it took between 4 to 14 seconds on average depending on what account was used. The NSX Manager is relatively busy and has about 84 Edges to keep track of.

I also noticed blow outs in response times (20-30 seconds) when multiple calls where made at the same time…which very much makes sense, but I am wondering which mechanisms can be put in place to increase the overall response performance of these calls…also wondering if the Chrome Rest Client is contributing to the times I’m seeing.

I’m doing more testing with different tools (cURL and PowerCLI) to see if I can get a more definite outcome..in the mean time if anyone has any input based on a greater experience than mine playing with APIs, please feel free to comment.

So this post doesn’t so much answer the question or get into the inner workings of API authentication mechanisms…but using the SSO Admin account should result in the quicker API response.

References and Further Reading:

http://www.routetocloud.com/2014/12/working-with-nsxv-api/

http://stretch-cloud.info/2015/01/experimenting-rest-api-tools-can-use-part-iii/

http://pubs.vmware.com/NSX-61/topic/com.vmware.ICbase/PDF/nsx_61_api.pdf

http://wahlnetwork.com/2014/10/28/nsx-api-calls-powershell/

http://vbyron.com/blog/nsx-rest-api-primer-1-connecting-curl/

https://networkinferno.net/creating-nsx-objects-with-rest-api