Monthly Archives: June 2018

Adding Let’s Encrypt SSL Certificate to vCloud Director Keystore

For the longest time the configuring of vCloud Director’s SSL certificate keystore has been the thing that makes vCD admins shudder. There are lots of posts on the process…some good…some not so good. I even have a post from way back in 2012 about fronting vCD with a Citrix NetScaler and if I am honest, I cheated in having HTTPS at the load balancer deal with the SSL certificate while leaving vCD configured with the self signed cert. With the changes to the way the HTML5 Tenant Portal deals with certs and DNS I’m not sure that method would even work today.

I wanted to try and update the self signed certs in both my lab environments to assist in resolving the No Datacenters are available issue that cropped up in vCD 9.1. Instead of generating and using self signed certs I decided to try use Let’s Encrypt signed certs. Most of the process below is curtesy of blog posts from Luca Dell’Oca and it’s worth looking at this blog post from Tom Fojta who has a PowerShell script to automate Let’s Encrypt SSL certs for us on NSX Edge load balancers.

In my case, I wanted to install the cert directly into the vCD Cell Keystore. The manual end to end the process is listed below. I intend to try and automate this process so as to overcome the one constraint with using Let’s Encrypt…that is the 90 day lifespan of the certs. I think that is acceptable and it ensures validity of the SSL cert and a fair caveat given the main use case for this is in lab environments.

Generating the Signed SSL Cert from Let’s Encrypt:

To complete this process you need the ACMESharp PowerShell module. There are a couple of steps to follow which include registering the domain you want to create the SSL cert against, triggering a verification challenge that can be done by creating a domain TXT record as shown in the output of the challenge command. Once submitted, you need to look out for a Valid Status response.

Once complete, there is a script that can be run as show on Luca’s Blog. I’ve added to the script to automatically import the newly created SSL cert into the Local Computer certificate store.

From here, I exported the certificate with the private key so that you are left with a PFX file. I also saved to Base-64 X.509 format the Root and Intermediate certs that form the whole chain. This is required to help resolve the No Datacenters are available error mentioned above. Upload the three files to the vCD cell and continue as shown below.

Importing Signed SSL from Let’s Encrypt into vCD Keystore:

Next, the steps to take on the vCD Cell can be the most complex steps to follow and this is where I have seen different posts do different things. Below shows the commands from start to finish that worked for me…see inline for comments on what each command is doing.

Once that has been done and the vCD services has restarted, the SSL cert has been applied and we are all green and the Let’s Encrypt SSL cert is in play.

Quick Tip: Let’s Encrypt ACME Powershell Ownership Challenge Can’t see Challenge Data

I’m currently going through the process of acquiring a new Let’s Encrypt free SSL Certificate against a new domain I registered. For a great overview of what Let’s Encrypt is and what is can do for you, head over to Luca Dell’Oca’s blog here. I was following Luca’s instructions for getting the new domain authorised for use with the Let’s Encrypt service via a DNS challenge when I ran into the following.

After running the PowerShell command to generate the challenge, it was not returning the Handler Message as expected form the direct output…well obviously anyway.

After scratching my head for a bit, I checked to see if the data was contained withing the returned PowerShell command.

From here I was able to create the DNS TXT entry and complete the challenge.

Just in case it wasn’t obvious this very quick post will save you a bit of time.

Released: vCloud Director 9.1.0.1 – API Tweaks and Resolved Issues

There was a point release of vCloud Director 9.1 (9.1.0.1 Build 8825802) released last week, bringing with it an updated Java Runtime plus new API functions that allow additional configuration of advanced settings for virtual machines. There was also a number of bug fixes from the initial 9.1 release earlier in the year. Some of the issues that are resolved are significant and worth looking into if you have 9.1 GA deployed.

I haven’t been able to find an exact list of the new API functions, however traversing the Org Admin rights API call I did spot something new relating to Latency as show below.

And when I granted this right through the API mechanism I was able to allocate the right to the Org Admin via the administrator web interface.

I’m trying get a list of all the new API rights that where added as part of this release and will update this post when I have them.

Some of the bigger issues that where resolved are listed below:

  • In vCloud Director Tenant Portal, the Configure Services tab is disabled for Advanced Edge Gateway. In vCloud Director Tenant Portal, you cannot configure Advanced Edge Gateway settings as an administrator with any of the Gateway Advanced Services rights.
  • When importing a virtual machine from vCenter Server, vCloud Director relocates it to the primary resource pool. When you import a virtual machine created on a non-primary cluster in vCenter Server to vCloud Director, the machine is always relocated to the primary cluster.
  • In the vCloud Director Tenant Portal, the administrator of one organization can see virtual machines that belong to other vCloud Director organizations. When you configure the organizations in vCloud Director to use an LDAP server for authentication, an administrator of one organization, who is logged in vCloud Director Tenant Portal, can see virtual machines that belong to other organizations.
  • Importing a virtual machine from the vCenter Server deletes the original virtual machine after cloning it. When importing a virtual machine from the vCenter Server to vCloud Director involves changing its datastore, the process consists in cloning the source virtual machine and deleting it, while effectively changing its Managed Object Reference (MoRef).
  • Enabling High Availability for existing edge gateways in a data center with installed NSX Edge 6.4.0 fails.  In a data center with installed NSX Edge 6.4.0, you cannot enable High Availability for existing edge gateways that belong to a datastore cluster with enabled Storage Distributed Resource Scheduler (SDRS).
  • vCloud Director Tenant Portal does not display existing organization virtual data centers. When you use a self-signed SSL certificate for vCloud Director and you log in to vCloud Director Tenant Portal, you do not see a list of the existing organization virtual data centers.

The rest can be found here.

Just to finish up, there is still a lingering issue from the GA release that changed the behaviour of the HTML5 Tenant UI in scenarios where the SSL self signed certificates are used which is covered in this VMwareKB. Even though (as shown above) it’s been listed as resolved…I have run into it again in two different installs.

Obviously, if you are using legit SSL certificates you won’t have the issue, however the work around is not doing it’s thing for me. Hopefully I can resolve this ASAP as I am about to start some validation testing for Veeam and vCloud Director as well as start to test out our new functionality coming in Update 4 of Backup & Replication for Cloud Connect Replication.

For those with the correct entitlements…download here.

#LongLivevCD

References:

https://docs.vmware.com/en/vCloud-Director/9.1/rn/rel_notes_vcloud_director_9-1-0-1.html

Released: Veeam Availability Console Update 1

Today, Veeam Availability Console Update 1 (Build 2.0.2.1750) was released. This update improves on our multi-tenant service provider management and reporting platform that is provided free to VCSPs. VAC acts as a central portal for Veeam Cloud and Service Providers to remotely manage and monitor customer instances of Backup & Replication including the ability to monitor Cloud Connect Backup and Replication jobs and failover plans. It also is the central mechanism to deploy and manage our Agent for Windows which includes the ability to install agents onto on-premises machines and apply policies to those agents once deployed.

What’s new in Update 1:

If you want to get the low down from the What’s new document can be access here. I’ve summarised the new features and enhancements below and expanded on the key ones below.

  • Enhanced support for Veeam Agents
  • New Operator Role
  • ConnectWise Manage Plugin
  • Improved Veeam Backup & Replication monitoring
  • New backup policy types
  • Sub-tenant Accounts and Sub-tenant Management
  • Alarm for tracking VMs stored in cloud repositories
  • RESTful APIs enhancements

RESTful APIs enhancements: VACs API first approach gets a number of enhancements in Update 1 with more information stored in the VAC configuration database accessible via new RESTful API calls that include:

  • Managed backup server licenses
  • Tenant descriptions
  • References to the parent object for users, discovery rules and computers

As with the GA, this is all accessible via the built in Swagger Interface.

Enhanced support for Veeam Agents: VAC Update 1 introduces support for Veeam Agents that are managed by Veeam Backup & Replication. This adds monitoring and alarms for Veeam Agent for Microsoft Windows and Veeam Agent for Linux that are managed by a Veeam Backup & Replication. One of the great features of this is the search functionality which allows you to more efficiently search for agent instances that exist in Backup & Replication and see their statuses.

New Operator Role: While not the Reseller role most VCSPs are after this new role allows VCSPs wanting to delegate VAC access to their own IT staff to take advantage of the new operator role without granting complete administrative access. This role allows access to everything essential to remotely monitor and manage customer environments, but restricts access to VAC configuration settings.

ConnectWise Manage Plugin: ConnectWise Manage is a very popular platform used by MSPs all over the world. VAC Update 1 includes native integration with ConnectWise Manage. The integration allows VCSPs to synchronize and map company accounts between the two platforms, integrated billing, enabling you to use ConnectWise Manage to generate tenant invoices based on their usage and the plugin allows you to create tickets based on triggered alarms in VAC. The integration is solid and based on VACs strong underlying API driven approach. More importantly, this is the first extensibility feature of VAC using a Plugin framework…the idea is for it to just be the start.

Alarm for tracking VMs stored in cloud repositories:  A smaller enhancement, but one that is important for those running Cloud Connect is the new alarm that allows you to be notified when the number of customer VMs stored in the cloud repository exceeds a certain threshold.

Scalability enhancements: Finally there has been a significant improvement in VAC scalability limits when it comes to the number of managed Backup & Replication servers for each VAC instance. This ensures stable operation and performance when managing up to 10,000 Veeam Agents and up to 600 Backup & Replication servers, protecting 150-200 VMs or Veeam Agents each.

References and Product Guides:

https://www.veeam.com/vac_2_0_u1_release_notes_rn.pdf

https://www.veeam.com/documentation-guides-datasheets.html

https://www.veeam.com/availability-console-service-providers-faq.html

https://www.veeam.com/vac_2_0_u1_whats_new_wn.pdf

Installing and Managing Veeam Agent for Linux with Backup & Replication

With the release of Update 3 of Veeam Backup & Replication we introduced the ability to manage agent from within the console. This was for both our Windows and Linux agents and aimed to add increased levels of manageability and control when deploying agents in larger enterprise type environments. For an overview of the features there is a veeam.com blog post here that goes through the different components and the online help documentation is also helpful in providing an detailed look at the ins and outs.

Scouring the web, there has been a lot written about the Windows Agent and how that’s managed from the Backup & Replication console, but not a lot written about managing Linux Agents. There theory is exactly the same…Add a Protection Group, add the machines you want to include in the Protection Group, scan the group and then install the agent. From there you can add the agents to a new or existing backup job and manage licenses.

In terms of how that looks and the steps you need to take. Head to the Inventory menu section and right click on Physical & Cloud Infrastructure to Add Protection Group. Give the group a meaningful name and then to add Linux machines select Individual or CSV method under Type. In my example I chose to add the Linux machines individually and added then added the machines via their Host Name or IP Address with the right credentials.

Under Options, you can select the Distribution Server which is where the agent will be deployed from and choose to set a schedule to Rescan the Protection Group.

Once this part is complete the first Discovery is run and all things being equal the Linux Agent will be installed to the machines that where added as part of the first step. I actually ran into an issue upon first run where the agent didn’t install due to the following error shown below.

The fix was as simple as installing the DKMS package on the servers via apt-get. Asking around, this was not a normal occurrence and that it should deploy and install without issue. Maybe this was due to my Linux server being TurnKey Linux appliances…in any case, once the package was installed I re-triggered the install by right clicking the machine and selecting Install Agent.

Once that job has finished we are able to assign the Linux agent machines to new or existing backup jobs.

As with the Windows Agent you have two different Job modes. In my example I created a job of each type. The result is one agent that is in lock down mode meaning reduced functionality from the GUI or Command line while the other has more functionality but is still managed by the system administrator. The differences between both GUIs is shown below.

From the Jobs list under the Home menu this is represented by the job type being Linux Agent Backup vs Linux Agent Policy.

Finally, when looking at the licensing aspect, once a license has been applied to a Backup & Replication server that contains agent licenses, an additional view will appear under the License view in the console where you can assign or remove agent licenses from.

From within Enterprise Manager (if the VBR instance is managed), you also see additional tab views for the Windows and Linux Agents as shown below.

References:

https://helpcenter.veeam.com/docs/backup/agents/introduction.html?ver=95

https://helpcenter.veeam.com/docs/agentforlinux/userguide/license_vbr_revoke.html?ver=20

https://helpcenter.veeam.com/docs/backup/agents/agent_policy.html?ver=95

vBrownBag TechTalks at VMworld 2018 – The Power to Catapult!

VMworld 2018 is fast approaching and in the last 24 hours, notifications where sent out to those lucky enough to have their session submissions accepted. Having been on the wrong side of that email multiple times I understand the disappointment that comes with not having a session accepted. The great news about VMworld is that there is another way to get your session seen and heard…and that is through the vBrownBag Techtalks.

The TechTalks have been a staple at VMworld’s (and other industry conferences) for a number of years now. Last year saw a stepping up of the vBrownBag game by having the TechTalks listed in the VMworld Content Catalog. I’ve had the pleasure of presenting tech talks at three VMworld’s over the years. The first one was back in 2014 but I remember it being a significant milestone in my career…regardless of the fact it was just a TechTalk it meant a lot to present at VMworld.

Make no mistake…these talks have the power and potential to catapult careers!

While the TechTalks offer the opportunity for folks that have not had sessions accepted, the real power of the talks is in offering a platform for the community to step up and present relevant, thought leading content that generally isn’t driven by marketing. In many ways I see more value in these sessions than in the VMworld sessions proper and there is a lot that can be taken away from the sessions.

That said, it’s great to see a number of vendors sponsoring the TechTalks and as per usual, Veeam is leading the way in our support of community at VMworld. As of last week there where around 50 TechTalks submitted and the team expects to have space for over a hundred TechTslks between the both VMworld conferences.

There is still plenty of time to submit your session, more information is in this post.

Here is a Playlist of the 2017 VMworld TechTalks. For those interested, there is a blog post by the vBrownBag team on what it takes to get get a presentation up live streaming and onto YouTube so fast…I found it a fascinating read.

Using Terraform to Deploy and Configure a Ready to use Backup Repo into an AWS VPC

A month of so ago I wrote a post on deploying Veeam Powered Network into an AWS VPC as a way to extend the VPC network to a remote site to leverage a Veeam Linux Repository running as an EC2 instance. During the course of deploying that solution I came across a lot of little check boxes and settings that needed to by tweaked in order to get things working. After that, I set myself the goal of trying to automate and orchestrate the deployment end to end.

For an overview of the intended purpose behind the solution head to the original blog post here. That post was mainly focused around the Veeam PN component, however I was using that as a mechanism to create a site-to-site connection to allow Veeam Backup & Replication to talk to the other EC2 instance which was the Veeam Linux Repository.

Terraform by HashiCorp:

In order to automate the deployment into AWS, I looked at Cloudformation first…but found that learning curve to be a little steep…so I went back to HashiCorp’s Terraform which I have been familiar with for a number of years, but never gotten my hands dirty with. HashiCorp specialise in Cloud Infrastructure Automation and their provisioning product is called Terraform.

Terraform is used to create, manage, and update infrastructure resources such as physical machines, VMs, network switches, containers, and more. Almost any infrastructure type can be represented as a resource in Terraform.

A provider is responsible for understanding API interactions and exposing resources. Providers generally are an IaaS (e.g. AWS, GCP, Microsoft Azure, OpenStack), PaaS (e.g. Heroku), or SaaS services (e.g. Terraform Enterprise, DNSimple, CloudFlare).

Terraform supports a host of providers and once you wrap your head around the basics and view some example code, provisioning Infrastructure as Code can be achieved with relatively no coding experience…however, as I did find out, you need to be careful in this world and not make the same initial mistake I did as explained in this post.

Going from Manual to Orchestrated with Automation:

The Terraform AWS provider is what I used to create the code required to deploy the required components. Like everything that’s automated, you need to understand the manual process first and that is where the previous experience came in handy. I knew what the end result was…I just needed to work backwards and make sure that the Terraform provider had all the instructions it needed to orchestrate the build.

the basic flow is:

  • Fetch AWS Access Key and Secret
  • Fetch AWS Key Pair
  • Create AWS VPC
    • Configure Networking and Routing for VPC
  • Create CentOS EC2 Instance for Veeam Linux Repo
    • Add new disk and set size
    • Execute configuration script
      • Install PERL modules
  • Create Ubuntu EC2 Instance for Veeam PN
    • Execute configuration script
      • Install VeeamPN modules from repo
  • Login to Veeam PN Web Console and Import Site Configuration.

I’ve uploaded the code to a GitHub project. An overview and instructions for the project can be found here. I’ve also posted a video to YouTube showing the end to end process which i’ve embedded below (best watched at 2x speed):

In order to get the Terraform plan to work there are some variables that need modifying in the GitHub Project and you will need to download, install and initialise Terraform. I’m intending to continue to tweak the project and complete the provisioning end to end, including the Veeam PN site configuration part at the end. The remote execution feature of Terraform allows some pretty cool things by way of script initiation.

References:

https://github.com/anthonyspiteri/automation/aws_create_veeamrepo_veeampn

https://www.terraform.io/intro/getting-started/install.html

 

Quick Look – Backing up AWS Workloads with Cloud Protection Manager from N2WS

Earlier this year Veeam acquired N2WS after announcements last year of a technology partnership at VeeamON 2017. The more I tinker with Cloud Protection Manager the more I understand why we made the acquisition. N2WS was founded in 2012 with their first product shipping in 2013. Purpose built for AWS supporting all types of EC2 instances, EBS volumes, RDS, DynamoDB & Redshift and AMI creation and distributed as an AMI through the AWS Marketplace. The product is easy to deploy and has extended it’s feature set with the release of 2.3d announced during VeeamON 2018 a couple weeks ago.

From the datasheet:

Cloud Protection Manager (CPM) is an enterprise-class backup, recovery, and disaster recovery solution purpose-built for Amazon Web Services EC2 environments. CPM enhances AWS data protection with automated and flexible backup policies, application consistent backups, 1-click instant recovery, and disaster recovery to other AWS region or AWS accounts ensuring cloud resiliency for the largest production AWS environment. By extending and enhancing native AWS capabilities, CPM protects the valuable data and mission-critical applications in the AWS cloud.

In this post, I wanted to show how easy it is to deploy and install Cloud Protection Manager as well as look at some of the new features in the 2.3d release. I will do a follow up post going into more detail about how to protect AWS Instances and services with CPM.

What’s new with CPM 2.3:

  • Automated backup for Amazon DynamoDB: CPM provides backup and recovery for Amazon DynamoDB, you can now apply existing policies and schedules to backup and restore their DynamoDB tables and metadata.
  • RESTful API:  Completely automate backup and recovery operations with the new Cloud Protection Manager API. This feature provides seamless integration between CPM and other applications.
  • Enhanced reporting features: Enhancements include the ability to gather all reports in one tab, run as a CSV, view both protected and unprotected resources and include new filtering options as well.

Other new features that come as part of the CPM 2.3 release include full cross-region and cross-account disaster recovery for Aurora databases, enhanced permissions for users and a fast and efficient on boarding process using CloudFormation’s 1-click template.

Installing, Configuring and Managing CPM:

The process to install Cloud Protection Manager from the AWS Marketplace is seamless and can be done via a couple different methods including a 1-Click deployment. The offical install guide can be read here. The CPM EC2 instance is deployed into a new or existing VPC configured with a subnet and must be put into an existing, or new Security Group.

Once deployed you are given the details of the installation.

And you can see it from the AWS Console under the EC2 instances. I’ve added a name for the instance just for clarities sake.

One thing to note is that there is no public IP assigned to the instance as part of the deployment. You can create a new Elastic IP and attach it to the instance, or you can access the configuration website via it’s internal IP if you have access to the subnet via some form of VPN or network extension.

There is an initial configuration wizard that guides you through the registration and setup of CPM. Note that you do need internet connectivity to complete the process otherwise you will get this error.

The final step will allow you to configure a volume for CPM use. With that the wizard finalises the setup and you can log into the Cloud Protection Manager.

Conclusion: 

The ability to backup AWS services natively has it’s advantages over traditional methods such as agents. Cloud Protection Manager from N2WS can be installed and ready to go within 5 minutes. In the next post, i’ll walk through the CPM interface and show how you backup and recover AWS instances and services.

References:

https://n2ws.com/cpm-install-guide

https://support.n2ws.com/portal/kb/articles/release-notes-for-the-latest-v2-3-x-cpm-release