I’m ok admitting that I am still learning as I progress through my career and I’m ok to admit when things go wrong. Learning from mistakes is a crucial part of learning…and I learnt a harsh lesson today! That Infrastructure as Code is as dangerous as it is awesome…and that the public cloud is an unforgiving place!

Earlier today I created a new GitHub Repository for a project i’ve been working on. Before I realised my mistake I had uploaded a Terraform variables file with my AWS Access and Secret Key. I picked up on this probably two minutes after I pushed the contents up to the public repository. Roughly five minutes later I deleted the repository and was about to start fresh without the credentials but then realised than my Terraform plan was failing with a credential error.

I logged into the AWS Console and saw that my main VPC and EC2 instances had been terminated and that there was 20 new instances in it’s place. I knew exactly at that point what had happened! I’d been compromised and I had handed over the keys on a silver web scraper platter.

My access key had been deleted and new ones created along with VPCs and Key Pairs in every single AWS region across the world. I deleted the new access key the malicious user created locking him out from doing any more damage, however in the space of ten minutes 240 EC2 instances in total where spun up. This was a little more than the twenty I thought I had dealt with initially…costing only $4.50…Amazing!

I contacted AWS support and let them know what happened. To their credit (and to my surprise) I had a call back within a few hours. Meanwhile they automatically restricted my account until I had satisfied a series of clean up steps so as to limit any more potential damage. The billing will be reversed as well so I am a little less in a panic when I see my current month breakdown.

The Bad Side of Infrastructure as Code and Public Cloud:

This example shows how dangerous the world we are living in can be. With AWS and alike providing brilliant API access into their provisioning platforms malicious users have seen an opportunity to use Infrastructure as Code as a way to spin up cloud resources in a matter of seconds. All they need is an in. And in my case, that in was a moment of stupidity…and even though I realised what I had done, all it took was less than five minutes for them to take advantage of my lack of concentration and exploit my security lapse. They also exploited the fact that I am new to this space and had not learnt best practice for storing credentials.

I was lucky that everything I had in AWS was just there for demo purpose and I had nothing of real important there. However, if this happened to be someone running business critical applications they would be in for a very very bad day. Everything was wiped! Even the backup software I had running in there using local snapshots…as ever a case for offsite copies if there was one! (Ergo – Veeam Agents and N2WS)

The Good Side of Infrastructure as Code and Public Cloud:

What good could come of this? Well, apart from learning a little more about Terraform and how to store credentials the awesome part was that all the work I had put in over the past couple of weeks getting a start with Infrastructure as Code and Terraform was that I was able to reprovision everything that I lost within 5 minutes…once my account restriction was lifted.

That’s the power of APIs and the applications that take advantage of them. And even though I copped a slap in the face today…I’m converted. This stuff is cool! We just need to be aware of the dangers that come and the fact that the coolness can be used and exploited in the wrong way as well.