Tag Archives: Service Providers

Attack from the Inside – Protecting Against Rogue Admins

In July of 2011, Distribute.IT, a domain registration and web hosting services provider in Australia was was hit with a targeted, malicious attack that resulted in the company going under and their customers left without their hosting or VPS data. The attack was calculated, targeted and vicious in it’s execution… I remember the incident well as I was working for Anittel at the time and we where offering similar services…everyone in the hosting organization was concerned when starting to think about the impact a similar attack would have within our systems.

“Hackers got into our network and were able to destroy a lot of data. It was all done in a logical order – knowing exactly where the critical stuff was and deleting that first,”

While it was reported at the time that a hacker got into the network, the way in which the attack was executed pointed to an inside job and all though it was never proved to be so it almost 100% certain that the attacker was a disgruntled ex-employee. The very real issue of an inside attack has popped up again…this time Verelox, a hosting company out of the Netherlands has effectively been taken out of business with a confirmed attack from within by an ex-employee.

My heart sinks when I read of situations like this and for me, it was the only thing that truely kept me up at night as someone who was ultimately responsible for similar hosting platforms. I could deal and probably reconcile with myself if I found myself in a situation where a piece of hardware failed causing data loss…but if an attacker had caused the data loss then all bets would have been off and I might have found myself scrambling to save face and along with others in the organization, may well have been searching for a new company…or worse a new career!

What Can Be Done at an Technical Level?

Knowing a lot about how hosting and cloud service providers operate my feeling is that 90% of organizations out there are not prepared for such attacks and are at the mercy of an attack from the inside…either by a current or ex-employee. Taking that a step further there are plenty that are at risk of an attack from the inside perpetrated by external malicious individuals. This is where the principal of least privileged access needs to be taken to the nth degree. Clear separation of operational and physical layers needs to be considered as well to ensure that if systems are attacked, not everything can be taken down at once.

Implementing some form of certification or compliancy such as ISO 27001, SOC and iRAP will force companies to become more vigilant through the stringent processes and controls that are forced upon companies once they meet compliancy. This in turn naturally leads to better and more complete disaster and business continuity scenarios that are written down and require testing and validation in order to pass certification.

From a backup point of view, these days with most systems being virtual it’s important to consider a backup strategy that not only looks to make use of the 3-2-1 rule of backups, but also look to implement some form of air-gapped backups that in theory are completely seperate and unaccessible from production networks, meaning that only a few very trusted employees have access to the backup and restore media. In practice implementing a complete air-gapped solution is complex and potentially costly and this is where service providers are chancing their futures on scenarios that have a small percentage chance of happening however the likelihood of that scenario playing out is greater than it’s ever been.

In a situation like Verelox, I wonder if, like most IaaS providers they didn’t backup all client workloads by default, meaning that backup services was an additional service charge that some customers didn’t know about…that said, if backup systems are wiped clean is there any use of having those services anyway? That is to say…is there a backup of the backup? This being the case I also believe that businesses need to start looking at cross cloud backups and not rely solely on their providers backup systems. Something like the Veeam Agent’s or Cloud Connect can help here.

So What Can Be Done at an Employee Level?

The more I think about the possible answer to this question, the more I believe that service providers can’t fully protect themselves from such internal attacks. At some point trust supersedes all else and no amount of vetting or process can stop someone with the right sort of access doing damage. To that end making sure that you are looking after your employee’s is probably the best defence against someone feeling aggrieved enough to carry out an malicious attack such as the one Verelox has just gone through. In addition to looking after employee’s well being it’s also a good idea to…within reason, keep tabs on an employee’s state in life in general. Are they going through any personal issues that might make them unstable, or have they been done wrong by someone else within the company? Generally social issues should be picked up during the hiring process, but complete vetting of employee stability is always going to be a lottery.

Conclusion

As mentioned above, this type of attack is a worst case scenario for every service provider that operates today…there are steps that can be taken to minimize the impact and protect against an employee getting to the point where they choose to do damage but my feeling is we haven’t seen the last of these attacks and unfortunately more will suffer…so where you can, try to implement policy and procedure to protect and then recover when or if they do happen.

Vote for your favorite blogs at vSphere-land!

Top vBlog Voting 2017

Resources:

https://www.crn.com.au/news/devastating-cyber-attack-turns-melbourne-victim-into-evangelist-397067/page1

https://www.itnews.com.au/news/distributeit-hit-by-malicious-attack-260306

https://news.ycombinator.com/item?id=14522181

Verelox (Netherlands hosting company) servers wiped by ex-admin from sysadmin

Looking Beyond the Hyper-Scaler Clouds – Don’t Forget the Little Guys!

I’ve been on the road over the past couple of weeks presenting to Veeam’s VCSP partners and prospective partners here in Australia and New Zealand on Veeam’s Cloud Business. Apart from the great feedback in response to what Veeam is doing by way of our cloud story I’ve had good conversations around public cloud and infrastructure providers verses the likes of Azure or AWS. Coming from my background working for smaller, but very successful service providers I found it almost astonishing that smaller resellers and MSPs seem to be leveraging the hyper-scale clouds without giving the smaller providers a look in.

On the one hand, I understand why people would choose to look to Azure, AWS and alike to run their client services…while on the other hand I believe that the marketing power of the hyper-scalers has left the capabilities and reputation of smaller providers short changed. You only need to look at last week’s AWS outage and previous Azure outages to understand that no cloud is immune to outages and it’s misjudged to assume that the hyper-scalers offer any better reliability or uptime than the likes of providers in the vCloud Air Network or other IaaS providers out there.

That said, there is no doubt that the scale and brain power that sits behind the hyper-scalers ensures a level of service and reliability that some smaller providers will struggle to match, but as was the case last week…the bigger they are, the harder they fall. The other things that comes with scale is the ability to drive down prices and again, there seems to be a misconception that the hyper-scalers are cheaper than smaller service providers. In fact most of the conversations I had last week as to why Azure or AWS was chosen was down to pricing and kickbacks. Certainly in Azure’s case, Microsoft has thrown a lot into ensuring customers on EAs have enough free service credits to ensure uptake and there are apparently nice sign-up bonuses that they offer to partners.

During that conversation, I asked the reseller why they hadn’t looked at some of the local VCSP/vCAN providers as options for hosting their Veeam infrastructure for clients to backup workloads to. Their response was, that it was never a consideration due to Microsoft being…well…Microsoft. The marketing juggernaut was too strong…the kickbacks too attractive. After talking to him for a few minutes I convinced him to take a look at the local providers who offer, in my opinion more flexible and more diverse service offerings for the use case.

Not surprisingly, in most cases money is the number one factor in a lot of these decisions with service uptime and reliability coming in as an important afterthought…but an afterthought non-the less. I’ve already written about service uptime and reliability in regards to cloud outages before but the main point of this post is to highlight that resellers and MSP’s can make as much money…if not more, with smaller service providers. It’s common now for service providers to offer partner reseller or channel programs that ensure the partner gets decent recurring revenue streams from the services consumed and the more consumed the more you make by way of program level incentives.

I’m not going to do the sums, because there is so much variation in the different programs but those reading who have not considered using smaller providers over the likes of Azure or AWS I would encourage to look through the VCSP Service Provider directory and the vCloud Air Network directory and locate local providers. From there, enquire about their partner reseller or channel programs…there is money to be made. Veeam (and VMware with the vCAN) put a lot of trust and effort into our VCSPs and having worked for some of the best and know of a lot of other service provider offerings I can tell you that if you are not looking at them as a viable option for your cloud services then you are not doing yourself justice.

The cloud hyper-scalers are far from the panacea they claim to be…if anything, it’s worthwhile spreading your workloads across multiple clouds to ensure the best availability experience for your clients…however, don’t forget the little guys!

Differentiate…Or Die?

I spent the last week on holiday in the Wine Region of Western Australia’s South West. I’ve been holidaying down south since I was a teenager and I’ve seen the region transform over the years…I can’t speak for the years prior to my time spent around Margaret River, Dunsborough and Yallingup, but I had a thought as I was visiting one of the newer Wineries/Breweries that, in some ways… the Wine Industry down south shares similar traits to the Hosting/Service Provider Industry.


Winaries of the South West View Larger Map


Cloud Hosting Providers of Perth View Larger Map

So what has wine and tourism got to do with Hosting and Cloud?

I remember a conversation with a local Microsoft SPLA guy (those in Australia know there is only really one guy who fits that bill…@PhileMeAU) during a Hosting Partners dinner at TechEd 2010 where by the group was talking about the possible impact of BPOS/Office365 and what it meant for traditional hosters. Out of that conversation the strong advice given at the time was that we had to Differentiate, or Die…That was to say, there was really no future in hosting vanilla applications like Exchange or MSCRM because commodity based public clouds will eventually swallow all before them. Three years on and the same could be said for those doing IaaS and the thought that traditional Virtual Machine hosting is now the realm of the bigger players.

In some ways the rise of AWS, Azure and other public clouds has shifted the industry closer to a Demolition Man style Taco Bell monopoly. But there are enough alternative Service Providers competing against the big guns and winning that proves that, for all the marketing money aimed at perpetuating FUD…somewhere along the lines those smaller players are doing something right? Have they taken on the differentiation threat? Or is something else responsible for their continued existence and success?

Back to the Wine Industry example, going back 20-30+ years there might have been 5-10 Wineries that dominated the industry until the smaller players starting buying up land and producing their own vintages. Pretty soon the market became flooded with Margaret River wines and competition was at it’s peak. For those wineries lucky enough to be not too far off the Caves Road (the main road running parallel to the coast) there was a guarantee of a steady stream of customers…What I have seen over the last 5 years or so is a number of Wineries trying to differentiate themselves from the others by bringing out more exotic vintages and even branching off into brewing of Beers and Spirits. The region was trying to become as famous for it’s liqueur’s as it’s vino’s.

With that going on, its still the more established wineries that attract the majority of the tourist dollar…this is as much due to reputation, and market muscle as it is for the quality of their product. Differentiation hasn’t worked…at the end of the day, people visiting will find their ways to the bigger players and the smaller players will continue to exist to serve their own particular market niche.

The same can be said for the Hosting and Cloud industry…lots of service providers have tried to differentiate their services so as to try and ward off the threat from an AWS, or an Azure…but in doing that I’ve seen (and been part of) companies loosing focus on getting the simple things right. Being a jack of all trades and a master of none is dangerous in the Service Provider industry…unless you have a bottomless pit of resources (both money and people based) there is no way you can achieve an excellent standard across a number of product sets. You also risk not focusing on the key areas of automation and process that goes hand in hand with a successful product set.

Small to Medium Service Providers can still thrive if they stick to core competencies and strive to excel within those narrower, but focused areas. The key that I’ve found of late (and am of the strong belief) is that you just have to keep it simple and do what you do well. That is to say…pick a course and stick with it. If you do IaaS well, why try to offer Platforms or Applications? If your strength lies in Hosting .NET…why try to branch out to a LAMP platform? All that’s achieved in my experiences in a thinning out of the quality of service leading to a situation where brand name is impacted.

As with the wineries down south Service Providers need to be wary of trying to keep up with the big boys…just because Winery BXT has released an updated blend why try to match that? Similarly core focus will be lost if Service Providers try to keep up with “new/justfixesforpoorinitialrelease” features AWS and Azure and others seem to be releasing every month or so to keep on looking like they are adding value add…when really all they are really doing is filling gaps.

So, take away here is to not take the differentiate or die message literally…Service Providers should focus on being excellent at what made them strong in the first place…the differentiate message may have been perpetuated by those that would want to see SP’s lose focus and die…a slow death!