Tag Archives: Metrics

CloudPhysics: Rightsizing Intelligence and Cost Calculator for Private Cloud

CloudPhysics have been a little quiet over the past twelve or so months with focus shifting from presenting data via Cards to Dashboards and also focusing on delivering on boarding solutions for managed service provider partners that has resulted in their channel business growing successfully. Before VMworld they announced the release of their Cost Calculator for Private Clouds in addition to releasing a couple more dashboards for their SaaS based landing page as well as adding a tagging feature for VMs and other objects.

CloudPhysics roots is all about data science and what can be achieved with literally billions of data points…so it’s no surprise that they are starting to put that front and center when it come to their new feature capabilities. Rightsizing at the 99th and 95th percentile usually cuts off the top 5% or 1% of metric peaks, and then presents the data at the nearest metric rate. In this way infrequent peaks are ignored, and the data is better suited to making decisions against. Now CloudPhysics rightsizing can be applied with intelligence to virtual machines and compute/storage infrastructure and capture savings by reducing workloads to match actual demands and reduce over provisioning.

The CloudPhysics Cost Calculator for Private Cloud lets you apply basic costing models to determine your actual costs per virtual machine (VM) in terms of power, compute resources, memory, storage, licensing, and more to generate a cost baseline.

As you can see below the new Card gives you the option to enter in cost points for most input items in a typical private cloud situation. They have not only included standard costs of servers hardware, memory and storage but also given you options to enter in depreciation terms, hypervisor cost details, environment costs relating to power and cooling but also additional 3rd party license costs that could be used for backup or acceleration software.

Once entered in you can filter through your platform as seen by the CloudPhysics Observer and get an understanding of what each individual VM is costing you in relation to your inputs. You also get a Cost as Configured amount that can be adjusted for the 99th and 95th percentile as well.

This view really gives you an understanding of what VMs are costing you the most and then get an idea of how to plan for any move to a public cloud where rightsizing based on more than just maximums is key. There is an option to click on the Compare Cloud Costs button which takes you to a new sister Card that displays the side by side cost of hosting your private cloud on AWS or Azure and again lets you manipulate the data with rightsizing.

In talking with the CloudPhysics team I’m hopeful that they will add to this card to include vCloud Air Network service providers running vSphere based IaaS platforms. I’m sure the 4000 odd vCAN SPs would appreciate a direct comparison for potential new customers looking to make a choice between the hyperscalers and their on platforms.

New Dashboard Items and Tags:

As mentioned in the opening paragraph CloudPhysics also added a couple new dashboards that can be configured to look at a number of different VM and Host metrics and show a trend over the last one, seven for thirty days. These new dashboard items as shown below are extremely handy for being bale to pick up problem objects in your infrastructure.

Also added is the basic ability to add Tags to VMs for easier searching from withing the CloudPhysics interface. In future these will be possibly integrated with vSphere tags which would be a welcome feature as more and more people are implementing tags for Storage Based Policy Management and Backup Management.

All in all another great set of enhancements to the CloudPhysics platform and I can tell you all that you need to keep an eye on what the team has in store for the next 6-12 months as I believe they are ready to take their offering to the next level and expand well and truly beyond anything they have done up to this point.

They have a free edition which you can tryout here: CloudPhysics Free Edition

Additional Content:

Chris Schin from goes through some of the new features during VMworld.

Resources:

https://en.wikipedia.org/wiki/Percentile

http://vmblog.com/archive/2016/08/25/cloudphysics-unveils-cost-calculator-for-private-cloud-with-public-cloud-comparison-tool.aspx#.V9au3Lh94-W

CloudPhysics Exploration Mode – New Host View

Late last year CloudPhysics released their VM Exploration mode feature which allowed for a detailed look into what was happening holistically to a VM with the ability to view key metrics and VM related events over an extended period of time. Last weekend CloudPhsyics extended this to also include Hosts. Extending Exploration Mode to include Hosts further improves the proactive monitoring and analysis capabilities of the CloudPhysics platform as it looks to break away from its roots of Card Views.

With Exploration Mode now encompassing both VMs and hosts, administrators can focus in on a workload performance issue and “replay” the environment to correlate events, resource utilization patterns, and environment changes in the seconds, minutes or days leading up to a problem in application performance or availability.

To view a Host with Exploration Mode, you use the new Search Virtual Machines and Hosts bar at the top of the CloudPhysics Web Console.

Once the Host has been selected you get taken to a dashboard that gives you configuration details of the Host, any changes (Power Operations, Snapshot, vMotions) that have been done against that VM in the provided date range and a performance graph that covers CPU, Memory, Network and Storage. There is also an Issues section which alerts you to any possible configuration issues or mismatch.

There is also the introduction of a Tab View which allows you to have open multiple Hosts and/or VMs to compare against…what would be nice would be the ability to overlay both Hosts and VMs to try and pinpoint events or key metrics points as they happened.

Below is a YouTube video from a recent webinar where the CloudPhysics VP Product Management Chris Schin walks through the way the platform uses Exploration Mode to identify root causes of VM Performance issues.

If you are interested in giving CloudPhysics a try, they have a free edition which you can register for and download here: CloudPhysics Free Edition

Firstlook: CloudPhysics Exploration Mode

During VMworld CloudPhysics released their new Dashboard Feature which saw a change of direction in the way CloudPhysics customers get presented with their data and was the first time Card Based analytics was not used to allow access to the wide array of metrics CloudPhysics stores in their data warehouses.

I’ve been working closely with the CloudPhysics team for a number of years now and they are great at listening to feedback around how to improve the platform. One of my biggest gripes (if you could call it that) over the years was that there was no way to view in detail (and historically) what was happening to a particular VM. One of the other issues was the time it took for data to show up in the CloudPhysics UI which meant that you could get access to data after about thirty minutes.

With the release of Exploration Mode there is more a case for proactive monitoring and analysis of VMs and their issues and the data refresh rate has been brought down to about 15 minutes which allows for more real time troubleshooting as well as allowing us to go back in time a number of days to try and correlate issues and try to look at patterns that might have occurred over the course of those days.

With Exploration Mode, administrators can go back in time, correlating events, issues, and changes that are associated with any selected time range in the vSphere environment, making it possible for users to see exactly what transpired in the seconds, minutes or days leading up to an application performance or availability problem.

To view a VM with Exploration Mode, you use the new Search VMs bar at the top of the CloudPhysics Web Console.

Once the VM has been selected you get taken to a dashboard that gives you configuration details of the VM, any changes (Power Operations, Snapshot, vMotions) that have been done against that VM in the provided date range and a performance graph that covers CPU, Memory, Network and Storage. There is also an Issues section which alerts you to any possible

CloudPhysics have always been a personal favorite of mine and I’m legitimately excited with what the team has got in store to further develop the platform into an extremely powerful analytics tool for VMware based platforms.

They have a free edition which you can tryout here: CloudPhysics Free Edition

Sources:

http://www.marketwired.com/press-release/cloudphysics-releases-exploration-mode-which-lets-vmware-users-identify-root-cause-2069851.htm

#VMworld: First Look – CloudPhysics New Release

Over the past year the guys at CloudPhysics have been relatively quiet compared to the proceeding 3 years since they burst onto the Scene at VMworld 2012. The reason relative radio science has been had was that they have been busily working away on a revamp of their SaaS based Analytics platform…and the results are impressive.

The new release has the following highlights.

  • Always-on diagnostics: Continuous diagnosis of infrastructure with changes continuously captured, recorded and reflected. Unique data derivations, correlations, mashups and filters reduce “noise” and identify true hazards.
    • Configurable dashboards: Rich contextual views expose hot spots and potential risks before problems form and impact operations. Trending analysis consolidates multiple objects and views, enabling multi-dimensional correlation.
    • Groundbreaking exploration capabilities: Interactive ability to analyze changes over time through easily manipulated exploration mode, using time slices with zoom in/out capabilities to evaluate correlations and causation. Users can “correlate in context” to troubleshoot application disruptions with data drawn from VM performance/resource consumption; change/event log; configuration history; and known issues associated with operational hazards and best practices.
    • 20+ new analytics for managing health and preempting hazards, available in our extensive library of “cards.”
    • Platform innovation: Time series data is uniquely handled by the CloudPhysics platform to enable a user to analyze multiple dimensions of the infrastructure around the same time axis.

With the new features, CloudPhysics delivers unique, meaningful insights, giving vSphere teams the confidence to act boldly to reduce risk and waste without compromising safety of the virtual infrastructure or the applications it supports. Building on its ease of use, intuitive user interface and deep visibility across multiple vCenters, CloudPhysics now:

  • Reduces disruption and incidents with always-on diagnostics that surface hot spots and emerging problems, enabling admins to get ahead of nascent performance problems• Improves mean-time-to-resolution (MTTR) with directed exploration, enabling admins to zero in on root cause and resolve application disruptions more quickly• Generates insights for realigning misconfigured infrastructure to prevent future performance and availability issues and improve efficiency

CloudPhysics have always been a personal favourite of mine since I first chatted to Irfan back on the Solutions Exchange floor in 2012 and I’m legitimately excited with what the team has got in store to further develop the platform into an extremely powerful analytics tool for VMware based platforms.

They have a free edition which you can tryout here: CloudPhysics Free Edition

While at VMworld, head over to Booth #2346

vCloud Director SP: VM Metrics Database Configuration Part 2

vCloud Director SP 5.6.x has the ability to export VM statistics to an external database source which can then be queried via a set of new API calls. I’ve gone through a couple of different posts on how to configure the Cassandra/KairosDB data platform.

vCloud Director SP: VM Metrics Database Configuration Part 1
Installing and Configuring Cassandra and KairosDB on Ubuntu 14.04 LTS

This post finishes off the series and goes through the configuration of vCloud Director to start exporting VM metrics and then how to query the vCloud APIs to get those metrics. One thing to mention before continuing is that the vCD SP Documentation KairosDB v 0.9.1 is referenced as the version to install. Even though there are newer builds 0.9.1 is the only one tested and verified…other versions cause bugs and I have seen some strange results.

Configuring vCD SP Metric Database Connection:

Data for historic metrics is stored in a KairosDB database backed by a Cassandra cluster. Cassandra and KairosDB are configured you then use the cell-management-tool utility to connect vCloud Director to KairosDB. To create a connection from KairosDB to a vCloud Director, use a command line with the following form:

cell-management-tool configure-metrics options

Those familiar with the vCD cell.log entries you will notice a couple new startup entries…there is a new port (8999) thats bound for KairosDB communications and the tail end of the log above appears after the config..though I’m not sure what it actually does.

Once the cell services have been restarted the VM metrics should start to be collected by KairosDB. Wait about 5-10 minutes and then enter this URL into a web browser : http://IP-KairosDD:8080/api/v1/metricnames You should see the following displayed:

You can see that the results show metrics relating to VMs…this is a good thing! Using a RestClient you will see a much prettier output

API Calls to Retrieve Current and Historical Metrics:

The following API Calls are used to gather current and historical VM metrics for vCD VMs. The Machine ID required used the VM GUID as seen in vCenter. The ID can be sourced from the VM Name. The vCD Machine ID shown below in the brackets is what you are after.



CloudPhysics: Enhanced Storage Analytics Cards [Part 1] – Datastore Contention

The guys at CloudPhysics have been busy behind the scenes of late working on improving an already great Analytic and Monitoring platform and recently I was able to preview a new enhanced set of Storage Analytic Cards. These cards are currently in Preview and with an official write up here by @esxtopGuru Which goes through the different cards on offer.

Coming from Service Provider land, I am always extremely interested in being able to find out how my datastores are performing and which VMs are causing or have caused trouble…I am also interested in SnapShots and if any have the potential to do harm on our platform. In this post i’ll be going through the Datastore Contention v2 Card…followed by Part 2 which will go through the Snapshots Gone Wild v2 Card.

Below is the new interface to the Datastore Contention v2 Card and you can see off the bat that there is a lot more going when compared to the v1 Cards

The initial Card View will show you Datastores across your environment that need Attention and those that are of interest. This is based on an algorithm that CloudPhysics have created that dictates acceptable levels of contention on VMs on datastores. You will get an overview of Throughput, IOPS and Latency metrics as well as total VMs and how many are potentially affected by storage contention.

While the actual metrics haven’t changed here from the v1 Card the way in which you can manipulate the data has been enhanced. For a period going back the lat 24 hours (It would be nice to go back further…something I’ve mentioned as a feature request) you can dynamically change the graph to display Bandwidth, Latency, IOPS, Outstanding IOs and choose to display Average, Read and/or Write Values.

As you click on the Active Red Zones in the Graph the list of Culprit VMs and Victim VMs changes to match the time period. [UPDATED] You now have a side by side view of Culprit VMs and Victim VMs giving quick and easy access to affected instances…By Clicking on the Blue View Details Button you can further drill down into the list and view VM specific storage metrics for that period as shown below.

You also now have the ability to export each graph in a variety of usable formats which is excellent for reporting purposes. In fact exporting has been enabled at all levels and you can export to CSV the entire Card View of data by clicking on the ALL Tab that lists all datastores in your environment. As a side note…the left hand search menu drop box sorting has been added to by a dynamic search bar which allows you to search for datastores…it even takes regular expression for those that are that way inclined!

Heading back to the top of the Card View you get an overall Aggregate picture of the currently selected datastores in the main presentation area…this dynamically adjusts based on what datastore(s) are in focus at the time.

Wrapping Part 1 up the new Datastore Contention card is brilliant…not only does it better give you access to potential problem datastores and VMs but it’s able to let you visualize and export data quickly and efficiently…The enhancements in the visual representation of data shows that the guys at CloudPhysics are looking at more dynamic style of data view which makes the overall experience and usability that much more enhanced.

http://blog.cloudphysics.com/blog/whos-minding-your-storage-zoo

http://blog.cloudphysics.com/blog/2014/4/7/noisy-neighbor-where-art-thou-performance-culprit-and-victim-analysis-using-cloudphysics-storage-analytics

 

First Look: CloudPhysics Card Designer

The boys at CloudPhysics are working hard behind the scenes at adding new features to their current stable of Analytic Cards based on data collected from their Probe VA’s hooked into vCenter environments.

Check out this post on their DataStore Contention Card:

For a general overview, go here: I am a massive fan of analytics and trend metrics and I use a number of systems to gain a wide overview of the performance and monitoring of our Hosting and Cloud Platform.

A few weeks ago, the CloudPhysics team released to a limited number of users a Custom Card Designer. This pretty much lets you construct custom cards based on a huge number of metrics presented via a builder wizard.

Cards you design and save are listed on the page above. From here you can view your custom cards and edit them if they require tweaking. Once you click the Create Card + button you are presented with a list of property data metrics from which to construct your card.

Properties fall under four main categories and there are a large number of available metrics under each category. The wizard lets you drag and drop items into the builder window. From there you can preview and then save your custom card for future use.

As a quick example I needed a quick way to see which datastores where connected to their respective hosts in each cluster so that consistency in datastore availability was maintained. It was as simple as dragging across Host:Name and Host:Datastore, putting in a filter to only view hosts of a certain name it was ready to go.

You have the option to preview and continue editing, or saving to the Card Designer main page. From that page you can execute the query. The results of my quick test card are shown below.

One thing I would like to see is an option to export the results to a csv or excel document…but other than that it’s a great example of what CloudPhysics is all about…data and how to get the most out of it as efficiently as possible.

First Look: CloudPhysics – Datastore Contention Card

I first came across CloudPhysics just before VMWorld 2012. For a general overview, go here: I am a massive fan of analytics and trend metrics and I use a number of systems to gain a wide overview of the performance and monitoring of our Hosting and Cloud Platform…as well as extending out to client systems.

I love the deep/complex analytics of VMware Operations Manager but sometimes I feel a sense of being overwhelmed with the sheer amount of data presented by the default views of vCOPs and working with the Custom Dashboards can be a frustrating exercise if you don’t have a heap of time and patience.

This is where I have found CloudPhysics comes into it’s own…via it’s brilliant presentation of things that matter. I’m not going to go through the setup and config, but in a nutshell…from the site, register, login, download and deploy the VMware Probe Appliance, give it an IP and enter in your email address as it relates to your CloudPhysics login. It’s one probe per vCenter, but you can deploy multiple probes to multiple vCenters and links them back under the same username and CloudPhysics App.

When you log in, you are presented with the home screen below:

From relatively humble and basic default cards released around the VMWorld launch the team has been adding more complex and useful cards. HA Cluster Health and SnapShots Gone Wild are my personal favourites and offer a view into key areas of vSphere management. What’s also great about these cards is that they offer external jump links to VMware KB’s and offer basic information about subject matter. The organisation and presentation of the data pulled by the probe is simple yet effective in allowing you to get an understanding of how your environments are performing and which areas are under stress.

Released today was the DataStore Contention Card which looks at the performance of VMFS Datastores in your environment. The Default view selects the DataStore that needs the most attention. In my case I was surprised to see the Datastore below exhibit combined read/write latency that was off the chart!

The interface allows you to select a block of time at any level and see which VM may be contributing the most to the Performance Metric selected. Those metrics are shown below and include Latency, Outstanding I/O’s, IOPS and Bandwidth. You also have the ability to  Filter the view by vCenter, Datastore Cluster and Datastore.

The screen grabs don’t do the CloudPhysic’s Web Application interface justice so head over the site and download the probe to get started. It must be said that the product is only in BETA so use at your own risk, but I’ve had no issues with the Probe VM who’s specs are 2vCPU, 4GB of RAM and 16GB of storage.