Mapping Virtual Machines to Datastores to Storage, Part 2

In a previous post, I walked through the challenges facing one of my customers during a recent outage, during which they needed to understand the mapping of virtual machines to datastores to LUNs.

Due to time constraints, I didn’t have the time to properly test and wrap everything into a nice little package, so the solution I ended up with actually used 2 different scripts… one to collect information on the virtual machines and their datastores, and another to collect information on the SCSI LUNs underlying the datastores.

I wasn’t satisfied, however, and knew I should be able to do it all in one script.  After a bunch of fiddling and testing, I was able to come up with the following:

  • I start by prompting the end user for the name of a target export file.
  • Then I capture information on all the virtual machines for the vCenter in question.
  • Entering a loop, I assign a few variables (more for ease of reference than anything else)
  • Then I capture information on the datastore – and here’s the key – using the name of the datastore as reported by the virtual machine as the filter parameter.
    • Get-View -ViewType datastore -Filter @{“Name”= “^$dsname$”}
  • Then using a Select-Object statement I capture a bunch of properties on the datastore and the virtual machine and
  • Export to a CSV file.

Once I figured out how to use the name of the datastore as reported by the virtual machine as the filter parameter, the rest sort of solved itself.  Here is the completed script:

It should be noted that this hasn’t been tested and probably doesn’t account for virtual machines that have VMDK files across multiple datastores, but then that wasn’t relevant for the customer in question.

I openly admit I am not a programmer, and only a novice PowerShell / PowerCLI scripter, so I am open to comments / suggestions!

Note: this was a script that worked in my environment. There is no warranty or support with this script, please use at your own risk.

Posted in Adventures in Scripting, Automation, From the field, TechInfo, Virtualization - Server | Tagged , , , | Leave a comment

Mapping Virtual Machines to Datastores to Storage, Part 1

summary of the problem

This past week, I had the (unfortunate) opportunity to help provide some on-site support to a customer suffering through an outage (as a matter of fact, I am again this week).  The customer in question lost a large amount of storage due to an array failure.  This, of course, caused their DR plans to kick in, they declared failover for their critical workloads, and they began the process of recovery.  Along the way, they discovered a large number of workloads did not, in fact, actually have plans for backup and/or DR.  As a result, they needed to quickly establish the scope of the impact, which virtual machines were on which storage, etc.  They didn’t / don’t have any tools appropriate to that ask – ideally they would have some sort of storage resource management view that informs them – QUICKLY – of which VM is on which datastore and therefore which LUN and array.  Yes, you can get this information out of vCenter, but it is a top-down view rather than a storage centric view… mapping datastores to LUNs and arrays isn’t quick, intuitive, or a native feature of vCenter.  They also didn’t have the appropriate vCenter plug-ins from their storage vendor(s) to tie that information together quickly.

Aside from the fact that the ESX hosts and vCenter were all seriously compromised due to the sudden loss of BOOT LUNs, datastores (separate post to come later), etc., their only option was to:
1- pull a list of WWNs for the impacted LUNs off the array
2- run a report of some kind from vCenter to pull the vms and their datastores, and then
3- run a report of some kind from vCenter to pull the datastores and their NAA ids…

My first thought was of course to try to pull the appropriate information from vCenter (even though it is not an accurate source of truth in a compromised environment (separate post to come later)…  for this customer, SSH is not enabled on their hosts, only a select few people have the right to do so, and many of the hosts were in such a compromised state that you couldn’t gain any useful information from them.  Therefore our only choice was to pull information fro vCenter, which had a view of the state of the datacenter at a particular point in time, prior to the event.  To compound matters, this particular customer has quite a few vCenter servers, and a subset of them represent the environment in question.

Therefore, from the superset of all vCenters, we needed to then pull ALL virtual machines and their datastores, and then pull ALL their datastore / LUN information, and cross reference it all against the WWNs pulled from the array to arrive at the information in question.

option #1 – try PowerShell

Knowing that we had to pull a LOT of information from (perhaps) dozens of vCenters, it seemed to make sense to me that this was a perfect opportunity to practice a little PowerShell.  No being a guru in that field myself, I did a little research, pulled in a couple favors, and put together the following few lines of code – very quick a dirty, but it looked promising:

1 – Get information on virtual machines and their datastores:

Once we have the information on the virtual machines, I then needed to…

2 – Get information on datastores and their LUNs:


No Love…

I decided to just run each of these little scripts against the vCenters in question (taking the time to put it all into one script and then parse through a list of vCenters wasn’t really an option, as the list of vCenters in question changed), and then once the data was gathered, use some functions in Excel to merge it all (which would be quicker in the short term for me, as my Excel skills far exceed my PowerShell!).  Well, as it turned out in a couple of test runs, it took hours for the above scripts to run… I mean, really, really horrible…  Abysmal.

In the end that first day, we ended up just going into the vSphere Web Client and exporting data into CSV files from the GUI…  It was neither the time nor the place to try to ‘root cause’ the sluggishness of the scripts, but in any case, it was not auspicious…

option #2 – try PowerShell, but better this time!

Well, this bothered me, needless to say.  Not only had it taken longer than I had wanted / hoped / expected, I ended up wasting both my and my customer’s time on a method that ultimately yielded poor results.  So I did some more digging in my off time, and discovered the PowerCLI cmdlet “Get-View.”  Whereas the Get-Datastore and Get-Datacenter cmdlets I used above actually talk to vCenter over the vSphere API and then query the vCenter database for some of the information (and really, it was the Get-Datastore loop above that was incredibly slow), the Get-View cmdlet just queries vCenter and returns the in-memory .NET object, and therefore doesn’t need to parse through individual hosts in the inventory.

Ultimately I ended up with the following lines of code in 2 different scripts:

Get-View command will return information on multiple types of vSphere objects:

  • ComputeResource
  • ClusterComputeResource
  • Datacenter
  • Datastore
  • Network
  • DistributedVirtualPortgroup
  • DistributedVirtualSwitch
  • Folder
  • HostSystem
  • ResourcePool
  • VirtualApp
  • VirtualMachine
  • VmwareDistributedVirtualSwitch

The really cool part of get-view is that when you run it on a certain type, you can parse through the object in question further by drilling down into additional properties, since you are basically just walking down through the API.. For example, in the above example, you will see I set a variable as follows: n=’NAA’;e={$}.  In Powershell, the notation $_ is a self-referrential notation, meaning it refers to the object that has just been gathered (in this case, Get-View -ViewType datastore).  Then, we ask for “.info.vmfs.extent.diskname” on the object, which asks specifically for the:

  • disk name assigned to the
  • extent assigned to the
  • VMFS volume for the
  • datastore.

The statement following then requests for the disk layout information on the virtual machine, which returns the name(s) of all the VMDK files assigned to the virtual machine (including the path to the datastore and VMDK), and the size of the VMDK.


This provided the required results… data from vCenter that was as fast as / faster than the C++ or Web client…

Along the way, I learned a bunch about PowerCLI, and in a future post I hope to how how to pull this information together into a single script.

Note: these were scripts that worked in my environment. There is no warranty or support with this script, please use at your own risk.

Posted in Adventures in Scripting, Automation, Best Practice, From the field, TechInfo | Tagged , , , , | Leave a comment

EMC Private Cloud Reference Architecture – references

Documentation Links

Download Links:

Posted in Automation, Journey to the Cloud, Virtualization - Server | Tagged , , , , , , , , , | Leave a comment

Graphy Goodness…

In my previous post, I tried to discuss how the traditional RDBMS, while being very good at storing and tracking certain types of data and transactions, fails to provide value for others, and how graph databases are coming to the forefront of modern architectures for understanding the connected nature of stuff.  Kind of funny, really, since whole books have already been written about the topic, and others have blogged about it (here), and written papers about it (here), but I guess I am just publishing my own process of working through it for myself….  🙂

nosql-logoTherefore, having addressed the reasoning behind the ‘raison d’etre’ for graphy goodness, my thoughts turn toward use case.  Clearly, just the fact that a wicked cool new db technology is starting to gain popularity is not enough, by itself, to displace traditional RDBMS…  Graph theory and implementation will have a hard time completely replacing the prototypical use cases like a customer-order tracking system.  There are plenty of discussions out there already on what NOSql databases are good for…

BTW – you can visit for an exhaustive list of NoSQL systems…

Neo_Technology_LogoNeo4J has also listed their own use cases here:

To my own mind, however, I believe that graph theory is still undervalued and as of yet not well understood… to be fair, I am still learning about it myself.  Nonetheless, when I read that a graph database is only good or something like, “Rock complicated graph problems. Fast” on the High Scalability  blog site above, or the handful of use cases on the Neo4J site, I am surprised that its impact on social graphing is not more self evident.  Personally, I have found that “graph problems” exist everywhere, yet up until now I simply haven’t had the tools or understanding to capture them in a meaningful way.

We, as humans, understand this inherently – we are frequently confronted with the implications of our ‘network’ – the intersection of our business with our social interactions.  I personally believe there is an application for graph theory throughout our lives.

hot-water-heaterSeveral years ago, we developed a leak under our hot water heater.  Water was leaking from all around our house, and since our house sits on a slab, with no crawl space or basement, we had no way to easily detect where the leak was.  Fortunately we found a very clever plumber, who by selectively turning specific faucets in the house on and off was finally able to get a clear sound of the leak – we could literally hear it dripping from the pipes.  Through his clever work , we were able to determine – through sound – that the leak was probably within a few feet of the water heater, rather than under the kitchen, near the sink or someplace else.  This localized where we had to jack hammer through the slab, and prevented thousands of dollars worth of damage.  Sure enough, the leak was in the hot water line, no more than a foot away from the water heater.

What’s my point?  Well, the next year, when we had some elective plumbing work to accomplish, we wanted to use the same plumber – but he had moved on.  It took a while for us to track him down and discover which outfit he had joined up with.  We had to force the issue with the previous plumbing company, ask a bunch of friends, call a few plumbers until we ran across a plumber from another company that knew the guy, and then we were able to track him down.  There’s a graph problem in there somewhere…

Multi-tenant Cloud

Multi-tenant Cloud

I work for EMC as a Presales Systems Engineer, and I interact with customers every day.  My job is all about assembling infrastructure solutions to solve business problems for our customers, and I happen to be focused on virtualization solutions.  It is frequently necessary – desirable, in fact – to coordinate my efforts with my VMware or Cisco counterparts.  However, the sales and support models for the three companies – EMC, VMware, and Cisco – are significantly different.  I work in the Enterprise Mid Atlantic South division of EMC, but any given customer may be a “Strategic” customer for VMware, while being a “Commercial Select” customer for Cisco ( don’t ask – account designations per vendor can be somewhat arbitrary).  If I want to work with the VMware SE on a particular account, certainly it would be easiest to just ask the customer who their VMware SE is.  However, there are times you might want to work with them before approaching the customer, and for that reason I might need to navigate the VMware presales organization to figure out who the correct SE is.  Today, I maintain my own business network in my head – I already know the VMware and Cisco SE’s in the area, and I can usually reach out to them directly.  But what if there was a kind of Facebook for customer / vendor relationships?  LinkedIn doesn’t quite accomplish this – it allows you to designate if you have worked with someone at another company, but not necessarily the nature of that relationship.

Those are just two examples from my own life… Consider the following possible uses cases as well:

  • Org charts – I have worked with many organizations that have both functional and hierarchical requirements for their organization – employees are “matrixed” to multiple business units.  A graph makes this extraordinarily easy to represent and navigate.
  • CRM – if you work for a sales organization, you are familiar with the problem of the “business” structure of a customer versus their “political” structure…  that is to say – who are the political and influential people at a customer, and that doesn’t always map cleanly to an org chart.  Employees way down on the org chart frequently carry influence, this manager might have the ‘juice’ to get a decision made where another might not, etc.   Again, a graph makes this easy to capture and represent, while many traditional CRM packages fail to do this seamlessly.
  • Career management – what happens when you leave a role?  You want to work with others that you know will help make you successful…  My gut tells me there’s a graph problem there, too.
  • Conflict Resolution – Consider the problem of Rock-Paper-Scissors (or Rock-Paper-Scissors-Lizard-Spock) – modeling conflict scenarios and possible outcomes…  I am thinking of a conflict modeling engine to help navigate problem resolution….  I would think there would be tremendous geo-political application.
Homage to Big Bang Theory and Neo4J

Homage to Big Bang Theory and Neo4J

As humans, we deal with graph problems daily – we are social creatures, and networks permeate every aspect of our lives.  I am sure I don’t (yet) have the capacity to anticipate every way in which graph theory might impact us, but I am sure it is going to.  Does this mean we need a graph database for everything in our lives?  Of course not, but for something so pervasive, I am guessing new and interesting ways to use graph theory are going to be popping out of the woodwork for years to come…

What do you think?  What are some of the interesting ways you see graphs being used?

Posted in Miscellaneous, Next-Gen Apps, NoSQL | Tagged , , , , | Leave a comment

Home Lab v. 4.0

Another lab post!

… I love it!

Having assumed the role of a manager this past summer (July, 2013), I find my opportunity to really dig into ANYTHING and satisfy my technical cravings are severely limited.  There is a lot of travel around the division for customer meetings, team meetings, internal meetings, etc., and hardly any for training.  I get that as a manager that is part of the gig, but I must admit that part of me rails against the idea that I have to drop my technical chops at all.

The lab setup described in earlier posts (here, here, and here) has actually spent the past year or so in the EMC lab in Columbia, MD, under the care and feeding of my partner in crime, Larry.  However, feeling more and more like I am losing my technical abilities (and ultimately my credibility), I decided to retrieve those systems and impress them into use once more as my proving grounds.

Home LabThe gear hasn’t changed much…

  • Western Digital Caviar Black WD1001FALS 1TB 7200 RPM SATA 3.0Gb/s 3.5″ Internal Hard Drive
  • Crucial 16GB (4 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10600) Desktop Memory
  • AMD Phenom II X4 965 Black Edition Deneb 3.4GHz Socket AM3 125W Quad-Core Processor
  • Diablotek PHD Series PHD750 750W ATX12V / EPS12V Power Supply
  • SAPPHIRE 100293L Radeon HD 5570 1GB 128-bit DDR3 PCI Express 2.1 x16 HDCP Ready Video Card
  • MSI 790FX-GD70 AM3 AMD 790FX ATX AMD Motherboard
  • Antec Nine Hundred Black Steel ATX Mid Tower Computer Case

The on-board REALTEK NIC drivers gave me a problem for a while, but I found this blog post that describes how to use esxcli to install new drivers, and it worked like a charm (clearly something I probably should have learned before now, but there you go).

I have added an additional couple of drives to each computer – a 100GB SSD as well as a 256 GB 15K RPM SAS drive.   Now each of the systems above has 3 tiers of storage – SSD, SAS, and SATA.  I have also included an Iomega ix4-200d for network storage of templates, bin files, ISOs, home directories, etc.    My plan is to use the Iomega for data at rest, or read activity only…. I have tried to use this particular device for active, running vms, but it doesn’t seem to be up to it.

Instead, I plan to have all 3 systems boot off a USB stick with ESXi installed, and use the 3 internal drives in an implementation of vSphere vSAN.  That will give me 48 GB RAM, 12 cores, and 3 TB of storage with SSD acceleration without shared storage!    Very cool…  I already have vSphere 5.5 installed on all 3 systems, without vSAN, and have tried using the vMotion capabilities between vSphere hosts without shared storage and it works great.  I am using the vSphere vCenter 5.5 appliance for all of this.

Once I have all the cluster stuff settled (HA / DRS / vSAN), my ultimate goal is to get the whole rig installed and running with CoudFoundry…  More on that later.

Posted in Miscellaneous | Leave a comment

Five Minutes on PaaS…

Is your company is trying to change?  Trying to update from a legacy architecture?  Are your applications tightly coupled?  Written on outdated systems?  Would you like to take advantage of newer application capabilities on state of the art infrastructure?

Skip the nonsense…

Why would you bother trying to update your legacy infrastructure?  Why would you even look at trying to build a private cloud from scratch?

telephone-polesTake a lesson from Africa…  Over the past 10-15 years, the world has watched while countries like Nigeria and Kenya have experienced an enormous increase in connectivity and access to the internet through  cell phone technology.  Arguably, there was an existing physical telephone infrastructure, but it was rigid, required people to go to one of the few places there was a phone, and subsequently access to telephones and internet was limited to a very few… exposure to the connected world didn’t exist.

Enter the cell phone…

According to CNN (, a little more than a decade ago there were only 100,000 phone lines in Nigeria, but today Nigeria has close to 100 million subscribers.  Cell phones have revolutionized life in Nigeria, Kenya, and South Africa, providing access to online banking and the ability to move money around, access to the internet and new forms of entertainment, communications capabilities for huge waves of activism and critically required disaster management.

They didn’t bother trying to expand on the existing rigid POTS network, they skipped right over that step and went straight to cell towers. Yes, I am sure I am vastly oversimplifying the reality of this shift for those countries, but there is a lesson to be learned.

product-2block-backgroundThe VCE Company trumpets much the same message, albeit just for datacenter  infrastructure.  why would you even bother trying to build your own private cloud?  The underlying components (servers, networking, and storage) are well understood, every single vendor has a highly reference-able installation base, the institutional knowledge required to operate those infrastructure domains has been baked into our organizations over the past couple decades… Why build it yourself?  We all (at least 99.999% of us) just buy a car from a dealer, rather than building it ourselves.  Why?  Because we want to DRIVE the car… we have companies that can BUILD the car far more reliably and efficiently than any of use could.  Hence the Vblock.

The Same is True for PaaS and ITaaS

Many IT organizations today are trying very hard to overcome the organizational challenges of figuring out how to operate in a cloud business model, where it’s all about service delivery, rather than project deadlines.  They see the cloud model as a strategic differentiator, rightly so, for becoming / remaining more relevant to the business.  Along with a cloud operating model, there is a whole slew of technology that can be put in place to enable the infrastructure to become agile enough to match a cloud operational model, but again I ask the question…. Why would you bother?

If your goal is IT as a Service (ITaaS), the first thing you should be asking yourself is whether or not that is REALLY your goal…  Is your business trying to become more agile and competitive?  Are they trying to bring new applications to market faster?  The answer from many of my customers is a resounding YES.  If so, then you probably need to skip ITaaS and move straight to Platform as a Service (PaaS).  Developers want to write and deploy their applications, not have to monkey around with operating systems and software revisions.

Yes, it is true that some companies offer a suite of products designed to help you BUILD a PaaS, such as VMware vCloud Automation Center WITH vFabric Data Director AND vFabric Application Director… AND vCloud Director….

…AND vCloud Networking and Security (or soon, NSX)…

… AND vCenter Orchestrator…

did I forget anything? Or right, … AND PUPPET… AND ViPR, et cetera.

You get the picture.  Don’t mistake me – these are very cool technologies that solve some very difficult, real-world challenges.  But I have to ask myself – if my customer is working towards PaaS, do I really want to ask them to try to stand all this up?  Or would I rather point them towards a pre-built solution that gives them everything they need?

PaaS_PivotalCycleHave you seen the updates on Pivotal CF (Cloud Foundry)?  CloudFoundry comes prebuilt with all the automation, application services, etc., that a group of developers needs to start coding, and without all the pain of setting up the infrastructure and the software stack necessary to do so.  If you prefer to run it in-house, with the recent announcement made (see link above), if can all be run in your private datacenter.

I personally believe the future of the datacenter is going to look a lot like the following:

angry-monkeyIf your goal is to move to the next generation of applications, then do it – don’t bother trying to build legacy infrastructure for next-generation applications.  Build the infrastructure that matches the operational model of the applications.

Of course, there are always exceptions…

All of my customers have client-server applications that were written sometime over the past 20 years (or more) that still provide tons of value to the business, and they are not going to just ditch those apps.  So, legacy infrastructure models will persist for a while, perhaps a long while, perhaps indefinitely.  I am not so naive to believe that many companies will be able to make this kind of a leap easily.

But to those that do – be bold and embrace the change!  With great courage comes great results!


Aside | Posted on by | Tagged , , , , , , , , , | Leave a comment

Roshambo, the Big Bang, and metaphysics …

I have been struggling for a while with trying to understand and depict in an easy-to-relay fashion the organizational structure of my employer – it may sound silly, but it turns out it is not all that easy to describe or represent our rather 3-dimensional management structure in EMC presales.  I found that Neo4J is easily capable of not only constructing the necessary relationship models, but also of allowing easy traversal of the graph to ‘walk’ the EMC presales organization.  Our presales organization is not only somewhat 3-dimensional, it is also not strictly uniform from division to division, theater to theater, or business unit to business unit.  Asking a question such as “who is my peer in that other division?” (or worse, “who is my peer’s manager in that other division”), while unnecessarily convoluted in an RDBMS, is relatively straightforward in Neo4J.

Then, while reading an article on How to Use Neo4J for Natural Language Search, the author made the point the value a graph brings to a complex data modeling problem isn’t just that it can model relationships in a more agile fashion than an RDBMS, it is actually the metadata that holds the value.   That is to say, the questions a user will be asking of a graph database are answered by the nature of the relationships themselves. Put another way, it is not necessarily the objects / nodes that are of interest, but the relationships themselves.

Let’s consider a common game that many folks are familiar with- Rock Paper Scissors (also known as Roshambo, though i will refer to it hereafter as RPS).  The game is played using hand symbols to represent objects (ummmm…, a rock, paper, and scissors, maybe?).  For an understanding of how the game is played, ease see Wikipedia here- Rock Paper Scissors.

Ready to get a bit metaphysical?

(Bet you never thought you would read that in the context of Rock Paper Scissors…)

While it is necessary to have representative objects for the players to use to play the game, the objects themselves are immaterial it is the relationships between the objects that imparts significance to them… That is to say, it is how the objects are used that we are interested in, rather than the rock, the paper, and the scissors.

If we wanted to try to represent this in a traditional RDBMS, what would we need? Well, we would need a table, and that table would have one or more rows of data, each row representing a single record. It would be easy enough to use a single table with 3 rows, one each for the rock, paper, and the scissors. How would we then represent the relationships?

Relationships in a RDBMS are formed through the use of identifying ‘keys’ to tie records in one table to a record or records in another. The ‘key’ is used to avoid duplicating data. In an effort to represent the relationships for RPS using a RDBMS, consider the following diagram: rps-as-rdbms

Note the complexity inherent in trying to force the issue – every record needs an identifying ‘primary’ key in the ‘object’ table, and a corresponding ‘foreign’ key is used in the ‘relationship’ table to tie the object to another object. Using a single table for the objects is the cleanest choice, reducing structural redundancy in the database, but it is necessary to hen create a ‘join’ or ‘bridge’ table to construct the relationships. Furthermore, all we have been able to accomplish thus far is establish that there IS a relationship, but since this type of ‘transaction’ doesn’t inherently carry the NATURE of the relationship with it, the solution falls short of being able to represent the transactions that occur during the game. We haven’t even addressed the nature of the relationships themselves yet – how would we denote that Rock ‘crushes’ the scissors, and so on? We would have to add another field in the existing table, or perhaps even another table…


In its full instantiation, it might appear as follows:


On my favorite show, The Big Bang Theory, Sheldon and the gang use a ‘nerdified’ version of RPS that they call “Rock Paper Scissors Lizard Spock” (click the link for an explanation).  In this game, the rock, the paper, and the scissors are complemented by the addition of a lizard, and by Spock (hence the name of the game!).


So what’s the point?

The point, briefly, is simply to drive home that while some systems fit into a relational management scheme well, others don’t.  In the case of the game above, a relational system just doesn’t capture the important information – the relationships – in a fashion that makes sense or are useful to humans.  Consider the following diagram instead:

In this diagram, we can clearly capture the relevant information about the relationships between the objects.  What’s more, this can be represented easily in a graph database, because the database was designed to be ‘schema-less’, inherently flexible and capable of representing complex relationships as first-class citizens.

Furthermore, in the aforementioned article on Natural Language Search, the author makes the point that the value of the graph database is in being able to identify and cache the relationships between objects, as queries to the database will be in the form of “what is the relationship of X to Y,” rather than the types of queries you might see in a traditional RDBMS (“how many objects of type X are there?”)…  Since the nature of the questions being asked differ, the nature of the system used to store and retrieve the data must necessarily be different as well.

That’s enough for now…  I will pursue the nature of the above in a future post.  Thanks for reading!

Posted in Miscellaneous, Next-Gen Apps, NoSQL | Tagged , , , | 1 Comment