VCIX6-NV Passed

After studying on and off for a couple months I was able to Pass the VCIX6-NV exam yesterday.

We will work on putting together a study guide soon but I just wanted to share my exam experience.

1. Follow the blueprint…if you can check off every item on the list as something you know how to do you’ll be fine.
2. Don’t get stuck or hung up on any one piece that you are unsure of, even if you might think it’s something you’ll depend on later in the lab. Just move on if you aren’t going to get it, you can also come back to it later if you figure it out or if you find something did depend on it. There is partial credit for things.
3. Accomplish as much as possible in the given time (beating your head against something isn’t earning you points)
4. Try not to get up and use the restroom mid-exam like I did…too much coffee..

Clearing AWS Solutions Architect – Associate

OK. So during the downtime between now and when I plan to take the VCIX Design exam at VMworld I decided to take a look at the AWS offerings and certifications.

My past hands on experience was fairly limited with AWS solutions.  I had previously done the following.

  • Created EC2 instances.
  • Created VPC’s.
  • Created both public and private networks/subnets.
  • Created VPN connections from VPC’s and VPG’s to on premise Customer Gateways.
  • Created and manipulated Security Groups, ACL’s and deployed NAT instances for Private/Public routing.
  • Deployed and tested Amazon Storage Gateway.
  • Used S3 for some static content.

So to prepare for the exam I did the following.

PluralSight training

Amazon Web Services (AWS) Fundamentals for System Administrators

Architecting Highly Available Systems on AWS

Optimizing and Managing Distributed Systems on AWS

I found the PluralSight training to be very well done.  The second two links provided above dive into deploying an actual distributed system using many of the AWS services offered.

Printed Documentation

I also read all the documentation I could get my hands on.  AWS is really good about supplying massive amounts of material in PDF/HTML and Kindle format. I simply downloaded lots of material to my Kindle.  And then read it.  Well…most of it.

Amazon Search for “AWS Guide”

AWS Documentation Site

AWS Lab work

I utilized the AWS platform and tons of their Free Tier resources to build and test small environments and familiarize myself with all the material I watched and read.  The Free Tier of resources is a great way to be able to get your hands dirty at a low cost.  I highly recommend this.

Test Format

The test itself was multiple choice, 60 questions and I had 80 minutes to complete it.  I completed it with about 25 minutes to spare after reviewing anything marked for further review.

The test will state to choose 1, 2, etc.  It will not state “choose all that apply” and it will not let you mark for review w/o selecting the minimum required selections.

AWS Certified Solutions Architect – Associates Info

AWS Certified Solutions Architect Exam Blueprint

Passing score is 65% (I believe).  I cleared with 72% after doing the above over a 3 day period of time.  Yes 3 days.  I am dedicated.  You can easily spend more time reviewing and studying but I also believe having a fairly strong virtualization background aided in the fast tracking of this certification.  I plan on continuing with the Professional Level.

There are 3 AWS Certification tracks you can go with.  I decided to do the Solutions Architect to start with.

AWS Certification Tracks

Solutions Architect


SysOps Engineer


Hope this helps.  Ping me with any questions that I may be able to help with!

VCP6 Exams now available!

VMware released 3 new exams associated with the new VCP6 yesterday.

vSphere 6 Foundations Exam
As far as I can tell this exam does not provide certification of the VCP6 but is a stepping stone to the VCP6-DCV which is not required. This exam covers the fundamentals of installation and administration.

VMware Certified Professional 6 – Data Center Virtualization Exam
This is the full VCP6-DCV exam which provides certification.

VMware Certified Professional 6 – Data Center Virtualization Delta Exam
This is the delta exam for existing VCP holders that want to upgrade to the VCP6 with a testing mainly focusing on what’s new.

Additionally if you would like here is the full certification roadmap for version 6.
VMware’s version 6 certification roadmap

Overlapping VMware Virtual Guest Disk Space alarms applied to an entity fail to trigger EITHER alarm in Veeam One v8

This particular issue was present in Veeam One v8 pre Patch-1 and exists post Veeam One v8 Patch-1.

The issue is that you cannot have multiple overlapping VMWare Virtual Guest Disk Space alarms applied to the same entity in your infrastructure.

For example.

You cannot have 2 VMware Virtual Guest Disk Space Alarms configured and applied to the same entity in your infrastructure.

If you configure one VMWare Virtual Guest Disk Space alarm to trigger to warn at 10% and error at 5 % you cannot add another VMware Virtual Guest Disk Space alarm for the same entity with different values.

If you do apply overlapping VMWare Virtual Guest Disk Space Veeam One v8 will not know what to do with both alarms and will simply not report any/either problem at all.  That’s right.  No warning, no error, nothing.  Applying multiple VMWare Virtual Guest Disk Space alarms effectively nullifies both alarms and you get no alerting.

Well why would you do this anyway?

To begin with, because I want to and I could just fine in Version 7.

The use cases for us are as such.

  1. In a multi tenant environment it is possible that a customer or a department wants to receive a warning or error based on their own dictated values while you as the admin still want to see the same alarm with your enterprise wide standard values.
  2. Veeam One does not by nature update its own alarms as values change.  So if a disk is filling up and your warning is set to trigger (below) 10%, your error is set to trigger (below) 5% you will only get alarms at those marks.  You manually have to “resolve” alarms and have them re-trigger to see what the new or current values are.  Because of this we had implemented a **Critical** disk space alarm that was escalated to a higher severity once tripped at (below) 3%.  This allowed us to know (without manually resolving alarms) when disk space had gone past what we considered both warning and an error state and reached an immediate severity one issue.

Is there a work around?


You simply cannot configure multiple VMWare Virtual Guest Disk Space alarms and apply them to the same entity in your infrastructure.  So look out for this and plan your alarm management accordingly and do not accidentally apply overlapping alarms.

Good luck!

vSphere HA initiated a virtual machine failover action

I received an warning at the Cluster level recently after putting a host into maintenance mode and rebooting it. No virtual machines were actually failed over and there was no problem to be seen. After doing some research I’m still not sure what triggered the error message but in order to clear it you will need to disable HA on the cluster then re-enable it.


Dell EqualLogic PS6210X Controller Panics and Failures

This is more of a heads up post and to see if anyone else is experiencing the same issues.
We have one particular environment with 4 Dell EqualLogic PS6210X arrays in a dedicated Storage Pool.
Connectivity is provided by Force10 S4820T’s and connected to a Dell M1000e blade chassis with Force 10 MXL switches.

3 of these 4 arrays have experienced multiple random controller panics or outright controller failures within a very short time frame (multiple within 30 days of ownership and they continue today).  No log data can pinpoint the cause and Dell has been unable to provide any definite fix. Dell always pushes the “update the controller firmware” card each time. We keep the firmware very much up to date with these units and the issues have persisted through controller updates.

The controller failover happens quicker in the PS6210X units vs past PS6110X units. The older PS6110X units come very close to the 60 seconds time frame. These newer PS6210X units failover much quicker. We have seen pings tests drop for about 4-6 pings. With each controller panic of failure we have maintained datastore connectivity but the VM performance experience was briefly impacted.

The impacted PS6210X units are currently running controller firmware versions V7.0.9 (R400081).

Versions V7.0.10 and V7.1.2 are both available but I doubt they will fix this issue as we have communication from an EqualLogic Engineer stating that the controller failover is a known issue and will be addressed in V7.1.3.

Both releases (V7.0.10 and V7.1.2 ) state: “provides some key maintenance fixes for enhanced stability and performance of your EqualLogic SAN.”


“While running firmware version 7.0.x, an unexpected controller failover, or restart, may occur at 248 consecutive days of uninterrupted operation. In rare circumstances, this may cause you to encounter a brief disruption to existing iSCSI connections.”

Based on past experiences we evacuate our PS series arrays prior to controller updates.  We have experienced much longer than anticipated failover times during the controller upgrade process.  Many EqualLogic engineers have also warned us that to avoid almost certain disruption of service the evacuation is necessary prior to the controller updates.

I will update this article with results and stability utilizing the V7.1.3 firmware release once its available and been in operations for a few months.

I have no known workaround for the issue and no data to support any preemptive action can be taken to mitigate the risk/failover. Transitioning critical workloads to other storage platforms would be my suggestion for now.

Example of the dump we get sent via e-mail.

I wrote this long after the last event and don’t have EqualLogic Array logs at this time to post. I will append them once this happens again.

WARNING event from storage array PS6210X-3
subsystem: SP
event: 28.3.51
time: Sun Jan 18 06:12:50 2015

Warning health conditions currently exist.
Correct these conditions before they affect array operation.
Active control module cannot communicate with secondary control module. Failover cannot occur.
There are 1 outstanding health conditions. Correct these conditions before they affect array operation.
ERROR event from storage array PS6210X-3
subsystem: SP [secondary]
event: 15.4.10
time: Sun Jan 18 06:12:55 2015

NVRAM contains valid data. This is an EXCEPTION RECOVERY due to an exception on a qrq processor.
ERROR event from storage array PS6210X-3
subsystem: SP [secondary]
event: 15.4.5
time: Sun Jan 18 06:12:55 2015

Saved CPU registers, CPU 4
at 0000000000000000 v0 000000000db43800 v1 0000000000000000
a0 000000000db43880 a1 0000000008022d60 a2 000000000db43868 a3 0000000000014622
t0 000000000db13d00 t1 ffffffffb41eb8c9 t2 0000000000000001 t3 0000000000000000
t4 848ba40000000000 t5 000000000302fb38 t6 0000000000000003 t7 00000000090c0000
s0 0000000000000000 s1 0000000000000000 s2 0000000002018960 s3 0000000000000000
s4 0000000000000896 s5 0000000000001000 s6 0000000000000000 s7 0000000000000000
t8 00000000090b9530 t9 00000000090b9558 k0 0000000000000000 k1 0000000000000000
gp 0000000008027ac0 sp 000000000302feb0 s8 0000000000000895 ra 000000000401aef0
ERROR event from storage array PS6210X-3
subsystem: SP [secondary]
event: 15.4.17
time: Sun Jan 18 06:12:55 2015

Saved CP0 registers, CPU 4
badva 0000000000000000 epc 000000000401af08 errorepc 0000000000000000
sr 748000b3 cause 4000900c errctl 00000000 cacheeri 00000000 cacheerd 00000000
buserr 0000000000000000 cacheerrdpa 0000000000000000

Veeam One Version 8 – VMware guest disk space alarms not working

Ok folks.  Stick to Veeam One V7 if you can.  V8 needs some patching.  The one big one that is going to bite you is that in many cases (post V8 upgrade) the VMware guest disk space alarms are not being reported correctly.  There is a patch for this but you have to call Veeam to obtain it.

The issue will be pretty evident.  The guest disk space alarm for VMware guest disk space will not trigger any warnings or errors.  Go ahead and make a test VMware guest disk space alarm and set it to warn at something ridiculous like 90% free or less.  (Just don’t enter a valid e-mail when making the test alarm or you will get spammed).

If this test alarm does not work then you have Veeam One V8 VMware guest disk space cooties and this patch is for you.

I am not sure how long the FTP sites Veeam provided me will remain valid so I also uploaded required files to Dropbox.

Link to Veeam One V8 Guest Disk Space Patch (provided by tedg)

Link to Veeam One V8 FTP Guest Disk Space Patch (provided by Veeam)

If you want to Roll back to version 7 you can use the links provided below.

Link to Veeam One V7  Downloads (provided by tedg)

Link to Veeam One V7 FTP (provded by Veeam)

All patch instructions are included but I urge you to take a Veeam One DB backup and a snapshot of your Veeam One server  prior to patching V8.


Hope this helps!


**Update on 02.17.2015.  Post patch issues still exist.  We have VM’s not tripping VMWare Guest Disk Space alarms.  Built in alarms for disk space as well as custom disk space alarms that we made are failing to trigger properly.  Case re-opened.  Insert sad face.  Insert devil horn face.

vSphere 6.0 notable features

Ok, maybe I should call this “notable for me”.  I am sure someone will chime in with “Hey-You fat bloated idiot! – What about X feature?”.

VMware published White Paper on this topic.- “VMware vSphere Platform What’s New”


  • Clusters:  64 Hosts.
  • Single Host:  480 Physical CPU’s, 12TB of RAM and 1,000 virtual machines.  (whoa!)
  • VM’s:  128 vCPUs and 4TB of vRAM.  64Tb datastore limit remains.

These numbers are getting out of control or I simply don’t live in the world where a VM and the application architecture requires these maximums. At what point do these figures surpass customer driven requirements and exist to maintain paper number data with competing products (Redmond)?  There has to be a graph owned by some department at VMware for the high maximum requirements vs actual demand vs RD costs vs marketing collateral.  Tangent!   Move on.


  • When hot adding memory to a vNuma machines, the memory is allocated across all NUMA regions.

vCenter Installations

  • vCenter, Inventory Service, Web Client and Auto Deploy are all installed on the same server now with no option to split these components up.  VUM can still be installed on another server.

vCenter Appliance

The vCenter appliance is looking like the way to go and has been looking that way for a while.  Now the vCenter Appliance can support the same numbers as the Windows vCenter Server.

  • Hosts per vCenter:  1,000
  • Powered on VM’s per vCenter:  10,000
  • Hosts per Cluster:  64
  • Virtual Machines per Cluster:  8,000
  • Linked mode is supported.

vSphere Web Client

  • Publicized to be faster.  I really need to get on board with this and move away from the C# client.

vSphere vMotion

There have been great enhancements to vMotion.

  • Live migration of VM’s across virtual switches as well as vCenter Servers.
  • Long Distance vMotion over links up to 100ms!  (Too Cool for School)
  • Migration from Windows vCenter Servers to vCenter Appliance is also available.

All VM information is retained during migration across vCenter Server systems.  Including-

  • UUID and MAC
  • Event Data
  • Alarm Data
  • Task History
  • Resource Settings (shares, reservations, limits)
  • DRS settings (affinity, anti affinity, startup priority)

vSphere FT and HA

Good, sounds like I may have to read another clustering deep dive book from Duncan and Frank soon.  (How do they find the time?  Do they sleep?  Are they human?  Can I sit next to them at the lunch table?)

Enhancements include-

  • FT can now support VM’s with up to 4vCPU and 64GB of memory.
  • FT can protect up to 4 VM’s per host.
  • FT VM’s require secondary storage.
  • VM’s can be backed up via DP
  • VM’s can be backed up via snapshot-based solutions.  I will need to check on Veeam interoperability.
  • FT 6.0 now supports all disk formats.

HA now detects storage failures, PDL and APD events and will restart VM’s on other hosts not impacted by the storage or network event.  There was some limited advanced settings support for this in the past.  I am glad they addressed it with 6.0.  Call it limited self healing maybe?

Virtual Volumes (VVOLS)

If you have ever sat in on any sessions with @punchingclouds you will feel his passion for VVOLS and he has the shirt to prove it.

Allowing the VM disk to reach though the stack and live on the storage device is a much more VM top down approach to a SDDC.  Having to hand storage up from the bottom to the Hyper Visor is becoming a backwards way of thinking and handling individual VM storage.

Removing the complexities of datastores, datastore clusters, RDM’s and other architectural design requirements is a welcome change.  Designing datastore volume sizes and datastores per datastore cluster to align with your VM sizing needs, backup and replication architecture, RPO, RTO, etc. may become less of an issue or not an issue at all in the future.  I am eager to start working with this technology.  Many storage vendors currently support VVOLS.

Due to the nature of VVOLS being an API and providing VM’s reach through access to the storage device you will still need to work around storage vendor limits such as number of LUNs supported.  A very small number of VM’s will require many NFS directories or LUNs.  A very large number (~>1000) of VM’s will require 1,000’s of NFS directories or LUNs in the same way a large number of datastores would as well.

vSphere Replication

  • Ability to compress data before being sent across the network.
  • Replication traffic can be managed by Network I/O control

vSphere Networking

vSphere 6.0 brings the ability to reserve/limit bandwidth on a vNIC or distributed port group via Network I/O control.  This is pretty cool to ensure bandwidth for VM’s and to limit the impact of events such as network noisy neighbor VM’s.  This is a great feature for folks using a few 10Gb uplinks for most or all of their VM traffic.

 Wrap Up

I have touched on the features and changes that I think are pretty awesome.  I am looking forward to working with it in depth and want to say thank you to all the folks at VMware who develop the magic we use everyday.  I see lots of lab time in my immediate future.

vCenter Converter fails to convert server with software mirror

Heads up my friends.  If you are working on converting a P2V, be sure to double check that there is not a software mirror in place.  If so break it.  Then convert.  Otherwise your P2V conversion will not work and you will get the generic error-

“A general system error occurred: unknown internal error”.

Be sure to verify your backups prior as you will be running on that single boot partition/disk.


Software mirror?  GEESH.




VMware vExperts for 2015 announced.


The VMware vExperts for 2015 list has been announced.

I am a little late to relay the news but its worth mentioning for nothing else than further program promotion.
The vExpert designation is awarded to those who contribute to the VMware community through blogs, technical assistance, education, community support, etc. The members of the vExpert community are passionate about the VMware product line and assisting others with adoption and support. They are VMware evangelists, champions or maybe even fanboys/gals!
Congrats to this years list!

Check the official announcement here.