I was doing an upgrade from 4.0 to 4.1 this week on a two node cluster. This cluster is owned by an SMB and its a fully contained VMware setup, basically it has two DL380 G6 servers each with 8 – 146GB 10k SAS drives, dual Nehalem processors, and 24GB of ram. We have HP’s P4000 VSA software installed on each node to form a redundant two node SAN, so each server has all 8 drives in a RAID5 and a single VMware VMFS volume on them. Inside of that volume we have a single virtual machine (the VSA) and it consumes about 90% of the space in that datastore. Inside of the VSA is where all of the production VM’s live, but the problem is that the local datastores are in an alarm state because they are above the threshold set at the vcenter level. I suppose I could just change that threshold to like 98% or something and the alarms would go away, but that wouldn’t let us much time to react if the VSA volume ever got full. So the better solution would be to somehow ignore alarms on local datastores but still keep the alarms for shared datastores. Below is what the problem looks like… Local datastores are in an alarm state… but the “real” data which is in “VM Storage Repository” is not full yet.
After doing a little research I was able to come by one other blog post that used this same method on ESX to fix the errors on the service console volume, but I could not find anything related to local and VSA shared volumes. The process is the same for both though, so I figured I would share.
The first step is to log into vcenter (or esxi, whichever your using) and goto the Datastore Inventory tab. Next create two folders, one for local datastores and another for shared. Then drag your local datastores to the local folder and your shared datastores to the shared folder.
Note that the pictures shows how it will look after we delete some alarms and recreate them.
After putting your datastores in the proper folders click on the vcenter, or esxi object (whichever is the top level) and go to the alarms tab (you will need to click on the Definitions button as well. Find the ‘Datastore usage on disk’ alarm and go into it and take some screen shots of how it is setup, we will use these later to recreate the alrm, then delete it. (Or at least disable it) Then go down to the shared datastore folder that you created, and then into the alarms tab again (then click Definitions). In here we will want to recreate a ‘Datastore Usage on Disk’ alarm so that we still get alarms for our shared storage. Right click and Add new. and then refer to the screenshots you took in order to create it properly. Just for reference here is what it looks like inside of the alarm definition:
Now you should have something that looks like the last screen shot… you will have a ‘Datastore Usage on Disk’ alarm that has been created in “This object” and your local datastores are no longer monitored. If you wanted to you could create a disk usage alarm for the local folder and set its thresholds much higher just to be safe.
Pingback: Tweets that mention Fixing ‘Datastore Usage on Disk’ Alarms when using Virtual SAN Software | Justin's IT Blog -- Topsy.com