Skip to main content

VMWare ESXi - Hardware monitoring - vSphere host disconnected


Recently a client of mine experienced a sudden disconnect of one of their hosts from vSphere Client. A quick check quickly confirmed that the VMs were still running happily along, however the host was "tagged" as disconnected. Forcing a reconnect did not work, so it was time to get my hands dirty and start digging. 

I connected to the host in question with SSH and changed directory to the location of the syslog files (you do store your logs on a SAN disk right and not on a local drive?). After some quick searches I noticed this entry in the hostd.log file:


2012-12-24T19:01:01.309Z [FFFC2B90 info 'VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {
-->    dynamicType = <unset>,
-->    key = 1348822881,
-->    chainId = 761820257,
-->    createdTime = "1970-01-01T00:00:00Z",
-->    userName = "",
-->    datacenter = (vim.event.DatacenterEventArgument) null,
-->    computeResource = (vim.event.ComputeResourceEventArgument) null,
-->    host = (vim.event.HostEventArgument) {
-->       dynamicType = <unset>,
-->       name = "nameofesxhost.local",
-->       host = 'vim.HostSystem:ha-host',
-->    },
-->    vm = (vim.event.VmEventArgument) null,
-->    ds = (vim.event.DatastoreEventArgument) null,
-->    net = (vim.event.NetworkEventArgument) null,
-->    dvs = (vim.event.DvsEventArgument) null,
-->    fullFormattedMessage = <unset>,
-->    changeTag = <unset>,
-->    eventTypeId = "esx.problem.visorfs.inodetable.full",
-->    severity = <unset>,
-->    message = <unset>,
-->    arguments = (vmodl.KeyAnyValue) [
-->       (vmodl.KeyAnyValue) {
-->          dynamicType = <unset>,
-->          key = "1",
-->          value = "tmp:/auto-backup.17245931/etc/ssh/ssh_host_rsa_key",
-->       },
-->       (vmodl.KeyAnyValue) {
-->          dynamicType = <unset>,
-->          key = "2",
-->          value = "tar",
-->       }
-->    ],
-->    objectId = "ha-eventmgr",
-->    objectType = "vim.HostSystem",
-->    objectName = <unset>,
-->    fault = (vmodl.MethodFault) null,
--> }
2012-12-24T19:01:01.326Z [FFFC2B90 info 'ha-eventmgr'] Event 46 : The root filesystem's file table is full.  As a result, the file tmp:/auto-backup.17245931/etc/ssh/ssh_host_rsa_key could not be created by the application 'tar'.

This made me suspect that a volume was running out of space. Some quick commands later (vdf -h and stat -f /) showed that sufficient space and inodes was available. Google to the rescue as I search for "esx.problem.visorfs.inodetable.full" the following article from VMWare popped up:


Apparently the hardware monitoring service (sfcbd) is flooding the /var/run/sfcb directory with files (>5000) and creating the problem. Now you could just do a /etc/init.d/sfcbd-watchdog stop, delete the files in /var/run/sfcb and then run /etc/init.d/sfcbd-watchdog start, however I logged in on the console of the host in question and did a full restart management agent. That did the trick.

Comments

Popular posts from this blog

Developing PowerShell modules for REST APIs – Part1

Over the years I have developed different PowerShell modules for different web APIs. I thought it would be a good idea to write a 2 series post about how you could go about to do this. This will be a 2 part blog series where we will run through the entire process of building a module for a REST API. I will try my best to keep this as simple as possible and leave more advanced stuff for a follow up post if the interest is there.What you needDepending on your experience with source control and PowerShell in general, you might want to use GIT or some other software repro for the code. In addition we are going to create a test REST API using the splendid UniversalDashboard PowerShell module created by Adam Driscoll. It is available on the PowershellGallery. Other prerequisites are built-in to Powershell. I will assume that you will be following along using at least PowerShell version 5 or greater.
What is HTTP metods for REST API.The primary or most common HTTP verbs used are POST, GET, PU…

Serialize data with PowerShell

Currently I am working on a big new module. In this module, I need to persist data to disk and reprocess them at some point even if the module/PowerShell session was closed. I needed to serialize objects and save them to disk. It needed to be very efficient to be able to support a high volume of objects. Hence I decided to turn this serializer into a module called HashData.



Other Serializing methods

In PowerShell we have several possibilities to serialize objects. There are two cmdlets you can use which are built in:
Export-CliXmlConvertTo-JSON
Both are excellent options if you do not care about the size of the file. In my case I needed something lean and mean in terms of the size on disk for the serialized object. Lets do some tests to compare the different types:


(Hashdata.Object.ps1)

You might be curious why I do not use the Export-CliXML cmdlet and just use the [System.Management.Automation.PSSerializer]::Serialize static method. The static method will generate the same xml, however we …

Developing PowerShell modules for REST APIs – Part2

This is part 2 of the REST API blogpost. In part1 we successfully setup two REST API endpoints using the UniversalDashboard PowerShell module. In this part we are going to create a simple module that support some CRUD operation against our API. As we are trying to keep things as simple as possible, we will not use any fancy framework (like Plaster) to build our module. We are also going to skip a very important step you should familiarize yourself with, Pester tests. Lets get to it.


The moduleWe will build a module called FilesAPI. The module folder will look like this:



In the functions folder I have already added the 2 helper functions from part 1, Get-AuthorizationHeader and ConvertTo-Base64. The other folders are just placeholders for important stuff like classes, private functions that you do not want to make available for the module consumer and tests for Pester tests. For such a small module that we are going to create, one could argue that it is much easier to just add the functi…