VMware vCenter 7.0.0c high CPU usage

Update:

After an upgrade to vCenter Server 7.0.0c, you see high CPU usage.

Patch for VMware vCenter Server Appliance 7.0.0d

After you upgrade your vCenter Server system to vCenter Server 7.0.0c, CPU usage continuously stays high. On a single-core, CPU usage might spike up to 100% for hours. The Workload Control Plane service causes the issue, even if you do not have Workload Management enabled in your environment.

This issue is resolved in this release.
https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-vcenter-server-700d-release-notes.html

The issue

After upgrading my vCenter server appliance to the latest patch, 7.0.0.10600 Build 16620007, I found out that the CPU usage was abnormally high than before. Specifically on “observability”. Constant 4 to 5Ghz usage with just 4 hosts connected.  Top shows java being the pain and vimtop shows observability hogging the CPU.

After some troubleshooting and consulting in the VMware Slack channel for vCenter we saw that the WCP service is consuming a lot of CPU. Ended up stopping that service and everything was back to normal. Thanks, Martin Gustafsson for pointing it out 🙂

Be aware!

When disabling the WCP Service you will be unable to put a host in maintenance mode. For a lab, this is not a problem but disabling it in a production environment is not recommended at all.

You will be warned with “Host cannot enter maintenance mode because it failed to enter Namespaces maintenance mode on the node.”

So now what?

To disable the Workload Control Plane service login to vCenter on the management page with https://vCenterFQDN:5480 and go to services. Then on the bottom, you will find the Workload Control Plane where you can disable it. If you are running Kubernetes then you might need to contact GSS for another fix because the Workload Control Plane is kinda an important part of that 🙂

After disabling the service you can see that my CPU usage went down to normal.

The bug has been reported and waiting on the fix. Bugzilla 2615223 for who has access to it as a VMware Employee or PSO consultant.

Disabling it via 5480 is just a one-timer. After reboot, it’s back on hogging CPU cycles. Want to disable it permanently? Follow this guide:

vCenter Appliance 7.0 – Disable Services

or

How to change the startup setting of services in vCenter Server Appliance 6.x using command line

Provided by Sidney Laan

So 3 ways to go with this issue.

  1. Restore to a previous backup.
  2. Disable the Workload Control Planes service and lose maintenance mode.
  3. Wait for the fix and ignore the CPU usage.

18 thoughts on “VMware vCenter 7.0.0c high CPU usage

  1. Mike M

    Thank you for this posting workaround. I’m on vCenter 7.0c and ESXi 7.0 GA and I just stopped the WCP service in the GUI (did not do anything more permanent). Now I’m seeing an error blocking maintenance mode – “Host cannot enter maintenance mode because it failed to enter Namespaces maintenance mode on the node.”

    Just wanted to bring awareness in case anyone else encounters this problem. Hoping VMware is able to patch this soon. In the meantime, I’ll just stop and start the service as needed to get things done.

    Reply
    1. LaurensvanDuijn Post author

      Tested again and i can confirm it too.. changed blog with remarks thanks. Weird that it worked for me before:(

      Reply
  2. Ronald de Jong

    Not able to disable services from the GUI anymore (at least not in my three VCSA’s), easiest way for me is using: “vmon-cli -U wcp -S MANUAL” in the bash shell.. That way I am also able to start it (and stop it again) to put hosts in or out of maintenance mode.

    Reply
      1. Ronald de Jong

        Yeah, I can stop and start them there, but I can’t disable them there or set them to manual (which I have done with this service, to be able to put hosts in maintenance mode, when necessary).

        Reply
  3. Pingback: vCenter Server 7.0.0c で CPU 使用率が高騰する | kurokobo.com

  4. Franck

    Hi there,

    I can confirm it works for sure, but from my end, I reverted my home lab vCenter to 7.0b as 100% CPU just consumes too much ressources (and power at the end). Having to restart the service to put hosts in maintenance was too painful! 🙂

    On the other end, that it takes more than 3 weeks to VMware to release a fix… not so good… 😉

    Reply
  5. LaurensvanDuijn Post author

    The issue has been solved! Patch for VMware vCenter Server Appliance 7.0.0d is available.

    Reply
    1. Stuart Ryan

      As a user of mostly a HomeLab for learning purposes, VERY happy to have a fix released (and also hugely grateful for your workaround tips that lessened the impact until the fix was released)!

      Thankfully all resolved with the patch (thank you for the heads up that there was a fix, had been watching this post closely!)

      Reply
  6. Chris Cheung

    HI there,

    Any fix after i have stopped WCP service in VCSA 7.0.0.c. The host can’t go to maintenance mode – “Host cannot enter maintenance mode because it failed to enter Namespaces maintenance mode on the node”
    How can i fix that issue?

    Thanks.
    Chris

    Reply
      1. Chris Cheung

        Thanks for reply.
        I have already update to 7.0.0.d to fix high CPU usage problem. But the host still can’t go to maintenance mode and show – “Host cannot enter maintenance mode because it failed to enter Namespaces maintenance mode on the node”.

        Any solution to fix that problem?

        Many thanks.

        Reply
        1. Simon

          We’re having the same issue in 7.0c and after upgrading to 7.0d issue still persists across both vCenters in ELM. We upgraded from 6.5 a few weeks ago (lots of issues…)

          Logged call with VMware Support, lets see what they say.

          Reply
        2. Franck

          Don’t know exactly if I had the same issue, but I’ve seen many vMotion problems that I didn’t had in the past since I’ve upgraded to vSphere 7.X. Sometimes, to put a host in maintenance mode, I’ve to vmotion manually some VMs. With 6.7, I had zero issues (and nothing changed in configuration)

          Reply
  7. Pingback: Technology Short Take 131 - s0x

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.