GZ "There is no connection with the host security service. Please restart your system."

jkbd

Something is going on with GZ beginning last week. We have 200 hosts across 15 separate client sites running GravityZone, and between 11/18 and today three production servers (each at a different site) have come up with the same issue -

Cloud console reports "update disabled"; antimalware and power user modules are OFF; desktop console very slow to load, and says "There is no connection with the host security service. Please restart your system."

In each case, we rebooted and the problem cleared, and two days later has happened again.

Tech support suggested running a repair install. We've done that and are waiting for results.

Is anyone else having similar issues, and have you had any substantive response from support?

Alexandru_BD

Hello @jkbd,

This behaviour could be related to a known issue reported here and already acknowledged by the development teams:

https://community.bitdefender.com/en/discussion/94226/massive-issue-with-web-console-gz-on-prem

Regards

jkbd

Hello and thanks for following up. I don't think this is the issue, as the Cloud Console is functioning perfectly fine. The problem is the client on the host computer, I see repeated event 7011 in the system log, timeout in the EPSecurityService service. Server response slows to a crawl, GZ client stops communicating and says ""There is no connection with the host security service. Please restart your system."

As I said, 200 hosts, but only these three servers - three different sites, separate networks, separate BD licenses and accounts, different hardware, different network architecture, different OS and different apps - are experiencing this. One site 11/18, 11/22 and 11/23; one site 11/18 and 11/23; and one site 11/21 and 11/23.

Rebooting fixes the issue temporarily. All we've had from GZ tech support is "check your ports and run a repair install".

TnyCi

Hello,

We have the exact same problem since the 11/18 at midnight on many server.

It seems to concern only Windows 2012 R2 OS.

We are actually trying to repair install on those server.

Alexandru_BD

Hello @jkbd,

Can you provide me with the tech support case number please?

Thank you

Alex_Dr

Hello @TnyCi,

I highly recommend getting in contact with the Enterprise Support team and providing them with a STL to be analyzed by the support engineers. Also, please leave the case number here on the post so we can monitor the situation from our side as well.

Best regards,

Alex D.

jkbd

https://community.bitdefender.com/en/discussion/comment/318437#Comment_318437

Thank you @TnyCi for confirming we are not the only ones with this issue. One of our affected sites is 2012r2, two are 2008r2.

jkbd

https://community.bitdefender.com/en/discussion/comment/318438#Comment_318438

Hello Alexandru, we have these case numbers, one for each affected site -

00721717

00721718

00722076

However, we were informed by a tech that

"Cases 00721718 and 00722076 were opened for the same reported situation and were closed as duplicates. 

We kindly advise you not to open multiple cases for the same issue."

As these are three separate accounts and three separate sites, we've requested the closed cases to be re-opened.

ehinkle

Was this resolved I have similar issues on some windows 10 host. Tried doing a repair install and getting this message now. Prior to doing the repair install it was stuck on running a full scan.

jkbd

issue is not resolved -

So far support has recommended: repair install, update ServicesPipeTimeout (service timeout value) in registry, gather logs for review.

site 1 CASE 00721717 - Froze 11/18, rebooted. Froze 11/22, rebooted and ran repair install. Froze again today; unable to gather logs, "error 2147483646". Will be updating ServicesPipeTimeout and rebooting tonight.

site 2 CASE 00721718 - Froze 11/18, rebooted. Froze 11/22, rebooted and ran repair install. Froze 11/23, rebooted.

site 3 CASE 00722076 - Froze 11/21, rebooted. Froze 11/23, rebooted and ran repair install. Froze 11/26, updated ServicesPipeTimeout and rebooted.

site 2 and site 3 are ok as of this morning...

Does anyone have any similar experience, or any advice?

jkbd

brief update -

site A, Windows 2012 server - 12/3 we rebooted, uninstalled BD, created a fresh package and installed. Server has been running normally.

site B, Windows 2008 - 12/4 we rebooted, uninstalled BD, created a fresh package and installed. Server froze again on 12/5. We uninstalled BD and installed MSE. Server has been running normally without BD.

site C, Windows 2008 - 12/4 we rebooted, uninstalled BD, created a fresh package and installed. Server froze again on 12/6. We uninstalled BD and installed MSE. Server has been running normally without BD.

Flexx

@Alex_Dr can you please look into this.

Alternatively, you can also contact the bitdefender business support by visiting https://www.bitdefender.com/support/contact-us.html?last_page=BusinessCategory  

Regards

jkbd

Hello Flexx, thanks for following up. You can see above that there are currently support tickets for each of the affected sites. The tickets for sites B and C have been "escalated", whatever that entails. Hopefully a review of the requested logs will shed some light on this issue.

One poster earlier noted they were experiencing the same issue. Do you know if there have been other reports of issues with BD GZ client freezing servers following 11/18 update?

Flexx

https://community.bitdefender.com/en/discussion/comment/319038#Comment_319038

@Alex_Dr @Alexandru_BD @Mike_BD can you check on this on urgent basis.

Regards

Alex_Dr

Hello @jkbd,

Our internal policy regarding technical support cases is that the situation is being handled on one case per situation and the other cases are linked to the earliest open case. This way confusion is avoided and multiple replies per multiple cases lead to inevitable confusion regarding what is effectively happening. Also, being the same reported situation, every reply we send and every reply you give us allow us to better monitor the information and sift through it more thoroughly.

Our engineers are awaiting the logs requested in the e-mail exchange, as more information is required for troubleshooting. Kindly reply to our support teams and provide the necessary details, so they can move forward with the investigation.

Thank you for your understanding and cooperation, we are looking forward towards a swift resolution.

@ehinkle

At the moment our engineers our looking into the cases opened by @jkbd. Have you gotten in contact with the Enterprise Support Team? If so can you provide me with a case number to monitor?

Best regards,

Alex D.

jkbd

Hi Alex - as a consultant I have to account for my actions to each client. This issue has appeared at three different sites, three different clients. The sites are not connected or related in any way. They are three separate issues and will be treated as such. That's all there is to it.

I don't know if you have access to the tickets cited above, but please be assured that I have responded to each. I have provided the logs requested by the techs. The techs appear to be capable of reviewing the three different cases, with no confusion.

In fact, I had a response on 12/8 regarding case 00722076:

"We shall escalate this case to our senior colleagues for further analysis.

You shall receive an update with discoveries or further instructions as soon as there is any progress on the investigation."

and another just this morning for the same case:

"We are still investigating the situation, once we finish our analysis we’ll reply via email on this thread with our findings."

I had a response on 12/12 regarding case 00721718. I had a response on 12/6 regarding case 00721717.

Everyone is up to date on the status, everyone has received the information they requested, and apparently techs are working on these three separate tickets. Would it be any different if these three separate cases were reported by three separate people?

My question on 12/11 was

"One poster earlier noted they were experiencing the same issue. Do you know if there have been other reports of issues with BD GZ client freezing servers following 11/18 update?"

If you don't have any further information, or are not permitted to discuss it, that's fine. I only note that one other poster, who is not known to me, reported the same issue on the same date.

thanks

Jeffrey

Alex_Dr

Hello @jkbd,

As for the question you raised, there are no other client's that have experienced this issue besides @ehinkle.

I do appreciate the clear-up for the different clients and I apologize if i caused further confusion. Our internal policies state that the cases raised by the same client with the same issued to be monitored under the first case that was raised. The 3 cases were raised with the same account and the Tech Support Engineers closed as "duplicate" when there should have been a clarification that there are 3 different clients and be handled accordingly and again, i do apologize for the way it was handled.

As per the latest update from 00721717 it seems that one of the sites functions as intended. Can you please let me know if Bitdefender is working properly on the other 2 as well?

Happy holidays!

Alex D.

jkbd

Hello all, more bad news to report - all three of the previous tickets were closed by support as unresolved, with absolutely no useful feedback. GZ has been removed from all three affected servers and another antivirus solution implemented. No issues so far with those servers.

Today another Windows 2008r2 server has been affected, same symptoms as the others: GZ stops communicating with the update server, performance slows to a crawl. Rebooting affected server will restore function, update proceeds normally. In the previous cases, the issue would reappear after 2 - 7 days.

I've submitted another ticket, hopefully it will get a proper response.

adrian-datatech

It seems that for some reason the communication between the GravityZone server and the server Bitdefender is installed on fails. What is puzzling is that after a reboot, this connection is established, but after some point, the communication will stop and grind everything down to a halt. A remedy to this that has worked on a couple of different servers is to configure the server to only communicate using TLS 1.2.

For me, this meant installing IIS Crypto, taking a backup of the current registry, and disabling any protocols that are below TLS 1.2 in both the Server and Client protocols. It may not be necessary to do it for both the client and server protocols, but that is what I was able to get it working with.

I haven't had any issues with Bitdefender since making those changes, but certain client applications may need to be modified to ensure they are using TLS 1.2.

Two main things were puzzling for me:

1) The communication obviously established after a reboot, so why does it stop?

2) Task manager didn't show an excessive consumption of resources, and event viewer didn't show any anomalies. Bitdefender logs also didn't show anything unusual when I submitted them.

In any case, disabling protocols below TLS 1.2 on the server and rebooting has fixed the issue for me without any signs of it cropping back up.

Flexx

https://community.bitdefender.com/en/discussion/comment/322695#Comment_322695

Kindly contact the bitdefender business support by visiting https://www.bitdefender.com/support/contact-us.html?last_page=BusinessCategory  

Additionally, @Alex_Dr _Dr or @Andra_B can have a look into this for you.

Regards