Hi.
Just seeking
some advice out here to see if I can start to get some suggestions.
I do not have much information just yet, so apologies for some of the detail lacking here.
If the following is a problem with Bit Defender, then hoping there might be something that someone may be able to suggest.
Over the past week or 2, I have had some a number of customers calling in where a custom program they use (a Windows Desktop application) is crashing displaying an Out Of Memory Error.
When we access the system with the problem happening, we find that the memory usage of the PC is quite high, on the occasions I have seen it is 80%-90% in use, and the most memory is being used by a Bit Defender process. Have seen usage of 500MB-800MB.
Now I understand this information is circumstantial, but every customer that has called, with the exact same memory usage errors, are all running Bit Defender and the issue started around the same time, which was around 2 weeks ago.
The Bit Defender appears to be the centrally managed version which is looked after by multiple IT service agents. So the customers I have contact with use a different IT support service agent managing the Bit Defender software for them.
So essentially for now, just seeking to know if there has been any general issues going on with any changes and/or updates to the Bit Defender software that may relate to experiencing this Out Of Memory issue for a Windows Desktop application.
Any suggestions are appreciated.
I'm also experiencing this and a Google search came up with this post.
In my Event Log (System), filtered by I get a lot of Resource-Exhaustion-Detector, I was getting a lot of errors about running low on Virtual Memory and it highlights the epsecurityservice.exe (every time) as the main memory consumer every time, e.g.
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: epsecurityservice.exe (17096) consumed 1117134848 bytes, w3wp.exe (18724) consumed 990916608 bytes, and w3wp.exe (15640) consumed 845512704 bytes.
In the last week or two, I have noticed .NET windows services running out of memory (Which coincides with the update to 7.4.2.130, with the Last product update: 16 December 2021 08:23:20).
However, in the logs, these errors have been occurring since at least the 22/11/2021.
There was another reddit post with similar issues.
Here's the current situation in Resource Monitor:
Thanks for your information GBSM.
The Windows Desktop Applications, that I have been experiencing this issue that I posted, are C# Applications using the Dot Framework.
The date you listed seems to coincide with the timing of the reported problems.
Again this is circumstantial, but interesting this seems to be that Dot Net Framework applications, and instance of Bit Defender and timing of issue coinciding with this update appear to be a common factor here.
I am hoping to get some further feedback from our customers IT service providers soon.
I will post any updates on this problem.
We have exactly the same problems at two customers (Accounting solution from Datev in Germany), WCF Services crash with OutOfMemory Exceptions, also since about 2 weeks. It took us a week to determine the cause, more than 1,500 users were affected.
Thanks Datev_eG for the information.
I am now more convinced that Bit Defender is the main culprit here with regards to this error.
Were you able to fix the problem with Bit Defender?
Yes and no, we disabled the scan engine (but leaved it on the VMs), after that the errors disappeared.
We will open support cases next week. Bad thing.
Additional to the post of Datev_eG (we are one of the customers mentioned...):
We disabled the modules On-Access and ATC (no other modules installed). It seemed that a system reboot was necessary to take full effect.
None of the Bitdefender Logs contained useful information to see what happened. Unfortunately there is still no possibility in BEST to do some kind of Realtime-Monitoring to see what is going on inside of Bitdefender. I already asked Bitdefender support about such a tool in a previous case, but they just wondered why someone could need this. Here's the answer...
Following closely.
we have many 100's of agents deployed, and this all started around the recent software upgrade rollout. Only client it is currently impacting uses Fujitsu Scanners to scan to a DMS system, which was coincidentally updated around the same time. We ripped out Bitdefender and the problems went away, and just redeployed, even with all the scan modules disabled, came back to haunt the client. Have an open support ticket, but all that is recommended is to disable all modules and enable one by one, which is crippling to the end client, and quite intermittent.
For now we will rip out Bitdefender at that location, and push out an alternative product to see if the problem is resolved, and let Bitdfender work out their issues before we move back over.
This is only impacting approx a dozen endpoints currently, out of 100s so not sure how time consuming the troubleshooting process will be if we just stick with the product and the proces.
Please keep this thread alive, along with updates, and if anyone wants the provided troubleshooting steps provided by my support ticket, happy to pass along, or post upon request.
Hi guys
We are a software house in Greece and we are developing .Net (C#) applications. From about mid December and on, we are experiencing many uses with clients having BitDefender installed. Out of Memory errors occur on a daily basis. As our program is a email management solution, it's hard to convince the IT depts of the companies to disable the Access Scan modules. Hope Bit Defender solves this.
I did notice on mine that Windows Defender was also still running on that server. Is that something that is happening in your situations too?
No
Has a resolution been provided?
Has anyone implemented any changes to bypass the out of memory errors outside of disabling the program?
No, interesting to see that the vendor (Bitdefender, hello.... ) does not monitor their own forums.
My personal recommendation: Switch to DEFENDER.
The clients with these issues we were forced to move them over to a different AV product for now, not the right solution, but the problems ceased concurrent with the move from Bitdefender to a competing product/vendor
It also appears they also do not monitor their support page either.
I have sent 5 requests for information or assistance, and heard nothing back from Bit Defender at all.
We run many terminals with Bitdefender installed on them. Ever since 1st January (when the Bitdefender client was updated) we continuously get errors with our .NET SQL Framework based Windows applications. I had spent a great deal of time checking what the issue was. I even upgraded the RAM on many terminals. Issue still exists.
Bitdefender needs to be aware of this issue and fix it! Disabling vital AV modules is not the solution to this issue!
20220111 02:01:39.3990 SendData error - 4 Exception = System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at proto_15(Object , ProtoWriter )
at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Write(Object value, ProtoWriter dest)
at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
at ProtoBuf.Meta.TypeModel.SerializeCore(ProtoWriter writer, Object value)
at ProtoBuf.Meta.TypeModel.Serialize(Stream dest, Object value, SerializationContext context)
at ProtoBuf.Serializer.Serialize[T](Stream destination, T instance)
at IPSCommunication.Packet.Base.Encapsulate()
at IPSCommunication.TCP.Connection.SendData(Stream s, Base p)
I have disabled On-access Scanning and Advanced Threat Control. Will report back to advise if applications are still crashing.
I have confirmed that Bitdefender is the culprit because when I uninstalled Bitdefender on another terminal, the logs indicate the error is gone.
I have now submitted a support ticket. I will update you all in the interest of sorting this horrible issue out for all.
Hello everyone and thank you for your feedback regarding this.
My advice would be to get in touch with the Enterprise Support Team, for them to collect the necessary logs for troubleshooting. If I'm not mistaken, this is related to the following:
You can use one of the contact channels below, to reach our engineers:
Stay safe.
I've uninstalled my instance from the server a couple of days ago to see if it has an effect. No crashes yet, but it usually took a couple of days so I'll review by the end of the week.
After disabling these modules, the out of memory error causing apps to crash has disappeared.
Bitdefender, you have some questions to answer (urgently)
My support ticket remains unanswered.
Hello @A_Guy ,
We're not hiding anything 😊. Thanks for reposting here, I will leave it active and merge the two. That post was in a Category which we archived, hence the imposibility of posting anymore. But we will keep this one active.
Support for Enterprise is (yet) limited in the Community, as @Alexandru_BD posted there, but we'll try anyway to help as much as we can.
I see that you've already contacted Support, can you leave here the ticket number so we can follow-up on it?
cheers,
Mike
We have the same issue with our application. We have tried to disable BitDefender on several customers machines and issue go away.
Info which may help to find the reason:
1) As a target framework our application uses Net framework 4.8. WPF is used as UI framework.
2) According to our logs, there is no specific place where this exception occurs (it can be triggered in any place of code).
3) At the time of fail system has more than 2.5 Gb of free RAM, our application uses about 80 Mb of RAM, epsecurityservice uses about 220 Mb of RAM. So system has enough memory resources.
We've had the same issue and raised a ticket, started mid December and affected the 3CX voip client that I think uses .net.
Following BD advice we stripped the policy back to on-access only and disabled all other modules including ATC and that has stopped the random OOM crashes.
The current advice from BD is to setup a process type exclusion and see if that resolves, but I'm going to give it a few more days to make sure the crashes have stopped before I apply the exclusion and start re-enabling modules one at a time.
Hi all,
We too have been impacted by this. WPF application .NET 4.6.2. Infrequent and intermittent crashes often reported as out of memory exceptions but not always. It took us a long time to establish the cause which started around the 9th December.
We have been able to prevent the problem by using scan exceptions for the path where the executable runs from.
You can use wildcards as well if the .exe doesn't run from the same location on each PC. like this:
c:\**\test-app.exe
or like this
%userprofile%\appdata\etc\etc\*:
I think BD might still be looking into the root cause.
Dear " Alexandru_BD (admin)" -
first of all, this does not seem to affect only Enterprise customers - I am a Single User licensee and am facing the very same problems!
Secondly, it appears as quite strange to me that you suggest that all your customers would open a separate ticket themselves (if it wouldn't be for the mere reason to create a track record) instead of tossing this over to them yourself - with some sort of a raised priority.
Best.
Adding to what "VP_NZ" has written earlier today, I can confirm that the recent point release update from 26.0.7.41 to 26.0.10.45 (Bitdefender Internet Security, single user license) did NOT solve the OOM issue: my .NET based app in question continues to reproducibly die six to eight hours after invocation - on an almost entirely idling 16 GB machine.
And as with others, also the vendor of "my" app has AFAIK already been in touch with your - but unfortunately not with any satisfying success yet.
Just came here to add my experience. We've also experienced the same OOM issues on our .NET applications, causing us to have to remove our Enterprise BitDefender installation from our servers. This has been a nightmare of troubleshooting and down time for my company and clients.
Just like the others, it's been taking 7-8 hours to burn through 16GB of RAM. We're desperately awaiting a resolution on this.
BitDefender, you've got a real issue on your hands here. What are you going to do about this?
A little ray of hope here.
I installed optional .net cumulative update for jan 22 KB5009467 ( W10) on a pc that was crashing an app ( Boxcryptor ) every 24 hrs. So far it has run for nearly 48 hr without the app crashing. It is very early days. The pc spend a very varied amount of time in sleep mode so it may be luck that it has lasted this long.
The info on the update talked about apps crashing but seem to relate to using a special path ?
Though I built my first pc in 1980 I am not a programmer and did not understand the jargon.
It also occurs to me that it may be something to do with the optimisation process that runs after the .net update and the next update to Bitdefender may break the system again.
I agree with the above comments about the response from Bitdefender. I know these issues can be difficult to fix but we have all been left with the impression the have done nothing and don't care
[Single-user license Bitdefender Internet Security 26.0.10.45 on a fully-patched Win 8.1]
To whom it may concern: Disabling the "Advanced Thread Defense" module via the control panel did the trick for me, too.
I now have a working Boxcryptor application again, smoothly and continuously running in the back without any crashes every six to eight hours as it's been the case since mid November 2021.
Our problem application stopped misbehaving after adding the process to exclusions. So, we are running the same settings as before without having the need to disable modules (Basic few days test).
Same for us, I've got the process exception configured and I've had ATC and AAE turned back on for a few days with no issues.
Good morning - at least on this side of the globe.
Thanks everyone posting here for the feedback regarding the out-of-memory issue. We investigated and found that some .NET apps will create and delete strings when running .NET. This process will create an out-of-memory exception without following Microsoft guidelines.
We have addressed the issue and created a fix that takes these guidelines into account and is available in all releases in the January 28, 2022 update. We apologize for the inconvenience.
Mike, do you know when the Jan. 28 update will roll out? We are a GravityZone MSP. We have not seen this a lot, but are seeing this with more frequency, even a couple today. The last update on my PC shows "Product version: 7.4.3.146. Engines version: 7.91195 (10851086)."
We've linked it to other symptoms such as non-Bitdefender services and video drivers crashing and/or unable to start, or in my case with my home PC this morning, the PC was on (caps lock light toggles, drive light flashes) but I was unable to wake up the monitors, or "soft" shut down via the power button, and had to power off.
@debakelit glad to hear this.
@Steve_ITS update was released. You're describing some strange behavior and, mixed with non-BD services anomalies.
@Mike_BD
Hmm. I just had to restart my home PC because programs started randomly crashing. Task Manager only showed 77% RAM usage (out of 8 GB) but I found event 2004/Resource-Exhaustion-Detector:
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: epsecurityservice.exe (5104) consumed 714006528 bytes, firefox.exe (1240) consumed 316162048 bytes, and qw.exe (15188) consumed 289820672 bytes.
After reboot, pagefile.sys is 20 GB, set to auto size, which seems a bit large.
Across several hundred PCs, the size of epsecurityservice.exe ranges from about 200 MB to 800+ MB, and the latter stays at that level even after a restart. Is that a normal size? I didn't count but it looks like a fairly even spread of PCs across that range, in terms of number of PCs per each memory size (200, 250, 300, etc.).
I can't explain why this happens then, only that epsecurityservice.exe is normally at the top of the list when the event triggers.
There was a change...if I look back, we had this event 2004 on various random client PCs from October to the first week in January, then a gap, then since Jan. 29 it's only been a select few of our own PCs, but not all. Possibly, we do leave our PCs logged in while we log out most managed PCs nightly (we figure we can install our own browser updates when prompted :) ).
The other services and programs crashing I believe is simply due to Windows being out of memory.
Just following up on this thread to say we did eventually track our issue down to the new (January) version of EPOS Connect headset software. It has a secondary DSEA SDK software it installs that has a massive memory leak. They are releasing an updated version in April-ish 2022, but have a manual update procedure in the meantime. Nothing actually showed this program using as much as 40 GB virtual memory, since Task Manager and other tools showed it as 50-75 MB, but ending that process freed up the memory.
Dear Steve,
We faced problems with crashing dwm.exe (Windows Desktop Manager) and other programs on PC's that have DSEA SDK installed. I opened a support ticket at EPOS / Sennheiser, and they don't want to know anything about they're software having such a problem. Could you please let me know how you got informed about this SDK that they have such a problem with Memory Leak? did Sennheiser / EPOS confirm this to you?
Same issue here right after upgrading the EPOS software - DSEA seems to be the corrupt. Can't find anywhere about an update though......
EPOS does not have a public update yet. They said it should release in late April or May as I recall. Our EPOS support rep immediately knew it was a known issue back on March 3 but they have a workaround with a beta update to the DSEA. Have EPOS look up our ticket 207180 if they can't figure it out. The "quick fix" was to log off and on again, every 48 hours.
On the subject of memory leaks I happened across this yesterday while working on a PC:
"Install Intel® Graphics – Windows* DCH Drivers driver 30.0.101.1191 or a newer version to fix the issue."
I have some more info - support was able to assist - The update is scheduled for May
The fix for DSEA is below:
Note: This fix won't be needed after the EPOS update is installed
Here are the notes from EPOS
"For now please follow the below steps to fix the issue.
- Open the task manager and kill DSEASDK and EPOS Connect.
- Download and unzip the new SDK and install.
- Launch EPOS connect.
- Observe if the Memory leakage issue is observed again."