Wednesday, 12 August 2009

CA eTrust goes nuts with StdWin32 and other false positives

CA eTrust ITM has gone completely nuts today, with a load of seemingly random false positives mostly for StdWin32 in a large number of binaries, including some components of eTrust itself.

The core problem seems to be a signature update from 31.6.6672 to 33.3.7051, there seems to be little consistency in what is being detected as a false positive although there are multiple occurrences of Nokia software, VNC and event DLLs and EXEs belonging to eTrust's core components.

Probably the best thing to do is block the update or change the Realtime scanning behaviour to "disabled" or "report only".

Update: problem seems to have started at about 0525 GMT when the new signature pattern applied. There no consistent pattern to the infected files, it looks like it happens at random. Several other people seem to be having the same issue!

Update 2: Signature pattern 34.0.6674 appears to fix this problem. You can then enjoy repairing your faulty machines.. thanks CA!

Update 3: Amusingly, CA eTrust seems to have deleted its own key components in many cases. I don't know if this is the first recorded case of an anti-virus application mistaking itself as malware!

Update 4: CA have released a statment as follows:

Last night, CA released a new updated antimalware engine. This new release has resulted in false positive detections of a number of files. CA Threat Manager customers are the only customers being affected by this issue. This is not a result of signature updates and does not impact CA consumer Internet security products.

To resolve the issue, CA has rolled back the new engine and re-released its previous antimalware engine. CA customer support representatives are on call to answer customer questions and to provide remediation support. A remediation tool to rename the quarantined files is now available through CA support and will soon be accessible online.

CA is aggressively working to resolve the issue, assist any customers who have been affected, as well as identify the root cause of the incident. We apologize for this inconvenience and look forward to the roll out of our new antimalware engine, which will ultimately offer our customers many benefits including enhanced malware protection and improved performance.

Update 5: Got a mention on El Reg.. funny thing is that I went in to work today wearing my El Reg T-Shirt. Coincidence? Consiparacy? Cockup?

PS: Please remember to read the comments if you are still having problems!

71 comments:

PeeGee said...

Thank you Dynamoo. I thought I was going mad this morning. I've logged a call with CA so we shall see.

Funny enough, when I logged the call the rep asked me if I thought it was a false positive. Maybe she reads your blog too.

Adrian said...

I have the same problem today here.

I think CA is doing very bad things with the eTrust ITM Antivirus. I am looking for a different Antivirus, Have you any idea ?

Scott said...

Same problem here. Isn't this twice in one month? I don't recall having these issues with eTrust 7.x...What's going on with their quality control???

Jason said...

Yep I had 235 workstations updated and started deleting files.

Examples:
[time 8/12/2009 7:40:27 AM: ID 14: machine PC4137.cei-dom.ceicmhb: response 8/12/2009 7:45:29 AM] The was detected in C:\WINDOWS\SYSTEM32\SPOOL\DRIVERS\W32X86\3\CNFR0MUI_DB1E4.DLL. Machine: PC4137, User: NT AUTHORITY\NETWORK SERVICE. Status: No cure for this infection.


[time 8/12/2009 7:40:50 AM: ID 14: machine PC2537.cei-dom.ceicmhb: response 8/12/2009 7:46:24 AM] The was detected in C:\WINDOWS\SYSTEM32\DESK.CPL. Machine: PC2537, User: CEI-DOM\Innes. Status: No cure for this infection.

Consumer said...

i have the same problem today. What I have to do??

Jason said...

We have changed the realtime to report only, disable the update, and did a discover now on phone home to get policies.

Tony said...

I have the same issues as well, its annoying that theres nothing on the CA site to acknowledge this.

A final nail in the coffin anyway as were moving to Trend Micro's Worry free over this weekend. Testing as we speak.

Jason said...

Yeah - same here we have 650 workstations. And will be switching to kaspersky shortly. In testing phases too.

Luke said...

Completely Atrocious Protection Suite does it again?

Cannot say I am surprised, caused havoc at some sites today.

Luke said...

Also, echoing above comments - Last Straw for CA.

Consumer said...

But my problem is that the Ca console installed on the IsaServer is not accesible now!
Instead My workstation works correctly.

Jason said...

Online with CA support now. They have fixed the problem - er deleted the new sig and revert back to old one. Have your ITM server update and do the same to workstations. It is the 33.3.7051 sig thats bad

Jason said...

New Sig file that is "working" 34.0.6674.0

Luke said...

"But my problem is that the Ca console installed on the IsaServer is not accesible now!
Instead My workstation works correctly."

Same, Console completely in-accessible due to "Invalid Username/password" It is most certainly correct, re-install time *groan*

PeeGee said...

Updated with the rollback version 34.0.0.6674.
All I need to do now is put back the 40 odd quarantined files.
I wonder if CA appreciate the total man-hours lost with dealing with this. I doubt it...

Consumer said...

"But my problem is that the Ca console installed on the IsaServer is not accesible now!
Instead My workstation works correctly."

Same, Console completely in-accessible due to "Invalid Username/password" It is most certainly correct, re-install time *groan*

MY PROLEM IS THAT THE iTRUST SOFTAWER DOESN'T WORK ON THE SERVER NOW. I'LL TRY TO REINSTALL IT...

Jason said...

Luke - restart the etrust RPC service and you will be able to log back in.

Andrew said...

Luke - I had similar issues on my BES server. To fix disable & stop all eTrust (trust??!!) services on the ISA server. Map to the local drives on the ISA server & search for *AVB. If your realtime policy is set to cure files all cured files should be returned. Just rename the files back to their original name.

eTrust is a terrible product. Currently testing Nod32 corporate edition as a replacement.

Consumer said...

"Luke - restart the etrust RPC service and you will be able to log back in."

Please, suggest me how.

Jason said...

On the ITM Server
Click Start
Choose Run
type services.msc
Find Etrust ITM RPC Service
Right Click and Choose restart

You should be able to logon after that without reinstall but probably only 1 time.

Luke said...

All services restarted, Server restarted, everything restarted. Nothing worked.

1200+ Pages of "Security breach - Database Editor Edited one or more of the bla bla bla"

This CA install committed Seppuku out of shame.

Consumer said...

On the ITM Server
Click Start
Choose Run
type services.msc
Find Etrust ITM RPC Service
Right Click and Choose restart

You should be able to logon after that without reinstall but probably only 1 time.


I have not that services in my list. thank you

Consumer said...

I can see a 2CA pest patrol realtime protection services"..

Consumer said...

Ok. I run the CA agent. it said the signatures installed is the 33.3.7051. how can I remove it?

Terry Dooher said...

Hi folks.

had the same problem this morning with it randomly isolating files from Incredibuild and Visaul Studio.

To those that have lost the management console, Try restarting the Apache Tomcat service as well as the etrust services. (Tomcat powers the web interfgace to the MC) It worked for us.

Jerad said...

CA obviously knew this was going on before most people did. They had already pushed out a new update to our AV server. A correct Sig number is now 34.0.6674.0

Problem went away after we put that sig on all affected servers

Conrad Longmore said...

Pattern 34.0.6674 should fix this problem.. then you can put your broken clients back together!

Eoin said...

the 6674 update on our server now , when i try and force an update from PC they say "component is not available for download.." all pcs on 33.3.7051 - any ideas ?

Terry Dooher said...

We've pushed 6674 out and it does seem to have resolved things. I've re-enabled the on-access scanning on affected machines and they're ticking over nicely.

No email from CA, though (as was promised) and no official word of any issue on their website, still.

Anyone else think this is something other than a simple false positive? given the randomness of files affected and the fact that the virus name listed was (in 99% of our cases) an empty string, it looks like someone is going to get drawn and quartered for this.

Eoin said...

terry , 6674 wont push out for me from server , when i discover , any ideas ?

Andrew said...

I agree Terry, looks like more than a false positive. On just one of our servers a whole host of *.dll & *.exe files were renamed resulting in problems with the following:

1. eTrust itslef (realmon.exe, vete.dll & other files were renamed)
2. MSDE
3. Veritas Netbackup
4. WSUS
5. BES
6. Java Run-Time

Someone at CA is for the chop!!!

David said...

Hi All,

CA have tools to roll back quarantined files. They attached them to our support case.

Asger said...

Is there any other way to get hands on that tool?

Terry Dooher said...

I've just had a similar email. An FTP link to a password-protected rar file comtaining a .CMD script that invokes the client console with options to restore all quarantined files.

Stripping it down, the cmd file just seems to open IE at the following link:

http://localhost:5250/spin/ITMClient/Quarantined_items.csp?action=RestoreAll&impersonate=false&user=%computername%%5C%username%

Which seems to do the job without all that cmd wrapping. (For me, anyway. I of course accept no responsibility if it 'Unquarantines' your system32 directory)

(Eoin, it just came down fine for me. All I can sugegst is restarting the RPC and Job services on the client and server and trying to force an update from the client)

David said...

I can post them somewhere if somebody has space

Jason said...

Here's the tools.... virus free:
http://bleucube.com/Restoretools.zip

Asger said...

I got 250 clients. Our eTrust setting for “Action to perform if cure fails” is not to “Quarantine file”, but to “Rename file”. Therefore I need a tool to restore all renamed files. i.e. remove the 0.AVB extension. Can “Renameavb2exe_with_date” from Restoretools.zip do the job?

Scott said...

Jason, thanks a million for posting that! It saved me a lot of time.

Cheers,

Scott

Scott said...

Asger,

Based on the ReadMe file for the utility, it looks like it should work. Do you have a relatively expendable affected client to try it on?

Scott

Asger said...

Now I have tested the “Renameavb2exe_with_date” from Restoretools.zip, but it does not work. I still have all my 0.AVB files. I think I have a problem understanding what exactly the date parameter does. Does anyone know anything about that?

Martin said...

Date format is American - uses the date to strip the extension off dates on & after the last accessed date.
Create a text file, rename it with the extension and test.
You can reduce the drive letters by editing the executable.
We are going to roll this out shortly - we have thousands of affected files...

Jarl said...

The rename util does not rename all files for us it gets so far and then stops.

Also we can not download the roll back update our AV server downloads then the update process stalls.

Martin said...

Does it end, or stop (i.e. fail)?

We haven't sent this out yet - we're just collating the affected client list... Also a reboot would be advised as services might have hung due to .dll rename???

Consumer said...

My problem is similar to your. My ISA server doesn't run the CA agent and i can't dowload de 6674. Is it possible to run a shadow copy of the server?

Travis said...
This comment has been removed by the author.
Steelergrl said...

Does anyone have a fix for machines that won't boot? So far, I have about 20 and that's going to grow.

Travis said...

Over 1000 computers in our company are infected with what we are calling the "ETrust update virus". Tons of help desk calls and countless hours of reinstalling software ahead... Only one more year under contract with CA and we'll be free. Hooray!

Terry Dooher said...

@Consumer: If you haven't already, try shutting down Apache Tomcat and all of the etrust services on the server. Then go to your Program Files\CA folder and search (F3) for *.AVB. Rename any files that pop up in the results menaully and then restart the services. You should be able to reach your Management console then.

@Steelgirl. Two possible options: Boot from the relevant CD and run an automatic repair. This should (hopefully) restore the missing files. If it works, run a windows update to get them current again.
Failing that, boot to a repair command prompt (either via safe mode or via the boot CD) and (deep breath) manually rename the affected files. (CD to C:\windows\ and use 'dir /s *.AVB' to locate them. Hopefully there won't be too many.) Once you're booting again, you can use the repair tools mentioned earlier. Good luck...

dennismsean said...

Taken from the ca support forum:


A CA ITM engine update (engine v33) released at 1:04 AM ET on 8/12/09 has been found to detect multiple clean files as malicious in certain circumstances . If you are running CA ITM software and experiencing a false positive condition after upgrading to engine v33 please initiate an update immediately to resolve the false positive issue. An updated engine package engine v34) was created and released the same day, 8/12/07, at 7:21 AM ET.

For the files which are already renamed or quarantined, we have uploaded the rename and un-quarantine tool to below mentioned link.
ftp://ftp.ca.com/outgoing/8888888/17943192-01
File name: Renameavb2exe_with_date.rar
File Name: CA_Unquarantine.rar
File Name: Password.txt

Please download and run the rename tool or un-quarantine tool first to restore the files and then update the machines to version 34.0.0.6674.

Jarl said...

The rename stops on the

"Downloaded Progam Files" folder on our machines.

Arrgh

Consumer said...

@Terry

I opened the consol. What should I do now?

Thank you for your support

Consumer said...

I tryed to upgrade the server but I'm still waiting...

Consumer said...

@Terry

It works!
Thanks a lot for your support.

Terry Dooher said...

@Consumer: Glad to hear it :)

The Reg have picked this up, now:
http://www.theregister.co.uk/2009/08/12/ca_auto_immune_update/

I'm amazed that their support site still lists nothing related to this problem, and not even an official response to the forum post.

@Dynamoo: Thanks for letting us hijack your blog as an impromptu support group. :)

Consumer said...

Now my problems are with the clients. They didn't succeed downoading the fix.

Martin said...

Non-booting machines have had crucial OS files renamed - we are arming our engineers with the tool on a bootable USB drive to allow them to rename the files as was.
Alternate you could use a PE Builder CD with network capability to run patch from CD or other network source.

Jarl said...

Now my problems are with the clients. They didn't succeed downoading the fix.

same problem here

Consumer said...
This comment has been removed by the author.
Steelergrl said...

@Martin - Could I ask what you are using on your USB drives? I'm having issue running the tools because of wScript errors and the scripts asking for IE.

laurin1 said...

How do I force eTrust clients to update? The schedule is set for 1:30AM tomorrow. I need it to happen now?

Martin said...

@Steelergrl

Not sure - I didn't create them. Will find out tomorrow - at home now.

Have a read - http://www.thepcspy.com/read/bootable_usb_flash_drive

Martin said...

@laurin1

We ran a "Client Policy" from the CA console.
Forces an immediate update from your deployment server or www.

Scott said...

Laurin1,

If you haven't already setup the client policy that Martin spoke of, CA tech support should be able to assist with this. Good luck and thanks to Dynamoo for the impromtu forum.

Jason said...

Put this in the login script or bat file and have the clients run it:

c:\Program Files\CA\SharedComponents\ScanEngine\ITMDist.exe

that will make it update. We use a Kbox1000 to roll out scripts.

Conrad Longmore said...
This comment has been removed by the author.
Conrad Longmore said...

If your non-booting system is a standard IDE or SATA drive, then often the easiest way to fix it is to put the HD from the victim machine into an external drive enclosure and slave it to a laptop or desktop.. I've always found that a lot easier than mucking about with bootable CDs, USB keys and recovery consoles.

Nick said...

Pretty much what I'd expect from the company that gave the world ArcServ

JC said...

First get the updates in your CA-eTrust-AV Server... Then make a GPO to run InoDist.exe;

Or just use a VBS with this code:

SET objShell=CreateObject("Wscript.Shell")

strPrograms = oBJsHELL.ExpandEnvironmentStrings("%PROGRAMFILES%")
strPath = strPrograms + "\CA\SharedComponents\ScanEngine\InoDist.exe"
objShell.Run(strPath)


set objShell = nothing


Shame on CA!!!

Maz said...

CA is crap. We finally got fed up with their bull crap and ditched them over a year ago. Sounds like a lot of you are considering doing the same. We switched to Sophos (Ya I hadn't heard of them before either) but I got no problems recomending them. We tested Kaspersky, Nod32 and Trend, they were OK but not great.

JC said...

I have a organization with 800 computers and some of them just crash! the system don't boot.

So we used ERD Commander and made a rollback on system restore. Seem working...

What about AVAST? I think is a good choice.

Lucas said...

Epic fail CA!
I've spent hours trying to troubleshoot problems with our software at a client site before I noticed that a bunch of DLLs in the .NET framework had .0.AVB appended to them.