Wednesday 12 August 2009

CA eTrust goes nuts with StdWin32 and other false positives

CA eTrust ITM has gone completely nuts today, with a load of seemingly random false positives mostly for StdWin32 in a large number of binaries, including some components of eTrust itself.

The core problem seems to be a signature update from 31.6.6672 to 33.3.7051, there seems to be little consistency in what is being detected as a false positive although there are multiple occurrences of Nokia software, VNC and event DLLs and EXEs belonging to eTrust's core components.

Probably the best thing to do is block the update or change the Realtime scanning behaviour to "disabled" or "report only".

Update: problem seems to have started at about 0525 GMT when the new signature pattern applied. There no consistent pattern to the infected files, it looks like it happens at random. Several other people seem to be having the same issue!

Update 2: Signature pattern 34.0.6674 appears to fix this problem. You can then enjoy repairing your faulty machines.. thanks CA!

Update 3: Amusingly, CA eTrust seems to have deleted its own key components in many cases. I don't know if this is the first recorded case of an anti-virus application mistaking itself as malware!

Update 4: CA have released a statment as follows:

Last night, CA released a new updated antimalware engine. This new release has resulted in false positive detections of a number of files. CA Threat Manager customers are the only customers being affected by this issue. This is not a result of signature updates and does not impact CA consumer Internet security products.

To resolve the issue, CA has rolled back the new engine and re-released its previous antimalware engine. CA customer support representatives are on call to answer customer questions and to provide remediation support. A remediation tool to rename the quarantined files is now available through CA support and will soon be accessible online.

CA is aggressively working to resolve the issue, assist any customers who have been affected, as well as identify the root cause of the incident. We apologize for this inconvenience and look forward to the roll out of our new antimalware engine, which will ultimately offer our customers many benefits including enhanced malware protection and improved performance.

Update 5: Got a mention on El Reg.. funny thing is that I went in to work today wearing my El Reg T-Shirt. Coincidence? Consiparacy? Cockup?

PS: Please remember to read the comments if you are still having problems!

71 comments:

  1. Thank you Dynamoo. I thought I was going mad this morning. I've logged a call with CA so we shall see.

    Funny enough, when I logged the call the rep asked me if I thought it was a false positive. Maybe she reads your blog too.

    ReplyDelete
  2. I have the same problem today here.

    I think CA is doing very bad things with the eTrust ITM Antivirus. I am looking for a different Antivirus, Have you any idea ?

    ReplyDelete
  3. Same problem here. Isn't this twice in one month? I don't recall having these issues with eTrust 7.x...What's going on with their quality control???

    ReplyDelete
  4. Yep I had 235 workstations updated and started deleting files.

    Examples:
    [time 8/12/2009 7:40:27 AM: ID 14: machine PC4137.cei-dom.ceicmhb: response 8/12/2009 7:45:29 AM] The was detected in C:\WINDOWS\SYSTEM32\SPOOL\DRIVERS\W32X86\3\CNFR0MUI_DB1E4.DLL. Machine: PC4137, User: NT AUTHORITY\NETWORK SERVICE. Status: No cure for this infection.


    [time 8/12/2009 7:40:50 AM: ID 14: machine PC2537.cei-dom.ceicmhb: response 8/12/2009 7:46:24 AM] The was detected in C:\WINDOWS\SYSTEM32\DESK.CPL. Machine: PC2537, User: CEI-DOM\Innes. Status: No cure for this infection.

    ReplyDelete
  5. i have the same problem today. What I have to do??

    ReplyDelete
  6. We have changed the realtime to report only, disable the update, and did a discover now on phone home to get policies.

    ReplyDelete
  7. I have the same issues as well, its annoying that theres nothing on the CA site to acknowledge this.

    A final nail in the coffin anyway as were moving to Trend Micro's Worry free over this weekend. Testing as we speak.

    ReplyDelete
  8. Yeah - same here we have 650 workstations. And will be switching to kaspersky shortly. In testing phases too.

    ReplyDelete
  9. Completely Atrocious Protection Suite does it again?

    Cannot say I am surprised, caused havoc at some sites today.

    ReplyDelete
  10. Also, echoing above comments - Last Straw for CA.

    ReplyDelete
  11. But my problem is that the Ca console installed on the IsaServer is not accesible now!
    Instead My workstation works correctly.

    ReplyDelete
  12. Online with CA support now. They have fixed the problem - er deleted the new sig and revert back to old one. Have your ITM server update and do the same to workstations. It is the 33.3.7051 sig thats bad

    ReplyDelete
  13. New Sig file that is "working" 34.0.6674.0

    ReplyDelete
  14. "But my problem is that the Ca console installed on the IsaServer is not accesible now!
    Instead My workstation works correctly."

    Same, Console completely in-accessible due to "Invalid Username/password" It is most certainly correct, re-install time *groan*

    ReplyDelete
  15. Updated with the rollback version 34.0.0.6674.
    All I need to do now is put back the 40 odd quarantined files.
    I wonder if CA appreciate the total man-hours lost with dealing with this. I doubt it...

    ReplyDelete
  16. "But my problem is that the Ca console installed on the IsaServer is not accesible now!
    Instead My workstation works correctly."

    Same, Console completely in-accessible due to "Invalid Username/password" It is most certainly correct, re-install time *groan*

    MY PROLEM IS THAT THE iTRUST SOFTAWER DOESN'T WORK ON THE SERVER NOW. I'LL TRY TO REINSTALL IT...

    ReplyDelete
  17. Luke - restart the etrust RPC service and you will be able to log back in.

    ReplyDelete
  18. Luke - I had similar issues on my BES server. To fix disable & stop all eTrust (trust??!!) services on the ISA server. Map to the local drives on the ISA server & search for *AVB. If your realtime policy is set to cure files all cured files should be returned. Just rename the files back to their original name.

    eTrust is a terrible product. Currently testing Nod32 corporate edition as a replacement.

    ReplyDelete
  19. "Luke - restart the etrust RPC service and you will be able to log back in."

    Please, suggest me how.

    ReplyDelete
  20. On the ITM Server
    Click Start
    Choose Run
    type services.msc
    Find Etrust ITM RPC Service
    Right Click and Choose restart

    You should be able to logon after that without reinstall but probably only 1 time.

    ReplyDelete
  21. All services restarted, Server restarted, everything restarted. Nothing worked.

    1200+ Pages of "Security breach - Database Editor Edited one or more of the bla bla bla"

    This CA install committed Seppuku out of shame.

    ReplyDelete
  22. On the ITM Server
    Click Start
    Choose Run
    type services.msc
    Find Etrust ITM RPC Service
    Right Click and Choose restart

    You should be able to logon after that without reinstall but probably only 1 time.


    I have not that services in my list. thank you

    ReplyDelete
  23. I can see a 2CA pest patrol realtime protection services"..

    ReplyDelete
  24. Ok. I run the CA agent. it said the signatures installed is the 33.3.7051. how can I remove it?

    ReplyDelete
  25. Hi folks.

    had the same problem this morning with it randomly isolating files from Incredibuild and Visaul Studio.

    To those that have lost the management console, Try restarting the Apache Tomcat service as well as the etrust services. (Tomcat powers the web interfgace to the MC) It worked for us.

    ReplyDelete
  26. CA obviously knew this was going on before most people did. They had already pushed out a new update to our AV server. A correct Sig number is now 34.0.6674.0

    Problem went away after we put that sig on all affected servers

    ReplyDelete
  27. Pattern 34.0.6674 should fix this problem.. then you can put your broken clients back together!

    ReplyDelete
  28. the 6674 update on our server now , when i try and force an update from PC they say "component is not available for download.." all pcs on 33.3.7051 - any ideas ?

    ReplyDelete
  29. We've pushed 6674 out and it does seem to have resolved things. I've re-enabled the on-access scanning on affected machines and they're ticking over nicely.

    No email from CA, though (as was promised) and no official word of any issue on their website, still.

    Anyone else think this is something other than a simple false positive? given the randomness of files affected and the fact that the virus name listed was (in 99% of our cases) an empty string, it looks like someone is going to get drawn and quartered for this.

    ReplyDelete
  30. terry , 6674 wont push out for me from server , when i discover , any ideas ?

    ReplyDelete
  31. I agree Terry, looks like more than a false positive. On just one of our servers a whole host of *.dll & *.exe files were renamed resulting in problems with the following:

    1. eTrust itslef (realmon.exe, vete.dll & other files were renamed)
    2. MSDE
    3. Veritas Netbackup
    4. WSUS
    5. BES
    6. Java Run-Time

    Someone at CA is for the chop!!!

    ReplyDelete
  32. Hi All,

    CA have tools to roll back quarantined files. They attached them to our support case.

    ReplyDelete
  33. Is there any other way to get hands on that tool?

    ReplyDelete
  34. I've just had a similar email. An FTP link to a password-protected rar file comtaining a .CMD script that invokes the client console with options to restore all quarantined files.

    Stripping it down, the cmd file just seems to open IE at the following link:

    http://localhost:5250/spin/ITMClient/Quarantined_items.csp?action=RestoreAll&impersonate=false&user=%computername%%5C%username%

    Which seems to do the job without all that cmd wrapping. (For me, anyway. I of course accept no responsibility if it 'Unquarantines' your system32 directory)

    (Eoin, it just came down fine for me. All I can sugegst is restarting the RPC and Job services on the client and server and trying to force an update from the client)

    ReplyDelete
  35. I can post them somewhere if somebody has space

    ReplyDelete
  36. Here's the tools.... virus free:
    http://bleucube.com/Restoretools.zip

    ReplyDelete
  37. I got 250 clients. Our eTrust setting for “Action to perform if cure fails” is not to “Quarantine file”, but to “Rename file”. Therefore I need a tool to restore all renamed files. i.e. remove the 0.AVB extension. Can “Renameavb2exe_with_date” from Restoretools.zip do the job?

    ReplyDelete
  38. Jason, thanks a million for posting that! It saved me a lot of time.

    Cheers,

    Scott

    ReplyDelete
  39. Asger,

    Based on the ReadMe file for the utility, it looks like it should work. Do you have a relatively expendable affected client to try it on?

    Scott

    ReplyDelete
  40. Now I have tested the “Renameavb2exe_with_date” from Restoretools.zip, but it does not work. I still have all my 0.AVB files. I think I have a problem understanding what exactly the date parameter does. Does anyone know anything about that?

    ReplyDelete
  41. Date format is American - uses the date to strip the extension off dates on & after the last accessed date.
    Create a text file, rename it with the extension and test.
    You can reduce the drive letters by editing the executable.
    We are going to roll this out shortly - we have thousands of affected files...

    ReplyDelete
  42. The rename util does not rename all files for us it gets so far and then stops.

    Also we can not download the roll back update our AV server downloads then the update process stalls.

    ReplyDelete
  43. Does it end, or stop (i.e. fail)?

    We haven't sent this out yet - we're just collating the affected client list... Also a reboot would be advised as services might have hung due to .dll rename???

    ReplyDelete
  44. My problem is similar to your. My ISA server doesn't run the CA agent and i can't dowload de 6674. Is it possible to run a shadow copy of the server?

    ReplyDelete
  45. This comment has been removed by the author.

    ReplyDelete
  46. Does anyone have a fix for machines that won't boot? So far, I have about 20 and that's going to grow.

    ReplyDelete
  47. Over 1000 computers in our company are infected with what we are calling the "ETrust update virus". Tons of help desk calls and countless hours of reinstalling software ahead... Only one more year under contract with CA and we'll be free. Hooray!

    ReplyDelete
  48. @Consumer: If you haven't already, try shutting down Apache Tomcat and all of the etrust services on the server. Then go to your Program Files\CA folder and search (F3) for *.AVB. Rename any files that pop up in the results menaully and then restart the services. You should be able to reach your Management console then.

    @Steelgirl. Two possible options: Boot from the relevant CD and run an automatic repair. This should (hopefully) restore the missing files. If it works, run a windows update to get them current again.
    Failing that, boot to a repair command prompt (either via safe mode or via the boot CD) and (deep breath) manually rename the affected files. (CD to C:\windows\ and use 'dir /s *.AVB' to locate them. Hopefully there won't be too many.) Once you're booting again, you can use the repair tools mentioned earlier. Good luck...

    ReplyDelete
  49. Taken from the ca support forum:


    A CA ITM engine update (engine v33) released at 1:04 AM ET on 8/12/09 has been found to detect multiple clean files as malicious in certain circumstances . If you are running CA ITM software and experiencing a false positive condition after upgrading to engine v33 please initiate an update immediately to resolve the false positive issue. An updated engine package engine v34) was created and released the same day, 8/12/07, at 7:21 AM ET.

    For the files which are already renamed or quarantined, we have uploaded the rename and un-quarantine tool to below mentioned link.
    ftp://ftp.ca.com/outgoing/8888888/17943192-01
    File name: Renameavb2exe_with_date.rar
    File Name: CA_Unquarantine.rar
    File Name: Password.txt

    Please download and run the rename tool or un-quarantine tool first to restore the files and then update the machines to version 34.0.0.6674.

    ReplyDelete
  50. The rename stops on the

    "Downloaded Progam Files" folder on our machines.

    Arrgh

    ReplyDelete
  51. @Terry

    I opened the consol. What should I do now?

    Thank you for your support

    ReplyDelete
  52. I tryed to upgrade the server but I'm still waiting...

    ReplyDelete
  53. @Terry

    It works!
    Thanks a lot for your support.

    ReplyDelete
  54. @Consumer: Glad to hear it :)

    The Reg have picked this up, now:
    http://www.theregister.co.uk/2009/08/12/ca_auto_immune_update/

    I'm amazed that their support site still lists nothing related to this problem, and not even an official response to the forum post.

    @Dynamoo: Thanks for letting us hijack your blog as an impromptu support group. :)

    ReplyDelete
  55. Now my problems are with the clients. They didn't succeed downoading the fix.

    ReplyDelete
  56. Non-booting machines have had crucial OS files renamed - we are arming our engineers with the tool on a bootable USB drive to allow them to rename the files as was.
    Alternate you could use a PE Builder CD with network capability to run patch from CD or other network source.

    ReplyDelete
  57. Now my problems are with the clients. They didn't succeed downoading the fix.

    same problem here

    ReplyDelete
  58. This comment has been removed by the author.

    ReplyDelete
  59. @Martin - Could I ask what you are using on your USB drives? I'm having issue running the tools because of wScript errors and the scripts asking for IE.

    ReplyDelete
  60. How do I force eTrust clients to update? The schedule is set for 1:30AM tomorrow. I need it to happen now?

    ReplyDelete
  61. @Steelergrl

    Not sure - I didn't create them. Will find out tomorrow - at home now.

    Have a read - http://www.thepcspy.com/read/bootable_usb_flash_drive

    ReplyDelete
  62. @laurin1

    We ran a "Client Policy" from the CA console.
    Forces an immediate update from your deployment server or www.

    ReplyDelete
  63. Laurin1,

    If you haven't already setup the client policy that Martin spoke of, CA tech support should be able to assist with this. Good luck and thanks to Dynamoo for the impromtu forum.

    ReplyDelete
  64. Put this in the login script or bat file and have the clients run it:

    c:\Program Files\CA\SharedComponents\ScanEngine\ITMDist.exe

    that will make it update. We use a Kbox1000 to roll out scripts.

    ReplyDelete
  65. This comment has been removed by the author.

    ReplyDelete
  66. If your non-booting system is a standard IDE or SATA drive, then often the easiest way to fix it is to put the HD from the victim machine into an external drive enclosure and slave it to a laptop or desktop.. I've always found that a lot easier than mucking about with bootable CDs, USB keys and recovery consoles.

    ReplyDelete
  67. Pretty much what I'd expect from the company that gave the world ArcServ

    ReplyDelete
  68. First get the updates in your CA-eTrust-AV Server... Then make a GPO to run InoDist.exe;

    Or just use a VBS with this code:

    SET objShell=CreateObject("Wscript.Shell")

    strPrograms = oBJsHELL.ExpandEnvironmentStrings("%PROGRAMFILES%")
    strPath = strPrograms + "\CA\SharedComponents\ScanEngine\InoDist.exe"
    objShell.Run(strPath)


    set objShell = nothing


    Shame on CA!!!

    ReplyDelete
  69. CA is crap. We finally got fed up with their bull crap and ditched them over a year ago. Sounds like a lot of you are considering doing the same. We switched to Sophos (Ya I hadn't heard of them before either) but I got no problems recomending them. We tested Kaspersky, Nod32 and Trend, they were OK but not great.

    ReplyDelete
  70. I have a organization with 800 computers and some of them just crash! the system don't boot.

    So we used ERD Commander and made a rollback on system restore. Seem working...

    What about AVAST? I think is a good choice.

    ReplyDelete
  71. Epic fail CA!
    I've spent hours trying to troubleshoot problems with our software at a client site before I noticed that a bunch of DLLs in the .NET framework had .0.AVB appended to them.

    ReplyDelete