16
abril
2007

AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

[News] 
I have read today this piece of news:


"An experiment conducted at the end of March by independent security-industry benchmark website VirusTotal.com attempted to simulate a malicious attack using a long-known source of malicious code on computers. Competing with 32 rivals, only Finjan's Vital Security Web Appliance detected and blocked the malicious code in VirusTotal's tests. The computers running other products were all comprised [sic] - resulting in potential data loss and theft."


This paragraph may lead to confusion, whether that was the result intended or not, and that is why we feel compelled to declare the following at Hispasec:


- VirusTotal has not conducted any experiment or test related to AV comparative analyses.

- VirusTotal has no notice whatsoever of the malicious code they refer to in this piece of news.

- VirusTotal has never tested nor tried Finjan's security solutions.


Generally speaking, even though it may seem obvious, we must state that all anti-malware products have detection problems due to the tremendous proliferation and diversification of malware nowadays. Likewise, any product may detect a new sample on its own, either because of its heuristics or because they are the first ones to generate a specific signature. This is why it seems totally inadequate and opportunistic to claim the superiority of a product based on the result of a sole malware sample.


We are rather tired of repeating that VirusTotal was not designed as a tool to perform AV comparative analyses, but as a tool that checks suspicious samples with several AV programs and helps AV labs by forwarding them the malware they failed to detect. Those who use VirusTotal to perform AV comparative analyses should know that they are making many implicit errors in the methodology, the most obvious being:


- VirusTotal AV engines are commandline versions, so depending on the product, they will not behave quite like the desktop versions: for instance, in such cases when desktop solutions use techniques based on behavioral analysis and count on personal firewalls that may decrease entry points and mitigate propagation, etc.


- In VirusTotal desktop-oriented solutions coexist with perimeter-oriented solutions; heuristics in this latter group may be more aggressive and paranoid, since impact of false positives is less visible in the perimeter. It is simply not fair to compare both groups.


In general, it is not an easy task to perform a responsible and reliable AV comparative analysis; it requires having a malware collection that is both representative (nowadays it should be larger than the In-The-Wild collection) and authentic (ZOO collections are riddled with false viruses and corrupt executables). Besides, given the implementation of new AV technologies, in the case of desktop AV products, it would be necessary to execute those samples one by one in real environments with each of the resident products to see their detection capabilities and their prevention. As of today, there is no AV comparative analysis in the world that meets these basic requirements.

Sent by bquintero @ 18:51 | Permalink | Comments (11) | Trackbacks (0)
Comentarios
Re: AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

I saw that the article you referred to appears to be on a a site devoted to helping Israeli businesses. The site is even asking for cash contributions as a nonprofit organization. It appears to me to be spam--perhaps not the malware type, but spam nevertheless.

Many users helped Finjan to develop their "black box" by using the old Finjan behavior blocking software they incorporated in it, so Finjan should be very much aware of the limitations of antivirus/antimalware programs. That's why I'll bet that Finjan didn't have very much to do with this news.

VirusTotal should contact the publisher of this news and inform them of their error. I tried to do so but couldn't find a contact point.

Regards,

RWS

Posted by: GuitarBob at abril 17,2007 04:20
Re: AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

# GuitarBob: "That's why I'll bet that Finjan didn't have very much to do with this news."

http://www.prnewswire.co.uk/cgi/news/release?id=194028

Posted by: marketingrulez at abril 17,2007 09:54
Re: AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

You're kidding? It's a Finjan sponsored press release! It even has a "About VirusTotal" section... Shame on Finjan!

Posted by: killav at abril 17,2007 10:59
Re: AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

The link isn't proof that Finjan knows about the false press release (although they may). There are also several other links, but they are closed--guess they didn't pay up! I did send Finjan email referencing this page here at VirusTotal.

Regards,

Posted by: GuitarBob at abril 17,2007 18:08
Re: AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

What 'false press release' your are talking about? This is marketing kung-fu style marketing foobla:

http://www.finjan.com/Pressrelease.aspx?id=1402&PressLan=1230&lan=3

Posted by: Me at abril 17,2007 21:26
Re: AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

"Malicious Page Under Benchmark

Finjan benchmarked a page from a long-known source of malicious code against 32 web security products, using an independent online security benchmark website. Finjan’s Vital Security™ Web Appliance was the only product that managed to proactively detect and block the code without any product update or signature, illustrating the difference between real time code inspection versus other security products and technologies."

http://www.finjan.com/Content.aspx?id=1367

Posted by: Me2 at abril 17,2007 21:29
Re: AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

Finjan's "Malicious Page under Benchmark" PDF here:

http://d.turboupload.com/d/1722962/FinjanMPUB-March2007.pdf.html

Posted by: Not Me at abril 17,2007 21:43
Re: AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

I'm not convinced (isn't that a surprise [grin]). Here are my counter-counter arguments:

- I'm 100% with you that not all capabilities of all products are present. I'm just arguing that (empirically) this makes up a very small percent of the detections and AFAIK nobody was able until now to quantify these features.

- Getting back to the "how are those scanners configured" question - well, most of the tests omit the exact configuration of the products. Again, given how VT is not in the "testing" business, there isn't any reason why they wouldn't configure the engines the way companies ask them, which is much better than most of the testers do (who usually use "default" configurations to test).

- An other argument in favor of VT is the flux of samples they get. Lets say that they process 10 000 malware samples a day (a conservative estimate). Av-comparatives works with a collection of ~1 000 000 files spanning the last 6 months. During the same period VT would have seen ~1 800 000 samples. Of course some of those are duplicates, some of those are damaged and so on, but still, the numbers favor VT.

I'm still of the opinion that the numbers VT sees (but doesn't publish for political reasons) are very relevant and probably close to the ones seen by organizations like av-comparatives or av-test. Of course, uploading a couple of random samples and using the results to declare "AV is dead" is not valid, but uploading known malware by SANS incident handlers and seeing the poor detection rate is a good indication of the reaction time for AV products.

Posted by: web hosting at agosto 09,2009 21:34
Re: AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

I remembered that when I grew up with virustotal in dating areas of dating london, it became useful to find a bad combination.

Posted by: Josh at agosto 13,2009 21:05
Re: AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

highly appreciate your effort in bringing forth this great analysis.no doubt that it is helpful.nothin more to say

Posted by: kelly web at agosto 21,2009 10:37
Re: AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

I have never been able to Analise a file yet. Starts with queuing and zero bytes but never goes beyond that. Have waited 1 hour and it does not progress.

Posted by: hunhan at agosto 21,2011 13:40
Trackbacks
Please send trackbacks to: http://blog.hispasec.nospam/virustotal/22/tbZ3ping
Replace "nospam" with "com"
There are no trackbacks.
Post a comment