14-08-2012, 02:51 PM
Issues in Benchmarking Intrusion Detection Systems
[attachment=31501]
IDS Benchmarking?
How hard can it be to benchmark intrusion detection systems?
Very!
There are lots of ways to get it wrong
Accidentally
Deliberately
Avoiding doing it wrong does not necessarily mean you’ve done it right
What’s an IDS?
IDS = Intrusion Detection System
Primary criterion for measurement is the IDS’ ability to detect intrusions
Secondary criteria for measurement are other issues:
False positives - false alarms
False negatives - real attacks that are missed
Performance impact - thruoughput delay or CPU usage on host processor
Properties of: Network IDS
Collect packets in promiscuous mode
Issues:
Packet collection rate - what is the maximum throughput?
Reassembly/defragmentation/reordering - what about traffic spoofing?
Selective analysis - is the IDS choosing to ignore some traffic in order to optimize?
Properties of: Load-Balanced Network IDS
Use a load-balancing pre-processor to “spread” load across multiple NIDS
Issues:
Can scale to “infinite” bandwidth
Total cost of solution is not single unit pricing (requires switch + multiple NIDS)
Fragment Re-assembly
Re-assembling fragments takes significant CPU time as well as memory to buffer packets
IDS can be negatively impacted by faked fragments intended to consume extra memory
How does IDS handle fragmented attacks? Simply alert “I see fragmented traffic” or de-fragment then apply IDS logic?
[attachment=31501]
IDS Benchmarking?
How hard can it be to benchmark intrusion detection systems?
Very!
There are lots of ways to get it wrong
Accidentally
Deliberately
Avoiding doing it wrong does not necessarily mean you’ve done it right
What’s an IDS?
IDS = Intrusion Detection System
Primary criterion for measurement is the IDS’ ability to detect intrusions
Secondary criteria for measurement are other issues:
False positives - false alarms
False negatives - real attacks that are missed
Performance impact - thruoughput delay or CPU usage on host processor
Properties of: Network IDS
Collect packets in promiscuous mode
Issues:
Packet collection rate - what is the maximum throughput?
Reassembly/defragmentation/reordering - what about traffic spoofing?
Selective analysis - is the IDS choosing to ignore some traffic in order to optimize?
Properties of: Load-Balanced Network IDS
Use a load-balancing pre-processor to “spread” load across multiple NIDS
Issues:
Can scale to “infinite” bandwidth
Total cost of solution is not single unit pricing (requires switch + multiple NIDS)
Fragment Re-assembly
Re-assembling fragments takes significant CPU time as well as memory to buffer packets
IDS can be negatively impacted by faked fragments intended to consume extra memory
How does IDS handle fragmented attacks? Simply alert “I see fragmented traffic” or de-fragment then apply IDS logic?