27-06-2012, 02:00 PM
Using Hash Table to Extract Real-Time Online Network Traffic Features for
Hardware IDS
[attachment=25754]
ABSTRACT
This work introduces an efficient algorithm for extracting set
of features from raw network traffic. Network traffic is captured directly from
a Network Interface Card (NIC). The proposed algorithm is used to build
an efficient real-time Network Based Intrusion Detection/Prevention Systems
(NBIDPS). NBIDPS rely on network traffic as their primary data source, so
there is a great need for a reliable, fast algorithm to extract required features
for hardware intrusion detection system from available network traffic. It minimizes
search time for extracting statistical features from connection records
stored in connection queues to memory references. NBIDPS need to implement
this algorithm in a high bit rate network such as a gigabit network, ten
gigabit or higher.
INTRODUCTION
Due to the rapid spread of computer networks, with their capabilities and
speeds, securing these networks require more and more research. Building hardware
Intrusion Detection Systems (IDS) is an essential task. Many approaches
have been taken in building IDS as a software application, but most of
them suffer from security problems and processing limitations. The alternative
option of software IDS is hardware IDS. The hardware IDS needs a fast,
reliable algorithm for extracting features from network traffic to analyze and
make decisions. Features are directly observed from raw traffic and statistically
calculated. Statistically calculated features needs buffering for a time window
and then calculate statistical function over the buffered data. To calculate statistics
over buffered data requires a search. Collision-free hash tables are known
to have zero search time. An organization for buffered traffic in a memory
and proposed hash function is introduced in this paper to build an efficient
collision-free hash table. An algorithm to maintain data in memory and to
extract feature is also introduced.
RELATED WORK
Nguyen, Memik, Memik, and Choudhary (2005)
have presented real-time feature extraction for high
speed networks. They used sketches for data stream
modeling. It uses a constant amount of memory and
has constant per-record update and reconstruction
cost. They focused on the features extracted to detect
only Denial of Service (DoS) attacks, which are simple
and needs no huge processing. Estan and Varghese
(2003) presented a traffic measurement and accounting
algorithm. They treated traffic as a collection of
flows that need to be measured, use multistage filters
and samples, and hold algorithms for measuring
the flows; these algorithms takes a constant number
of memory references per packet and uses a small
amount of memory. Krishnamurthy, Sen, Zhang, and
Chen (2003) have designed a variant of the sketch data
structure, k-ary sketch, which uses a constant, small
amount of memory and has a constant per-record
update and reconstruction cost. Its linearity property
enables summarizing traffic at various levels.
KDD CUP 1999 DATA
KDD cup data is the dataset used for both the third
and fifth International Knowledge Discovery and Data
Mining Tools Competitions, held in conjunction with
KDD-99 (http://kdd.ics.uci.edu/databases/kddcup99/
kddcup99.html) (Lippmann et al., 2009). The competition’s
task was to build a network intrusion detector,
a predictive model capable of distinguishing between
“bad” connections, called intrusions or attacks, and
“good” normal connections (Zakia & Sobh, 2005). The
KDD Cup 1999 dump network traffic into DARPA
database. This database contains a standard set of data
to be audited, which includes a wide variety of intrusions
simulated in a military network environment.
CONCLUSION
This paper presents a model for using a hash
table and an algorithm for extracting statistical features
for network streams. This algorithm/data structure
minimizes search time for extracting statistical
features from connection records stored in connection
queues to only three memory references. This
proposed algorithm/data structure can be used effectively
with hardware implementation of IDS especially
when dealing with high bit rate networks
such as 1Gbps or 10Gbps. This high bit rate network
needs high processing capabilities and resources.
Consistency/aggregation algorithm is used to control
and coordinate the access to the hash table by using
a connection queue to keep track of received connections.
Hardware IDS
[attachment=25754]
ABSTRACT
This work introduces an efficient algorithm for extracting set
of features from raw network traffic. Network traffic is captured directly from
a Network Interface Card (NIC). The proposed algorithm is used to build
an efficient real-time Network Based Intrusion Detection/Prevention Systems
(NBIDPS). NBIDPS rely on network traffic as their primary data source, so
there is a great need for a reliable, fast algorithm to extract required features
for hardware intrusion detection system from available network traffic. It minimizes
search time for extracting statistical features from connection records
stored in connection queues to memory references. NBIDPS need to implement
this algorithm in a high bit rate network such as a gigabit network, ten
gigabit or higher.
INTRODUCTION
Due to the rapid spread of computer networks, with their capabilities and
speeds, securing these networks require more and more research. Building hardware
Intrusion Detection Systems (IDS) is an essential task. Many approaches
have been taken in building IDS as a software application, but most of
them suffer from security problems and processing limitations. The alternative
option of software IDS is hardware IDS. The hardware IDS needs a fast,
reliable algorithm for extracting features from network traffic to analyze and
make decisions. Features are directly observed from raw traffic and statistically
calculated. Statistically calculated features needs buffering for a time window
and then calculate statistical function over the buffered data. To calculate statistics
over buffered data requires a search. Collision-free hash tables are known
to have zero search time. An organization for buffered traffic in a memory
and proposed hash function is introduced in this paper to build an efficient
collision-free hash table. An algorithm to maintain data in memory and to
extract feature is also introduced.
RELATED WORK
Nguyen, Memik, Memik, and Choudhary (2005)
have presented real-time feature extraction for high
speed networks. They used sketches for data stream
modeling. It uses a constant amount of memory and
has constant per-record update and reconstruction
cost. They focused on the features extracted to detect
only Denial of Service (DoS) attacks, which are simple
and needs no huge processing. Estan and Varghese
(2003) presented a traffic measurement and accounting
algorithm. They treated traffic as a collection of
flows that need to be measured, use multistage filters
and samples, and hold algorithms for measuring
the flows; these algorithms takes a constant number
of memory references per packet and uses a small
amount of memory. Krishnamurthy, Sen, Zhang, and
Chen (2003) have designed a variant of the sketch data
structure, k-ary sketch, which uses a constant, small
amount of memory and has a constant per-record
update and reconstruction cost. Its linearity property
enables summarizing traffic at various levels.
KDD CUP 1999 DATA
KDD cup data is the dataset used for both the third
and fifth International Knowledge Discovery and Data
Mining Tools Competitions, held in conjunction with
KDD-99 (http://kdd.ics.uci.edu/databases/kddcup99/
kddcup99.html) (Lippmann et al., 2009). The competition’s
task was to build a network intrusion detector,
a predictive model capable of distinguishing between
“bad” connections, called intrusions or attacks, and
“good” normal connections (Zakia & Sobh, 2005). The
KDD Cup 1999 dump network traffic into DARPA
database. This database contains a standard set of data
to be audited, which includes a wide variety of intrusions
simulated in a military network environment.
CONCLUSION
This paper presents a model for using a hash
table and an algorithm for extracting statistical features
for network streams. This algorithm/data structure
minimizes search time for extracting statistical
features from connection records stored in connection
queues to only three memory references. This
proposed algorithm/data structure can be used effectively
with hardware implementation of IDS especially
when dealing with high bit rate networks
such as 1Gbps or 10Gbps. This high bit rate network
needs high processing capabilities and resources.
Consistency/aggregation algorithm is used to control
and coordinate the access to the hash table by using
a connection queue to keep track of received connections.