Solutions Showcase Theater – Strata + Hadoop World San Jose 2016

Sobre este curso

okay in this talk what I'm going to go over is a customer solution for advanced threat detection on streaming data the customer was solution airy and their

managed security services provider and they provide threat intelligence as a service and their objective was then you wanted to provide real-time threat intelligence and they wanted to be able

to have machine learning anomaly detection store and process unstructured data to provide threat alerts on customer data so they had a lot of data and it was coming in fast their existing

solution was storing in Oracle relational database and it was expensive and it wasn't scaling for all the data that they needed to store and process so what did the solution need to do they

needed to receive a lot of data from customers networks and applications they need to collect this data process the data store the data and then serve the data and they need to do this really

fast divide real-time insights so in order to do this really fast they need to do it in parallel and have it partitioned so for collecting the data they chose kafka for fast distributed

messaging with Kafka you have topics which are a logical collection of your events and this allows you to categorize them for example they had a topic for raw data and rich data and alerts and

also the topics decouple the producers from the consumers the producers can be load balanced and that they published to the topics and a consumers subscribe from subscribe to the topics that

they're interested in then physically these topics are partitioned so that they're spread out across your cluster then the producers are load-balanced to publish in parallel and the consumers

can be grouped also to read in parallel this makes it really fast so parallel partition messaging for fast scalable messaging then for the processing of the messaging

they chose storm which provides us to distributed real-time processing they have storm spouts to read from Kafka to parse the data then some of the then they would arrange this data with

geographical information and then this data would be stored but then the data would also be sent to kaska for another topic for complex event processing and correlation so next we'll look at the

complex event correlation with Esper so what the complex event correlation allows you to do is detect a set of related events for patterns so for example a pattern could be excess login

failure to detect that the same user has logged in multiple times and this is what it could look like it looks like an SQL query so it's checking that there was a login failure within a time window

and a certain amount of time within a time window so it looks like an SQL query but it's it's querying a sequence of events to detect these patterns so again here we to do this fast and

scalable we have parallel partitioned processing and this is some examples of detecting real-time events correlation so for example an employee accessing a database server unusual hours or another

one could be a user logging in from two different countries within a short window so next for storing all this data they had casket bolts that were storing into HDFS HBase and elastic search so H

to fsnh space were being used then for cert for machine learning and then also for querying for an analytics and then elastic page for indexing for real search analytics

a space was chosen because it scales with the relational model what you do is you normalize your data which means that you have little pieces of it spread across the cluster and then you bring

that back together inquiries for joins and that could be a bottleneck what with HBase what you do is you store the data that together the way that you're going to read it that the data that's read

together is stored together also with age space it was designed to scale what it does is it's automatically partitions your data across the cluster by key range and then you access this data by

the rookie which gives you really fast reads and writes so again for fast storage we have parallel it's parallel and partitioned then this data was made available for machine learning threat

modeling anomaly detection and security analytics so for example for a forensic and investigation what type of queries would you perform in this data things like what happened within a time and age

how did the threat get in what are the activities associated with an specific IP has this occurred somewhere else in the past so this is showing then the complete streaming solution architecture

that we went over so the key to the scale was to have parallel partition messaging processing and storage and you can build all this with the complete architecture with we saw with

distributed messaging distributed storage and then also this distributed processing so why for the Duke for security analytics because it provides a cost

effective for storing and analyzing large volumes of data and also you it provides things like machine learning and we saw a complex event processing and if you'd like to learn more about

this we have some free on-demand training at learn map are calm we have free trading for HBase spark and for nap our streams which is an implementation of the kafka API so if there's any

questions I have a couple of minutes okay thank you