Strata + Hadoop World – Singapore 2015

Sobre este curso

thank all of you for joining us here at the very first ever strata hadoop world in asia i will say that i'm absolutely thrilled with the news you just heard

the fact that the first time we've ever come to this part of the world with this conference we attracted a sellout crowd now it's not a surprise to me it's fantastic that we've got this level of

interest in the region but I think it really highlights the excitement that we all feel for big data across industries and in order to commemorate this moment i'm going to ask all of you guys to join

me in a photograph that I want to tweet for our inaugural event so everybody say hi Internet whoo all right and please retweet that when I get it up after the talk we're seeing this explosion of

interest in Big Data around the world because of the value that data and analytics are bringing to all sorts of businesses now those of you who know about Cloudera will know that we

specialize in delivering Apache Hadoop and the rich suite of ecosystem projects around Hadoop to enterprises that want to transform themselves that wants to do business in better ways and we've been

working on this for more than seven years now seven and a half years ago we founded the company in the summer of two thousand eight the market and our business have grown dramatically in that

time we're very very excited about how to it surprises some people to hear me say that we expect that Hadoop will disappear in the next couple of years that's the strange thing for a guy who

runs a business that's built on this incredible open-source technology to say but let me explain exactly what I mean when I say that it's not that Hadoop won't be important it's not that you

won't be using Hadoop and the rich ecosystem of projects around it in your businesses what I mean is that that foundation is going to become so ubiquitous it's going to be so common

that we won't need to talk about it anymore rather than talk about the foundation of the building we'll talk about the amazing structures that get built on top

the applications the analytics the new ways to evaluate and extract insight from data that were previously impossible Hadoop will be important but it won't be what we focus on any longer

we're going to see the emergence of a rich collection of applications and analytic solutions and data exploration tools that are native built to run on the platform we are seeing that happen

around the world now and I've been able to present with partners and with customers about some of the fantastic work that they are doing I'm fortunate today to be able to introduce to all of

you Quinton Anderson Quinton is the head of engineering at the Commonwealth Bank of Australia and he's here to talk to you about CBA's journey into big data and the fantastic and innovative work

that the CBA is doing with Hadoop and friends Quinton welcome thanks Mike take it away so our journey with the dupe it kind of started with analytics being the primary use case we hide a data science

team and who do kind of came along for the journey so analytics has kind of been the key tenant from the beginning but kind of if you think about the logical endpoint of why do you do

analytics well it's there to fundamentally impact the business in some way so we started out with bullying some models competing with traditional modeling techniques and showed some

really good uplift and the next thing you need to do is operationalize analytics and to do that you've got to have all the data flowing in one place you've gotta be able to promote your

experimentation into production so on and so forth and so we landed up with our Hadoop instance in front of our warehouse and we'll end up with this really nice splits of kind of internal

bi regulatory type workloads existing on our warehouse and analytics existing on a Hadoop instance and the next logical step is really focusing on operational attributes and once you've done that the

next logical step is kind of this evolution into affecting the decisions and the way the product systems within the organization work so it doesn't help building a predictive model coming up

with some kind of a analysis and having that not affect the way customers interact with your business so the real next step in the evolution for us is taking analytics in

this truly operational form and making a entirely analytics or data driven in surprise so moving from just analysis operational analysis and into really decisioning and have that effect

frontline have that affect our channels and have that affect our core business processes so if you unpack what that looks like kind of from an architectural point of view huge a number of channels

within a bank like house front line outbound marketing lots of touch points with our customers lots of extremely rich data we're really really privileged in the kind of data that we have

internally and Hadoop kind of starts out as the one place to store all of that deep copies of it subsetting it across the warehouse but increasingly it becomes a way to move breed work led off

the the systems of record it becomes a way to do experimentation and it becomes a way to fundamentally affect the product systems and the way we interact with our customers so today interactions

what is the next best offer how we're going to interact with you via the website how we're going to interact with you via the mobile and to extend some of the more core business processes like

credit decisioning start to form part of our Hadoop ecosystem so I think the main point from my side is that there's this definite evolution from you need to have one place to manage data and you've got

data centric views of the world you step into doing analytics and once you've done that you have to be able to operationalize analytics and that's a non trivial exercise and once you've

done that you've got to be able to Arthur eight quickly put these things into production and have them actually affect your fundamental business processes and so you become an analytic

centric enterprise we're that's fantastic thank you for the detail you've given I know that one of the real advantages of this event is that we're able to bring together new folks to the

technology people are interested in working with big data with real practitioners people who've got some experience for the benefit of this group many of whom may be just getting started

love to ask you a little bit about the bank's experience in your experience in making big data happen at CBA you talked about operational I analytics providing these tools to

business users how has that gone have there been cultural challenges or changes that you folks needed to meet in order to proliferate the technology sure so I mean the obvious one is skills

you've got to find the right kind of people for this kind of technology there's lots of subtle things like the analyst within the organization have very much a traditional view on Heidi

analytics and so as you bring more advanced modeling techniques notions like is your model explainable start to become real friction points so you're getting more advanced analytics rolled

out in the group and then also the risk appetite for just the sheer amount of change we introduced was quite difficult to get across the line I'll be glad to talk to people about why this is easy

and how it goes well I think one of the really useful things that you can share as a practitioner is what hasn't gone so well have there been challenges or difficulties that you've had to overcome

in rolling out this technology across your organization sure so we initially try to roll the start in an IT organization that look like a traditional IT or enterprise IT

organization so a lot of kind of what Gardiner calls mode one style thinking infrastructure etc drives a lot of higher price points and makes it very difficult to roll out scale

infrastructure so that's one significant challenge and then we started this journey fairly early relative to a lot of our enterprise competitors so a lot of what we needed for analytics things

like ml Liberty sure didn't exist at the time so a lot of the tools that we built originally we had to build ourselves so storage layers for features for machine learning we had to build ourselves data

science pipelines concepts that exist in things like ml lab today we had to build from scratch ourselves and indeed today we we have data science pipelines used by a lot of art and lists and we're

migrating a lot of that functionality often on to things like ml loop excellent well is it Quentin thank you very much for stopping by appreciate your sharing your story thank you thank

you you've heard about one practitioners experience in rolling this technology out for real the advantage of this conference is you get to hear that story

from many of your peers here in the region and around the world Hadoop began as a very simple collection of technologies one big scale-out storage system HDFS and a single data processing

framework called MapReduce Hadoop today years later is much more interesting much more powerful much more capable while you're here today you're going to hear a lot about apache spark a powerful

new framework for working with data at scale Cloudera recognized the opportunity in the apache spark project and embraced it more than two years ago now pulling it into our distribution and

allowing users to work with data in spark as well as in MapReduce and the other engines in the system we made major forward investments in spark launching our one platform initiative an

effort to drive security scalability manageability stability into that platform we've just announced our embrace of ML lib and our extension of ML lib in the spark framework for the

benefit of our customers one of the technologies that you heard Quinton talk about for machine learning for analytic applications spark is going to be a very powerful way for you to work with data

spark is one example of a tremendous change that we've seen happen over the seven and a half or eight years that we've been around as a business spark began as HDFS and MapReduce one

processing engine one storage engine today it is a multi-engine collection of tools aimed at data analysis at scale impala for high-performance sequel analytic queries spark of course

MapReduce of course search technology built on lucene the Apache Solr cloud technology lots of different ways to process transform and analyze a single shared data set this is why Hadoop has

been so successful it's grown because the open source community has continued to innovate you can be sure that that innovation hasn't stopped spark is great Impala is great searches great MapReduce

is great but what's even more great is that as a community we get to work together and continue to make this thing better continue to allow it to take advantage of new analytic frameworks in

order to make this work we've had to drive enterprise-grade capabilities into the platform consistently and across the board and I want to talk about some of the things that we've announced in the

last month or two and that you've got an opportunity to learn about while you are here at strada hadoop world first of all we've introduced a new open source project called record service the idea

with record service is to provide a consistent security model across all of your data strong encryption good key management user and role based authentication and access control the

tools that if you're a bank or a hospital or a telecommunications provider you need to live up to the regulatory and compliance constraints that are on your business you can't put

big data in a big data platform if that platforms not secure record service is a new open source project that you can participate in that's aimed at delivering the same consistent security

no matter whether you spark your data or MapReduce your data it's a critical piece of the ecosystem and one that we're very proud to be driving forward we've also been driving innovation at

the storage layer now you've heard me talk a couple of times about HDFS this is the Hadoop distributed file system one big massive scale out store system land any kind of data in there that you

like for as long as you like very very cost-effective good fault tolerance and reliability and availability you can land data much more inexpensively than was previously the case and traditional

enterprise software and because you get all of that data in one place you can process and MapReduce you can process and analyze it in lots of ways HDFS was great for the original workload that

Google described its law large sequential transfer law uggs and web activity where you land a bunch of data and then read through that data in order doing your analyses but

not every workload looks like that the open source community created HBase a very powerful no sequel engine that runs on top of that same framework a way to deliver records in real time to support

web-based applications that need very fast record fetch put get support HBase is awesome and it meant that we could take on new workloads we're close that were not otherwise accessible HBase does

single record access HDFS does large sequential in order record access but there was an important missing part of the puzzle an important class of applications that didn't run well in the

framework we've announced a new project called kudu and we've sponsored it for incubation and the apache software foundation kudu is a third storage alternative eight space is great for

single record access HDFS is great for log structured access kudu is aimed at Internet of Things time series traditional relational aggregation queries scanning through tables

aggregating on records very very fast aggregation support and good update support it fills in a gap in the storage platform that we believe is going to open up new consumption for Hadoop and

big data we're very excited to release kudu through the asf and we invite all of you to participate in the beta to get involved in using and developing the software the third innovation that I

want to talk to you about today and again you can learn more while you're here at strata hadoop world is what we call cloudera navigator optimizer now this is new technology that we've just

released to beta in the last several weeks it is a hosted service that allows you to upload queries that you run against your data the engine analyzes those queries and helps you optimize

them so that they run extraordinarily well in a big data framework helps you lay out your data in a sensible way helps you reformulate those queries so you'll be able to run them in

of parallel it's a powerful tool for expanding the analytic surface of Apache Hadoop again we think this is going to allow Big Data technology to support a broader collection of workloads and

allow you more choices in what you do with your big data all of this is happening all of the progress that we're making is driven by the open source community and that again is why Hadoop

has been so successful a single vendor could never have created this technology and made it proliferate in the way that it has but the fact that we can tap the global skill of the entire planet in

order to build innovate deploy use this technology means that it goes faster and gets better faster than it would possibly do if it were proprietary and I want to invite all of you to join us in

that journey on behalf of Cloudera and on behalf of the broader apache software community we'd love your contribution we'd love your adoption of this technology you've got a chance to get

involved and make the ecosystem better by your participation we've been doing this for a while now I told you Clara is seven and a half years old but this is the 10th anniversary of the Apache

Hadoop project Doug cutting whom you saw just a little bit ago and Mike kafir ella created Apache Hadoop in 2005 in that 10 years as a global community we've gone from HDFS MapReduce fairly

limited very powerful but constrained set of big data capabilities to the unbelievable fast-moving ecosystem you see today including spark and Impala and all of the rest of these capabilities

with security and data governance and so on that is enormous progress for enterprise software to have made in 10 years please join me in congratulating Doug and Mike and the global Hadoop

community on that 10th anniversary and happy birthday hadoo we look forward to working with all of you in the months and years to come let me thank you for attending and let me

encourage you to use your time at the conference to see some of the fantastic talks that have been arranged the great submissions that have come in and network with one another build your

networks and build your best practices thank you very much and welcome to Singapore