The O’Reilly Jupyter Pop-up

About this course

my name is ing espaço said we we're from the spark Technology Center we just announced that rebranding so now we are called code

eight which is a sorry stuff up to here can you hear better okay so we just got renamed so we're now being called code eight which is a Center for open source data AI technologists so we are

basically expanding the scope and doing more things more towards AI think that is a trend that we are seeing in the industry so that's kind of like where we're going I've been working with open

source for a lots of years more than ten I guess and have been doing a lot of like a patch of stuff and recently might be any more powers like data science infrastructure and raise it on Jupiter

if you need to connect me here are all the information it's gonna be available just reach out hi my name is Kevin Bates and I'm I – a colleague of Luciano's at the spark Technology Center or the code

a location I've been in the enterprise software space for my whole career over 30 years however I'm relatively new to the open source and Jupiter frameworks so this is a good learning experience

for me but also I feel like I can add some insight from an enterprise perspective to this environment so again there's my contact information and I'll hand it back to Luciano and talk to you

in it so I think we're gonna divide this talk like in three pieces we're just gonna go through a couple of slides where we just want to make sure everybody get on the same page and know

most of you knows Jupiter but because we're gonna be talking about the infrastructure we're gonna go a little bit of more details on that then we're gonna talk about enterprise gateway and

then the actual demo that we're gonna do and how we build that for those that are new to Jupiter I believe very little guys but what what it Jupiter brings to you is like an interactive development

environment that allows you to start adding your code and adding surrounding information to the basically we can document be very interactive enables you to have

plottings of your data so you can have insights and a lot of tools all available on that environment for you to start building your algorithms your applications and getting insights of

data when we look into the architecture and how all of that work we see mainly three pieces here you have your browser where you actually see the UI that is being served by what we call the Jupiter

server but then when you try to execute code we have what we call kernels and kernels is an abstraction which is responsible for executing code and gives you kind of like a understanding of the

language that you're using so you have a kernel for Python you have a kernel for Scala and and so forth that we have I think over 100 kernels available today and they keep growing in terms of like

how they communicate this is kind of like a more breakdown of the architecture you have the notebook server where it takes care of the communication with the kernels you have

the kernels running on a separate process and they will be communicating by what we call the socket quartz which part of like the kind of like the protocol that is using for communication

we have five ports that are use it and each one have its own kind of like responsibilities and we're going to be talking about some of those in the next few slides for doing the communication

they need to kind of like know what part you connect you in couple of other informations on how like to sign those messages and things like that and Jupiter has this notion of like

connection profile that will have the information necessarily so that the server can connect you to the car to the to the kernel most of the time when you look into Jupiter in itself they are

kind of like grounded into the same machine local process but we're gonna talk we're gonna talk a little bit about also having the ability to have remote car

nose so I mentioned about the ports here are kind of like a little bit of what goes on on each port so you have like a shell port where you start sending requesters

I hope hope is kind of like to communicate back some of the results you have the sender input like if you need to pass input for the colonel's control is most likely for sending control

commands like if you want to do a shutdown interrupts and things like that and heartbeat monitors if you have your kind of your kernels still up otherwise Jupiter will try to restart the kernel

and things like that and when you actually execute try to execute a cell kind of like this is what you're gonna see you have a flow of message going on but the only thing I want to mention

here where we're gonna use later on is you can have two types of results coming back from your request so if you're doing like a computation like for example one plus one or something like

that you get a result back but if you're doing kind of like a streaming where you're doing like a print or you have like a large data set and you just streaming some data that is gonna come

like in multiple batches as what we call is streaming and that makes a little bit of difference when we programmatically trying to communicate and talk to the kernel which we'll see a little bit

later on with that let me pastor Kevin he's going to talk a little bit of like a enterprise gateway thanks Luciano so I wanted to come back to this slide and point a couple things out one is when

we're doing analytical processing especially in a spark environment the kernel Python process or the kernel process is what's known as the spark driver and spark drivers have a tendency

to consume lots of resources in addition the the current design of this architecture is that it's geared toward a single server environment so you got your notebook server and your kernels

all running located that presents some challenges in an enterprise-wide space where you want to have lots of Colonels running for lots of users and so we kind of get

bogged down especially if it's doing heavy analytical processing it can it can kind of take your server consume too many resources on the server so one of the things we wanted to do with

enterprise gateway is and we'll get to what enterprise gateway really is but we found that in IBM many of the product groups were embedding SPARC on yarn so there was a big embracement of SPARC

across the product space at IBM and they all wanted to use two of their notebooks in this in this space and so that that presents the challenges and these are enterprise applications so you have your

your enterprise tenets of multi-tenancy and scalability and security to be addressed and we felt that the way things were right now we we couldn't really address all those aspects

appropriately some groups tried to scale up their their notebook server to support multiple notebooks but then you get into multi-tenancy issues others try to launch per user notebook servers but

then you get into scalability issues with that but one thing that resonated across the administrators and developers at IBM was Jupiter kernel gateway and what jupiter kernel gateway brings to

the table is this notion of a bring your own notebook and so you can essentially remote your notebooks and get the notebook files that the data scientists are using off of your enterprise servers

and that that's a that's a pretty compelling thing to think about so it solves some of the issue you get some of this stuff off and you let the commodity server drive the notebook and they do

this through the the WebSocket personality and so by by adding the extension this.mb 2kg extension which is notebook the kernel gateway into your notebook or jupiter lab it all it all

just works you're essentially remote in your notebook to your enterprise servers so so that's one thing that we felt we needed to build upon so we did some prototyping with Jupiter

Colonel gateway and you know yeah we were able to push our notebooks off off the servers we have Bob in blue and Alison red but we still ran into issues we still didn't really solve the the

real enterprise issues that were happening and as you can see here when you go to hit kernel gateway your kernels are still co-located with the with the process that's launching the

kernels and so your spark drivers are still consuming the resources of a single server yeah you get your workers out in your in your garden cluster but really the the heavyweight resources are

still pinned that one server and and that wasn't a good thing for us to deal with in the enterprise the other thing is your spark jobs are all running as the the gateway user the service user

that launches that the Gateway is associated with and and that's not a good thing either you don't have any sandbox control Bob could go bang into the shell and see Alice's processes and

likewise for Alice and it just wasn't a good thing to even you know send out in an enterprise capacity so as you can see here this isn't a scalability diagram that you'd want to take home to your

mother this just isn't gonna fly you're completely gated by the kernel gateway server as to how many kernels you can launch that's not going to work no bueno so we set out to build enterprise

gateway and so we had some initial goals the three being to address scalability multi-tenancy and security and so we knew we wanted to leverage the the capacity of the resource managers so you

know we focused on yarn initially and by launching the kernel as a yarn cluster application instead of a client mode we let yarn determine where that kernel is going to land all right so it now the

kernel is running off on some worker nodes somewhere and we've established a way to communicate back with that kernel so now we're talking to a colonel and remote in a remote format so as we we

implemented this we knew we wanted to plug in other resource managers along the way and and we've got a couple more that are coming soon that we're really excited about and in doing so we've

essentially built a general framework for remoting kernels and I think this is going to be very helpful in the enterprise space now by leveraging yarn one of the primary ways security is

addressed in these yarn clusters is through Kerberos and so by kerberized in your yarn cluster we can take advantage of sparks proxy user flag and actually launch the kernels in impersonation

so we're impersonating the users for each of the kernels that's establishing a isolation from a user level but also because HDFS is ingrained in this ecosystem we can also associate each of

the kernels with an HDFS home folding and we get the file downloads and stuff per HDFS and some sandboxing capabilities with that the user ID now is alice and for alice's kernels and

it's Bob for Bob's kernels and likewise if they were to do a batch job they would also see in the yarn management UI they would see their jobs associated with their user IDs so administrators

are able to decipher whose job is what and whose job has taken too many resources and needs to be terminated in doing so we because the kernels are now remote we needed to secure that

communication between the kernel and the in the Gateway server so we do that as well as any front in security concerns that come in from the front side so as you can see we're built right on top of

the Jupiter stack so we we derived directly from kernel gateway and we we kind of reached down and and override some of the kernel management functionality in both Jupiter client and

in notebook so so with that we're all all part of the same group and we've essentially enterprise gateway is essentially a remote kernel manager it's really the core of the code that's in

enterprise gateway with this plug ability to plug in different resource managers so here's that diagram before but now we we have enterprise gateway running and as you can see that we've

still got Bob and Alice off remoting using MB 2kg to get to the server but now the kernels are running across the worker nodes and running as the users that they are intended to be running as

so we get the sandboxing we get the user isolation and we've removed the the heavy duty spark driver resources off of our gateway servers so it can be lessened from a resource perspective and

you can focus on the the worker nodes as being your resources so here's that same scalability graph it's a little bit better if we needed to add more kernels we can just add more nodes and we should

expect to see that kind of increase in in scalability this is one that you you could take home to your mom if you wanted to so here's that original diagram that I started to talk with but

we've done a couple things here we've inserted enterprise gateway in the middle here and then I've color-coded where you could have division of nodes so we could have our we could have our

kernels running on a separate node Enterprise gateways in the middle and then your notebook server is really off on your desktop now and your enterprise machines are what are servicing the

kernels which is where most of the work is happening so with that I want to hand it back over to Luciano he's going to talk about the analytical micro service that we developed for this presentation

the moment you've been waiting for thanks Kevin and I think you guys now have another standing of like a lot of like the things that we're going to be using by

building this enterprise gateway the the socket personality give us a little bit simplified environment for managing the kernels it also gives us a scalability so we can run more kernels machining in

a distributed environment but let's take a look at like a what is all about building the micro service so let's start with kind of like a use case that we kind of like took as an example we

wanted to work on like a real data so we were looking for kind of like what dataset we could use one of the things that we found out is like the there is this Yelp data set from Cagle which

basically has the business has like reviews from those businesses and a couple of more information so we decided to do some example with that we wanted to calculate the sentiment so we use

I've seen I don't know if you guys are familiar with that but there is a library in Python that can help us calculating the sentiment and Athene is kind of like I think a big dictionary

with kind of like scores for those words in terms of like sentiment and that is used to calculate the score for for kind of like what we are using we are using PI spark for like a training and doing

data manipulation and things like that and we are using the Jupiter kernels to really like interface and give us the flexibility to easily connect back to a cluster in a very simplified way in

terms of like components what you're gonna see in the code we have kind of like the front end the front round is gonna be any client that can talk rest you can be using as a part of your like

solution where you're building aggregating multiple micro services you can just may be connecting from a curl command or you name it like a web app or a mobile app so we have a sentiment

resource we in this case we are leveraging a flask to handle some of the kind of like research exposition for us we they've created a sentiment provider that kind of like goes into just like

being able to the logic around the sentiment and we have the we create a quick utility that wrap all that communications so like knowing how to start how to to stop kernels and things

like that and we go into more details in a little while but then in in the on the right side kind of like in the server side will be launching kernels remotely and all in kind of like a spark cluster

here so let's take a quick look into kind of like initial things so the first thing we want to be familiar with the data so nothing better than using Jupiter notebooks so we use that just

you kind of like load the data play a little bit with that do any kind of like manipulations that we need it or just you get familiar with the data and so this is what kind of like the only very

little things that we did one of the things we just for the purpose of the demo we filtered a little bit the size of like the scope of the data so it could be easier for us to like be

interacting and not having to wait a little while to to get this data process doing kind of like development so the sentiment REST API we can take a look on the right side you'll see mostly of the

code here you can see it's very simple base it on flask flask restful in what you need is you define like a sentiment resource class and you have like what do you want to do when you respond to when

you get a a call request on that URI that you're exposing and that logic is incorporated there and then you just don't flash like how you want to expose that so the REST API is basically

sentiment slash and you can pass a parameter which is kind of like which customer or which business you want to retrieve kind of like the sentiment for that

and when we are kind of like booty booty strap the server we kind of do kind of like the start of the kernel and stop sorry just start in terms of like the

sentiment provider that encapsulates all interactions of the kernel so he knows like what in what commands he needs to send for the kernel imagine this like things will go in the cells of the

notebook so the the logic behind of like how to calculate the sentiment and the I think the most important thing here you will see kind of like towards the bottom is mostly like how we actually integrate

the the spark to calculate on the fly kind of like the the new column from the data frame and put the sentiment for that review there and we actually doing this when you do the invocation so you

can get like the greatest latest and greatest information available we also created like a kind of like two classes for help in dealing with the kernel we have a kernel launcher that has methods

for starting and stop the kernel the way you do this with like the api's that are exposed into j jupiter karna gateway and enterprise gateway basically there is a call a rest call that you can invoke and

that you you will start the kernel for you and then there is another one that you base it on the kernel idea will stop that current for you and we have encapsulated that and what it returns

back in this case it's kind of like a a kernel object and the kernel object then knows how to send code it will you basically call evolve into a piece of code that you have like let's say in

your cell and we send that to the kernel we then look into all those socket message that we are receiving back so that we a granade the results that are coming and

and send that as a return and that gets flown all the way to to kind of like you know the rest that is the whoever is consuming that rest api so right now I'm just starting

to research this is basically my logs you can see that we are having the kernel started this is on the enterprise gateway server this blue yeah and we see here on my console here what codes we

are sending and basically when we start up in the resource you can see that I'm basically loading the data so I can then have the sentiment escort and if we go now to our kind of like front and UI so

here is kind of like a just a very simple web page and what I'm doing on each one is basically each of these links invoke one of the rest calls for a given business and you will see that

then data is being sent there we are doing the lookup for that business at that moment and also calculating the sentiment and when you get back so we we got the business details so we know like

which which business you requested and we get for each sentiment we get a score from – of like what actually the sentiment is and we wanted to make this kind of like just a real scenario mostly

like as an example I think the idea is like you can start building adding kind of like analytics capabilities to your application in a programmatic way and exposing that as rest or just if you're

just building an application integrating that and leveraging all the power of kernel or the power of like the Jupiter platform to being doing some of these interactive development and gaining

analytics and insights from your data I think that's most love of what we wanted to get with you guys today let me just go back to slides where we have some additional information so further

readings here just if you guys are wanted to get more details so two things if you for example wanna have the requirement to connect directly to the kernel there is a couple of other things

that you can use here you can use Jupiter client but also I've done actually client which in the past which is very similar to what we did today is just that instead of like going through

the personality we actually go directly to the kernel we've talking with directory of the five courts also we have another type of personality it's positive through jkg which is kind of

the HTTP personality what that gives you it's kind of like launch the gateway into like a single notebook mode and expose a rest your eye for kind of like each cell of your notebook so that's

probably another way that you can think about exposing some of your analytics and we are open source project we welcome contributions we welcome questions and anything here are where we

are we are currently a Jupiter incubator project eagerly looking to get to Jupiter project documentation czar here the link for all the code that I show today and we look look for us we

probably getting a really as out sometime this week we were hoping that we're gonna be doing that before the the conference but either the release or all of them working so of the forever but

yeah thanks and any questions we are here to answer those