Introduction to TensorFlow (Cloud Next '18)

[MUSIC PLAYING] LAURENCE MORONEY: All righty Hi, everybody, and thanks for coming to this session My name is Laurence Moroney I work as a developer advocate on the Google Brain team I’m primarily focused on TensorFlow, machine learning, and AI I’ve just come from another session, so I’m doing back-to-back sessions And if you were at the previous session, some of this material you probably have seen already So forgive me for the duplicate And because I know there’s a lot of folks waiting outside– so if we can kind of shove up And maybe if there’s some room for the folks to come in– or if you’re outside and you can hear this, we’re repeating this session on Thursday morning So take a look, and we’ll see And if you want to hear the session, there’s a repeat of it being done on Thursday So I’m here today to talk a little bit about TensorFlow and to introduce you to TensorFlow But I always like to begin with the question, what is AI anyway, right? You know, you turn on the news, you open a newspaper, you browse the web, and you see people talking about AI And you know, you probably see pictures like these, of CGI androids, or predicting the future, or maybe things that are even worse And you read a lot of doom and gloom scaling from AI is taking our jobs to AI is the next Arnold Schwarzenegger movie And it’s all of these things And I find that maybe we, as an industry, have done a pretty bad job of explaining what AI actually is We talk a lot about what it does But we rarely talk about what it actually is, and particularly from a programming perspective So actually, quick question– how many people here are developers who write code? OK, a lot of you, cool So you’re the right audience So what is AI anyway? I like to talk about it in terms of writing code And I like to talk about machine learning in particular in terms of writing code So say, for example, if I wanted to write an application which is a classifier of cats versus dogs OK, how would you do that, being a coder? We all write code that’s like, if this, then that, while this, do that, all these kind of things But how would you do that for a classifier of cats versus dogs? Like, say I gave it this image And this image is– as a human being, you’re going to take a look at that, and you say, that’s a cat Why do you say that’s a cat? There’s probably some features on it that you recognize that are very catlike, like the pointy ears, right? So you go, OK You look at that, and you say that it’s a cat And you think about that as a programmer You might say, if has pointy ears, then cat But in order to get that, you have to process the image somehow, find where the ears are in the image somehow, and determine that they’re pointy somehow It’s very complex, very hard to do But then I’d show you this image OK, is this a cat? It’s not, right? It’s a dog But it’s got pointy ears And as a result, we go, huh, OK, I can’t just say if has pointy ears, then cat And then I show this image And it’s like, OK, this has got really pointy ears, OK? Now, is that a cat, or is it some kind of weird science-fiction creature? I don’t know It’s got very strange ears to me And so I go, OK, so there’s where I’m confused as a developer, OK? It’s going to be very hard for me to write all the image processing and write all the if-thens and all these kind of things to figure out, OK, which is a dog and which is a cat And then I go, OK, there’s another feature of a dog that we all know, and it’s like this, all right? So it’s like, if it’s got a big, long, floppy tongue like that, it’s probably a dog And in my case, I usually write the rule, like, if it’s currently plotting your murder, then it’s a cat And if it will love you for the rest of your life, then it’s likely to be a dog And then when I’m looking at image like this and trying to classify them, then I come up with a beauty like this, which, to me, is this is what happens to a mogwai when you feed them after midnight But apparently it’s a dog as well But as a programmer, it’s really, really hard for me to write some kind of an application that would determine this OK, not everybody needs to write an application of cats versus dogs But maybe you do need to write some kind of classification application– spam versus not spam, hot dog versus not hot dog for “Silicon Valley” fans, and that kind of thing And how do you write these classifiers? And in traditional code, that’s very, very difficult Because traditional code looks like this, right? You’ve got your classical programming You input the rules to that And your rules are the code that we all make our bread and butter off of, you know, writing Java, or C++, or C#, or Kotlin, or Objective-C, or Swift, or how many other languages there are out there You know, that’s the rules that we write And usually, in classical programming, we write rules, and we have data And out of that comes answers So we write an application with lots of rules in it to say, it’s pointy ears, it’s a cat– you know, those kind of things But the idea is that, then, there are a number of scenarios that are really easy to do with this, but there are scenarios that are practically impossible to do with this You might be able to write a cats versus dog classifier But think about the amount of code that you’d have to write, the amount of code that you’d have to maintain And then suddenly, you get another picture of a creature, and you don’t know if it’s a cat or a dog You’d probably have to rewrite all your code again And it becomes infeasible and difficult to do that Now, one of my favorite things, I wear a smartwatch And on my smartwatch, it has activity detection It knows when I’m running, it knows when I’m walking, it knows when I’m swimming– or in my case,

sinking And it knows all of these things based on– its being trained, based on data, to say– you know, the people who built the smartwatch took thousands and thousands of people and said, let’s make them walk and say this is what it looks like when you’re walking when you’re wearing a watch Let’s make them run This is what it looks like when you’re running when you’re wearing a watch Let’s make them swim This is what it looks like when you’re swimming when you’re wearing a watch, and to actually train a system off of that So it changes the rules completely So instead you inputting rules, and inputting data, and getting out answers, the concept of machine learning, and the whole revolution of machine learning, is very simply, you input answers, and you input data, and you get out the rules So the idea behind this one is you get lots and lots of data to show something So for example, I get thousands and thousands of images of cats And I say, this is what a cat looks like So the data are the images The answer is me saying this what a cat looks like Same for dogs, same for mogwai, same for all these other things Or if I’m building a watch, I get lots and lots of data of people running, I get lots and lots of data of people walking, and all of these things And I train it So as a result, it’s opened up all these new scenarios for types of applications that you can build that would be very difficult to do with if-then kind of rules You know, some of the early spam detectors used a lot of if-then If the body contains these particular words, it’s probably a spam If it’s sent to you Bcc, it might be a spam If it’s sent from somebody in your contact list, it’s probably not a spam– all of these kind of things And you could kind of get a good spam detector with if-then rules But what if– many of us receive thousands of emails a day And we categorize spam, not spam Well, what if that’s us giving the answers? And then a machine learning algorithm could take the data, this is what a spam looks like, this is what a not spam looks like And as a result, you end up getting vastly improved spam filters, because they’re trained in that way So then the question becomes, well, how do you train them? So just show the diagram again first So it’s like, classical programming, machine learning– very different We’re just flipping the axes And the nice thing about flipping those axes is that it’s opening up new scenarios that I thought– and I think the industry has shown– were previously close to being impossible So I like to use an example And it’s one thing that I’ve been working on since I started on machine learning And I’ve only been working on this at Google for about six months But the first project that I took on when I was doing it was to take a look at breast cancer data Now if you think about breast cancer data– and in this case, the data set that I was looking at for breast cancer data had about 30 different data points for cells in a biopsy around breast cancer And these data points would be things, for example, such as the size of the cell, the density of the cell, the shape of it, things like that And so with these 30 different data points, you could conceivably write lots of if-thens You know, if it’s above a certain size, if it’s a certain density, if it’s a certain shape, then there’s an increased likelihood of it being a breast cancer cell And you’d do something that looks like this, right? So the red ones on the right are malignant Big ones are malignant, green ones are benign Wouldn’t it be really nice if it was that easy? You know, if x is greater than 30, then it’s malignant Else, it’s benign But unfortunately, it’s not that easy, right? Because then, if you add a second axis to it, it kind of looks like this And you’ve got a little bit of overlap You can’t just draw a straight line down the middle and say if x is greater than 30 But you could probably draw a line like this and then say, OK, the ones above and to the right of the line are malignant And the ones below and to the left of the line are benign And if you remember your high school geometry, you know the equation of a line– y equals mx plus c And if you can figure out what the m is and you can figure out what the c is, then you can start saying, ah, now maybe I can start making a decision about what’s malignant and what’s benign This is great That line, in machine learning parlance, is called a decision boundary But this is the reality of the data This is what it looks like It’s not really that simple So I’ve plotted this This is only three dimensions that I’m plotting, instead of– there were about 30 in the data set And the three dimensions that I’m plotting are the two that I showed a moment ago, plus, a third one is the actual size of the cell So if you look closely, the size of the blob is proportional to the size of the cell So you see the ones over on the right are really big And the ones over on the left are really small So even just plotting in three dimensions of data, then we’ve got a lot of mixed up data right in the middle there So how do I draw a decision boundary? Do I draw it something like this, where most of the ones on the right are red and most of the ones on the left are green? You can’t really do that with something as important as cancer, right? You just can’t depend on that type of thing And also, if I zoom in, and I start looking– OK, it was the axes here, with the smoothness, the compactness, and the concavity You can see, even when I zoom in, it’s not really that simple, that there are reds amongst the greens, and there are greens amongst the reds It becomes very, very difficult for me to make a simple decision with an if this, then that type of algorithm So if I go back to this, and I start thinking about this in machine learning terms instead of traditional programming terms– and I’m going to say, OK, the simplest example– and I know this isn’t going to work, because I’m only training on one feature

So when I said– when you do machine learning, remember, you pass in the data, you pass in the answers So when I’m talking about features, this is one of the aspects of the cell I’m just going to pick one of them In this case, it was the size And if I only train on one, I know it’s going to fail But I just want to use this as an example of how the process works So here it’s a case of– OK, in the case of a neural network, I’m going to say– I have this if-then If it’s greater than 30, it was malignant If it’s less than 30, then it’s benign Well, what if I created what I call a layer in a neural network? And the idea of that layer is that it calculates something called a weight And then the weight– if remember y equals mx plus c, the weight is the m It’s the slope of the line But we only have one axis here, so it’s even simpler So we’d say, OK, when we think about this as a function, well, we saw 18 was benign, 17 was benign, 13 was benign, 31, 35, and 39 were malignant, it’s really easy for me to say, OK, in my layer, I’m going to call a function called ReLU ReLU is a common function used in machine learning And it basically is the same as writing the code, if x is greater than 0, then the output is x Otherwise the output is 0, OK? The idea is it just allows you to filter out negative values So I’m going to say, here– I’m going to subtract 30 from the radius and get the ReLU of that Greater than 0 is true 0 is false And now I can start going, ah, I now have made this decision I’ve created a really simple network, which isn’t really a neural network at all But I’m able to say the weight for this is 1, because it’s going to work all the time So the answer is going to be 1 times the ReLU of radius minus 30 This is all very mathy stuff that goes on in a neural network And fortunately, you’re not going to need to know most of this But I want to show what’s happening under the hood So then if I go, OK, so my second one is– here it is in two axes And my decision boundary is going to look like this, as we showed earlier And we have y equals mx plus c So in a neural network parlance, what’s that going to look like? Well, I got two layers And these two layers, I’m going to have one figure out my m, which is the slope of the line And I’m going to have one figure out my c And my output’s going to be y equals mx plus c So when I can figure out m, and I can figure out the c, or if a machine can learn what the m is, and a machine can learn what the c is, now the machine has that decision boundary that it’s figured out for itself And me, as a programmer, I don’t need to write any code for that It’s figured it out for me So now I can go, OK, what if I initialize it with a couple of random values, something like this where m equals 0 and c equals 5 We can visualize that the boundary looks like this But it’s not a very good one, right? There’s a lot of benign cells above the line that are showing up as malignant It’s not horrible Because you’re going to tell somebody that they have cancer when they don’t, which is not good But it’s not as horrible as telling them that they don’t have it when they do But we go, OK, we can learn from this Let’s see if we can do better So what happens then in your neural network is it’s going to say, OK, I’m going to take that value, draw that line with those initial values I’m going to calculate which ones are right and which ones were wrong And based off of that, I can figure out what my loss is And then I’ll have a function called a loss optimizer And all the loss optimizer is is there is a bunch of built in mathematical functions that say, you know what, when you lose this much, try tweaking your m to this and your c to that And then see how much you lose And if you lose less, we’re heading in the right direction If you lose more, we’re heading in the wrong direction, so we’ll tweak them the other way So those kind of things TensorFlow does for you automatically, all that optimization, all those loss functions So typically I’ll get my data, and I’ll test that And I know the answers, already, to this data So I’ll keep going until have a very low loss So when I come up with an m and a c that works for all of my data, I now know that I’ve trained a neural network to make a decision based on the boundary So for example, maybe it goes to this and says, OK, I’m going to try this value And this is a terrible value So even though my loss value– my benigns are great I have a couple of malignants here, but I’m telling the person they don’t have cancer So that’s if I try it with m equals 0 an c equals 12 And you just keep trying And the idea of a neural network, that’s what it is The process of training that we call it is just trying, and trying, and trying different values, calculating the loss each time, figuring out what you’re going to do to reduce the loss the next time, and then try that value And keep going until you get a decision boundary which gives you a very low loss or a very high result Now, in this case, obviously, with two dimensions, we say, hey, when m equals minus 1 and c equals 20, we’ve got a great decision boundary But that was only two axes And of course, the real data looks like this But the real data looks like this In higher-order math, instead of drawing a decision boundary like this, which is bad, the reality is it’s possible for you to draw a decision boundary through multiple higher-level dimensions, like, in this case, five dimensions of five features But if you’ve got 30 features, it’s possible for you to actually draw a line across 30 dimensions with an equation that’s very similar to doing it in two dimensions that we had here And so I’m not a mathematician I can’t figure out that math for myself But fortunately, the process is exactly the same as it was for the two dimensions that I showed earlier on

So we have your layers You have your weights And when you think about your weights on those layers, you’re just going to figure out what weight for this value, what weight for this value, what weight for this value will give me my output I’ll do my loss optimization on that And in this case, I’ve got three layers I could have 30 layers, or I could have more, those kind of things And as a result, it will– the loss optimizer feeds back to the weights That feeds back through And that cyclical process, as I’ve mentioned, is the process of training a neural network So in reality, this ends up looking like– in the case of the breast cancer, you pull a biopsy on a cell A doctor will pull the biopsy on the cell And then that biopsy is used The cell will be explored under a microscope And then the features of that cell will be extracted You know, we’re going to take a look at the radius of the cell, we’re going to take a look at the concavity of the cell, going to take a look at the size of the nucleus, the color, things like that And those features are then put into a database And then those features from that database are then extracted into, and fed into, a neural network We have an input layer in the neural network We’ve got a number of neurons We’ve got fancy diagrams like this But all that’s going on in that network is that constant cycle going on, going on, going on, trying to find the weights that will give you that line that you draw through a multi-dimensional space that gives you the optimum amount of values of one side on the line and the optimum amount of values on the other side of the line Now I do want to– I always like to stress here, and then ahead of that, you’ll have two neurons And the neurons will be an estimation of whether the cell is malignant or an estimation if the cell is benign So it might– you give it a particular cell, and it might say, hey, this cell, there’s a 5% chance it’s malignant There’s a 95% chance it’s benign I almost said “divine” there I don’t know why And so the idea, then, is that this is something that we want to build as a tool that will help doctors, not replace doctors So the idea is, think about the amount of time that it would take to explore this data to take a look at something if you were doing it manually But if you get enough samples, and you’ve trained those samples against ones that have been labeled by a doctor that a doctor says, this is what a bad cell looks like, this is what a good cell looks like, then you can speed up diagnosis So that’s the idea of how a neural network actually works And how that fits in, then, to the idea of artificial intelligence is that, if you think about an artificially intelligent application is an application that does everything It’s your user interface It’s your if-then code But it’s also taking advantage of machine learning and may have a number of different models of machine learning, one for analyzing the breast cancer, one for maybe– I don’t know– analyzing the patient’s ability to pay– I don’t know– all of these kind of models So it’s hybrid code where you’ve built models for making a decision combined with code that you’ve written yourself is your overall artificially intelligent application And then there’s a process called deep learning, which is part of machine learning And that’s when you start really getting into the deep neural networks for classifying and for understanding things My breast cancer one could be either a traditional machine learning or a deep learning I haven’t really gone that deep in understanding the network behind that yet I’ve only really just started working on it But I like to use that as an example to illustrate how the process works So all of this is available through what we call TensorFlow TensorFlow is your programming API that allows you to do these things This allows you to design your neural network This allows you to take your data and structure it so it can be run through a neural network And this allows you– all those things that I mentioned, like those mathematical functions for loss optimization and all those kind of things, are built into TensorFlow So you don’t have to be a mathematician to figure out how to do them yourself It’s a case of, you design your neural network in TensorFlow And then once you’ve designed your neural network, you run your code through your neural network, and you hope that it works A lot of it, of course, you’re going to have to optimize your network And it’s a lot of– just like if you’re writing Java code or C++ code, I would say, OK, for a particular task, what’s better, a for loop or a while loop Sometimes you’ve got to write both and figure out which It’s the same kind of thing with this You’re going to design your neural network based on your input data, train it with your data And then you can optimize, and optimize, and optimize to see what works So the goal is that we’ve been building this and open sourcing this to be a machine learning platform that’s for everybody to solve real-world problems So we started using machine learning many, many years ago at Google We’ve been constantly building, constantly refining it I mean there’s one thing– I joined Google five years ago There’s one thing that they told me when I first joined that I thought it was really funny, particularly at the time And that was that we have cafes in Google that serve food to Googlers And they actually have been using machine learning for many years based on people using the cafes, and based on wastage of food, and that kind of thing to predict how much food to cook And so as a result, they will overall reduce the wastage of food And little things like that have been going on internally for many, many, many years And it’s all of this learning, all of this code that we’ve been writing for our own systems is what we’ve taken and brought to the world as TensorFlow, which is this machine learning platform for everybody

So as a programmer, the steps to build a machine learning model are a little bit different than what you’re used to As a programmer, you might get a design document or a specification And then you decompose that And then you write some code around that And maybe you write some throwaway prototyping code And then that throwaway prototyping code, you’ll learn something from that And you keep going You keep iterating With machine learning, it’s different But in some ways, it’s very similar from the iterating part So the first thing that you need to do is gather your data, OK? You’ve got lots of different data sources that your data might be in There’s lots of different ways that your data might be represented Let me give an example Like in the breast cancer that I’ve been talking about, there’s the size of the cell Say that size of the cell might be a value from 10 to 30, an integer or a long value from 10 to 30 That’s probably not very good for me to train a neural network I might want to normalize that data where the smallest cell is a 0 and the biggest one is a 1 so I’ve got a span of data And then I can be a little bit more intelligent, that I know the ones towards the bigger end of the span are more likely to be malignant, and the ones towards the smaller end are more likely to be benign– things like that that you’ve got to gather your data And then you’ve got to ingest that data into the programming language that you’re using Most people, when building with TensorFlow, use Python But we’ve been expanding TensorFlow recently We’ve added Swift, and we’ve added JavaScript recently And so think about the data structures and the ways of representing data that are available in those languages And you have to ingest it into that But even further on, one extra step that you have to take as a TensorFlow developer is we’ve got something called tf.data where the idea is that, say, for example, if you’re a Python developer and you are using NumPy, that’s great for you to be able to train your models off of, but it’s not optimal It’s not optimized If you then take that and convert that into tf.data, which then convert that into tensors– and the idea of TensorFlow is the concept tensors are a mathematical construct And it’s all about flowing those tensors through your neural network, doing those calculations, and being optimized So your first step, gathering your data, you take it, you stick it into your programming language, and then you optimize that using tf.data And so then, the next part is processing the data, choosing your features, and choosing your labels In the case of the breast cancer that I’ve been speaking about, there were 30 features I was able to train a network that was just as good at predicting a malignant or a benign cell using only six features as I had from one that was using 30 And the one that used those six features took a fraction of the time to train To give another example, there’s a common data set out there for house prices in California that a lot of people use for learning in machine learning And one thing I learned when I was playing with that one that I thought was really interesting was that, usually when you’re buying a house, the number one thing that dictates the price is the number of bedrooms, right? A four-bedroom house costs more than a three-bedroom, costs more than a two-bedroom typically But when training a neural network to actually predict house prices, the number of bedrooms had no impact I trained a model, and I got a particular set of accuracy I trained it by removing the number of bedrooms I got the same accuracy, but it was quicker to train Because often, sometimes, one piece of data is hidden in another In this case, the number of bedrooms is hidden in the size of the house A four-bedroom house is bigger than a three-bedrooms is bigger than a two-bedroom And it was the size of the house that was the bigger signal in determining the price of the house So things like that is processing data, formulating it to optimize for ML– so a lot of trial and error And that’s really what we call “data science.” Then of course, there’s training your model That’s where you’re going to be writing your TensorFlow code I’ll show some in a moment, where you’re designing a network, you’re experimenting with it, and you’re optimizing it for accuracy And then finally, and probably most importantly, is where do you use your model So there are a number of different places you can use it There’s a serving environment We’ve got something called TensorFlow Lite that will allow you to run it on a mobile I’ll demo that in a moment And then it’s a course of continually optimizing, improving, and updating your code So let’s take a quick look, first, at gathering data, if we can switch to the demo machine Sorry, before we switch to the demo machine– sorry, go back to the slides for a second My screensaver’s there, my beautiful screensaver Can we go back to the slides? So a very famous detective, Sherlock Holmes, said “It’s a capital mistake to theorize before one has data.” And unfortunately, he’s fictional, but he’s also right And so getting good data and training your system with good data is very, very important So gathering data, it’s all about data science Data is the lifeblood of machine learning, I showed the diagram earlier on where it’s data in, answers in, rules out So you really– the job role of data scientist has grown massively in importance in the last few years, and this is the prime reason why Because you can write crap code that will compile and still execute, right? A programmer– the really good programmer is the one who writes the really good code that becomes easy to maintain, that executes quickly, and doesn’t have bugs in production It’s the same kind of thing in data It’s gathering your data You can train a neural network that’s going to work, but you could train it on all the wrong data And training could take a long time and be very expensive The model may not be optimized So it ends up being, data is the new code in many ways So one of the things that you need to help you get your data ready, of course, is good tools

So one of the tools that we have in Cloud that I like to use is BigQuery So the nice thing with BigQuery is it gives you this massive data warehouse if you’ve got tons and tons of data In my breast cancer example, I only had a few hundred records So I didn’t need that But say you’re training off of millions of records BigQuery is really great for that Because not only does it store it in a way that’s easily accessible and it’s quick for you to access, it also allows you to do some analysis on your data and then download it, for example, as something like a CSV that you can then turn into a training model And right now, even though CSV is quite primitive, it’s the number one data format that’s used in a lot of machine learning So now can we switch the demo machine please? And just to give a quick look at what BigQuery looks like, I have a query running here So there’s a number of public data sets that we’ve put into BigQuery And so these public data sets– and one of the one that I’ve been playing with and I’ve been having a lot of fun with is there’s a baseball data set So I love to play fantasy baseball And I’m now giving away all of my secrets of how I beat all my friends at fantasy baseball Because I’ve been doing a lot of machine learning stuff out of the baseball data set that’s in BigQuery And shh, don’t tell them And so then, the idea is, like, BigQuery, this is a browser-based interface I’m doing it as a live demo now So for example here, I’m going to search for pitchers And I’m going to search for my third-favorite baseball player Can anybody guess who it is from his name, for pitcher first name equals Felix? I’m a Seattle guy That’ll give you a clue But I run a query And there you can say, it just took like 1.2 seconds It’s gone through the entire data set This is just a standard query It’s a SQL query But it’s given me the data from that data set And because it’s SQL, I can start doing interesting things like that, like normalizing it or whatever Or I could just take the data, and I can just download it and then do all the fun normalization stuff using tools, or directly in Python or whatever language I’m using, to put it into TensorFlow So can we switch back to the slides, please? So the next step, then, is processing your data And processing your data is a little bit different than it would typically be because of how you sometimes need to encode your data to make it understood by a neural network So the first thing that you do, as I’ve mentioned earlier, is, whatever your language-specific encoding is– Python, JavaScript arrays, Swift data, all those kind of things– take a look at this URL, tensorflow.org/guide/datasets That will tell you the optimal ways to represent your data in these languages and then turn it into tf.data, which is a format of data which is tensor-based, which is then optimized for running in TensorFlow But more importantly is how you think about your data And let me give one example for your feature engineering And this is unintuitive as a programmer So say, take, for example, I’ve got something that’s got four different classes And my breast cancer was two, right, malignant or benign But say I want to classify something that’s got four different possible classes Now, as a programmer, I’d say, OK, if something has got four possible options, the optimal way for me to store that is probably 2 bits, right– you know 00, 01, 10, or 11 Or maybe slightly less optimal is it’s an integer, OK? I’ll put it as an integer So it’s going to be 0, 1, 2 or 3 But in machine learning, you actually use four integers to store it And you go, this seems crazy Why would I use four integers to store a value that could be between 0 and 3? So the reason for this is that– start thinking in terms of, we want to run classification on something Remember, we’re giving it the data We’re giving it the answers And it’s inferring in the rules and giving us something out So in this case, I would say, OK, there’s going to be four neurons at the end of the neural network And these four neurons are going to give me a percentage probability that it is what it is Like, I don’t know, pick something that there’s four of I don’t know– soccer teams in a group in the World Cup, right? So what was it? It was England, Belgium, Tunisia, and Saudi Arabia, I think, were in one group So imagine if I was building a probability that these ones are going to win that group in the World Cup Now instead of outputting a 0 for England, a 1 for Belgium, that kind of thing, it’s a case of, I have four neurons that are going to fire And I want this neuron to fire if I’m predicting England to win I want this neuron to fire if I’m predicting Belgium to win, that type of thing And as a result, I have to pass the data into it that looks like that So when I pass the data in for, these are the players, this is the form, this is how they play in a certain temperature, on a certain field, all of those kind of data, when I pass in the answer, then I also want the answer to be 1, 0,0, 0 when it’s England were good, 0, 1, 0, 0 when it’s Belgium are good, those kind of things So that’s the pattern for the neuron that I want So this process– and this is just one concept This is called One-Hot encoding But this is part of the process of feature engineering So as a software developer in TensorFlow, these are the type of terms and these are the type of concepts that you need to learn so that you can design your data, you can engineer the features of your data so that you can train neural networks around it So I want to give a quick look at feature engineering in Python So we can switch back to the demo machine So the best thing to do, I would say, is–

if you go to tensorflow.org/tutorials, this website teaches you how to get started with TensorFlow And if I scroll down here, these five links– basic classification, text classification, regression, overfitting, underfitting, and saving and loading– these will lead you to something called a Colab notebook And the Colab notebook looks like this And what’s really neat about the Colab notebook is that it’s something that executes completely in the browser, but it gives you the full TensorFlow runtime And it gives you the full TensorFlow code and API And you can even run it on a GPU if you want to do training on a GPU, which is more optimized for it And this is something that runs in the browser and is completely free And what we’ve done is we’ve built these number of tutorials that will say– for example, in this tutorial, it’s like, if I want to do a binary classification around movie reviews, it has all the code in there And it allows you to step through that code And it will do things such as the feature engineering that I’ve mentioned Here it’s preparing the data Here it’s building the model Here it’s putting in the loss function, those kind of things And this will take you through all the steps of, like, here’s a raw feed of data from IMDb And what we’re going to do with this raw feed of data from IMDb is we’re going to build this classification of good movie versus bad movie And it’ll take you through the steps of doing that So time is a little short, so I won’t go through those steps now But all of this is public And I’d strongly recommend to go take a look at it and step through these notebooks It doesn’t take long And you’ll find them all on tensorflow.org/tutorials OK, if we could switch back to the slides, please So the next step– remember gathering your data was the first step, repairing your data was the second The next step, and the most important from a programmatic step– from a programmatic point of view– is training your models So first of all, when you’re training your models, there’s three different types of chip that you can use in your computer We’re all familiar with the CPU And a CPU, to me, is like a Swiss army knife It’s a chip that’s been designed to handle everything in your computer, managing your windows, managing your network stack, all of this kind of stuff Your code compiles and it executes on that So as a result, it’s a general purpose chip for doing just about anything So therefore, it’s not optimized for any one particular thing It’s just like a Swiss army knife The second type of chip is a GPU And a GPU is kind of interesting, because it’s a graphics processing unit And a graphics processing unit, they’ve evolved greatly for games over recent years But the things that you do in a game such as 3D rotation, the matrix multiplication for that, all that kind of stuff is actually very similar to the math that you would do in a neural network If you remember earlier, I was talking about the math and calculating the weights It’s very, very similar to the kind of things that are done in a GPU And of course, a GPU is a type of chip that’s got lots of little cores And all of these little cores are optimized for doing math And as a result, for a long time, the researchers who have been trying to optimize what they’re getting out of machine learning have been using GPUs And it’s a chip that works really great, but it’s not designed for the task, like slicing a chili pepper with a razor blade And so then, the third one is a TPU And the TPU is a chip that we’ve built and we’ve made available It’s called a Tensor Processing Unit– so T for Tensor– with the idea being that this is absolutely optimized for machine learning It’s optimized for processing tensors And as a result, it’s like a specialized knife, like a sushi knife, in this case So this is a TPU and some of the numbers behind it We get about 180 teraflops per device Bandwidth memory– 64 gig The reason for that high-bandwidth memory is when you think about loading data, processing data, loading data So it really improves your data throughput It’s programmed via TensorFlow, and it’s now available in beta So if you use the Google Cloud Platform, you can get a machine in the Google Cloud Platform that uses a TPU And then you can just rent time on that TPU to do your training And one example that we’ve done was– there’s an image classification thing called ResNet where, as you can see here, we did a throughput of taking real images to train in ResNet We were able to, with TensorFlow 1.7, about 2,500 images per second with a TPU But then when we preprocessed those images to make them a little bit more optimal, that went up to 3,150 per second being able to process on a TPU But the important bullet point here is the third one, that the ResNet– for many years, researchers have been working on this And to train a model in ResNet used to cost in the tens of thousands of dollars, and then in the thousands of dollars, and then in the hundreds of dollars And we did this one benchmark back in March where we actually tested it It was about $85 in CPU time And I was actually working on this slide It was used in our keynote at the TensorFlow Summit that Jeff Dean gave And as we were working on the slide, they came back and said, they’ve actually slashed it even further, but they haven’t verified by how much exactly But I’m not allowed to say the number, but it’s significantly lower than this So we have a kind of a Moore’s law in effect for the actual cost for training, where it’s plummeting dramatically And using TPUs in particular, even though a TPU might be, on per second basis, more expensive than a GPU, when you can do that much more with it and train the model much cheaper–

and if you go to this URL at Stanford, you will see our first benchmark submission that we put around it So then the next part is actually using the models So you’ve gathered your data You’ve processed your data You’ve trained your model How do you use them? And so I’m very passionate about mobile and using mobile But there is three different things– three different ways that you can use machine learning models So first of all is what I’ve been talking about is TensorFlow TensorFlow itself, it’s designed so that you can train models There’s an API for you to train models And it’s a runtime for you to execute them on your machine But you know what, if you’ve trained a model and you’ve got a nice, big, fat GPU under your desk, and you’ve trained a model on that, and you can execute and you can infer on that, so what? Not anybody else can use the model, right? It’s great if you’re an academic, and you’re researching, and you want to show that, hey, I used this machine, and I have this model, and here’s my benchmark But it’s not very good for you to deploy the model for other people to use So we’ve been hard at work to build two runtimes The first of these is TensorFlow Lite, where the idea is that if you have an Android device or an iOS device– in an increasing amount of IoT devices, you’ve probably seen, we’ve put demos out for Raspberry Pi, where the idea is, if you’ve got a Raspberry Pi device, you can then run inference on those devices And by that I mean you build the model on your big, fat GPU under your desk, or you build it on a TPU And then you can actually crunch that model, flatten it, prepare it for mobile, and then deploy it to a mobile device And then on that mobile device, you do not need any connection to the internet to be able to run inference on the model I have a video that I’ll show later if we have time But it’s a terrific example of, there’s a crop in Africa– well, it’s a crop all over But it’s used– it’s part of the food group in Africa called cassava And we worked with a group in the country of Tanzania, where cassava was primarily the backup plant for farmers They had their primary plants that they wanted to grow, but they had cassava if their primary plant failed So if the primary plant failed, you wouldn’t have a famine But if the cassava failed, it’s a disaster Now the thing about cassava that’s really interesting is that there is a disease, there’s a blight that affects cassava But you don’t see this with the human eye until it is too late You’ll see splotches on the leaves It’s too late by that point The plant is blighted And the blight has spread to the other plants You just don’t see it yet So as a result, a blight coming in the cassava can actually destroy an entire crop before you know it’s being destroyed So we worked with a university, and we worked with some folks in Tanzania, to build a detector to say– to train it on leaves and say, OK, this is a leaf We don’t know if it’s diseased or not But in the future, we’ll label it if that plant becomes diseased, and train it on a bunch of these And it ended up being able to spot, on a disconnected device– most importantly, it’s a disconnected device, so no broadband connectivity– that farmers can take an old Android device, wave it over a leaf, and determine, much earlier than you could with the human eye, if the leaf is diseased or not And if it is diseased, then they can yank that plant out before it destroys the rest of the crop So little scenarios like that have been empowered using this kind of thing Train it You’re labeling You’re giving the data You’re creating a model You’re flattening that model And then, using TensorFlow Lite, you can then run it on an Android device or an iOS device And then the third of these is TensorFlow.js And this one, to me, is also particularly exciting Because now you can use JavaScript in the browser, not just to run inference on existing models, but to actually build the models So all those steps that I was talking about of gathering your data, preparing your data, training the model, and running the model can be done directly within a browser So think about the distributed scenarios that are available there If you go to youtube.com/tensorflow, we’ve got a lot of videos on there And I’m actually in the middle of producing a video series at the moment on doing this in JavaScript So if you’re a JavaScript developer, the basics of getting up and running with creating a neural network, training it, and doing inference in the browser are on there So go take a look at them So I’m going to switch back to TensorFlow Lite for a second, which is the mobile one that I was talking about So the idea of how that works is you’ve built your model using your big, fat GPU or your TPU And then we’ve created a converter for that So the process that I called flattening, that converts it to TensorFlow Lite format And then there’s an interpreter that we’ve built for iOS, another one for Android, another one for IoT devices, that will then execute that model for you And depending on the hardware that you’re using, then there are hardware acceleration delegates So more recent Android phones, for example, are able to use some of the hardware acceleration delegates that are on there, things such as the Pixel 2, so you can have even faster inference on those We’ve also tried to make it cross-platform as much as we could So that converted to TF Lite format will convert it so it can be run on an Android app using Java, Kotlin, or C++, on an iOS app, only using C++ at the moment– we haven’t yet built the extensions to allow Objective-C or Swift to be used– and then also on Linux and some other IoT devices And there’s some ongoing stuff on there and some announcements coming pretty soon about that to watch out for There’s some really exciting stuff happening in IoT And if you’re an iOS developer and you want to use CoreML, then of course you can use that too We’ve created a converter to CoreML format So you can go off and use Swift or Objective-C on CoreML

once you’ve built your model and been able to convert it One of the neat things that we always get asked about is the size What impact is this going to have on my Android device? Well, we found that the core interpreter to interpret a model is about 75 kilobytes Now TensorFlow Lite is in the very early stages right now So in the early stages, what we’ve done is we’ve built it, and we’ve optimized it for a number of scenarios And the first scenario that we’ve been working on is image classification So a lot of the common image classification models, all of the operations that they use are currently supported So if you train something, like for example, with MobileNet or ResNet, and then you convert that to TensorFlow Lite, it will work But if you train something custom that isn’t those, like for example, my breast cancer classifier, then that neural network might use some operations that are not supported And so we’re continuing to work on having operational support parity between TensorFlow and TensorFlow Lite But right now, with all supported apps on TensorFlow Lite, it’s about 400 kilobytes OK, let me give a quick demo of image recognition on mobile So this will be image recognition, if we can switch to the demo machine And I’m going to run this on an Android emulator, which is a little bit risky So I’m just running my emulator here right now And what this is doing is that this is capturing from the camera on the front of my laptop, which is emulated as the camera on the front of an Android device And it’s taking each frame And it’s passing that through a network that’s been trained to classify 1,000 different images So the output from that network is going to be 1,000 different numbers, with each number being a probability that what you’re looking at is the thing represented by that number Like, I think, for example, number 55 is a coffee cup, those type of things So of those 1,000 now, it’s looking at the dots behind me here And it’s classifying that as a shower curtain, or a window screen, or a strainer, which is pretty good But I’m going to do my banana demo So I’m just going to hold this up, because it sometimes gets a little confused And if I hold this in front of it– can never know the orientation There we go And you see it’s actually– well, it was a banana There we go, 76% chance it’s a banana Oh, syringe, wow I don’t know why it thought that’s a syringe My favorite is, I usually stand in front of it, and it gives me something different I was an Italian greyhound earlier I don’t know why It must be my hair There we go I’m a strainer But thinking about what’s going on there, there’s 1,000 different things Remember, earlier, I was talking about cats versus dogs, right? That was two different things But here, there’s 1,000 different things The model for this is just a few hundred kilobytes It’s running that because it’s been flattened to run on Android And it’s just a simple Android application that’s reading a frame from the camera, passing that frame to the model, getting the array back from the model of the 1,000 things, picking the top three, figuring out why those top three are, and then showing it on the screen And then that number is the time of inference So you can see it’s about 1,300, 1,400 milliseconds But that’s because it’s running in the emulator, OK? I have my Pixel 2 phone with me If you want to see a demo of it later on a phone, you’ll actually see it doing this classification in, usually, 30 or 40 microseconds It’s that fast at being able to do it The code for this is completely open-sourced It’s on tensorflow.org Go have fun Have a play with it Also, there’s some codelabs outside There’s one codelab called TensorFlow for Poets, which will teach you all about this and how you can retrain this to classify your own things Like maybe you want to classify different types of flower, or something along those lines, instead of the 1,000 classes that it gives you And it will show you how to do that So if we can switch back to my slides OK, so TensorFlow.js is the next one And so if we think about TensorFlow.js, the idea with TensorFlow.js, as I mentioned, is that you can train a neural network using JavaScript And you can then run inference on that neural network using JavaScript Now, I’ve been talking about JavaScript in the browser as one scenario But there’s a second scenario if you’re a JavaScript programmer, and that’s JavaScript on Node.js So if you have a Node.js server, that server can be used to run inference for you Or that server can be used to gather data and train things And where that, to me, gets particularly exciting is– any Firebase fans here? A few, cool So you are probably familiar with something called Cloud Functions for Firebase And the nice thing about Cloud Functions for Firebase is that they are a serverless environment that you can write code on And then that code will execute in response to triggers So there can be triggers in your application, like maybe a user logging in or a particular analytics event happening And then code could execute But now, instead of traditional JavaScript if-then code executing, that JavaScript could wrap a model And then you could run inference on that model, maybe to do a classification or something along those lines So that’s TensorFlow.js And if I take, a little bit, a look at the code again before I show a demo, just remember the ML paradigm It’s all about answers in, data in, and rules out If you take nothing else away from this, as a programmer,

start thinking in those terms Start thinking about what scenarios am I going to get if I put answers in, and I put data in, and I get rules out And I’m going to give one final demo of that And this one is a tough demo So let’s see if we can switch to the demo machine All right, so this is a public demo If you want to try it out for yourself, it’s on js.tensorflow.org And this is “Pac-Man.” So if you’re like me, if you like Google Doodles, if you remember, there was a Google Doodle where we actually created a “Pac-Man” maze out of the Google logo So what this demo does is, in the browser, you can see here, there are four boxes corresponding to up, down, left, and right So what I can do is I can start capturing images of me maybe pointing up, or looking up, or something along those lines And I’m actually going to start doing that now I’ll do it by pointing So I’m going to start capturing some images of pointing up I’ll move it around a little bit, because they all look a little bit different I don’t know, having the dotted background, if that’s going to ruin it or not We’ll see– so about 80 of them And I’m going to capture some images of me pointing right, la, la, la, la, get about 80 of them as well Down is always the hard one I’m going to catch some images of me pointing down, bleh, bleh, bleh, bleh I can’t see how many of them, because it scrolled off I’ll just guess it’s about 80 OK, and I’m going to capture images of me pointing to the left, try to get my head out of the frame, bleh, bleh, bleh, bleh And remember, this is all just running in the browser I’m using the webcam on my machine Capture a bunch of them OK, so now I’m going to train the model So when I train the model, after I press the button, you’ll see a number underneath, which is the loss So what that’s doing is– if you remember earlier, when I was showing– it’s calculating something, figuring it out, comparing it with the results, seeing how well it did, seeing it did really badly, try something else, and keep going And so it will keep going And you’ll see, the loss should keep reducing So there, now the loss is bad It’s going down, down, down So now the loss is really tiny So that’s telling me that of all the frames that I’ve captured, very, very– it’s built a model where it can predict like 99.999% accuracy, it thinks, that if I give it a frame, of what that is So let’s put that in action and try playing “Pac-Man.” So I’m going to play And I’m going to point to the right Come on, Pac-Man There, he’s going to the right I’ll point up He’s going up Ah, down, down, down oh, it was this hand I used, wasn’t it? Down– there we go I forgot I changed hands And now if I– ah, he got me But even you’ll see the highlights if you don’t look– if you look at the highlight on the images itself And you’ll see it’s actually detecting, there’s a right This is hard, because I have to switch hands, that kind of thing that narrows it down Come on, go left, Pac-Man Eat Go, go, go There we go Whoops Ah, I’m going to die Quickly switch No, too late But as you can see here, this is just– this is entirely open source So if you want to play with the code yourself– we can switch back to slides, please So what I’ve done here is I just captured those images live I trained the model with those images in the browser You can see it only took a few seconds to get a reasonable accuracy And then we ended up being able to play “Pac-Man” really badly OK, so finally then, the last part to think about– and particularly thinking about the future now is a real-world machine learning system– is that all this stuff that I’ve been talking about, building models and all that kind of thing, is that tiny little box in the middle when it comes to production So when it comes– to be honest, as an industry, we’re still in the infancy There’s still lots of opportunity out there And our ML code that we’ve built is that little box in the middle There’s all these other things that you’re going to need And I’m going to focus on the ones on the right here, which are the serving infrastructure and monitoring So if you want to build something that’s production-ready, of course you need a serving infrastructure that will allow you to do it at scale You need monitoring to make sure you’re doing it right, and all that kind of stuff And so I would recommend you go take a look at TensorFlow Serving So just a few weeks ago, we released TensorFlow Serving The idea, it’s a server infrastructure for deploying models and allowing people to access your models via the web in a scalable and secure way And the idea is keep your traditional server architecture, keep your traditional request-response model The demo that I just showed, of “Pac-Man,” that actually downloaded an existing model, MobileNet model, of image classification And then I retrained the final layer for those four classes But the original model that was downloaded that does most of the work was something that got served to me out of TensorFlow Serving And as a result, I can actually be able to deploy that model and retrain it as we just did So take a look at TF Serving for that So with that, this is the theme that we’re working on for TensorFlow We’re all about really trying to make machines intelligent and improve people’s lives We’ve open-sourced this so that you can build on it Think of the scenarios that you couldn’t previously do as a programmer Think of the things that you can’t do with if-then code

And you may be able to do that with data in, answers in, being able to infer rules out So with that, I just want to say thank you very much [APPLAUSE] [MUSIC PLAYING]