Rendering film, TV, and games media on GCP with OpenCue (DevFest 2019)

[MUSIC PLAYING] SHARIF SALAH: So yeah, do you like my warm-up pants? This is a thing that was made I think a couple of years ago at Google Next in San Francisco, and we’ve been using [AUDIO OUT] as parts of a kind of series of tutorials and learning resources I say ostensibly the talk is about Google Cloud, but really, it’s about a specific piece of open source software called Open [AUDIO OUT] how to run that on Google Cloud So I’m going to pause this and tinker over here and introduce who I am, what I do, that kind of thing So my name is Sharif Salah A technical writer at Google Cloud based in London, so fairly local to you I’m also open source contributor on OpenCue, the projects based on GitHub We have the site and we have the project itself I’ll just point out that what I’m planning to do– because this is like the first talk I’ve given on OpenCue I’ve done a couple of demos, and this is going to be more demo-y, to be honest But I’m probably going to post the links that I use in the talk on LinkedIn So there’s my LinkedIn link, if you want to get access to the resources And it’s just technical writing So that’s my [AUDIO OUT] spend a lot of time writing, also teaching slides and things like that So I didn’t want to give a talk where I was going to show you a lot of slides or even write a load of slides To keep it fun for everybody, I actually [AUDIO OUT] fairly interactive and probably going to ask some questions and kind of just show of hands, and show you lots of stuff So some sample codes and some demos of the software, and then we’ll do some Q&A. [AUDIO OUT] This was the end result. But if we scott over to here, this was actually generated from a [AUDIO OUT] How many people are familiar with the Blender project? So a good chunk of the audience I’d say probably about half So Blender is basically an animation rendering package It’s open source like [AUDIO OUT] and not affiliated OpenCue actually supports a bunch of rendering engines, depending on how you kind of bake them into the scheduler It’s fairly arbitrary As you’ll see, actually OpenCue is very flexible in the use cases beyond rendering for video, film, or gaming So this is the project, and artists on big budget films, they might have millions of hours of compute time to render, and they need something to manage that workload So you’ve got hundreds of artists all submitting jobs concurrently to a big cluster How do you manage that? Well, the big studio [AUDIO OUT] scheduler At Google, we’ve offered for several years a product called Zync, which is a managed scheduler, but that one is proprietary So the one I’m talking about today, OpenCue, is open source OpenCue is actually– started off as a collaboration between Sony Pictures Imageworks and Google Cloud, but then it got donated and handed over to the Academy Software Foundation So what’s behind me [AUDIO OUT], which you can check out, which gives you a kind of overview, high level overview of what the Academy Software Foundation is sort of managing in terms of projects and related projects So [AUDIO OUT] down into imaging color, assets, asset management, and workflow Rendering and cueing, which is the category for OpenCue, you’ll see down there on the bottom left hand side And the font is fairly small there– [INAUDIBLE] and simulation So that’s where OpenCue fits into the set of tools And there’s a lot there that is, I think, broadly applicable to lots of other domains It’s just that these are primarily ind– [AUDIO OUT] but there may be stuff on there that you might be very interested in, so do take a– some time to check that out after the talk But we’re going to focus on OpenCue and running OpenCue on Google Cloud So what is OpenCue? Well, OpenCue, as projects go, it’s a little– the contrast is a little hard to see on that slide This is in the documentation, by the way, so it will be included in the link But it’s a fairly kind of in-depth project, I guess

There’s lots of moving parts So at the top– actually, let’s start with the Artist Workstation So I just showed you Blender running on [AUDIO OUT] in this case So the artist is running their rendering package or their 3D modeling program, and they want to submit some work So they submit work to OpenCue over the network using a program called [AUDIO OUT] that talks locally to a API layer called PyOutline, which in turn talks to another API layer called PyCue The reason I’m just calling out PyOutline, or outline in general, is that you don’t have to use the [AUDIO OUT] use you There are command line options and there is extensive API as well So you could script submitting work to the cloud or to your rendering engine or to your cluster using it for something other than rendering So after you submit, you want to be able to monitor your job, so that’s where CueGUI comes in, and that’s the PySide2 app And so all of these talk to the brains of OpenCue, which [AUDIO OUT] Cuebot server up there And then the Cuebot server does basically its accounting on the Cuebot database, so that’s running there That’s the PostgreSQL server, typ– [AUDIO OUT] And then the Cuebot server, once it’s got its jobs and it’s lined them up and it’s ready to dispatch them, it sends them to the rendering agents And those rendering agents– I call them rendering agents, but they could be doing other types of work Primarily, for the purposes of this talk, we’re talking about rendering work So those agents are running something called RQD, and as part of the talk, I’ll also show how you can customize RQD so you can decide what you want to run on your– [AUDIO OUT] –ode and your cluster So as you can see, this is fairly involved stuff And if you wanted to start, like, trying out and testing this and kind of kicking the tires on it, it could take you– I would say for somebody who’s like pretty new to this stuff, it could take you the better part of a day, which is a big investment just to see if it’s useful to you So recently, we came up on OpenCue with the idea of coming up with a way to try [AUDIO OUT] really simply, really easily, and really quickly We thought, wouldn’t it be really cool if you could spin up OpenCue on a workstation? So no cloud, no big cluster, no server room full of machines Just you and OpenCue [AUDIO OUT] or your workstation and just see whether it sort of makes sense to you– whether you like kind of the features it offers So we created something called the sandbox environment So most of what I’m going to show you in the sandbox environment but the same principles apply You can very easily turn one rendering load into many and scale it out So this is a– how many people here familiar with Compute Engine in Google Cloud? Nobody’s familiar with Compute Engine? OK, so I will take it from that That’s good It means I can take it from sort of basic principles [AUDIO OUT] in Google Cloud is our part of– is the sort of foundational service of our infrastructure as a service offerings– essentially virtual machines in the cloud Each [AUDIO OUT] that you create is equivalent to a server, if you like, and typically they do not have a desktop These are primarily meant for running in a computing jobs, [AUDIO OUT],, machine learning, that kind of thing But for the purposes of testing out OpenCue and running the demo, what I’ve done is I’ve installed the desktop environment on this Debian Linux instance So I’m going to show you how [AUDIO OUT],, or how I got to there But before that, if you take for granted that this is a server running on Google Cloud, and I’m connected to it over Chrome Remote Desktop, this is the OpenCue graphical packages, but also the OpenCue sandbox is running on the same machine So everything I need to try out OpenCue is running on my Compute Engine instance So with this instance, I’ve done all the work Depending on time, I might have a go at trying to show you how you go about setting up the sandbox from scratch, but assuming I’ve finished my rendering work and I’m ready to submit, I use [AUDIO OUT] So this is CueSubmit running on my workstation, and I’ve got a bunch of attributes

here that I need to set I need to tell OpenCue what the name of the job is Tell [AUDIO OUT] In this case, I’ve just made up a username, Rubble 2 If I’m working on a kind of film or TV based project, I’d be setting the shots You could set this to any [AUDIO OUT] arbitrary, and then layer So the way layers work is they’re essentially analogous to subjobs So you’ve got a job, and then you can have subjobs within that job And you can have– if you want to get kind of more advanced and, I suppose, flexible about it, you can set up dependencies So you could have several jobs and one doesn’t start until the previous one finishes So what I’m doing here is single layer, so it’s essentially it’s just a job doing one piece of work And I’ve got this piece of code So this is saying command to run So the cool thing about this is I can run anything that [AUDIO OUT] supports It doesn’t have to be a rendering workload I could type “hello world” here I could run, like, just a shell script or I could run some piece of Python Anything I want to run on a distributed system So I’m going to go over here and see if I can find that piece of code I just blew it up so that we can look at it properly What’s happening– what I’m about to submit So this rendering node has Blender [AUDIO OUT] it It’s not going to use audio It’s going to ignore any audio There isn’t actually any audio in the project I’m telling it where the Blender file is So [AUDIO OUT] with Blender, the one that I– the footage that I started the talk with, that was kind of like the high res version For the purposes of the talk, I’m just going to switch it to the low res version just so that we see some actual activity happening and it’s kind [AUDIO OUT] submit the job I’m also telling it where I’m outputting the result of the rendering operation, too, and I’m tagging on the end of the frame number there So I’m going to render a [AUDIO OUT] And then the format– I need to tell it what format So I want JPEGs for this And again, the frame number This is the frame number for the ingests like which frame within the Blender file are you supposed to operate on So switching back to OpenCue, I’m pretty much ready to submit You can see I’ve run some– as I was kind of [AUDIO OUT] getting ready for the talk, I was running some previous jobs So I was basically just incrementing the number– test 01, 02, 03 So now I’m on test 07 And also, the other thing I’m going to mention is [AUDIO OUT] range that I’m rendering is 600 to 700 You think about most kind of video footage, you’re talking about 25 frames per second or something So I’m rendering about [AUDIO OUT] worth of footage, and we’ve picked some interesting footage So this is kind of part of the– a subsection of the video that you saw at the start of the talk So I’m just going to go ahead and click Submit and it pops up [AUDIO OUT] “submitted job” and it gives you an ID And if I go over to CueGUI– so this window is what the artist would use to monitor their work And it won’t appear immediately– [AUDIO OUT] have to prompt it– but there you see the job has appeared So that’s the– where we’re looking at it, it’s the fourth one down, and it says “Testing Roboto Test”– sorry, underscore test 07 And you can see it says “In progress” there So I can double click on that And then in the monitoring section below it, which we can refresh, we’re seeing it’s actually running, rendering nodes– or sorry, two layers at the same time concurrently And we can click on those and see the log of the output So that actually looks like [AUDIO OUT] working quite well Blender has quit, so Blender’s running in a container It spun up the container, did the work, and it quit So actually, if we go back to the monitoring window and we say Refresh, you’ll see that’s now succeeded Quick go and see if we can catch one running as it’s going So if we go over to the File Manager, we will have a look at the fin–

[AUDIO OUT] results So this is the low res version, remember, so this is just viewing in Gimp, which is the equivalent of Photoshop on Linux, for those of you who aren’t familiar with it So pretty straightforward, pretty easy, to actually submit the work OK, so I’m just going to leave that running No need to stop it having fun So if I scoot over, actually, just before [AUDIO OUT] the other machine that I have– it wouldn’t be a Google Cloud talk without showing you the Google Cloud Console So the Google Cloud Console here And you can see in my project, I’ve got a single project and I’ve got two Compute Engine instances So the first one, OpenCue Demo, is the one that you’ve just been looking at OpenCue sandbox is– honestly, I just picked the name arbitrarily They’re both running the sandbox The difference is that the demo one, I’ve kind of fiddled with a little bit It’s some stuff that I’m working on to publish for the open source project, which I’ll talk you through later on But the sandbox one is pretty vanilla All I’ve done is [AUDIO OUT] engine instance, and I’ll talk you through that in a second and you’ll be able to take pictures and whatnot if you want to be able to kind of recreate that And left it pretty much as is So see if I can find my script, basically I say– I call it a script This is actually one command, if you can believe it So this is using the Cloud SDK [AUDIO OUT] fairly familiar with Google Cloud’s This is the command line tool for manipulating resources So in this case, I’m using the Cloud SDK to create a Compute Engine instance So I say “instances create” and then the name of the instance, and then there’s a bunch of attributes and properties about this machine So I’m using an image, which is Debian 9 stretch in this case to [AUDIO OUT] I want to pull that image from The machine type– this is worth noting, OpenCue is– it needs some memory I think, off the top of my head, it might be an [AUDIO OUT] off the top of my head, but it– well, let me say this On the Mac, by default, it won’t run on using the default amount of memory that’s allocated to Docker So you do tend to have to [AUDIO OUT] some memory to this I’ve given it a boot disk, and then I’ve also placed it in Europe And I’m also reading– so this is important Right at the bottom there, you see, it says minus minus [AUDIO OUT] I’m reading some data from Google Cloud Storage, which is our blob store So what I’m reading is the very last line I’m saying in the metadata there that I want to use a startup script So I’ve got a startup [AUDIO OUT] for this particular Compute Engine instance Startup script in this case is just a shell script, which is telling it a bunch of stuff that it needs to install So it’s going beyond the default. So out of the box, remember, I said you don’t [AUDIO OUT] interface, you don’t get a graphical desktop So all I’m doing here is I’m actually installing a Chrome Remote Desktop [INAUDIBLE] and then I’m installing a bare bones graphical desktop So that’s all I’ve done So put that into my terminal I run it And that creates my machine, and that machine looks something like this when I boot it up I really haven’t done very much with this machine [AUDIO OUT] the few commands that I did issue So I was inspecting the log to look, make sure that it installed everything that was in that startup script And then I did some stuff around kind of permissions to make sure I could run Docker I made a directory, a couple of directories, and I also set up Chrome Remote Desktop The only other thing I’ve done, just in the interest of transparency, is I’ve set up a variable for the OpenCue database password, just so that I’m not typing it on a livestream But it’s a fairly ephemeral password, to be honest It’s just for the sandbox After the demo, I’m going to be blowing this machine away destroying it So I want to rearrange my tabs a little bit here What I want to do is bring this window so that it’s fairly close to this window And it looks like it’s just decided to disconnect I should say I’m running everything [AUDIO OUT] a hotspot, so it’s a little sluggish That’s why some of the connectivity

looks like it’s a little sluggish The machine has just gone to sleep So OK, let’s scoot over back to our web browser And I want to quickly show you where it is I am here So this is the OpenCue site This is not the main Google Cloud site, This is our Open [AUDIO OUT] site, And there’s some cool stuff on here If you want to learn more about the Academy Software Foundation, there’s a nice intro clip It’s about a minute Definitely worth looking at All [AUDIO OUT] of overview material of what OpenCue is, who it’s for, and some community resources So during the discussion forum, or stay up to date by subscribing to the blog and so on To the documentation And I mentioned that we’ve got this sandbox environment So the purpose of the sandbox environment is so that you can quickly get started So we’re looking at the quick starts I’m running OpenCue on Linux, because that was the operation system that I [AUDIO OUT] my Compute Engine instance But there is also a macOS version for those of you that want to try directly on your laptop And you need a few things to get started, and that’s part of what I’ve been [AUDIO OUT] startup script So you need Python installed and configured, you need the pip command, you need virtualenv, you need Docker, and Docker Compose How many people in the audience familiar with Docker? Show of hands So most people [AUDIO OUT] Docker Compose? OK, so much less I’ll be honest, I didn’t know– actually know anything about– I didn’t even know Docker Compose was a thing up until about three months ago, when some of the other engineers working on the open source project mentioned it Pretty cool It turns out it is really cool So I’ll show you how you use Docker Compose to open up the– or get the sandbox installed, and it’s kind of a cool neat thing anyway to learn about You [AUDIO OUT] So I need to grab a copy of OpenCue to run the sandbox So I’m just going to copy this, and I’m going to paste it into [AUDIO OUT] my home window, home directory And then I’ve done most of these steps So I’m just going to grab this one I’m going to open that [AUDIO OUT] and let’s set an environment variable And this is telling OpenCue where the logging directory is for OpenCue And then the password I’ve done before the talk And then the final one is this Docker Compose command So let’s start this, and then while it’s going, we’ll head over and I’ll show you what the Docker Compose file looks like So I’m just going to go ahead So now it’s pulling a load of data onto my server from the cloud OK, so let’s go and see if I can find some cool codes for the [AUDIO OUT] –pose component of OpenCue So this is the Docker Compose YAML for OpenCue, and it’s basically declaring a load of services And essentially, these are the services that are going to make up OpenCue And each of those is a container So you’ve got essentially a collection of containers that are running on your machine, and they make up OpenCue And this is describing the infrastructure components This is not describing the components, the graphical components that you saw earlier on So this is not like the stuff that would run on the artist’s workstation That doesn’t really need to run in an [AUDIO OUT] So we’ve got the database We’ve got Flyway, which is basically configuring the database and bringing all of the data in to run the demo Cuebot, remember, is the brains of OpenCue RQD at the bottom there This section– that is the rendering node And by default, the sandbox environment ships with the default vanilla image [AUDIO OUT] RQD for OpenCue So you can see that’s actually– what that’s doing there, the image– OpenCue/RQD– that’s pulling from Docker Hub So if you’ve used Docker before, that’s just– you can actually download those components from Docker You don’t need [INAUDIBLE] environment, but we also need some client packages that are going to talk to the server component so that we can render the robots So I’m going to go back to another terminal and I’m

going to– I’m just going to go into– it’s a little sluggish Go into the OpenCue directory and start a Python virtualenv This is essentially just a kind of Python sandbox It’s to run everything within a kind of virtual environment with Python It works really neatly, basically At the end of this, if this is my workstation, I could just blow the directory or I could delete it, and I’ve gotten rid of all of that, those [AUDIO OUT] So I’m just going to activate that environment And I’m going to tell it what version of OpenCue I’m running So for this, I’m [AUDIO OUT] go over here because we’ve got a slightly nearer release that we released about a week and a half ago, I think So that’s my version of OpenCue And I’m going to go ahead and run [AUDIO OUT] As part of the sandbox, we have an installation script that just goes off and installs all the packages Again, no kind of icky steps to wade through This is all very quick The idea behind the quick start is you could try this later today or tomorrow [AUDIO OUT] 15, 20 minutes max And I’d love to hear if you find it takes you longer than that, because it means we need to trim more stuff off OK, so it’s basically going to carry [AUDIO OUT] There’ll be a couple of steps that we need to just configure the client with I’ll talk you through those while we’re just waiting for the installation script to finish So the first one is we need to point it at a [AUDIO OUT] file, and then the other one is that we need to tell it where Cuebot lives, so we need to tell the client– the machine, the workstation that you’re running on– where the Cuebot server’s running, which is local host And then we can run [AUDIO OUT] So if I skip back to the installation screen, you’ll notice that’s finished So copy this and just run these I’ve already explained what they’re doing And [AUDIO OUT] quick start fairly clearly And so now the moment of truth is I’m going to run this command Let’s make this window a little bit bigger just so that you can read the output So we have a positive response So what’s going on here is I’m running a tool called CueAdmin CueAdmin is the command line variant of the graphical applications, managing an OpenCue deployment, and I’m saying list the hosts So it can see a single host in the cluster, and the operating system is the next, because remember, this is running inside the Docker container So [AUDIO OUT] why it’s reporting that operating system state It’s saying it’s ready, essentially, to run And at this point, I can also [AUDIO OUT] crossed here with the– demo-wise, everything works out So it’s installed CueSubmit, and that’s spinning up And I can run [AUDIO OUT] whoops– fat fingering that– and CueGUI So pretty much what you saw on the other machine And again, those took like five, 10 minutes to run Cool So the last [AUDIO OUT] to cover quickly is this is great This is pretty much where we leave you with the quick start But you can’t actually do anything very much that’s interesting with the RQD nodes, the rendering nodes, and your cluster At the minute, they basically just include the default Linux tools that come as part of a Linux container So I’m going to skip over here and talk you through something else that I’ve been playing around with, which is, remember the– I was [AUDIO OUT] the Docker Compose YAML file So before I get to– so I’ve got an updated version of the Docker Compose YAML file And I’ve got an updated [AUDIO OUT] so I just basically want to cover those two things to finish off So most of this has been left as is There’s not too much that was changed We’ve got the database, we’ve got the fly [AUDIO OUT] stuff to make sure the database is working as expected

Cuebot, I’ve left as is What I have done is I’ve changed some of this code down here for RQD, so that’s lines 46 through 60 Couple of things One is that on the volumes there, you can see I’ve set up a mount point, and the mount point is basically telling you where it’s going to find the Blender file and where it should out [AUDIO OUT] the output renders to Then under Build, originally this actually said “image.” As you can see on Cuebot line 32, it said “image” and then the container name– the [AUDIO OUT] Instead, what I’m doing is I’m building my own custom Docker image, which I’m adapting– I’m building on top of that OpenCue RQD bot So it says “build” and then the context, just where we are on the [AUDIO OUT] in the Docker file And then I’m pointing at where the Docker file is on my file system So this is what the Docker file looks like It says from OpenCue RQD That’s what we had before That was the [AUDIO OUT] And then we’ve got a bunch of packages that I’m installing So I’m doing YAML update and then install a bunch of dependencies So these dependencies are dependencies that Blender 2.79 [AUDIO OUT] to run And then I create a directory to run that in, download it, unzip it, untar it, and then get rid of it And that’s pretty much all there is So standard [AUDIO OUT] Nothing particularly fancy You can customize your own RQD instance using essentially this technique Again, haven’t got round to documenting this, so you’re seeing this before I’ve managed to actually document this for the project, which I hope is [AUDIO OUT] I’m interested, again, in feedback, if people have feedback about, like, this workflow So we can customize our RQD image, and the cool thing is this doesn’t have to be a rendering agent It could be anything you want to do It could be some sort of– for example, I’m into photography, and it occurred to me recently that I could use this to stitch huge panoramas Like, I could do a kind of gigapixel panorama Just [AUDIO OUT] something I’ve actually got around to, but in theory, if you can find the right open source packages, you can pretty much do that Couple of other things just to finish off That’s kind of pretty much it for the demo I’m going to [AUDIO OUT] couple of, like, pictures and resources that might be useful to you So on OpenCue, we’ve just published this This is the most recent document that we’ve [AUDIO OUT] on the OpenCue site, and it’s our first case study So this is a case study of how Sony Pictures Imageworks runs the pre open source release version of OpenCue internally Diagram just to illustrate how this works One thing that’s really cool about this diagram is you can see the render farm there and from the text, if you go through and read it, they’re actually using for their render [AUDIO OUT] installing RQD on their workstations And that makes total sense So you can actually use– you could use OpenCue to manage workloads within a workstation group It doesn’t necessarily have to be [AUDIO OUT] running in a data center or in the clouds Another resource I’ll quickly call out to before we go to Q&A is we have a more in-depth [AUDIO OUT] article, which we published kind of towards the beginning of the year And this is a much more in-depth version of what I talked through today This is creating a render farm from scratch on Google Cloud Platform user [AUDIO OUT] And down here– I may actually even have this as a– no, it doesn’t really matter I can just zoom in here, which is all I’d really done before [AUDIO OUT] is showing kind of a typical deployment for OpenCue on Google Cloud Platform So you have a bunch of Compute Engine instances that are running for the rendering hosts You’ve got Cuebot running on Compute Engine A managed Cloud SQL instance running the database And we have some cloud storage, which is where I said earlier on I was running my startup scripts, but we’re using that to actually collect all of the images and media that we’re rendering– any videos that we’re [AUDIO OUT] And we’re storing our Docker images locally within Google Cloud Platform in Container Registry So that’s the other component So again, I’ll include links to our lending [AUDIO OUT]

where you can find links to the solution And that’s pretty much it And I will just put this on screen for a second This is my kind of silly markdown file So this is the links that I’m going to post up on LinkedIn, but feel free to take a quick pic of that And with that, I think it’s probably a good time to say if anybody has questions, I’ll [AUDIO OUT] answer and say five minutes or so and that’ll be it [MUSIC PLAYING]