Puppet At Stanford University – Digant Kasundra -PuppetConf 2011

all right my name is Selena for those of you who haven’t seen me up here before I’m just here to help introduce our speakers and get things underway and I’m here right now to introduce bigint who is that Stanford University and he built and managed many of their systems in the configuration management infrastructure from 2005 to 2010 he brought puppet to Stanford in 2005 and helped spec many of the features that are used today he helped drive the server to admin ratio higher which I’m sure was wanted at Stanford and now he is I think officially in system administration recovery because he is now the lead middleware dev but he still is in charge of the central puppet infrastructure so please welcome ticket thank you guys for coming not a lot of you but I’m glad you guys came I’m so Cena saying my name is digging Cassandra I’m at Stanford University and we’ve been using puppet for a long time so we’ll give you a little bit of history so what I want to talk about today mostly is is kind of cultural based stuff you know what it is that we do and how we manage puppet at Stanford and how we built a culture around that so that some of these practices because as Luke talked about you know puppet is more than just technology so you guys heard a lot of great stuff on the technology and how people are using it and you know it’s really fabulous stuff but if you can’t integrate this into how you think and how the rest of your team thinks and you can really get into trouble so so that’s exactly where we started and is this mic cutting out in and out er are we good okay cool so so what we wanted to do you know when we first started out we had a real problem so the problem is what you have when you have and he kept a large shop and for us it was only like 15 people at the time we had this problem where our servers weren’t configured consistently right so we have I kind of call it like the artisanal model where you know we have some and they’re you know they’re John’s and John has this particular bouquet of server then over here we have Fred you know and Fred likes his server spicy you know they’re a little bit spicier and different and weird you know it’s like that’s cool but it doesn’t really work doesn’t really scale well so we wanted to get rid of that and we also needed to have a consistency of practice right so one of the things you have to kind of do as a team when you come together is you got to all agree to to kind of do things the same way you know you gotta kinda you know it’s like not everybody’s gonna win everybody’s gonna lose but you got to go you know okay this is the way we’re all gonna configure Apache this is the way we’re all gonna configure that’s safe because if you don’t you get a real scaling nightmare right because what happened when Fred you know gets hit by a bus right now Jon’s got to take over the server and he’s not gonna have a really easy way to kind of go forward so we can go back a little bit so we wanted to so we wanted to look at something like puppet so we looked at configure configuration manager systems we look at cfengine and you know a lot of what I’m going to talk about is a repeat a lot rather people have talked about so I’m gonna just you know mix it up with some personal stories so because I think Luke is not in here right okay so cool so this is what happened this is what really happened we wanted to look at configuration managed systems we put out in RFC and we got some we had a lot of companies that said yeah here’s what you can do with CF engine CF engines awesome and I already decided to see if engine was pretty much crap I didn’t want that I wanted a different way to look at stuff I wanted models where I could say this is the way my world should look and I want a system that builds that we got this one really interesting RFC from this guy in Tennessee and I when I got that I was like really Tennessee do they have computers in Tennessee is anybody here from Tennessee now okay good they apparently have three computers in Tennessee and Luke had one of those computers and so he was doing cool computer with it and and so so we got this guy in and he comes in and we brought him in a contractor this was five years ago when one of the oldest users of puppet at Stanford we brought him in on exactly this day actually this is exactly the day our puppet you know it’s kind of like when in Terminator when Cyberdyne goes lie right this is the day our puppet instructure went live this is the minute or puppet instructure went live okay so we brought Luke in who’s reductive labs of the time there’s only one guy named Luke he comes in this is day we brought it and the reason I know this okay the reason I know this is because this is the exact time five years ago that our puppet see a sign our puppets see a cert was signed does anybody else use puppet for coming up on five years now okay so as soon as your puppet see a cert comes up and it expires everything dies so that’s how we figured this out so I got

a call on August 24th 2011 which was just a little while ago at 2:10 p.m. and everybody said what the hell is happening I said I don’t know what do you mean and they said nothing is doing anything all the puppet systems are down and that freaking out little ready to tread downs because our CA cert head surprise it was a five-year cert and expired so we know to the day when when puppet went live we resigned the cert everything went okay so to this day we have about 73 thousand lines of code in puppet that’s a lot of manifests to read right so once again we get everybody that’s going to use the same system now you have to deal with how do we all work together on this thing so it’s not a big bloody mess all right we have 1784 classes there’s a lot of classes right we’re a small organization but the reason we have such a diversity of classes is because at Stanford we’re decentralized organizations so we have the service that we run internally right which is the right way to do it and then we have clients who have their own right way to do things and then they have other needs and so we have some very different models right we can’t just say this is the way database server works throughout Stanford that doesn’t fly everybody here work at higher education institutions okay so you guys know what I’m talking about right everybody has their own IT shop and everybody’s IT shop has the one right way to do things so you have to have different models to kind of spec for all these things so we have a lot of different classes but we have only twenty sucide means we’ve 26 admins they don’t just do full-time sysadmin we do Project Work we do a lot of other stuff and we have about 500 systems which is not a lot these days but at the time was considered pretty amazing so we have four puppet there’s two puppet queue servers I don’t think anybody cares about when a puppet master put the cue servers we have so so where to talk to you literally about its kind of coding practices and what I mean by that is isn’t you know what manifests you ride or here’s some cool puppet tricks but just how you write we know what’s the way to think about things I have this little spiel on puppet you know packages versus puppets that I want to give which is not really related but it’s something I feel strongly about team practices how do you work it together as a team what are your team practices should be what are the kinds of things you should you know culture you should try to cultivate server practices and then ITIL and crap that management cares about so anybody here heard of ITIL right okay have your bosses heard about ITIL right okay good so there’s ways that you can do and I know puppet enterprise is looking more towards this direction too of how to interface with that so we’re at Stanford where we focus mostly on the open source stuff we are not a puppet enterprise consumer at this point we’re big supporters of the open source software so anyway so we’ll start with that so let’s talk about coding practices when I talk about coding practices I think there’s there’s three major things a style guide I wrote one of the original style guides was based on what I came up with with Luke and and he came up with internally when we had a bunch of people writing code right so you’ve got hundreds of lines of code if you’re coding anything else you want to have a style guide right so the same thing happens in puppet there’s a couple of reasons one of them is because it just makes it easier to find things but the second thing is it’s actually more taxing on your brain to read stuff that’s formatted improperly so it’s not just that I’m OCD I swear to god it’s the style guide so let me give you some examples what is legible easy to parse code right if you’re looking at a hundred lines of code things matter right so if you’re looking at something that’s not that big then it’s okay but the bigger your group gets right the worse your code is going to get the uglier the code gets does anybody else seen this happen right if you’re writing by yourself your code looks pretty good right but then the next guy comes in and he likes his curly braces at the end of things not at the beginning of things right okay that’s cool you got two different ways right now your third guy comes in he’s got a different way of do things and the fourth guy in the fifth guy and then you get this one guy he was just lazy he just doesn’t give a where anything goes I do apologize I do tend to cuss I hope I don’t offend anybody and if I do then go no just kidding so the code it’s uglier and uglier so that was one of the problems we have to deal with early on is we have to say okay we got to agree on a coding style puppet Labs has a style guide it’s on the web it’s based on some of the earlier work that I did there’s got some new stuff for some of the new features in there I suggest everybody take everybody looked at the style guide who in here has not looked at the style guide okay so I would suggest you look at it you don’t have to adopt that style guide make your own it’s okay but it’s a good starting point and if you’re working with multiple people it will really help the other thing is the bigger the manifests get you know the harder they’re to parse so like I said we have about 71 thousand lines of code total so when I’m scanning through that trying to figure out you know where is this thing misconfigured or where am I you know in what manifests am i installing this particular package if it’s parsed out nicer it makes it so much easier to read

so here’s a just kind of a quick example I’m just gonna I’m for this thing’s for a couple of examples I’m just gonna talk about something simple right these little arrow things and shame on Luke for not following the style guide in his earlier talk I’m gonna talk to all about that but this is okay right this is this is a valid manifest this will work this will compile just fine arrow run and and this is actually not that hard to read but if you had you know 300 lines of this stuff it could get a little taxing right I just want you to just compare this to this right look how much nicer that is to read right you get your tango it’s alpha you can look down you can easily parse if it’s present everything is just nice it makes it easier on the eyes to do so these are the kinds of things I’m talking about when your whole group says okay these are gonna be our style guides and we start coding to that it makes it easier to work together and you don’t have that annoying guy right again come together as a team at off the style guide because then you don’t have to get mad at one dude who’s going to eventually get punched it happens these are just this is just the industry we work in people I think we all know this right people get hit a lot it’s a very violent industry packages so here’s another one this one actually if you this one I have a you know a slight disagreement with some of the ways that that the current style guide talks about this one you know we’re talking about putting packages is a different element everything I kind of like to group things together right so if you’re looking at this you know and it’s going on and on and on and on forever and you’re trying to figure out one particular package in this manifest isn’t being installed or not I mean you can do a grep you know there are tools for that but that’s just a mess to read right but if you had it all I’m gonna blocked nice and easily like this this makes it a lot easier to read and I swear to you this will make a huge difference you know and if the chicks see that she will get laid they love this stuff you know okay I made that up I can make up stuff it’s okay yabba question so the question was how do I feel about using square brackets to make it into an array so that it’s all insured present I love it I think it’s a great idea you know and I and I recommend it to and that kind of stuff is in the style guide and I do and I think it I think it works great you might have to think about it you know if you had 500 things in an array why would you have 500 package in your manifest I don’t know but maybe you do we actually have one manifest that has about 200 packages that were installing do I want to scroll down to the end of the array to figure out if I’m ensuring present or if I’m sure you’ve all right it’s a small thing right it cost you about two milliseconds of your life but those two milliseconds could change everything it could know it so yeah I think square brackets are great so legible and easy to parse I think is a good thing sensible chunks of code right the cooling about puppet is you can put anything anywhere right and does a graphing model and it all works out so it’s got this magic and it’s like this thing green depends on this thing and that thing depends on the other thing and it just runs and everything this kind of works beautifully right but I don’t have a graphing model in my head so I’m looking at a manifest it’s awesome if everything is put a put into chunks right here’s what I’m doing my packages here’s what I’m doing my services ensure what’s present at the very bottom is where I put my files things like that you know you can develop an ordering scheme you can say you know okay we’re gonna always define users at the top you know we’re gonna always define you know systems that we’re doing overrides on towards the top because that’s makes it here so these are things that you should just think about but if you just do it randomly what you can do it becomes a mess right because if I can’t figure out why this package is installing it’s not near the other hundred packages you’re installing it’s actually towards the bottom for no damn reason you know it gets annoying reusable code there’s about I think 22 people talking about reusable code at this conference so hopefully you’ll catch one of those car people modules for instance I think I think puppet has a great what is the module repository thing called who uses it the forge yes Forge is awesome when modules first came about and puppet you know the whole idea of kind of encapsulating things was great so here’s the problem that happens in your team right you’ve got one guy that thinks you know modules are awesome so I have a module for a server then you have another guy it’s going you know what everything should be a module you know every package should be a module every single letter should be a module how do you balance that out right how do you go from having one giant monolithic module to having you know way too many modules right so one of things we have to do internally is we have to start saying okay what is actually reusable what is this you know who is actually gonna install this package in multiple places in different ways that necessitates a module if it’s just you man you can’t alone create 23 modules for your project you know it’s just not

that special but it becomes useful to do so so finding that logical barrier you know sometimes you have modules that now have to get subdivided into smaller modules a patch is a great one one of the things we found is that Apache is configured slightly differently for things like Red Hat and things like Debian and we’re a big ready out in Debian shop and that’s pretty much all we do so now we have to start breaking things down even further so there’s a trade-off between modules right and some of these things are solved with things like parametrize classes and defined types but the problem is how many instances do you are you gonna have right one of the things you have to look into is not having sixteen different modules for sixteen different flavors of the same thing so having a strategy around that is a good idea and and you really want to kind of think I think of that stuff up modules into the subclasses in other classes so one of the other things we found like I said we have thousands of lines of code one things we found at Stanford is if we have a module that is over like say five hundred lines of code that’s probably too hard to read at any one time right so one of the things we started doing is breaking this down into subclasses so we would have a subclass that’s not really an independent class all it is this meant to be is an increment of a bigger thing right so now instead of looking at one giant class that’s 500 lines of code you would now include you know i underscore blank packages i underscore classes that’s just gonna help kind of you know break everything down into sub components use defined types how many people are using parametrized classes well how many people are using defined types okay how pre anybody using both okay what under what circumstances do you use one versus the other no answer that’s exactly right that’s exactly right I was just talking to Victor about this Victor victor everybody say hi Victor Victor’s from Stanford we were just talking about this defined types are pretty much the same as parameterised classes they have some different properties but the important thing here is to use them why because if you can break down what you’re doing repetitively into a well specified module of some kind and and encapsulated in a defined type it will make your life easier because you will be able to track we’re one change is gonna affect all your other systems so one of the reasons I I recommend using defined types is just way our usage pattern evolved oh that caffeine it’s amazing oh yeah I can feel it it’s right in my bones one of the ways that we use defined types for instance is we do a lot of overrides right so if we have a defined type that’s going to be for SSH server and we want to pass in what ports to run on if I have five hundred systems that need to run in one particular way but I only have two systems that need to run on a different port I can use an override and define type and I can’t necessarily do that as easily with the with a name class so that’s why we do it but it’s exact same principle right so a defined type is everybody knows because it Rios is something reusable variables I don’t even need to talk about that okay they’re all good use templates were possible one of these at Stanford we decided is we want to keep as very little files that we drop onto a system as possible we wanted to make sure early on we said you know what we’re gonna be really awesome and we’re gonna make native types for everything and we’re gonna make defining types for everything and it’s gonna be awesome and we’ll never drop files onto the system because everything will be modeled and that never happened because a lot of things are hard to model so you are gonna use files and we ended up using files in one of the ways that we learned to sort of condense the number of files we use this to use templates so we use templates for a lot of things like Apache configurations weasel templates for things where we want to break down what’s happening you know if it’s a master server it’s gonna have these lines of code if it’s something else we have these lines of code it still again gives me a better place to go and figure out what’s happening right I don’t have to pull up five files and go this is the file I put on this type of server this is if I put on that type of server and then do a diff right if it’s in a template I can just read the code makes it easier and using subclasses but big proponent of subclasses and I propose subclasses over parameterize types again because we had a very specific need we use our catalog a lot to figure out what happening on a system so if somebody wants to know how many systems do you have on your campus that you can connect to from off-campus we have a class for

that right it’s if it’s a parameterised class called SSH and I pass them as a variable I can’t just do a query with something as easy as looking at a class up text file right classes of text but if it’s a named subclass and it is for us it’s called SSH : : global then I can easily figure out what systems are using it and get that catalog and subclasses do overrides which i think is amazing overrides is one of the first features that I think I remember Luke was sitting at my office and I think anybody use puppet like geez what version was that even five years ago anyway the overrides was he was basically like now overrides are terrible nobody should do that and I was like yeah we’re gonna want to do overrides you’re gonna want to build that in I think so and we did keep things well named so this kind of goes back to my previous point if you are going to use something like a configuration management tool or a CMDB or anything ITIL related you’re probably going to want to use what you’re putting on your system is a way to figure out what’s happening on the system and communicate that with your bosses so if you keep things well named you can use these as triggers for what’s happening so again going back to my SSH global right if I want to know what’s happening on a system if I named all my classes stupidly I can’t figure out what’s happening I don’t think anybody does this so I’m gonna move on from that point but it’s useful when browsing a catalog like I said and you’re looking at what’s happening I mean I can tell from the name alone what the expected behavior is right this is another reason why I advocate for that if I can’t figure out why I can’t connect to SSH from off-campus if I look at the name and it says SSH global I can I know that it should be able to but if it’s a parameterised class I can’t do that from the name because what that class is supposed to do is hidden from me got passed in as variables and I don’t know what they are so we like at Stanford to name the classes based on the behavior that that you put this little piece in here and this is what it’s going to do it’s just it’s worked out well for us so things like LDAP and LDAP master as a subclass right now I know well that masters a-class applied to the system it is an LDAP master where if it’s LDAP replica it is a replica if I pass those things in as variables into a named class I would have no idea except to know that it works that it does all that well I probably knew that already because I’m looking at a directory server I was trying to figure out if this was supposed to be behaving as a master or a replica packages versus puppet one of our policies internally was we don’t distribute binary files over puppet we use puppet for configurations and configurations only this is really you know Luke was talking about you know kind of like heresy and things like that that’s all this is there’s no there’s no reason why you can’t do it you can be stupid all you want so I’m saying you know not in my turf man packages are anything compiled there’s actually good reason for this the reason you want to use packages for anything compiled is that packages are great for staging changes right so if you have a beta version of something you can name it that and it’s easier to shoot me that there are environments and we’re gonna talk a little bit about that it’s a good way to do change management but package is so much easier so if you have a binary file and you’re distributing through puppet it’s really harder to kind of stage that out but if you have a package you just tell puppet these dev servers you install this version of this package on a master server or on a production server used to all this version and it works all beautifully and packages handle dependencies better so again puppet is great at handling dependency dry it makes that magical graph web thing that I talked about that’s the sign for graph web if those it don’t know but why do you want to model that in puppet right if you have to install Apache and Apache needs these 15 libraries in these thirty two libraries why would you want to model that and Apache the same thing with their internal code right if your internal code has all these dependencies you can model that in puppet but if there’s just package dependencies you can just model that in your package right so you can even do things like a meta package that just says this is our and I think Martin talks a little bit about this and if you watch – tree from media temple they talked a little about this – or you can just have your package say you know these are my dependency this is what I need to do they can save you know a lot of lines of code so we have like timeshare servers where students log in and they expect to have 500 pieces of software on their you know we model all that stuff in puppy right now but if we just had one package and just told puppet install this package and then let the native tools under it do all the heavy lifting it would actually cut down a lot on our server times a team practice never make local changes Luke talked about this in his keynote and it’s one of the things he and I agreed on this is one of the problems that we had at Stanford it well you’ve got somebody it’s the middle of the night and something’s happening and Apaches down and they get called you know and they’re trying to figure out what’s happening so you get on the server right and you start making

changes and five changes later you figured out what it was but I don’t know about you but me at 2:00 in the morning I can’t know what the other four things I did were that we’re bogus right so if I make local changes I’m gonna have these problems right who problems over all right go back to bed next morning we got to reboot the server comes back up whoops it doesn’t work why I don’t know I did stuff what did you do it was 2:00 in the morning I don’t know I don’t know what I did and we have these problems right we go to rebuild the server right but it’s like okay we got this awesome puppet modeled server and everything builds and it’s awesome except this one little thing so I just tweaked it but I’ll remember that’s cool then I go to build a second one and it doesn’t work and I don’t remember what I did all right because people say stuff like this I’ll go put it in puppet later later later never comes and these are the people we all know right people they get punched at work these are the people punch them in the face later never happens so one of the rules we had and it was one of these I agreed with Luke was we let puppet change it back right so you make a change in the system and you didn’t commit to puppet puppet we’ll change it back and we like that I mean it’s one of the greatest things about puppet it actually really you know I say these mean things it really isn’t about bad co-workers it really is about the human problem right we forget these things and in puppet can fix these mistakes for you they can keep you from shooting yourself so so so going along with that right there’s a solution to that problem a puppet changing your systems right you lock puppet same what do you know this one it’s beautiful you just turn off puppet now puppet can’t screw with you so that’s a bad thing does anybody have a locking mechanism that they use for puppet cool two three people okay so we actually came up with a cool little locking mechanism at Stanford and I’m happy to share it it’s not that difficult at all there’s a run file if you if you that run file exists the puppet client won’t run so we have that lock file we created and we track who did it and why so if I need a lock puppet because I need to tweak something or you know I’m waiting for this package rebuild before it gets installed I can put that in there and when I forget three days later and somebody goes why is puppet lock they can look at that and go oh yeah digging did that and now he’s gone somewhere on some mysterious adventure and it’s okay we can change it back we also enforce some acts time so we have a nightly report that runs let’s see how many people get email reports okay just a few people how many people prefer dashboard to the email reports oh okay hardly anybody so more email users in dashboard okay interesting Thanks so we actually get email reports so one of the things we do is we have a talk about customer reports over later there’s something that watches and says ok the system hasn’t updated in a while you know it’s long it’s locked or it’s tangled or it’s broken or somebody blew it the up somebody go look into this thing and so we watch for that we watch for lock puppet clients and we reported that we can go and investigate so again because we want our systems up we want them configured properly at all times server practices Apache passenger I think there’s 25 people talking about Apache passenger it’s good use it it scales beautiful version control how many people are not using version control ok again how many people are not using pre-commit sin2x checks ok ok good pretty crude syntax checks are awesome because how many people have committed something and they left off the curly brace and he gets pushed out and now nothing compiles right ok that’s acceptable how many people about a co-worker view that that’s why we have the pre-commit syntax so less punchy more technology and use get because it’s awesome I get paid but yet every time I say that super practices pick a security model how many people do let’s see how many people have any kind of constraints around who can run puppet on your system okay couple people one of the things that we did that we decided upon is we only have route control those certain mounted users and those are the same users that can run puppet on that system so it’s a small thing but it’s useful on who can use puppet but who can commit how many have any kind of locks around who can commit to your puppet repository okay good good we don’t do that but we want to anybody that can commit on puppet can commit to any branch of our puppet that’s a problem but I think it’s a great idea to kind of lock that down certificates so certificate handling one of these that we decided internally is no system should be able to just auto sign a puppy right so puppet client turns on it makes a request for certificate you can configure the certificate authority to just go oh yeah everybody’s good that’s cool let him do

whatever right it’s a bad idea why because what is your puppet server called is it puppet domain.com yeah probably is right ours is ours was puppet the stand pretty to you that means when anybody in any department turned on their puppet said about puppet looks Gorman to try it out it sends a certificate request to us if we just thought assigned it it would start configuring itself and probably get some of our firewall rules things that we don’t want them to get also how do you revoke your certificates anybody doing certificate revocations cool anybody anybody ever cleaning out their puppet certs when you turn off servers so good okay I can’t say 100% good at it I don’t think we use the revocation list I think when we blow away a server we just delete this or difficut but we do have a report that says here’s all the certificates I have five of these systems don’t even exist maybe you do something about that customer report so this is kind of thing I wanna talk to you about the email reports is something that we found probably the most useful because if I have to go to a system and figure out what’s happening I’m probably not gonna get around to doing that but when I get an email in the morning it’s not that bad I can just check in pretty easily that’s my mom outside she’s singing she’s excited for me that the last check report is something that just says these puppet clients have not checked in the last 24 hours maybe you should look into that so we talked a little bit about that but here’s another cool one that we developed and this is not like hundred percent foolproof but it’s called the tangled report how many people have systems where a puppet is doing the same thing over and over and over again for days and you don’t even realize it right so we get that a lot too right let’s see there’s a service that puppet says should be running pop it runs goes out this thing should be riding it sock puppet tries to run it 30 minutes later puppy goes out this thing should be running it tries to turn it on this keeps happening well that service never gets on right but it puppet doesn’t realize never get on so we call that a tangled state where puppet is trying to do the same thing over and over again so we made a simple little tool where we track all the log reports from puppet and when a puppet client makes another log report because is this log the exact same as it was last time because if it is there’s probably something wrong somebody needs to get punched so so if this tangled report which will then let us know that the systems have been there’s something wrong with them and they need to get things so the ITIL stuff here’s the stuff I really wanted to kind of talk to you guys about because this is gonna make you look good so again why ITIL why ITIL simple is because your boss is heard of it and I know this is one of the directions again that puppet labs is looking to go into is to do more of seam DB type stuff and I know that we basically have that same measure right so we had somebody come to us and said we’re doing at CMDB and I said what’s the seam to be and they said we don’t know but we want one because that’s how management talks so but Luke talked about this right in his in his conference that what happens they give you an auditing tool the giving auditing tool that they want you to run in all your system so they can tell them what’s on your system what’s the make what’s the model blah blah blah well we have that stuff and puppet there’s no need to duplicate that and I certainly don’t want to run some third-party tool with root privileges on all my boxes that I don’t know about so there’s so there’s ways to deal with that but I think I think ITIL can be a good thing we have a larger organization you know you have to move up to that level where you’re not just doing you know it’s kind of moving from raw editing files to version control right it’s a good thing it’s a small change it’s a good thing I told the same thing it’s just making recommendations on how to run things and it’s trying to help people that don’t know anything about computers management figure out how people should be doing things to make things run better so one of the ways we found to address that is environment support is anybody using environment support okay not a lot of people so environment support is something that we we kind of had put into puppet because we wanted a way to stage changes so we said okay if we’ve got a patchy and we’re configuring it this way maybe this isn’t the best way to do it right yeah we kind of screwed up those manifests it really should look like this well if we just make all those changes that’s 500 systems we could bring down if we screw it up right so we really want to go through a change manager process that where we let everybody know we may bring down your systems because we may screwed up this new manifest but we’re letting you know right that’s change management it’s called you let people know ahead of time what you could be up and they tell you it’s okay someone does you go I told you I told you it could happen so we use environments to roll that out so you can say okay these dev servers I want to point to the master branch it’s gonna be you know whatever the latest changes are but these important production servers we’re gonna point out a stable branch and that stable branch we will test and we will roll out only when things are approved and they’ve gone through the clearance and that’s really kind of what environments are good for change management but there’s one leads we found is that there’s not a lot of divergence between our master and our debt or our death and our stable branch so there’s only that we didn’t really predict so this is one of our findings so if you want to look into

environment support there’s one that used to consider do you really need it do you really have a stable because one of the things we found is what changes a lot on a system well for us what changes is the users right we just hired a new guy he needs to be on the system we just fired a guy he needs to get off the system these are kind of changes that need to happen immediately it can’t be like cool well the next change of maintenance window is in two weeks so that done that’s like this guy’s got a sit ins cubicle and he needs to pretend like he’s being busy so you need to give him an account on everything so so these kinds of changes can’t wait so there’s not really a lot that we put in as stable and dev so that’s just one of our asides little findings so CMDB is another like in that that was that that was the big calling for us CMDB is a change management database it basically holds information about your system it’s the same way puppet holds major Batra system or store configs whole system’s information so we wanted something that would integrate store config so anybody using store configs here yes excellent very few people store configs I hope everybody is at least familiar it’s for configs sort configs is basically all that system information that your puppet master gets remember clients it can take that and store into a database so you can now use that for other information if you’re using exported resources you are using store configs by default so we started using store configs to get this information one of the things that we found was that store configs was not able to work if you had a big system so we came up with asynchronous store config so now your system so if you have a large manifest system and you’re looking to roll this out if you test this I swear it’ll work just fine but once you roll out into production and all your systems are trying to do store configs at the same time it’s going to blow up so use asynchronous store configs it’s helped us out a lot and there’s a technology out there it’s not technology this is a what’s the word I’m looking for it it’s a little it’s not really language it’s really kind of an XML spec but it’s aspect called seam DB F so if your bosses are telling about scene DB you need to do it look at the seam DB f it’s a it’s a way to kind of communicate change management information so it’s one of these we integrated in you can use custom data types if you need to integrate with the seam DB and you got store configs but you need to increase metadata or put metadata in about like the services that are running you can use custom defined types to do that they don’t do anything but that name will fill up in store configs that will help you finally one of the I think last few lessons that we learned at Stanford again a lot about management I say Luke talked about how there’s like this divide right between people like us who are running systems and management and we hate them they hate us and because we don’t talk one of these I found is as soon as I started making tools to help management to figure out what’s happening in our puppet environment they were very happy so there’s lots of ways to do this you can send them the email reports one of the things that we do is we copy our boss email reports about our systems that are locked or since are tangled because they can see you know hey look nightly stuff does happen and then nightly we do fix things I think they just get ignored but we send them puppet dashboard is another cool tool I would recommend you guys set this up if you haven’t we don’t use it we made it we made a tool I’m going to show you a little bit about it in a minute here but we made a tool similar to it called Malkovich mostly because I needed a job a project to work on just for fun here and working so so kind of demo so what this is kind of doing is just it allows my boss to go okay here’s our tangled servers today you know Wow geez the mirrors and the magic so here we’re kind of looking at a report here so on the right you know a boskie okay here’s otter tango reports and they can do a search all this type of search functionality is available I believe in puppet dashboard but it’s easy to do so you can design your own if you want for whatever reason like I did and then we can kind of show them a puppet reports so here’s our log so this guy was talking about taking old reports right the server for some reason is trying to do the same damn thing every time and they can look at that what this does this means nothing to them right this means nothing to management but what we found our experience was that they were happy to have this information so I’ve no idea how do I get back to my thank you what the stuff on it so what I found is that this allows management to feel like we are giving them visibility in what’s going on and that built sort of a trust around our culture puppet because today I’m we know puppet means nothing to them right if they don’t know anything about technology and we’re telling them we’re coming up this great system that’s gonna help us build you know and work faster they don’t understand that but by giving them just simple functionality and simple not functionality simple views

into what’s happening we were able to build that trust culture and doing that bought us a lot of leeway so when stuff does happen when things do break when they have visibility that there are a lot more there are a lot there a lot more understanding about the kind of problems yeah so work together so you know a kind of conclusion I think if you have a small group and it’s growing bigger and you’re trying to figure out how do you make puppet work you know the puppets not just a technology it really is sort of a new way of thinking about systems right if you have people that come on board that are using cfengine or they’ve been using chef for some other outdated model of configuring systems they’re gonna have to kind of think differently right because one of things that I found every time we get a new missus admin they want to kind of go oh where’s my for loop you know I want to do a for loop I’m ready these things it takes some time to kind of reframe the way you think in a puppet system right how you model something in a puppet world adopt a style guide I swear if you have a big team this is going to make a big difference help out a lot keep it in puppet do everything locally and pop it watch your logs watch your reports finally everybody’s been showing picture that kids I don’t have any kids but I made a movie yay all right so there’s a thing about my movie anyway um so that’s it are there any questions about puppet or how we do things at Stanford okay so the question was since we’ve unleashed all these reports do we end up with armchair admins who make unsolicited advice give us unsolicited advice on how to manage it now that actually hasn’t been a problem for us we we are I will say we release reports not to our end clients we report we release um only to where upper management so that’s like two or three managers in our group and they’re very good about you know their management we’re the technical people right what they want to do is management is identify problems in the organization and then let us work on them right so if we tell them look we’re already proactively looking for problems and working on them they just go sweet I can go golfing a little bit earlier today just helps out and you know what other questions do we and we have a lot of time good you feel that that’s fundamentally of so the question was I rail about people making local changes is that acceptable in a world where you can lock other scenarios where it is okay to kind of make local changes if you’re doing it in a write in a nice way I think I think ideally we always wanted to say don’t ever make local changes but what we really found is if I’m troubleshooting a problem I don’t know what I don’t know what’s happening right apache has like 500 configuration options and I don’t know what’s happening I need to try something out I don’t want to keep making a commit and pushing it out and then letting puppet pick it up and push it out there so we did find that we needed to lock puppet you know multiple times which is why we came over the infrastructure so what I would say is no it’s not it’s not a bad thing inherently to lock puppet and do changes but watch for that because what tends to happen this is what we have this is what was happening to us before we put this monitor is that people would lock puppet to work on something sometimes they wouldn’t remember to put the changes back in puppet or sometimes they wouldn’t remember to unlock puppet right so that’s the kind of problem you get into because it’s like once you’re done with the problem it’s 3:00 in the morning you just want to go to sleep so this is the way to do it you either remember to put it in puppet or you get a report that says puppet is locked and then you go oh I unlocked puppet then it undoes your changes and you go oh right and I also didn’t put it in puppet so good cool yeah yes and that’s the interesting way to do it that’s pretty cool actually Oh other questions back here up it tiny so interesting question now the first question was do we have something like puppet ID to keep everything clean no actually we don’t have a puppet ID so that would be that’s actually a good idea we probably should look into something like that to make it consistent right now we just sort of

spot-check and we vet and you so sevens and they come on you know we basically get the roll up newspaper it’s like that’s bad style bad you and then rub their nose in it and then they go we were to make write better next time the second question was how do we handle one off so if you’ve got a system that I just want to build do ten things on it and then send it off to somebody and that’s never gonna come back how do we handle that we actually don’t have that need so we haven’t had to do that Oh interesting so so the question was if we want puppet to do something to a system just once and then not worry about it afterwards there’s a couple of ways that we’ve done that we’ve done that in a few cases and we did something and I can’t rember this specific tag but you can do like for you an exec you have a tag that says and it creates this file and so if it created that file exists don’t ever do this again so we’ve done things by wrapping stuff in that way so we now we don’t have to go in there and go okay commit this to puppet run puppet once and now go yank that crap out of puppet again we just go look this is gonna run once it’s gonna create this file called I’m done if I’m done exists don’t do this again and then it just kind of goes on from there so small need that we haven’t had that a lot so any other questions one last question last question you get the honor so the question was because we have had some frustration with environments how are we are we still using them and how are we managing we are still using the environments and and they are still providing a little bit of use for us for instance I have anytime puppet-masters I want to upgrade like right now we’re looking at a two seven upgrade I have one test two seven server that’s on the master branch the rest are unstable and again this is where you know get really comes in handy for us because we do make a lot of changes that need to go right away that we make them right into stable and adamastor and using git merges is so much easier than doing subversion versions so you know I mean it’s been a lifesaver I don’t think we would have done an environment support until we moved to get I think doing with subversion was just a no starter so thank you very much if you’re in London next week check out my film okay so we have good okay so we have about ten minutes until our nice shock which is about automated deployment with seed bank and puppet and in the other room we have trust is the cornerstone of DevOps and I don’t think there’s anything going on in the breakout sessions but there is coffee upstairs if you want so see you in 10