David Maxwell: The Unix command pipeline – using Unix in the renewable energy era

math asked me to come percent again and I know all you people might be expected a certain type of presentation from me and I’ll do a little bit of that to begin with but then I’m going to do something different because I said to math I think people may be forgetting that I’m actually a technical person so I want to present something a little bit more technical at this time as well so I have some good news and some bad news for you which one would you like first and the bad news the bad news the demonstration you will see as a proof of concept and I’m not willing to give you the code just yet because it’s too ugly and the good news is I think that this is a tool that you might want to use every day which makes the bad news that much more frustrating realize who here in the room works on the unit’s command line every day so who didn’t have their hand up for the first lecture okay maybe maybe three people in them all right for everybody else I think that what I’m going to show you will be relevant but before I get into that you know this morning we have a presentation about a big platform attack and then we had the sorry application what’s up and then the platform presentation by Jordan and now I’m going to get down into you know the leads of the command line and what people do using a unique set of everyday so unique systems you know I learned them quite a while ago and the way that I learned them was a friend said I’m going to give you access to this dial-up EBS and I’m going to give you a few basic instructions to fault and essentially told me to do this not in script form but basically an echo this dollar path thing and you’ll see that that’s a list of directories and then you type ls’ and one of those directory names and you’ll see a list of programs and then you can type man and one of those program names and you’ll get instructions on how to use that that was how I learned the image now in doing that I said a lot of time reading about things like you know dynamic linkers that I didn’t understand at the time and I had no sense of why I wanted to know this but it’s still flowing into my head enough that later on when I was trying to solve a problem and I realized oh yeah there was a tool that had something to do with that you know figured out how to use apropos and find it again later it was very useful so what I want to start was you know asking some of you what was the first loop unix command you learned people at Dillard X so yeah some Mahler editors LS probably first man for almost everybody who just you know especially giving in without understanding other file systems with directories you can win you know that kind of introduction yeah let me ask you a different question what’s your favorite utility are Xers on your one 4xr what am i fine fine fine you know as Jacqueline as will Bachmann said when they’re going around the room at the beginning of the day you know the fact that UNIX has this tools philosophy of you know lots of individual components that do one thing and do it well and I can reassemble those and use them as I need to is a very powerful model that I think you know other operating system does as well as in Excel and in fact lots of them don’t really trying to accomplish that at all so you know which tools is your favorite probably depends on the kind of problems that you’re working on and what you have to do on a day to day basis but I know that I spend a lot of time in UNIX command line where somebody walks up to me and says I have this file and I needed to look something other than the way that it looks right now the the one that via you know struck my mind was somebody from the IC Development Group came over to me and they said I have this file it’s called a necklace thank you for laughing and it’s a you know I know you don’t really know what a necklace is but here’s what I need you

to do to it okay so there’s you know there’s many tasks like that and we learn different combinations of Unix commands that we can use to accomplish these destinies we figure out you know what’s the right tool for the job and of course that especially can include curl there’s more than one way to do it and everybody has their own favorite sometimes it’s fine like 2x cards sometimes it’s fine with – exactly you know there’s there’s more than one alternative way to accomplish the same thing in many cases and you know there are so many UNIX pants we don’t necessarily use them all on a regular basis and we have to look them up if we find some that we haven’t seen in a while even recently there are also a certain sets of Unix commands that I find it used frequently so if I said you have a file and you wanted to do a little bit of analysis sort of like that you know histogram or popularity measurements of commonality lines in the file there’s probably three commands you would string together to do that anybody talk beauty sort you need to and then sort again usually SNR right you have seen the sort that can are so you know we have learned these for the recipes as a unit user or system administrator that you know we can get from where we are to where we need to be with this particular combination of those recipes and they’re kind of passed down as oral history you know we we may read them in a book or on a blog posting or talking to some other men or we see somebody read something with the man line and we say how do you do that because you were faster at doing that but I would have been so you know I’d like to understand and you know because the UNIX utilities operate without any context you know the way the cranks work do command that’s being run runs independently it doesn’t know what came before what came after when you’re running something like unique – you have to run sort before unless you know that the input is already sorted because the way that the Niek works it depends on the input being sorted so you know that’s something that unique doesn’t do for us so there’s there’s a bunch of cases like that so as time went on you know I got to know the basic commands and then I learned about pipes you know and how data flows from one place to another and how you can use the old loop of one utility to start off something else does anybody else have other favorite sequences of commands you know I do a lot of you – ax-cut the sort are as well you know what other ones people know lots of double ampersand and just run this and this and this and this yeah and is this yes so we tune in our working environment you know if we have a particular recipe like that but we’re going to use a whole bunch we can put it into any at least in our shell of choice and make it faster to get out absolutely tail – a pan and wrap watching for things in a in a file as it’s coming in absolutely yeah so writing you know echo and somebody contract or printf and some contract and then I cannot – shell so metaprogramming yes yeah doing it with our yeah you do only the printf another person is inside of our head and of course you can do the same thing and generate other languages as well I write a lot of shell scripts that generate SQL scripts for example so you know it fundamentally being highly facile with UNIX is being able to manipulate data and so I said to myself you know I wonder when UNIX was originally being developed and you know there was this concept of pipelines you know what what were their constraints in the environment of time you know back in the late 60s early 70s when this was going on you know what are the sort of things that might not be true today because our computers are so much faster they have a lot more memory you know what might someone you have done differently if they were doing it today and herc probably knows but who knows who the two people on the screen are and one yes Doug McIlroy and Dennis Ritchie so lots of people know Dennis drinking on the right probably fewer people would recognize McIlroy you know from from a photo but Doug’s actually the person who suggested the concept of heights when units was originally being created and then this would be a useful thing and then also I believe suggested be the mechanism of the pipe symbol on

the shell command line as a way to tell this process is going to send its output into this next process so there’s there’s a course of a week computer are informed you can read up more about him and so in in looking at the way that this was being done and asking myself those questions about you know what what limitations were they working with and what design decisions did they make based on the systems that they had at the time I thought I would get in touch with professor McIlroy he’s working in at Dartmouth College now telling him about the project that I was working on and he didn’t get back to me when we’ve had some discussions of it so he’s interested to see where this goes as well so you know fundamentally the way the view fights hasn’t changed a whole lot since the early days of UNIX there are some things like if you use eggshell or corn shell you can do things like create more than one output and have sort of splitting and reemerging of streams so you can have a command you know mid pipeline that’s going to send its output to something that’s not consumed by the next pipeline stage and then something else they want to mix up again so there’s some you know different concepts like that but they’re not widely used and you know we do have new command-line tools I remember the day that I was helping somebody set up an internet service provider and I was starting them through a problem and I said okay what I want you to do is run you know LS dash L er for this and he said in doing that you said why didn’t you run located and I said well okay okay didn’t exist when I started using it and it was one of those things that have shown up they’ve been put in the net SD and I just never happened to observe it and got it certainly is a whole lot better I use it all the time of course so I expect that you know there are other tools like that that people use on a regular basis that maybe aren’t as widely known entry is another one for me then have a problem earlier in the year where somebody had a system that was modified from an original you know this was a work thing so is Linux you know there’s a Red Hat 9 and the modified version of it and they didn’t know how to go about identifying all of the changes between the two systems I said well that’s easy you just run em tree run entry on both systems and then you run em tree the member gives you the difference between the specification file and that’s on the local system so there’s a lot of power in these tools if you know that they existed you do not use it so you know let’s let’s look at how we’ve use the command line so I’m going to switch for a moment here and do sort of typical command line stuff so I have a couple files here like a you know the output of top and so I might and I’ll change I realized when I got to this room that the screen is very low for people refers it at the back so when I’m doing the demo I’m going to switch to an expert in this that’s just the top portion of the screen so I’m going to take this output from top and I might you know grab for something because I’m movin in certain lines of it and I might sort it americlean because I want to see that for some reason and then maybe I want to I don’t know not not that makes sense in this case but you know maybe I want to see that text on format and maybe I want to see line numbers on that and maybe I want to just you know know what the word kept other was yeah that line doesn’t necessarily make a whole lot of sense but it’s just a demonstrate sort of the ability of feeding one thing into the next and you’re all unit user so I know you don’t need to be told about that but often when you’re doing this kind of thing it’s not necessarily a matter of of just typing it I mean if you notice what I did I started doing more in the file and then I grabbed for something and in this particular case the output was small enough that are all thin on screen but if I was dealing with something bigger like assistant role the best many painters work and so now if I was going to go through a similar kind of process with this and actually I’m going to change the way that I do this just slightly out of habit so something like that I would typically bring my command line back delete the moorpark add in you know the grep that I’m looking for add the more back in look at that without that delete the more grep out of something go back and more so it’s this iterative process of Professor McElroy use the term of creating a pipeline as you go you know building up adding one more

and gradually debugging that it does what you want and who here has had the experience of you write the command line and then you start to run it and as you look at the output in a little bit more details you realize that one of the commands a little bit further back isn’t doing everything that you needed it to do you’ve made a mistake in a regular expression for example or you didn’t realize all the cases that you had handled and so you have to go back and probably delete the last few sections off of your command line figure out what’s wrong with that particular step in in the pipeline and then come back to it and then add the remaining fans back again is that a common experience so you know there’s certain things that I think you know it is there anybody here who would say that you know the example that I gave of sort of going through a history you know like that you know anybody disagree that that’s one of the most common things that we do if you’re using the command line so as I set myself to thinking about the fact that all of this was designed when it was I said you know what might we do differently today and I thought maybe it’s about time to bring a new perspective to working with pipelines and so I’ve created a proof-of-concept tool that I’m going to show you today it’s gonna be BSD license I I’m not ready to release the coders death because I know that it’s buggy I know that it’s got you know fixed size buffers and things like that that it’s going to go out in the world with so it needs a little bit of work but I share it with about ten people so far just discussing the concept and everybody has as basically wanted to copy of it now so after this huh I will get James to play bodyguard for me while I run out of the room so that you can’t get my laptop so everybody can see that on the screen so if I was going to doing sort of the same type of thing that I was doing before but I’m going to run in pipette and I’ll start with that top over now only so much of it is going to fit on the screen but what you’ll see here is that I have a sort of VI like interface they’ve got a sample of the file which is kind of what you would see from running more or less on your command pipeline and then I have cat talked Auto they file down the bottom stream and a few status flags in the bottom right I’ll get those now maybe the thing I’m interested in looking at isn’t right at the top of the file so I can put a page down to browse through you know further sections of the file but part of the the phrase tight cut comes from the fact that it’s going to take a sort of horizontal slice of a section of the file so that I’m only looking at one screen full at a time so right now I’m doing my development of the pipeline I’m just using a small buffer to do that which makes all the operations that are you doing very fast and it does mean that if I do certain types of operations so for example if I if I had a you know rep – II and and take something out then my – screen may not be full anymore backwards and at the moment that’s just the way the proof of concept is working later on I mean my back pressure and allow the later pipeline stages to pull additional data for the earlier stages as needed to make sure that the display is still up but let’s ignore after the moment you you I will be disappointed if you don’t see a craft in this demo because it’s like that when you’re supposed to have at least one I’ll bring the beer tonight for every crash going to have to examine this custom some screen corruption like here you see be the ogre that wrecked man – showing up on the display and it couldn’t so I’ll see if I kind of refreshed make that go away let’s go back in now in tightening the the unix command line we all get used to the commands so we get pretty fast out and if you’re working as a lunatic man like every day you probably get to be quite quick and you may have had the experience of people who are sort of professional cyclists looking at you and being surprised and although you don’t write memos and letters and things all day you’re actually quite fast on the keyboard the last time I did a typing test I just got 136 words per minute and I find that pipe cut makes me between four and fifteen times faster than that so I will start with some small examples and then I’ll build up to things that relieve the pain so in this I call it the eyelet interface I have a set of

single character commands or the most common UNIX utilities that I may want to make use of so as I was going to stainless let’s get rid of some of these lines like the lines with idle on them I’m just going to hit the letter X so that’s one key code and we see that it’s building up the UNIX command line as I go and now I’ll type idle and hit enter and the lines I want are gone and let’s drill that one with exact as well because now the interesting thing is not only can I do reps but I think at other commands so again that’s a single compress to get a short add-on to the pipeline now you know that’s okay and maybe it’s a little bit faster to type but now I’m no longer in my usual units command line edit environment and I have all kinds of familiarity with how I can jump around in the command line and change things in the middle and I don’t want to lose that capability so the second meaning of the term cut in pipe cut is to be able to take a vertical slice inside of a pipeline and look at what’s happening at different stages so if I press the left arrow key I can actually go back through the earlier stages of the pipeline and see what they look like looked like an except so here we are with the original file and then I take the items out and then I take the executive and then exported so the sort of thing that you know would be otherwise a process of waiting out of my pager you know bringing back my command line editing it go back to the pager several times in a row I think you know reproduce very quickly like that likewise as I said before if I put a page down to look at a different cut a different slice out of the file I can now go through the blades on those so happens that there was no reference to idle so there’s no pain when I go there but the sort bus paintings this sort probably should have been a sort back end because the majority of this file is starting with numbers so we’re not already have the stored on the command line at this point and and I’ll give you a terminology in a moment for what I’m using for these different pieces because it becomes useful to be able to refer to them now that now that you can interact with them the way that the school allows you to so sitting on me the sort I can press the minus key for often and then press the end and the sort becomes a sort at hand which I guess in this case didn’t make any difference because it was lexicographically equivalent but it allows me to modify that command that I’ve already put online as I said in this particular page the idle graph didn’t do anything so maybe I’ve decided that I go like that so I’ll press the Delete key and this happens quickly so whereas probably a lot of people don’t know this one of my hobbies is stage manager I’ve been doing after about thirty five years and one of the principles of stage magic is you don’t usually tell people what you’re going to do before you do it because you want to surprise them in this case if I don’t tell you what I’m going to do you’re not going to see if after I’m coming so I’m going to press Delete and it will take out that one stage of the pipeline so I just got rid of the grab – me Island that was there one thing that I haven’t built into the UI and I should emphasize at this point that because this is a proof of concept there could be a bunch of stuff that would change in the UI the the UI as is today is evolving as I think up additional functionality that I want to add and so it’s probably going to continue to change over time they’ll probably give more flags there later you know more options config file etc so right now it’s got some of the basics and I don’t promise to keep it exactly the way that the district now likewise I can go to the animal the command line and just press backspace to get rid of all of the stages that I’ve added to the pipeline put a dummy go back to the top of file and back where I started so note that this is the right sample data set for the sort of thing that we were talking about but if you were doing something like a sort – unique – need a sort – in our I’m gonna press just you for unique and unique knows that it’s input has to be sorted so because short wasn’t already the last thing in the command pipeline it added the sort for me because pipe crap knows about the context of different commands are going together and it realizes that that’s a requirement for unique to do its job correctly I think capital u is the unity that I was looking for and then of

course export – mr with a single keystroke as well because that one’s common enough that I gave it up so the kind of thing that I was doing on on the command line where I showed you even though it was sort of nonsense commands you know doing doing some graphs I can basically do the equivalent of that with that but quickly and and again I still have a look optionality to go through and look at the different stages so the terminology that I’m using here is that this command alignment that’s built up I call a tool set and each command within that tool set I refer to as a blade so I have the ability as I said to delete individual blades I have the ability to delete blades from the end and I have the ability to inspect at the point of any particular blade let’s let’s look at a different sample file so I’ve got a system full file here which I think they’re kissing fun to work with this is the the I find I’ve got demos better on a big screen because seeing more data at once is more interesting but I’ll keep it up to the top for visibility same so I took the cyst controlled output as an example because that’s what interesting words in it to two years of craft and so forth so let’s now I know I have a bug in the following mode or the motive is about to demonstrate let me demonstrate a different one first and then I’ll come back to a bug so let’s say that I’m going to exclude words with MCL bytes and I’m going to exclude words with an amine and which holy words file to include words with men and I’m going to go back to the beginning of the full set and I’m going to turn on one of these status lights that’s currently saying no la I’m going to turn that into la for look ahead and right now I’m sitting on the catch blade but the next blade in the tool set is the direct Ashby of M scale fights like that knows that so it’s giving me syntax highlighting of the regular expression matches that are in the screen that I can see if I want to browse around in the file I’ll see if there’s any other cell types references there aren’t and I have a I have a weird person saying if there’s somebody here in those purses you want to come and talk to me afterwards I don’t understand why when I say stand out a moment I sometimes get reverse and I sometimes get dim reverse so if you can help you with that I’d appreciate it as I move forward through the blades and I’ll go from highlighting MCL bytes the MCL bytes when went away because it was a grab – p.m. now I’m highlighting the NMB which is the next blade now I’m highlighting I’ll switch to the next blade now I’m highlighting them which is next way and so you can see exactly what’s happening what’s what’s about to happen to the file at any particular point as you’re going forward if you’ve got to look ahead mode turned on and processed that’s optional I should mention that all of the output at each of these stages is cached it’s a small amount of memory and we’ve got lots of memory in UNIX machines these days so it’s not hard to keep all of this thing and that allows me to go back and forth but quickly here I’ll also mention the black box in a moment but first let me give another example so this is the mode that I said I know has above right now that I just haven’t gotten around to fixing so I’m going to turn on the re moment the regular expression mode and I’m going to add well let’s say in buff through the sake of so I’ll press X again the graph and I’ll start typing mbox you’ll see that my command line is going to get messed up here you won’t see what I’m typing so all this to does ago M app so now I can get interactive highlighting of the mattress the regular expression as I’m typing up this I think the dress is one of the main pain points that I was talking about earlier when you’re editing a UNIX command line you type a regular expression it doesn’t match what we thought it was going to match this now gives you a way to you know browse the contents of the file as you’re typing that and see what’s actually going to happen when that in this case of grep – me was a and again I could look ahead with that but I think do that you know whatever I’m typing it yes sort of let the what the word is so I’ve got a little bit of a display issue there at the bottom when I’ve got to go the the caching that I mentioned on the blades of course can be turned on and off that’s not a big deal and the filter function isn’t perfect my deck at the moment so I’m not going to talk about that now let’s let’s say for a second that I’m looking at this file and I decided to grep – B which once once i

take off the gyrator thread so we grabbed actually out the perm and like I said there’s no back pressure so right now I’m ending up with all these empty screens because every single line in the system folders obliterated but that you grab that return and when I get down to some place in power there’s no references to permanent I start seeing it but now I built up this command line let’s let’s add a stork to it lets out a hex dump let’s add a hex dump to it just for sake of argument now let’s say that I got the only look at just one slice of the file and you actually want to see the entire output of what happens when you run this command line well then I can push the powerful symbol on the keyboard we call pipe and it will actually run this command line externally and pipe it to last so now I’m no longer in pipe cut I’m in less and I’m looking at the awkwardness or less foot maybe of the pipeline that I’ve built up and of course I can quit over there and go back to changing what I’m doing and I can go back to the last you can look at the output for any other state any other blade that I might care to to stop at and see what the output will be now likewise let me let me talk about a couple of different things here so behind the scenes there’s in pipeline an abstract syntax tree representation of these UNIX command line elements that the blades so high cut has the ability to look at things like say you know you didn’t do a sort right before this so when you ask for unique I’m going to add the sort because it can look at those stages in the abstract syntax tree and have conditions that have to satisfy it but one of the other things that you can do from that exact syntax trees you can do code generation so at this point the command line that you’re seen here of course is not something that I’m typing in editing as per normal right but as I’m typing these hotkeys I’m actually modifying the abstract syntax tree and then I’m generating the UNIX command line that you know is representative of that so that’s one thing that I can generate but another thing that I can generate currently mapped to pressing the exclamation mark is this particular line that I just had the taxes control rep – and turn sort sort NR has now been written out to a shell script so whatever I build up on the command line I can write into a shell script forgive side pipe that I want to do that additionally and I haven’t finished implementation of all of this yet but one of the things that wake up will do is of course have the ability to save and reload the abstract syntax tree representation so it has a sequel light database and you can apply a name and a description to any particular tool set that you build up so that you can come back and you later on what else be the code generation that I’ve shown you right now is kind of dull because it’s generating shell script code or UNIX command-line utilities in a pipeline but since since pipe that has an understanding of what each of these elements are what I’m going to do is provide a translation mechanism so that somebody can write a translator from unit utilities into other languages so they can generate for example a Perl script that executes functionally the same equivalent of what you’ve built up in your toolset through simple concatenation and mouth you know everything has to come in this dollar underscore everything has to exit this out underscore and then I can simply be composition to put all of these pieces together and give you a not marvelously efficient but basically functional Perl script that does the same thing that you just didn’t expand line and the advantage of that is that since we’re so familiar with working with UNIX tools we can very quickly do things in this kind of environment that you know might take a little bit longer if you had to write it out in Perl so being able to generate Perl code that at least lets you include it in that program that you’re already working on very quickly and easily it’s often a useful thing and of course that’ll be Perl Python Ruby MapReduce C C++ Java B B’s group are the babies crib Phoebe’s preeminent JavaScript you know I thought about trying to do push press commands so you know there’s there’s a lot that can be done and I’ll do two levels of API interface for the cogeneration I’ll do the one that I just mentioned which is the simple concatenation of functions that provide equivalent functionality but then I’ll also provide a second level API so if somebody is a dedicated Perl programmer animal to some of the people that I know that are and they want to spend a little bit of time doing this they can build a custom code generator that understands

how to work with B ASP model and then they can do a smarter code generation of throw instead of a simple concatenation they can actually make intelligent decisions to put things together and I’ll give you another example of something that becomes possible once you have this kind of representation of what’s going on in the command line is that in the example that we talked about a couple of times with the sort unit and then sort fashion are what this is doing in the UNIX command line is the following it’s running this cat of this file or in this case is not extremely large but you know let’s pretend it was larger data set that it is it is sorting all of that input so the sort program is running it’s being forked and exact is consuming all of the input from this file it’s been doing a quick sort or other efficient sorting algorithm on it it’s been generating the output for all of the lines in that dataset and then the unique program is running and unique is consuming all of the lines in that dataset you need to find them and then putting the subset that results along with the couch and then sort is consuming that and sorting it but that’s a smaller dataset so we’re not so worried about that if this is actually even efficient are people familiar with the you know half nap counting mechanism so some people is this is a kind of trick that’s used in things like MapReduce frequently because it’s distributable well but even in becoming a non distributed environment like this it still makes sense take the output from the top of the original file and instead of sorting it just make entries in a hash table for each line that you see an increment counter associated with that hack entry and now you’re doing the talent and the unique name all in one step and you’re not generating all of the output and you’re not consuming all the output again and generating it all again so you save you know a significant amount of work and if the data set is large it’s a significant speed up so you know that’s an example of one optimization that I know so the ASP as it currently exists is fairly little you know there’s one entry in the st for each line each each believe in the tool set in an earlier version of this I actually had been working so that he did a rep – be like that it would actually combine those two into one Egret – B max pipe host and I took that out when I added additional commands for the reasons I’ll skip right now so there will be things like that sequence of a short unique that she sort in are that there’s a more efficient implementation of and so as people you know feed suggestions about optimizations into whoever’s working on continuing to develop I thought you know we can add those kinds of things so that the code that’s actually generated run us can be faster than what it would be if you ran it on the command line in addition let’s suppose that I was working on something in unit 3 online they did when they find command that’s relevant so let’s take that 20 that I did before that that looked like that well perhaps I’ve started working on something on the command line and now I’ve decided that I want to use it in high pass well they would be paying to have to build that all up again even if it is only one keystroke per command in this particular case these are all single commands and I know that this isn’t going to work right it was working last night and I don’t know what I broke you know right before I lunch man I do like but what I can do is I can say history type to hike up and bite cut will harsh your command line history look for the last command that had pipes in it and turn that into a tool set so yeah the error message there is what I wasn’t yet the last night so it’s not actually running correctly but you can see that the tool set was built up they out of thing that the demand of stuff first from the command line history and that reports CSH tcsh back one shell at the moment they they’re all the two different versions of history formatting that would based on the time stamp that’s shown so this this brings me to talking about what is probably the last aspect that’s useful to understand for functionality like what I showed you with the regular expression syntax highlighting as you’re typing for you type that has to have its own internal implementation of graphing the graph – me and it’ll need a stat and it’ll leave a couple of other things anything where I want to be able to do highlighting a

regular expression it’s got to understand you know what that command looks like it which part of it is the regular expression and I also started they were doing that with an internal implementation of some of the tools because I didn’t know what the performance would be like if I was running things outside of cut cut but if I’m going to support the model that I just showed you where you can take any history that you typed and put it in the plate cut and by the way I’m sorry on that I’m going to see one other thing that you can do I can also take arbitrary commands on the fan line I can just type a single book and it will let me you know type something free form so it doesn’t have to dress the command so what goes into like advanced but as I was saying consuming from the history for example where typing something freeform on the command line here one of the things that the ast and pipette supports is a concept of a black box saying I know that this is a command I don’t know what it does you know maybe it’s pearl – me something and I don’t have a Perl interpreter built into Python and there will so you know how how many like let’s deal with that well as long as I can parse the line as well as any other shell would and understand the coding to tell you you know these are the demarcation this is this is a pipe that’s not inside some other kind of coded contract and it’s actually leading it to the next weight then I can go ahead and just stick that in and say I’m going to treat this as a black box I don’t know how it works but I know that if I exact it and my attack pipe stood standard in and standard ever I can run data through and get back what the user is expecting so that’s how actually a number of the things that I’ve been showing you like the implementation for me to add this hex dump taxi support took about three seconds in my development cycle because all I did was add a line that says when you type age create a new black box that says heck stop – capital C so any standard UNIX utility that I want to add to my alphabet of hot key commands is a very small amount of code to add and as you can see even though I’m running that well it’s cached now but if I turn the tassel off even though I’m running that because I’m only running it on one screens worth of data it’s quite fast I think that’s about everything that I wanted to talk to you about and I’m getting down to just a couple minutes left but I’ll take questions there Mike the right now the look ahead does not work two black boxes it only works for the built-in commands with the ring expressions like the breath and the graph Nephi was her particular command you were thinking it would be useful for okay no I suppose what I could do is take the vo put before and after running the black box and do a dip on it and then highlight it that way I have an implement of that take input from history but can we just get out so right now so the question is can can picked up pipe that takes standard in instead of a file in order to keep my life simple for the time being I’m starting every pipeline with a cat because if I don’t start it with a caste and I have to understand where does the file name or where the source of data is in the furry command that I’m consuming so just for the proven concept sake as a simplifier I start with cat one of the things that I mentioned was you’ll be able to name a tool set and call it back up later so one one example new space that I didn’t show was running pipe cut – T tool set name whereas it’ll pull that tool set from the sequel life database and then run as a filter so use an inline to execute a set of tool set that you built up previously and I’ve been asked to also be able to do that from a file instead of for the sequel like database so for portability reasons and you also probably have options to to say specify the input some other way what reason why I off is off using the box online time an option to send results set properly after a bike horn like this yeah so so the example is taking output from things like might equal and reformatting of vodka definitely I can add an option on the command line to say don’t read the input as history consumption I think right now as I said the UI maintained a whole lot right now I think the history consumption is the most intuitive format and then having some option to say the new processing on the input data instead right now the

implementation uses the fact that it can open the input as a file and sees around in it for the page up and page down in part so if it’s something that standard in I’ll have to consider buffering and like less does and that changes lushniak so I think about so the black box commands does a running in your shell or ATM and it calls exactly with it so if you do something that’s complex like or uses your aliases yeah this is so I am not certain what the interactions of everything will be there on optical yes all right I have a question observation first question is if you have something generating megabytes of alpha very headedness yeah how do you scroll horizontally through the chunks of data so you don’t got very long line lines no no let me say very long output Oh them so number of lines yeah mega continuous yeah yeah so so for example in this sit control volume which is sort of I always forget I go to hit word count and I forget that it’s only giving me the screens or worth workout so if I do the word count and then I get pipe that tell me there’s 630 line can that’s all together so if I put page down I see another screenful at a time the way that I’m doing that is I’m keeping track of where the file pointer is when I finished the previous eight and then I’m starting from there to show you have a further make the observation I want to make what by the way I think this is your equivalent or chosen kind of something you can do with the compositor with your ears but it’s also kind of ironic that you started from the perspective of simple tools and mechanisms that are being leveraged out because the first thing I thought when I saw this is we need like a pipe drawing system call that lets you actually add these into the biscuit cutter in other words quite should have multiple inputs and multiple language shouldn’t just be here a single segment pipefitters can do right you have that now your staff gets vertical as well as horizontal because I should pick any join point that I can create a different pipeline off of that specific point so that’s what IPS tree output right so one of the things that I thought about kind of along those lines are that sometimes the the data that you’re doing within UNIX is no I’m always lying or sometimes it’s record oriented and those records in either X number of line records like I’ve seen some tools that I put three lines of the time where they can have some sort of demarcation a header line and then in content lines so my thought there is the bank that will allow you to have an operation that’s a classifier you can either give it information about the record and saying these are three line records and then the classifier is implicit you know what Samad three the line number of this file or if it’s based on the d mark you can have a classifier that then what you’re saying I want to have the following tool set for first lines and following other tool set for second and third lines and so what you essentially have is a pipeline that force the data runs through it and then emerges back together for input closes so we’re a UNIX command line you typically get to a point where you have to write a Perl program and build a little state machine to consume that now you’ll be able to do it interactively in the command modifier any more questions yeah how soon can we see this as part of course it depends a little bit on my time to you know to get it cleaned up I feel like I haven’t spent that much time to get on get it to this state and things things like the – and to modify the options on command I wrote while I was waiting for the plane yesterday so you know adding features is pretty quick once you have the ASD built up and you’re just doing modifications on it so the things the things that I want to clean up before I put it out there are things like the fix buffer sizes and you know the occasional SDR popular sub skl cotton and stuff like that so I just want to get the code a little bit cleaner and and I think but I’m not going to wait until the you guys finalized or anything like that I will lend it out sort of in a similar state to what it is now understand one people you know famous things may change because it’s so they’re impatient customizable keybindings because my behind is absolutely I’ve already been asked for GMAC ski bindings I think occur to me for some reason but I understand why it’s upright and the ability to remap keys and text files and having bedded color codes color codes in

what you’re okay because that’s a data set that you work with frequently not okay now one haven’t occurred to me I will promise put it high on my list but it’s a fair suggest and I will start keeping up bug list and request those Christmas present perhaps I whom I will endeavour to get this a new state where I’m comfortable sharing it sooner rather than later I have a couple of friends who have offered to look at the code and help me you know clean things up a little bit so I’ll try and get them involved at this point I’m literally the only person who’s worked on the code I did want to get it to sort of certain level of completeness before I started you know accepting input or contributions from elsewhere right so the question is about JSON input and like treating other forms so kind of what I was saying about records and you know number of lines and classifiers another thought that I’ve had is that there are certain standard formats like m box or JSON that I might like to be able to use a tool like this with Professor McElroy actually played me in one of his graduate students who did a set of tools called actual tools which are UNIX tools for non line oriented data manipulation so I haven’t had a chance to look at that yet but that might be another way of doing part of it I realized I left a mention of a significant feature that’s not implemented yet that is on my list of things to do news related to this data file signatures so in the same way that the file command lets you look at a file and tell you for what it consists of if you feed something into pipe cut it already spawns off a thread that runs in the background that does little bit of analysis on the data file because I wanted to do some of that statistical analysis asynchronously there’s there’s a representation that’s not working at the moment where you get to see the command line in the top down view and what I want you to be able to see there is sort of the word count output of each stage of the pipeline so you can get a sense of what’s happening as data flows through the pipe but one of the other things in there is that if you are willing to do a little bit more analysis and the bin file command tries to be very quick and tell you what something is in a bath yes but in this pool I’m willing to do a little bit more analysis I look at your data file and every single line is it seem like every single line has 9 commas the stuff before the comma is always a numeric do you have a 10 field comma separated value file with numeric alpha date in or whatever and now pipe cut potentially warn you if you’re going to use the same tool cell phone and a data file that looks different you can say this may not do what you expect you’ve got a mark that refers to field number 10 and you only have four fields you know and when you go to add the arc it’ll add to that half comma for you because those associates yes for a lot of work here you already said pearl was there yes what you’re talking about here with pearly input reader was done a long time ago for four units and look with input mackerel by macros idea but when Michael I don’t mean Michael I came across do that Dorothy Ann 30 if you do something on the other side the templating you satisfy green content law which is that it’s like any program of any party service mr. badly that’s entirely possible I’m happy to live with it all right there’s there’s a command line commands appliance well yeah okay I should really stop because I’m going over my time but I’d be happy to talk to you after this