A Historical Architectural Tour of PowerShell
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Payette, Bruce
Join one of original PowerShell team members for a look at how PowerShell has evolved over time. We'll look at both the technical and environmental drivers behind many of the decisions that made PowerShell what it is today.
 Okay, hi, I'm Bruce Payette. In this talk, we're going to go through, we're trying to trace the evolution of PowerShell from the very beginning up until I run out of energy and we run out of time. Because this is a 45 minute talk with three hours of content. I want to look at some of the cultural, technical, and some of the business forces. There are artifacts in the source code in particular that have been impacted by some business decisions in the past. This is the first round of at least two rounds of this talk. The ultimate goal is to gather all this information together into a document that will go on GitHub so that when people come, it'll become part of the contributors package so that you can deeply understand what we're doing with PowerShell and why we're doing it that way. Hopefully that will help people more easily evolve the language in a coherent way. In the beginning, things always start there. PowerShell started in 2001. I joined Microsoft in December of 2001. There was a project that was being funded out of the India Development Center, IDC, in Hyderabad, India to investigate the idea of a new shell. The code name at the time was Kermit because the person who was organizing this project had a child who had a kid's book about Kermit, the Kermit crab who lost his shell. I think we kept the Kermit name for about six months. It does not show up in the source code anywhere. The project was staffed by people coming from the Services for Unix team. Part of Services for Unix was a product called InterX, which was an extended version of the POSIX subsystem for Windows that was almost fully Unix compliant. It made sense to have people who had some experience with the shell and utilities to actually do a new shell. Why did we do it? Because especially at the time, Microsoft was very GUI-centric. The command line was pretty deprecated for a lot of stuff. We had commissioned a survey that had gone out and asked people about doing automation on Windows. The survey had returned the result that it took ten times as much effort. It took ten times as long to do an automation, to automate something on Windows versus Unix, which is a bad thing. We started to care about Linux and Unix a lot at that time point. It was becoming a significant, it was clear that it was going to be a significant competitor in the future. Finally, there was an evolution in culture. We were moving away from the era where you could have one machine in your data center and you were good. The number of machines in the data center was skyrocketing and automation was very important. We have all of this going on. There's one thing that's really important that's missing, and that's a certain gentleman sitting over there, Jeffrey. Where is Jeffrey in all of this? Because he wasn't actually involved in the Shell project at the time. However, he was over in Building 42, madly prototyping a new type of command line based on some radically different set of concepts. A command line based on the composition of functional units, or FUs for short, with an extended viewable type system. You could describe a command line as being FU, FU, FU, and really get your anger out. This prototype was used a different model of composition than what eventually showed up in PowerShell. If you look at, if you can see the line there, the second last code line from the bottom, I think we were still doing, I think we were doing verb noun at the time, and the separator was a slash. But the idea, conceptually, it was more a case you were doing extensions on existing commands rather than composing a set of discrete commands. When we looked at it as part of the design review, we decided we would go with something that was more similar to the traditional Unix model. I don't know that it was, there are functional aspects that changed, but not so much from the prototype. And that's a discussion that's too long for now. So Monad begins. First thing we had was a bunch of guiding principles. If you haven't read it, I'd recommend reading the Monad Manifesto. It dates from that time, but it's surprisingly still very, very accurate. And if you really want to understand what Jeffrey was thinking at the time, then this is a great document to read. We had a bunch of principles. To read, to use is very important. Maybe more important than being simple. You needed to be able to get things done. You needed to be able to deal with sophisticated tasks in an effective way. And so if you needed C sharp, this is probably going to get me in trouble, C sharp versus visual basic. Visual basic is very simple, but you have to take a lot of steps to get something done. C sharp or PowerShell, you can get stuff done with fewer steps. But it requires a little bit more sophistication. So we wanted it to be easy to use for the sophisticated user. But learning is also very important, and it's hard, right? So we need to facilitate learning and protect that investment. This is the sacred vow that Jeffrey talks about. If you learn something, you won't have to unlearn it. And one of the big drivers on this was consistency. So if you learned a principle once, then you'd be able to apply that principle over and over and over again. The one thing we wanted to avoid was the foolish consistency where everything would look like itself. There are still a couple of cases in, maybe a couple of cases in the language where we were a little foolish. We also wanted to address the tension between whip it up a toad, which is a pearl word, and production coding. The survey had said that sort of ad hoc automation was very important for Windows. The ability to write small scripts or even doing things on the command line was really important, which implies the ability to do a lot of stuff on the command line versus production coding where you want to be able to support the code that you're producing down the road. This results in a very wide dynamic range in the language, and there's a lot of ands, so methods and commandlets, simple functions and advanced functions. One of Jeffrey's favorites, command line and GUI. We also have some features that let you sort of scale your syntax with things like aliases and short options. You can have the line at the top, which is the fully elaborated line, and it's very readable because everything's fully spelled out. Then the line at the bottom, which is, I think, a little less than half the length of the top line, is the same command but condensed, so much quicker to type. The bottom line is good for the command line, and doing quick things is not recommended for putting into a script. Reuse. It's super important. We wanted to leverage reuse as much as possible, sort of extreme reuse, if you will. So common argument processing for all commands, ubiquitous parameters, common runtime services like the way well cards are processed, the way paths are resolved, and so forth. All those services are made available by the PowerShell runtime to people who are writing commandlets or scripts. Common formatting and outputting. Nobody has to write formatting code in their commandlet. We take care of that for you. All time is more important than processor time or disk space. At the time, everybody was reasonable to have a whole system to yourself, so throwing away resources to make the user faster was a non-issue. That's changed a little bit with multiple VMs on a physical machine, with IoT and resource constrained environments. So some of the earlier decisions that we made regarding resource consumption are biting us a little bit now as we move into these more constrained spaces. You can see some of PowerShell v6 were evolving and reducing some of the footprint. We wanted to provide friction on negative paths to use a probably incorrect analogy. We'll let you shoot yourself in the foot, but we won't hold the gun. So we have dash force on destructive operations. We have the whole dash confirm mechanism to confirm, again, operations that are going to make significant changes to your system. Just in general, we wanted to have aspects of the system that let you do something but didn't make it easy if it was probably the wrong thing to do. You had to think about doing the wrong thing. We wanted to facilitate experimentation. It should be safe to try out commands. So we have the what if ubiquitous parameter, and then we have declared annotations for commands and their parameters. So the logic for binding parameters and doing checking and so forth is done declaratively through a set of attributes rather than everybody writing their own null checks. The project begins, and we acquire a team. I started out of IDC in Hyderabad. The architectural owners were all here in Redmond, but the dev team was going to be in India because, well, for a bunch of reasons. But it turns out that radical experimentation does not flourish with this level of geographic distribution. It's a 12-hour time difference. We were awake, and there were 12 and a half, depending on daylight savings time. We were just using mail at the time to communicate, and occasionally video conferencing. So it wasn't a great environment. I think it might be better now, now that we have better tools for communication, we have GitHub and so forth, that geographically distributed development might succeed where it didn't work for us then. So anyway, that didn't work. First we pulled back development with the idea that test could still be there, but then even that really wasn't working very well because of the day lag on everything, because things were changing so fast. So that didn't work out. We went shopping and we acquired another team. This one was built out of team members from the Windows Management Pack team, which was great because they had a lot of management experience for Windows. Now we have a team we can begin for real. Oh, yeah, and be ready to ship in two weeks. This was all in the Longhorn timeframe. Does anybody remember Longhorn? Yeah. So my impression of the Longhorn timeframe was that Microsoft was starting to dip into the idea of continuous delivery. I could be wrong, but that's sure what it looked like because we were supposed to be ready to ship every two weeks. So it was exciting. We were not, we did not have the infrastructure to support this kind of thing. I'm now lost in my slides. Okay. Anyway, this was clearly not an ideal situation. We would have shipped a PowerShell that had almost no features that would have been not much more than Jeffrey's prototype, which was disturbingly functional. But still, I would have been missing a lot of the elements of a shelf. You wouldn't have been able to write scripts in it, which shell scripting language kind of needs to do that. So it didn't happen. That's good. I'll talk about that a little bit more later. So now let's talk about some of the big ideas. Big ideas are, we talked about the principles. They're very sort of generic principles. The big ideas are more architectural characteristics of the engine, more concrete than abstract. So one big idea is the idea of domain-specific vocabularies. So PowerShell was going to have, you would basically write a set of words. If anybody's familiar with the fourth language, there's this notion of domain-specific vocabularies where you would write a set of words for your domain environment. And this is different from a domain-specific language in that there are no syntactic elements that are different. It's just a set of nouns and verbs within your application domain. So we encourage the use of a constraint set of verbs. We didn't enforce it at this point. We didn't start doing anything around enforcement until version two. Skyline's not much verbs to use, this made it very predictable with a small set of verbs you can, the user can infer if I need to get something, I probably need to use the word get instead of fetch or whatever. Yes, we did explore and, the set of verbs is going slightly over the years, but not very much. I think we've only added a handful of verbs since the very beginning, which is kind of amazing given how long ago it was that we were doing this. So we had verb pairings, start, stop, get, set, again prescriptive so that the user could, again if the user knew that there was a start verb then they can infer that there's a stop verb. And then command aliases for interactive use against the whip it up a to do thing. There are two types of aliases, there are the canonical aliases like GCI for get child item which are specific to power shell and then there are the convenience aliases like LS or DER. This worked pretty well in Windows, but kind of got us into trouble when we moved to Mac OS and Linux because we were now hiding some of the existing commands on those platforms, especially curl, curl was a big problem. So big idea universal command line parsing, unlike most shells the commandlets are not responsible for doing their own parsing, the common parser code does that and breaks everything into, breaks everything into data structures that then get passed to command type specific parameter binders. So advanced functions or commandlets each have their own parameter binder and you can find all of those things under the engine sub directory of the SMA directory in GitHub. So you want to take a look at how the parameter binders work. This approach gives us broad consistency across commands, commands always have a dash, they can always be separated by spaces and so forth. This works really, really well for all of our command types except native commands that is executables because they still do their own command parsing. They take the argument string that they're passed and then they parse it up into arguments. And so native command binder had to take his arguments and then turn them back into a string kind of like what we think the user meant when he typed them. This works okay sometimes, sometimes it doesn't. There are a couple of bugs in GitHub around this and we're trying to fix some. We fixed some but it's very, they're kind of hacks. They're not appealing from an architectural perspective but they do make the user's experience better. This is one of these cases where it's not quite the foolish consistency but being super consistent makes the end user's experience a little rough. Declarative parameter constraints rather than writing imperative code to do all the checking, we have attributes to deal with that. These attributes did a couple of things. One they reduced the amount of code you had to write and two they reduced the amount of error messages you had to write and internationalize and so on and so forth. And it also eliminated or reduced the number of error messages that the user had to learn. So there were, this was a win-win-win situation except when it didn't work. So you can get into situations that are suboptimal like the regular expression, validate pattern where it tells you the regular expression that was like, the regular expression for a phone number. It doesn't tell you your phone numbers in valid, it tells you it's some like strange sequence of characters, gobbledygook that is not right or you're not matching this pattern. So it's not a great user experience if you use that particular attribute. And then sometimes this whole idea of attributes just didn't work. This is a comment out of the process, get process, get process, start process, commands in the source code, basically process name, glob attribute was deeply wrong. I think it's pretty funny comment. I'm removing this comment from the code by the way. It does have the programmer's name associated with it so I'm taking it out. Providers, so we were sitting around in a conference room discussing how many get commands we were going to need. And then somebody, and I don't remember who it was exactly or whether it was simultaneous invention, decided that wait a second, if we had a standard set of command list and just created this sort of pluggable model for hierarchical data stores, then we only have to have one get command because it will work for all of the stores. And so we developed what are called the core command list, item, trial item, item property, et cetera. And these sit on top of the providers which are in the source code, they're called namespaces. So if you want to see how all the providers work, look at that location, sma slash namespaces. Again in the effort to be consistent, we decided that everything could be accessed through providers. So file system makes sense, registry is a hierarchical store, sure that makes sense, functions, okay, that's a little strange. And variables and the environment and things that like the WS man configuration, although it doesn't come until version two, could all be exposed to the same set of command list. So you could navigate into these stores using CD basically. And basic provider options are also available through the variable syntax. So this is kind of dual ported. You can, for example, define a function by assigning a script block into a name in the function drive. A big idea as an extended type system. We wanted to build a management oriented type system. And we were going to layer it on top of the existing.NET type system. We had a wild dream of this idea of a central type extension system where you'd be able to publish patches to the type system of the running code. It was very cool, it didn't happen. I leave this as an exercise to the audience. We worked with a variety of teams at the time and a lot of people, not just PowerShell team, a lot of people thought this was kind of a good idea, having this management layer of the type system. So we worked with them about doing some of the designs, but ultimately it got lost in the world of schedules, I suppose you could say. And so as near as I can remember, there's only one actual result, and that's the fact that arrays have a.count property in PowerShell and they don't in.NET. So you can always say.count on a collection and get a number back. We also had the synthetic type system. This supported the idea of deserialized objects. So objects that deserialized object remembers what it used to be, not what it doesn't deserialize as the original type because the original type might not exist on that system anymore, but you can still work with that type because it remembers what it used to be. And we also allowed you to create new custom objects that didn't have a type associated with it. You couldn't construct types. You couldn't construct types dynamically in the CLR in version one or version two, I guess it was, anyway. But we didn't let you construct types. Big idea, PS object. This has come up a bit lately on some of the GitHub issues. Jeffery's original type system only allowed you to do class-specific extensions. So we also had scenarios where we wanted to be able to extend an individual object attached information to an object. And so we have PS object. PS object is a class that wraps instances of other objects and holds all of their extensions. So no properties, script properties, code properties are all wrapped in the PS object. It simplifies a bunch of things because you always go through PS object. But it also makes some things complex because you have to always deal with the PS object. C-sharp code that has a parameter of type object has to deal with a PS object just in case and then get the underlying object if it needs to. PS object is mostly intended to be invisible to the script user but is not entirely invisible to the script user. So it is part of the PowerShell API, which is part of the overall PowerShell user's experience. It's to be really clear about that, but most of the time you don't need to worry about it. And then we have the typos objects I mentioned earlier, the PS custom object type. This ability to cast hash tables into custom objects is a V3 thing, I think, but you can still do it in V1 with add member. It was just harder. The implementation. So the initial design was highly componentized. Most of the components in the service code, they live in system.management.automation.engin. So you'll find pretty much all of the pieces of the core PowerShell runtime in that directory or subdirectories. Some of the pieces include, on version one, it was run space configuration, which was all a set of initial trends that you had in your run space when you started up. Well, yeah. Run space configuration, largely supplanted by initial session states, starting in V2. The language that tokenizer parser and all of the execution stuff was their execution context, which includes all of the stores. So aggregating all of the various provider instances into one single object. Command processor is the piece that runs pipelines, kind of obviously. And then there's all the command processors that are associated with that. And finally, there's the error handling subsystem. The problem with this design is that you basically had a bunch of Lego blocks and you basically had to build your own shell. And it was not super trivial. It required a certain amount of understanding of how the pieces went together. It meant you could do a very minimal shell. You could have, in theory, you could have the pipeline processor without the engine, or without the language at all. So you'd have a very minimal shell. You could have something that didn't have language. It didn't have most of session state. It didn't have formatting and outputting. And it would give you a relatively small footprint. Anyway, to deal with the complexity, we decided that we would bundle everything together into an Uber class called the run space. And what is a run space? A run space is a space where you run things. I've never understood why people find that hard. But people found it hard enough that we have a wrapper class called a session for promoting that sits on top of a run space. And the run space sits on top of something called automation engine, which so it kind of turtles all the way down. And then the run spaces are exposed to the APIs, but not necessarily through the shell level. Context objects. So execution context. I said, we had session that included a run space that included an automation engine that included, and it also included a context object. Context object were all the resources that the running engine needed. And it got passed around all over the place. So every node in the expression tree of the compiled language had to get this thing. So you got to the point where it was just too much trouble to keep passing it from class to class to class to class. So we ended up sticking it into a thread local storage, a local variable, and it's accessible through get execution context from TLS. Get a fairly straightforward name. But you'll see that in the code in a number of places. You'll see execution context all over the place. And then you'll see people calling this API to get their execution context. And the execution context that's inside the engine is not the same as the execution context object that you see in dollar execution context, because that is a facade object that only shows to part of the public parts of that object. We'll talk about design patterns in a minute. Provider context did the same thing for providers. It provided all the provider context. By wrapping everything up into the context objects, though, this gave us the ability to do multiple run spaces for process or for app domain if you're a stickler for details. This means that we can do parallel execution. We can do run space pools. We can do all those things. This was, I'm not sure how super deliberate it was. I mean, it was, yeah, I guess it was. It sort of did something we did because it seemed obvious rather than being, oh, yeah, we should do this. So lots of design patterns I talked about. We used design. It was the era of getting the four patterns. We used a lot of design patterns fairly informally throughout PowerShell, the PowerShell source code, PowerShell design. We have adapter pattern in the type adapters. So we have a type adapter for various, like, data stores, done net objects, synthetic objects. All have type adapters. We used the facade pattern a lot for making all of the commands look similar. We used delegation in the command binders. So the command processor orchestrates the commandlet, but it doesn't do the commandlet binding. It uses a separate delegated class to do that. And the delegated, the commandlet binder, for example, is used both by commandlets, obviously, and by advanced functions. We have factory pattern in the Runspace factory, strategy pattern in command discovery as to how we go about finding commands. So again, if you're going through the source code, you'll see a lot of the stuff should look familiar because of all of these patterns. It should be relatively easy to figure out the role of the different objects in the running process. OK, now we get to the PowerShell language. Now, as I mentioned earlier, in the original architecture, the language was actually conceived of as being completely separate from the PowerShell engine. You would be able to not have the language or even plug in another language. Somebody had, at one point, had expressed interest in doing a command.exe style language for PowerShell. But that didn't happen. I say that a lot. In practice, the language has become pretty deeply mixed into everything because it's recursively used. The language uses the pipeline, which uses the formatting system, which uses the language. So it's pretty difficult to separate now. I think one of the reasons for making it separate is we weren't sure that the language was going to be right. We had grave concerns about that early on. I think we're OK with it now. It's only been 16 years. I think we're fine. The language roots. Where did it start? We started with the IEEE POSIX.2 shell grammar, which is essentially the corn shell, which is more or less bash, which all come from the born shell. POSIX stood for portable operating system interface with an X on the end because it sounds cool. It's a real designation. There are big changes to the implementation, though, because I'll talk about that more in a second. But we tried to stick to the POSIX grammar as much as possible. We actually had a YAC grammar that we started with. For a long time, I think Jim Truer, who was the PM, was maintaining the language grammar in YAC and trying to update it. Of course, part of the problem is that PowerShell is not a context-free grammar. There are a lot of deals between the parser and the tokenizer. Tokens have different meanings depending on their context, so you can't really represent that type of grammar in YAC. After so many shift-reduced conflicts, he gave up. We tried actually using other grammar tools to capture the grammar, and so far we have not succeeded. What is written, the source code is written in such a way that you can infer the grammar from the code. It actually has comments. All of the different parsing rules have comments associated with them saying what they parse. You used to be able to do a grep through the source code looking for slash, slash, G, and that would pull out all of the grammar comments. I think it's drifted from accuracy. Anyway, we started with the PowerShell. The parameter syntax, though, was inspired to a large extent by DCL, that command language, courtesy of Jeffrey. We have the long options, use of colon. Those are the primary components. That's different from the Unix shell that uses one-character options or double dashes for long options and so forth. It makes for a very readable shell. Then there were a whole bunch of concepts that we wanted to have that weren't in the Plasic shell. We started by adapting elements of Perl. At the time, Perl was pretty much the dominant scripting language on the Internet. Perl had arrays, Perl had hash tables, it had regular expressions, all of the things that we wanted. Super significant is that Perl had CPAN, which was the archive of modules. The joke at the time was that CPAN was the language of the Internet and Perl was just syntax. It was critically important to the success of Perl at the time. We have retained some syntactic elements from Perl. We use dollar under bar, which is the pipeline operator that fell over. Just remember that pipeline falls over. You don't have to explain why it is the way it is. We use that in a bunch of places, so in arrays of expressions, in the null array, or the sort of empty array. We use the ampersand for our call operator. We use regular expressions all over the place. Not much else from the language. The reason is that in the end Perl is kind of icky as a language. It also grew up from a shell background, but it didn't sort of rethink its things terribly clearly at any point. I haven't looked at the Perl six grammar these days. It might be cleaner, but at the time with Perl four or five, you could basically dump arbitrary text into it, and it would be a valid script. Line noise should not compile. We wanted people to be able to distinctly tell whether their script was valid or not. We switched our syntax model to align with C sharp, which essentially means that we're aligning to everybody else who derives from C, which coincidentally also includes Perl. That's sort of ironic. Our value proposition became in part that if you learn PowerShell and you could move to C sharp, you should be able to reuse a lot of your knowledge in C sharp. Otherwise if you knew C sharp, then you should be able to pick up PowerShell pretty easily. Again, we want to protect the students' investments. Learning is hard. Not entirely sure how well this worked, because I see people who keep wanting to turn PowerShell into C sharp, which is a little weird. I mean, it's not like C sharp is great, but it exists. Why would we do that? This is not your parental unit shell. I'm being gender-agnostic here. So almost all shells are expanded parts. How many people know what an expanded part shell or what an expanded part means? I see a few hands. Yeah, so the idea of an expanded part shell is that the shell reads your script. It doesn't read your whole script. It read your script or it processes your script line by line. It reads a line. It goes through and it does variable expansion. So it turns all of the variables get expanded, and that gives you a new string back. Then it goes through and it tokenizes that string. So the first position in the tokenized string is always the command. So if you had a dollar sign, if you had a variable, dollar command, in the original source string during the expansion phase, that would have expanded into a command name. And so by the time you parse it to, that expanded name is the one you execute. So you can say dollar foo and just run whatever dollar foo points to in an expanded part shell. You can't do that in PowerShell. PowerShell, on the other hand, parses the entire script. So the whole script is read in. It is parsed from beginning to end. It is parsed once. Again, expanded part shells, expanded parts every time it hits a line. PowerShell reads holding once. It parses it once. It builds, in version one, it built an expression tree that is an executable representation of the script. In PowerShell v2, it builds a proper abstract syntax tree that has most of the elements of the syntax are captured in the tree. And that tree is used for execution. And so if the user types the same command, with an ampersand in the beginning because we need the call operator, it parses to call and variable command. So you have a node in the tree that says this is a variable expression. So you have these expressions in the tree. And the variable has not been expanded at this point. It won't be expanded until runtime. So then you take that representation and you execute it. And it goes through that there. And it recursively walks through the thing and says, OK, this is my command. These are my arguments. OK, let me see if I have to evaluate the arguments. Oh, yes, this is a variable argument. I'll evaluate that argument and substitute the string. And the string will become the argument all the way through the rest of the arguments. So this produces a bunch of things, a bunch of significant differences from a traditional shell, not the call operator is one. One of the ones that's kind of annoying is that it restricts what aliases can do. So aliases, again, it's all compile ones. So the aliases get expanded at runtime, not at compile time. Because you'd never be able to change the aliases in your compile script if you got expanded at compile time. And the other thing is it can only resolve to a command name. And you can't have fragments of grammar in there. And the example with the expanded part shell, I could have had something that said ls-l could have been the value in the variable command. And then it would actually parse into two arguments. So you're actually doing kind of an eval on the resulting string. But we needed to do this. We needed to have this different approach because we're dealing with objects everywhere. And we'll see that in a second. Splatting. This was designed to solve, or was added to solve commands calling commands problem. In a lot of places, you have a command that wants to then call a sub-command with the same parameters that were passed to it. And so, Splatting does that. You get your PS bound parameters variable, and you do app PS bound parameters to the called the child command. Splatting comes from Ruby. It's another language, as I know. I got it from Ruby. Just to clear that up, because it matters to me for some reason. We use the, but Ruby uses the asterisk to indicate a splatted variable. We use the at symbol because the asterisk is a glob, right? If I wanted to have, if I wanted to, I'm doing my command, my command space, asterisk, foo, well, that's a glob expression. If I wanted to pass that variable, we see at something or other. So we needed to use a different sigil character to distinguish it from globbing expressions. Script Lux and the call operator. I've talked about this a little bit. We needed the call operator because we're not an expanded part shell. I need the call operator to be able to do an indirect call through an expression. So, and this includes things like commands with spaces that have parans around it, because that's an expression. The script Lux grew in a strange way. There was a gentleman on the MMC team who came to me and said, well, we're going to want to use PowerShell, but we need to have this idea of isolated fragments of code. Is there a way you can do that? I thought, well, we don't have a way to do that now, but I've heard of this thing that might work. And so this seems like a job for a Lambda function. And so we introduced script Lux using per end notation, block notation. The notations are much more common now at the time they were sort of notoriously known for being part of hard to use languages. It was a strange exotic thing that nobody understood. That would be a bad thing to do to our less than technical users. So we wanted it to be more amenable. And so they became script Lux rather than Lambda. And take a look at an example with where reach or for. It looks pretty natural. Most users are not aware that they're using a script block there or that is anything strange. So script Lux are pervasive throughout the language runtime. Every piece of code gets compiled into a script block first. When you run a script, it gets built into a script block and then evaluated. And really, a script is nothing more than a script block on disk. You could read it into memory, compile it, create script block, and then evaluate it. They also allow things like delegates, which open up a whole bunch of areas of coding that we weren't able to do before. So we can do informed coding in PowerShell. You can write Goobys in PowerShell. You can do XAML coding in PowerShell. You can even do stuff with the DOM and HTML if you really want to. So that opens up a whole range of possibilities that wouldn't have been available otherwise. Yes? Yeah. Yeah. So the work command was very complicated and then, well, it was very complicated and then they got reduced down to here's a script block just executed. And then it got complicated again for the simple where. Which is funny. Be careful what you wish for. You might get it. Yeah. And so we had, again, this function example where, I mean, then there's a parameterized function because script blocks take parameters. And so you can do the fully equivalent of a function definition with parameters and everything by assigning a script block into a variable or into a, well, into the function drive. So there's a bunch of language design questions that we had to deal with. First off, PowerShell is an expression-oriented language. Again, does it ring a bell with anybody? Total silence. So expression-oriented language is everything returns a value, which is what happens in PowerShell. So even something like a foreach statement returns a value. The if statement returns a value, which is our argument against having the conditional operators. One difference from a lot of expression-oriented languages, though, is that it returns everything. Every statement, every line in a block of code returns a value. In a lot of languages, like Lisp, it's typically the value of a collection of code is the value of the last statement in that collection of code. PowerShell is not. It's everything. The original version of the interpreter actually worked the way I said the Lisp interpreter worked. It only returned the value of the last statement. But that's not very shell-like. So PowerShell's right to streams. Streams get aggregated from every line within a function or a script. Lexical or dynamic scoping? This is another one. Ring a bell? No. Lexical scoping is most common in programming languages these days. Dynamic scoping, the original Lisp interpreters were dynamically scoped. Basically what it means is when you're looking for a variable, you look in your scope, your media scope, and if it's not there, then you look in your color scope, and if it's not there, then you look in the color scope all the way up until you get to global scope. And this is basically how PowerShell works. It makes a bunch of things simple. You can pass a whole bunch of context through scoped variables rather than having to have a lot of parameters. But it also makes things fragile that you can have false coupling between functions because the variable has leaked from the wrong place. So good for small programming and small, not great for programming in the large. We had discussions with some of the architects in the CLR team, and they were like, oh, you're going to get it if you go with dynamic parameters. But it has largely worked. And one of the things that we did in V2 with modules is that we introduced this idea of module scope so that you don't leak into the functions in the module. Could the option character be dash or slash? That was easy. We're competing against the Unix shell, not command.exe, so we'll go with the dash. How to handle switches. Switches are parameters that are either present or absent. Easy to deal with in an expanded parse because if your variable is empty, if your variable is not empty and contains a switch parameter, then the switch parameter is there. And it's on. And if the variable is empty, then the switch parameter is not there and it's off. Again, that doesn't work in a fully parsed environment. And so we needed a different mechanism. We were doing, addressing this, we thought, well, they'll just be Booleans. But people wanted to use Booleans for a whole bunch of other things. It seemed like kind of a large space to co-opt. And so we introduced this new class switch parameter that has this characteristic of being present or absent and doesn't take an argument until it does. Because, again, command is calling commands. You do want to be able to pass through an argument to a script block or through a command through a switch parameter. Adding in PSBoundPurr and there's Matek, a little bit less difficult. But it's still a case where, in simple cases, where you want to be able to pass it through. And you'd think this would be obscure enough that people wouldn't use it by accident. But I've seen lots of people, well, lots of stuff online where people are putting in switch parameters, colon, dollar, false. I have no idea where that's coming from. More language questions. So expand in part shells. They only deal with strings. All the arguments to a command are strings on the command line. And so it's up to the shell to figure out what it is. It's up to the individual command to figure out what that string is for, whether it's a number or a string or a file name. But it means that everything just kind of, like, automagically works because the commands know what they want to do with strings. They have their own built-in parsers. In our environment, we don't want people writing all these custom parsers. We want to be able to pass real objects. And so the primary reason for having the fully parsed shell is that because the variables are not expanded into text, they're passed in as objects, then we can get objects passed through to commandlets rather than expanding into, some people thought this is what we were doing at first, is that we were expanding into this big-ass XML string and then turning it back into an object inside, which would have been horrendously slow. But we just passed in the object itself. But then we still had this idea that we want everything to just kind of work. You shouldn't have to worry about doing a lot of explicit conversions. How do we get the user experience with a type-based shell to be similar to that of a string-based shell? And the answer to that was super aggressive type conversions. So the runtime, when you're passing stuff around, the runtime makes a lot of implicit guesses at what you meant. There is a constraint on that. One of the things that we considered doing was having an automatically-chain conversion, and then we realized that that was a bad idea because everything can reverse the string eventually. And so we'd be right back where we started from. And so we don't do implicit chaining, but we can do explicit chaining. When we're doing this, we use an algorithm for searching for the converter between the source type and the target type. And we use an algorithm called the type distance algorithm so that in ambiguous cases, we compute a distance between the target types, and whichever one is closest to the target is the type conversion that we use. I think that has been tweaked a little bit over time, but I think we're still using the same basic type conversion algorithm. So that's how generic type conversion work. The parameter binder does not do this. So this is one of the things that we, in version one, said, well, we can ship it this way and we won't change it later. And 10 years later, we haven't changed it. So I don't think it's ever going to do. But anyway, binding in the parameter binder is done in two passes. The first pass goes through and tries to do an exact match on each of the parameters. So if the types are the type of the argument and the type of the parameter are the same, then it can simply bind the parameter. If that doesn't succeed, then it does a second pass where it looks for a type conversion. So there's a type conversion for the explicit target type, the source type to the target type, then it can succeed and proceed with a bind. So everything is done twice. If you ever wonder why it's slow, well, that's one of the reasons why it's slow is that it does this extremely complicated process using bitmaps or bitfields to determine the degree of closeness of the match and so on. It makes parameter binding very heavyweight. But hey, we said we were not going to worry about CPU cycles and stuff back in the beginning. So this was designed to give the best user experience at the cost of significant reserves of resources. We'd like to fix that. We have some ideas. One of the things that we don't do right now in the parameter binder is we don't do, so the DLR runtime does type caching or method caching. Once it figures out how to do something, then it caches that. So the next time through, it says, if I'm still doing the same thing, I can use this mechanism and I've cached. We don't do that for the parameter binder today. We could probably do that. So we do the expensive computation once and then cache the result. That would be fun. I'd like to work on that. Anybody else want to work on that? What am I doing? I have, oh, there's just no time at this point. So finally, we're near the end of what I was going to talk about today. So we have, we've built everything. It's all spiffy and cool. We have all these great ideas. Now we need to ship. Shipping is fun. Because PowerShell started as an independent project inside Microsoft, it had its own build system. It had its own everything. But later on, we shifted to be part of Windows. Windows has its own build system. So we switched from our build system to their build system and they have a bunch of rules about how stuff looks. So we changed our stuff to look like their stuff. And then there's the Longhorn Reset and the world explodes. We were a brand new.NET component and we were kicked in someplace in the anatomy out of Windows. Crap. Now what? Okay. So we're drifting aimlessly and Longhorn's the exchange team, our best friends, our new best friends. They were investigating an automation solution for managing exchange servers. And PowerShell pretty much matched all of their requirements just by simultaneous invention, which was very cool. So this was awesome. We have a new development partner, a great development partner because they gave us a lot of feedback, very constructive. And we have a new shipping vehicle. We can ship as an exchange shell. There's some impact on the product. First the fact that the Longhorn Reset kind of gave us a lot of time to work on things. And so PowerShell became much, much more refined than it might have been. So that was good. On the somewhat less good side was there was a focus on the exchange team features first. And that meant that some of the things that are applicable in general shells didn't get that much attention. Native command support is an example of that. So Native command processor was barely there. I barely had time to get anything working at the time frame. And it still sucks. And I know that Sergey Vorbev did a bunch of work to get to slightly unsuck it for PowerShell 6. But it still needs a bunch more work. It still has parameter binding issues. The pipe doesn't work the way it should. You are sort of just barely able to launch notepad. And that wasn't bad for exchange, but it was bad for a general purpose shell. And that's something I would really like to make work better. Shipping is fun. Three. So we finally get approval to ship as a Windows feature, not as an exchange feature. So now we're Windows PowerShell, not Exchange PowerShell, or Exchange Shell, or Exchange Monad or whatever. But we have to pass a bunch of Windows quality gates. So there are versioning requirements. There's a couple of ways you can go about doing versioning. You can have loose versioning, also known as DLL help. Or you can have rigid versioning, which is what the.NET team did with the GAC. And that is completely brittle. I mean, you'll never be wrong, but you might be useless. So as Jeffrey says, this versioning is not a problem. Problems can be solved. This can be only mitigated. Part of it had to do with how, the way we dealt with plugins. There was a no plugins requirement, because IE was, well, IE was IE. And it had lots of plugins, and it hung a lot. I mean, it got to the point where there were fewer crashes. There were more hangs than crashes in coming back from telemetry. And a lot of the hangs were in IE, not in the code itself, but in the plugins that people were writing. So there was a big push not to have new plugin architectures. And DevDev did a big effort to come up with, and I'll just describe it, a very resilient plugin architecture. It was also complicated and heavyweight. And so we looked at it, but it would have been wildly inappropriate for us. It was designed for the level of a component, not a commandlet. The PowerShell is all about plugins. So we had to do something to satisfy this requirement. So our first attempt was mini-shells. So there wouldn't be one PowerShell. There would have been a PowerShell, and there would have been an AD PowerShell, and there would have been an Exchange PowerShell, and there would have been everything. Because what you would have had to do is you would take the libraries for PowerShell, and you'd take your commandlets and some tooling, and out of this you would spit out a custom shell that included your commandlets hard linked in. And so the only way to call from commandlet to commandlet would be from shell to shell. And there is a feature in PowerShell. If you're within PowerShell and you type PowerShell, open brace, like a script lock, closed brace, then that will be evaluated as an expression in the child shell. That was actually added to make mini-shells work tractably well. I think it's the only part of mini-shells that are left. All the tooling and everything else got cut ages ago. Exchange looked at this and said, no. Are you guys idiots? This is not going to work for us. We need to be able to extend our shell with some kind of plug-in mechanism. So once again we return to the halls of Windows Core Architecture team, supplicants on vended knee begging for a solution, and since MMC already shipped with Windows and it had snap-ins, we thought, okay, this is shipping and seems to be okay, so can we do this? And I think there was a diverse set of opinions on MMC snap-in mechanism that are probably not publicly consumable, but they ultimately agree that we could do this tap-in mechanism. And so we went off to do the snap-in mechanism where you have to register the DLL and all that kind of stuff. And this still has dregs in the code. You can see it under system management automation, single shell, and there's another thing, single shell utilities. And these are left over from, I think, the reason it's called single shell is because we had mini-shells, which was mini-shells was mini-shells and single shell was one shell with plug-ins. So we got approval for snap-ins. It was not optimal, but we could ship. That was one of the first things we went after in PowerShell v2. Yes, the language reviews, because PowerShell is the only language product that ships outside of DevDiv. The developer division was a little concerned about this because at the time everybody was well. We were kind of siloed and everybody was protecting their products and making sure that we weren't confusing the user with the wrong thing and all the normal stuff that businesses do. It used to do anyway. So we had to have a review with the CLR and the relatively newly formed DLR team. One of the big issues was the architectural approach to passing objects through the pipeline. At the time DevDiv was producing link. Link works by composition of enumerators and it pulls the objects through their pipeline. On the other hand, the pipeline processor pushes the objects one by one through the pipeline. So we're a push model. They're a pull model. There was some idea that we should have one model. Fortunately, Anders Helzberg was part of the review and he pretty quickly figured out what we were doing and he understood why we were doing it. And interestingly enough, some significant number of years later, DevDiv has the reactive framework and the reactive framework works by pushing objects through a pipeline. So we felt slightly vindicated. And then we're stuck in the corporate layer. Oops. Somebody noticed that you could build software with PowerShell. Oh, my. So the MS build was new at that time. It was just coming out. Microsoft invested a lot of energy into building these new tools. Virtual Studio was notoriously weak on build automation. So it was a tool for automating builds. So mechanisms unknown. Someone saw a slide talking about how PowerShell could be used as part of a software construction tool chain. This was, I think, actually being used. The slide deck was used to justify the existence of PowerShell. Look at all the places you can use it. So of course, it tried on somebody's toes. And it's not like that idea is unique. I mean, the shelves have always been part of the programmers workbench on Unix. The programmers workbench is a set of utilities, make, and so forth, or building software. So it's not unexpected that PowerShell should be able to do this. But we had a review. And it was a review. In the end, it actually, we sort of clarified the relative roles of things by kind of like MS builders like Ant and PowerShell is like Shea and Nmake is like Make. Once people understood the relative roles of the tools, we got through that little hurdle without dying. And then finally, we're through all the hurdles, we're through the gates, we've had our reviews. Oh, yeah, I forgot the first Monad virus thing. It was the first Vista virus where some kid in somewhere, this guy wrote self-replicating code. He wrote it in every language. He liked using this to explore new languages. And so he wrote the same self-replicating virus that he wrote in C Sharp or VB or Unix Shell, he wrote in PowerShell just to see if he could do it. And somehow this got picked up by the security community in probably like on a Sunday morning after a big party when nobody's thinking clearly. And it got promoted in a big article. And this just started a huge fire store. So again, had to calm everyone down, publish the appropriate discussions, explaining what was going on. And again, another shipping hurdle was passed. But blood pressure, in that last period, blood pressure was going up and down like a yoyo. So finally we're at the end. One teeny thing left we have to fix. Everything has decided that we will rename Monad, which everybody loved, to Windows PowerShell, which was not as well received. And it had to be Windows PowerShell. It couldn't be PowerShell because there were other things that used PowerShell. If we wanted to get a trademark, it had to be Windows PowerShell. So we had to change it into code. We had to change it everywhere. Everywhere where the physical, visible, but also a lot of places in the code where things might be visible through reflection. And so with a bunch of scripts and a whole lot of scanning later, everything got cleaned up. Well, almost everything. You will still see in the code, like the fact that the PS object, the core of our type system, is stored in a file called MSH object. You'll see, you'll still see bits and pieces of that. So anyway, shipping is fine. And now we're ready to ship. Yay! And finally, we have shipped PowerShell 1, shipped in 2006, to great acclaim and huge size of relief. And we've been going steady ever since. And this is all the time I have, based on, oh boy, I'm way over. So again, I encourage you, based on what you've heard today, if there are things that I haven't answered or things that you'd like more information on, please let me know. I should have had all my stuff in the back. This is a long deck. Oh, yes, above me. So my email is here. There's there. Come on. Stay. Okay. Yes. Yes, I did it. My email, my Twitter handle, let me know. And stuff that you think I should cover, things stuff that you think I should cover in more detail, stuff I should cover less detail, stuff I should emit completely, please let me know. Anyway, so thank you.
 Okay, hi, I'm Bruce Payette.  In this talk, we're going to go through, we're trying to trace the evolution of PowerShell  from the very beginning up until I run out of energy and we run out of time.  Because this is a 45 minute talk with three hours of content.  I want to look at some of the cultural, technical, and some of the business forces.  There are artifacts in the source code in particular that have been impacted by some  business decisions in the past.  This is the first round of at least two rounds of this talk.  The ultimate goal is to gather all this information together into a document that will go on  GitHub so that when people come, it'll become part of the contributors package so that you  can deeply understand what we're doing with PowerShell and why we're doing it that way.  Hopefully that will help people more easily evolve the language in a coherent way.  In the beginning, things always start there.  PowerShell started in 2001.  I joined Microsoft in December of 2001.  There was a project that was being funded out of the India Development Center, IDC,  in Hyderabad, India to investigate the idea of a new shell.  The code name at the time was Kermit because the person who was organizing this project  had a child who had a kid's book about Kermit, the Kermit crab who lost his shell.  I think we kept the Kermit name for about six months.  It does not show up in the source code anywhere.  The project was staffed by people coming from the Services for Unix team.  Part of Services for Unix was a product called InterX, which was an extended version of  the POSIX subsystem for Windows that was almost fully Unix compliant.  It made sense to have people who had some experience with the shell and utilities to  actually do a new shell.  Why did we do it?  Because especially at the time, Microsoft was very GUI-centric.  The command line was pretty deprecated for a lot of stuff.  We had commissioned a survey that had gone out and asked people about doing automation  on Windows.  The survey had returned the result that it took ten times as much effort.  It took ten times as long to do an automation, to automate something on Windows versus Unix,  which is a bad thing.  We started to care about Linux and Unix a lot at that time point.  It was becoming a significant, it was clear that it was going to be a significant competitor  in the future.  Finally, there was an evolution in culture.  We were moving away from the era where you could have one machine in your data center  and you were good.  The number of machines in the data center was skyrocketing and automation was very important.  We have all of this going on.  There's one thing that's really important that's missing, and that's a certain gentleman  sitting over there, Jeffrey.  Where is Jeffrey in all of this?  Because he wasn't actually involved in the Shell project at the time.  However, he was over in Building 42, madly prototyping a new type of command line based  on some radically different set of concepts.  A command line based on the composition of functional units, or FUs for short, with an  extended viewable type system.  You could describe a command line as being FU, FU, FU, and really get your anger out.  This prototype was used a different model of composition than what eventually showed  up in PowerShell.  If you look at, if you can see the line there, the second last code line from the bottom,  I think we were still doing, I think we were doing verb noun at the time, and the separator  was a slash.  But the idea, conceptually, it was more a case you were doing extensions on existing commands  rather than composing a set of discrete commands.  When we looked at it as part of the design review, we decided we would go with something  that was more similar to the traditional Unix model.  I don't know that it was, there are functional aspects that changed, but not so much from  the prototype.  And that's a discussion that's too long for now.  So Monad begins.  First thing we had was a bunch of guiding principles.  If you haven't read it, I'd recommend reading the Monad Manifesto.  It dates from that time, but it's surprisingly still very, very accurate.  And if you really want to understand what Jeffrey was thinking at the time, then this  is a great document to read.  We had a bunch of principles.  To read, to use is very important.  Maybe more important than being simple.  You needed to be able to get things done.  You needed to be able to deal with sophisticated tasks in an effective way.  And so if you needed C sharp, this is probably going to get me in trouble, C sharp versus  visual basic.  Visual basic is very simple, but you have to take a lot of steps to get something done.  C sharp or PowerShell, you can get stuff done with fewer steps.  But it requires a little bit more sophistication.  So we wanted it to be easy to use for the sophisticated user.  But learning is also very important, and it's hard, right?  So we need to facilitate learning and protect that investment.  This is the sacred vow that Jeffrey talks about.  If you learn something, you won't have to unlearn it.  And one of the big drivers on this was consistency.  So if you learned a principle once, then you'd be able to apply that principle over and over  and over again.  The one thing we wanted to avoid was the foolish consistency where everything would look like  itself.  There are still a couple of cases in, maybe a couple of cases in the language where we  were a little foolish.  We also wanted to address the tension between whip it up a toad, which is a pearl word,  and production coding.  The survey had said that sort of ad hoc automation was very important for Windows.  The ability to write small scripts or even doing things on the command line was really  important, which implies the ability to do a lot of stuff on the command line versus  production coding where you want to be able to support the code that you're producing  down the road.  This results in a very wide dynamic range in the language, and there's a lot of ands,  so methods and commandlets, simple functions and advanced functions.  One of Jeffrey's favorites, command line and GUI.  We also have some features that let you sort of scale your syntax with things like aliases  and short options.  You can have the line at the top, which is the fully elaborated line, and it's very  readable because everything's fully spelled out.  Then the line at the bottom, which is, I think, a little less than half the length of the  top line, is the same command but condensed, so much quicker to type.  The bottom line is good for the command line, and doing quick things is not recommended  for putting into a script.  Reuse.  It's super important.  We wanted to leverage reuse as much as possible, sort of extreme reuse, if you will.  So common argument processing for all commands, ubiquitous parameters, common runtime services  like the way well cards are processed, the way paths are resolved, and so forth.  All those services are made available by the PowerShell runtime to people who are writing  commandlets or scripts.  Common formatting and outputting.  Nobody has to write formatting code in their commandlet.  We take care of that for you.  All time is more important than processor time or disk space.  At the time, everybody was reasonable to have a whole system to yourself, so throwing away  resources to make the user faster was a non-issue.  That's changed a little bit with multiple VMs on a physical machine, with IoT and resource  constrained environments.  So some of the earlier decisions that we made regarding resource consumption are biting  us a little bit now as we move into these more constrained spaces.  You can see some of PowerShell v6 were evolving and reducing some of the footprint.  We wanted to provide friction on negative paths to use a probably incorrect analogy.  We'll let you shoot yourself in the foot, but we won't hold the gun.  So we have dash force on destructive operations.  We have the whole dash confirm mechanism to confirm, again, operations that are going to  make significant changes to your system.  Just in general, we wanted to have aspects of the system that let you do something but  didn't make it easy if it was probably the wrong thing to do.  You had to think about doing the wrong thing.  We wanted to facilitate experimentation.  It should be safe to try out commands.  So we have the what if ubiquitous parameter, and then we have declared annotations for  commands and their parameters.  So the logic for binding parameters and doing checking and so forth is done declaratively  through a set of attributes rather than everybody writing their own null checks.  The project begins, and we acquire a team.  I started out of IDC in Hyderabad.  The architectural owners were all here in Redmond, but the dev team was going to be  in India because, well, for a bunch of reasons.  But it turns out that radical experimentation does not flourish with this level of geographic  distribution.  It's a 12-hour time difference.  We were awake, and there were 12 and a half, depending on daylight savings time.  We were just using mail at the time to communicate, and occasionally video conferencing.  So it wasn't a great environment.  I think it might be better now, now that we have better tools for communication, we have  GitHub and so forth, that geographically distributed development might succeed where  it didn't work for us then.  So anyway, that didn't work.  First we pulled back development with the idea that test could still be there, but then  even that really wasn't working very well because of the day lag on everything, because  things were changing so fast.  So that didn't work out.  We went shopping and we acquired another team.  This one was built out of team members from the Windows Management Pack team, which was  great because they had a lot of management experience for Windows.  Now we have a team we can begin for real.  Oh, yeah, and be ready to ship in two weeks.  This was all in the Longhorn timeframe.  Does anybody remember Longhorn?  Yeah.  So my impression of the Longhorn timeframe was that Microsoft was starting to dip into  the idea of continuous delivery.  I could be wrong, but that's sure what it looked like because we were supposed to be  ready to ship every two weeks.  So it was exciting.  We were not, we did not have the infrastructure to support this kind of thing.  I'm now lost in my slides.  Okay.  Anyway, this was clearly not an ideal situation.  We would have shipped a PowerShell that had almost no features that would have been not  much more than Jeffrey's prototype, which was disturbingly functional.  But still, I would have been missing a lot of the elements of a shelf.  You wouldn't have been able to write scripts in it, which shell scripting language kind  of needs to do that.  So it didn't happen.  That's good.  I'll talk about that a little bit more later.  So now let's talk about some of the big ideas.  Big ideas are, we talked about the principles.  They're very sort of generic principles.  The big ideas are more architectural characteristics of the engine, more concrete than abstract.  So one big idea is the idea of domain-specific vocabularies.  So PowerShell was going to have, you would basically write a set of words.  If anybody's familiar with the fourth language, there's this notion of domain-specific vocabularies  where you would write a set of words for your domain environment.  And this is different from a domain-specific language in that there are no syntactic elements  that are different.  It's just a set of nouns and verbs within your application domain.  So we encourage the use of a constraint set of verbs.  We didn't enforce it at this point.  We didn't start doing anything around enforcement until version two.  Skyline's not much verbs to use, this made it very predictable with a small set of verbs  you can, the user can infer if I need to get something, I probably need to use the word  get instead of fetch or whatever.  Yes, we did explore and, the set of verbs is going slightly over the years, but not very  much.  I think we've only added a handful of verbs since the very beginning, which is kind of  amazing given how long ago it was that we were doing this.  So we had verb pairings, start, stop, get, set, again prescriptive so that the user could,  again if the user knew that there was a start verb then they can infer that there's a stop  verb.  And then command aliases for interactive use against the whip it up a to do thing.  There are two types of aliases, there are the canonical aliases like GCI for get child  item which are specific to power shell and then there are the convenience aliases like  LS or DER.  This worked pretty well in Windows, but kind of got us into trouble when we moved to Mac  OS and Linux because we were now hiding some of the existing commands on those platforms,  especially curl, curl was a big problem.  So big idea universal command line parsing, unlike most shells the commandlets are not  responsible for doing their own parsing, the common parser code does that and breaks everything  into, breaks everything into data structures that then get passed to command type specific  parameter binders.  So advanced functions or commandlets each have their own parameter binder and you can  find all of those things under the engine sub directory of the SMA directory in GitHub.  So you want to take a look at how the parameter binders work.  This approach gives us broad consistency across commands, commands always have a dash, they  can always be separated by spaces and so forth.  This works really, really well for all of our command types except native commands that  is executables because they still do their own command parsing.  They take the argument string that they're passed and then they parse it up into arguments.  And so native command binder had to take his arguments and then turn them back into a string  kind of like what we think the user meant when he typed them.  This works okay sometimes, sometimes it doesn't.  There are a couple of bugs in GitHub around this and we're trying to fix some.  We fixed some but it's very, they're kind of hacks.  They're not appealing from an architectural perspective but they do make the user's experience  better.  This is one of these cases where it's not quite the foolish consistency but being super consistent  makes the end user's experience a little rough.  Declarative parameter constraints rather than writing imperative code to do all the checking,  we have attributes to deal with that.  These attributes did a couple of things.  One they reduced the amount of code you had to write and two they reduced the amount of  error messages you had to write and internationalize and so on and so forth.  And it also eliminated or reduced the number of error messages that the user had to learn.  So there were, this was a win-win-win situation except when it didn't work.  So you can get into situations that are suboptimal like the regular expression, validate pattern  where it tells you the regular expression that was like, the regular expression for  a phone number.  It doesn't tell you your phone numbers in valid, it tells you it's some like strange  sequence of characters, gobbledygook that is not right or you're not matching this pattern.  So it's not a great user experience if you use that particular attribute.  And then sometimes this whole idea of attributes just didn't work.  This is a comment out of the process, get process, get process, start process, commands  in the source code, basically process name, glob attribute was deeply wrong.  I think it's pretty funny comment.  I'm removing this comment from the code by the way.  It does have the programmer's name associated with it so I'm taking it out.  Providers, so we were sitting around in a conference room discussing how many get commands  we were going to need.  And then somebody, and I don't remember who it was exactly or whether it was simultaneous  invention, decided that wait a second, if we had a standard set of command list and  just created this sort of pluggable model for hierarchical data stores, then we only  have to have one get command because it will work for all of the stores.  And so we developed what are called the core command list, item, trial item, item property,  et cetera.  And these sit on top of the providers which are in the source code, they're called namespaces.  So if you want to see how all the providers work, look at that location, sma slash namespaces.  Again in the effort to be consistent, we decided that everything could be accessed through providers.  So file system makes sense, registry is a hierarchical store, sure that makes sense,  functions, okay, that's a little strange.  And variables and the environment and things that like the WS man configuration, although  it doesn't come until version two, could all be exposed to the same set of command list.  So you could navigate into these stores using CD basically.  And basic provider options are also available through the variable syntax.  So this is kind of dual ported.  You can, for example, define a function by assigning a script block into a name in the  function drive.  A big idea as an extended type system.  We wanted to build a management oriented type system.  And we were going to layer it on top of the existing.NET type system.  We had a wild dream of this idea of a central type extension system where you'd be able  to publish patches to the type system of the running code.  It was very cool, it didn't happen.  I leave this as an exercise to the audience.  We worked with a variety of teams at the time and a lot of people, not just PowerShell  team, a lot of people thought this was kind of a good idea, having this management layer  of the type system.  So we worked with them about doing some of the designs, but ultimately it got lost in  the world of schedules, I suppose you could say.  And so as near as I can remember, there's only one actual result, and that's the fact  that arrays have a.count property in PowerShell and they don't in.NET.  So you can always say.count on a collection and get a number back.  We also had the synthetic type system.  This supported the idea of deserialized objects.  So objects that deserialized object remembers what it used to be, not what it doesn't deserialize  as the original type because the original type might not exist on that system anymore,  but you can still work with that type because it remembers what it used to be.  And we also allowed you to create new custom objects that didn't have a type associated  with it.  You couldn't construct types.  You couldn't construct types dynamically in the CLR in version one or version two,  I guess it was, anyway.  But we didn't let you construct types.  Big idea, PS object.  This has come up a bit lately on some of the GitHub issues.  Jeffery's original type system only allowed you to do class-specific extensions.  So we also had scenarios where we wanted to be able to extend an individual object attached  information to an object.  And so we have PS object.  PS object is a class that wraps instances of other objects and holds all of their extensions.  So no properties, script properties, code properties are all wrapped in the PS object.  It simplifies a bunch of things because you always go through PS object.  But it also makes some things complex because you have to always deal with the PS object.  C-sharp code that has a parameter of type object has to deal with a PS object just in  case and then get the underlying object if it needs to.  PS object is mostly intended to be invisible to the script user but is not entirely invisible  to the script user.  So it is part of the PowerShell API, which is part of the overall PowerShell user's experience.  It's to be really clear about that, but most of the time you don't need to worry about  it.  And then we have the typos objects I mentioned earlier, the PS custom object type.  This ability to cast hash tables into custom objects is a V3 thing, I think, but you can  still do it in V1 with add member.  It was just harder.  The implementation.  So the initial design was highly componentized.  Most of the components in the service code, they live in system.management.automation.engin.  So you'll find pretty much all of the pieces of the core PowerShell runtime in that directory  or subdirectories.  Some of the pieces include, on version one, it was run space configuration, which was  all a set of initial trends that you had in your run space when you started up.  Well, yeah.  Run space configuration, largely supplanted by initial session states, starting in V2.  The language that tokenizer parser and all of the execution stuff was their execution  context, which includes all of the stores.  So aggregating all of the various provider instances into one single object.  Command processor is the piece that runs pipelines, kind of obviously.  And then there's all the command processors that are associated with that.  And finally, there's the error handling subsystem.  The problem with this design is that you basically had a bunch of Lego blocks and you basically  had to build your own shell.  And it was not super trivial.  It required a certain amount of understanding of how the pieces went together.  It meant you could do a very minimal shell.  You could have, in theory, you could have the pipeline processor without the engine,  or without the language at all.  So you'd have a very minimal shell.  You could have something that didn't have language.  It didn't have most of session state.  It didn't have formatting and outputting.  And it would give you a relatively small footprint.  Anyway, to deal with the complexity, we decided that we would bundle everything together into  an Uber class called the run space.  And what is a run space?  A run space is a space where you run things.  I've never understood why people find that hard.  But people found it hard enough that we have a wrapper class called a session for promoting  that sits on top of a run space.  And the run space sits on top of something called automation engine, which so it kind  of turtles all the way down.  And then the run spaces are exposed to the APIs, but not necessarily through the shell  level.  Context objects.  So execution context.  I said, we had session that included a run space that included an automation engine that  included, and it also included a context object.  Context object were all the resources that the running engine needed.  And it got passed around all over the place.  So every node in the expression tree of the compiled language had to get this thing.  So you got to the point where it was just too much trouble to keep passing it from class  to class to class to class.  So we ended up sticking it into a thread local storage, a local variable, and it's accessible  through get execution context from TLS.  Get a fairly straightforward name.  But you'll see that in the code in a number of places.  You'll see execution context all over the place.  And then you'll see people calling this API to get their execution context.  And the execution context that's inside the engine is not the same as the execution context  object that you see in dollar execution context, because that is a facade object that only  shows to part of the public parts of that object.  We'll talk about design patterns in a minute.  Provider context did the same thing for providers.  It provided all the provider context.  By wrapping everything up into the context objects, though, this gave us the ability  to do multiple run spaces for process or for app domain if you're a stickler for details.  This means that we can do parallel execution.  We can do run space pools.  We can do all those things.  This was, I'm not sure how super deliberate it was.  I mean, it was, yeah, I guess it was.  It sort of did something we did because it seemed obvious rather than being, oh, yeah,  we should do this.  So lots of design patterns I talked about.  We used design.  It was the era of getting the four patterns.  We used a lot of design patterns fairly informally throughout PowerShell, the PowerShell source  code, PowerShell design.  We have adapter pattern in the type adapters.  So we have a type adapter for various, like, data stores, done net objects, synthetic objects.  All have type adapters.  We used the facade pattern a lot for making all of the commands look similar.  We used delegation in the command binders.  So the command processor orchestrates the commandlet, but it doesn't do the commandlet binding.  It uses a separate delegated class to do that.  And the delegated, the commandlet binder, for example, is used both by commandlets, obviously,  and by advanced functions.  We have factory pattern in the Runspace factory, strategy pattern in command discovery as to  how we go about finding commands.  So again, if you're going through the source code, you'll see a lot of the stuff should  look familiar because of all of these patterns.  It should be relatively easy to figure out the role of the different objects in the running  process.  OK, now we get to the PowerShell language.  Now, as I mentioned earlier, in the original architecture, the language was actually conceived  of as being completely separate from the PowerShell engine.  You would be able to not have the language or even plug in another language.  Somebody had, at one point, had expressed interest in doing a command.exe style language  for PowerShell.  But that didn't happen.  I say that a lot.  In practice, the language has become pretty deeply mixed into everything because it's  recursively used.  The language uses the pipeline, which uses the formatting system, which uses the language.  So it's pretty difficult to separate now.  I think one of the reasons for making it separate is we weren't sure that the language was going  to be right.  We had grave concerns about that early on.  I think we're OK with it now.  It's only been 16 years.  I think we're fine.  The language roots.  Where did it start?  We started with the IEEE POSIX.2 shell grammar, which is essentially the corn shell, which  is more or less bash, which all come from the born shell.  POSIX stood for portable operating system interface with an X on the end because it sounds cool.  It's a real designation.  There are big changes to the implementation, though, because I'll talk about that more  in a second.  But we tried to stick to the POSIX grammar as much as possible.  We actually had a YAC grammar that we started with.  For a long time, I think Jim Truer, who was the PM, was maintaining the language grammar  in YAC and trying to update it.  Of course, part of the problem is that PowerShell is not a context-free grammar.  There are a lot of deals between the parser and the tokenizer.  Tokens have different meanings depending on their context, so you can't really represent  that type of grammar in YAC.  After so many shift-reduced conflicts, he gave up.  We tried actually using other grammar tools to capture the grammar, and so far we have  not succeeded.  What is written, the source code is written in such a way that you can infer the grammar  from the code.  It actually has comments.  All of the different parsing rules have comments associated with them saying what they parse.  You used to be able to do a grep through the source code looking for slash, slash, G, and  that would pull out all of the grammar comments.  I think it's drifted from accuracy.  Anyway, we started with the PowerShell.  The parameter syntax, though, was inspired to a large extent by DCL, that command language,  courtesy of Jeffrey.  We have the long options, use of colon.  Those are the primary components.  That's different from the Unix shell that uses one-character options or double dashes  for long options and so forth.  It makes for a very readable shell.  Then there were a whole bunch of concepts that we wanted to have that weren't in the  Plasic shell.  We started by adapting elements of Perl.  At the time, Perl was pretty much the dominant scripting language on the Internet.  Perl had arrays, Perl had hash tables, it had regular expressions, all of the things that  we wanted.  Super significant is that Perl had CPAN, which was the archive of modules.  The joke at the time was that CPAN was the language of the Internet and Perl was just  syntax.  It was critically important to the success of Perl at the time.  We have retained some syntactic elements from Perl.  We use dollar under bar, which is the pipeline operator that fell over.  Just remember that pipeline falls over.  You don't have to explain why it is the way it is.  We use that in a bunch of places, so in arrays of expressions, in the null array, or the  sort of empty array.  We use the ampersand for our call operator.  We use regular expressions all over the place.  Not much else from the language.  The reason is that in the end Perl is kind of icky as a language.  It also grew up from a shell background, but it didn't sort of rethink its things terribly  clearly at any point.  I haven't looked at the Perl six grammar these days.  It might be cleaner, but at the time with Perl four or five, you could basically dump  arbitrary text into it, and it would be a valid script.  Line noise should not compile.  We wanted people to be able to distinctly tell whether their script was valid or not.  We switched our syntax model to align with C sharp, which essentially means that we're  aligning to everybody else who derives from C, which coincidentally also includes Perl.  That's sort of ironic.  Our value proposition became in part that if you learn PowerShell and you could move to  C sharp, you should be able to reuse a lot of your knowledge in C sharp.  Otherwise if you knew C sharp, then you should be able to pick up PowerShell pretty easily.  Again, we want to protect the students' investments.  Learning is hard.  Not entirely sure how well this worked, because I see people who keep wanting to turn PowerShell  into C sharp, which is a little weird.  I mean, it's not like C sharp is great, but it exists.  Why would we do that?  This is not your parental unit shell.  I'm being gender-agnostic here.  So almost all shells are expanded parts.  How many people know what an expanded part shell or what an expanded part means?  I see a few hands.  Yeah, so the idea of an expanded part shell is that the shell reads your script.  It doesn't read your whole script.  It read your script or it processes your script line by line.  It reads a line.  It goes through and it does variable expansion.  So it turns all of the variables get expanded, and that gives you a new string back.  Then it goes through and it tokenizes that string.  So the first position in the tokenized string is always the command.  So if you had a dollar sign, if you had a variable, dollar command, in the original  source string during the expansion phase, that would have expanded into a command name.  And so by the time you parse it to, that expanded name is the one you execute.  So you can say dollar foo and just run whatever dollar foo points to in an expanded part shell.  You can't do that in PowerShell.  PowerShell, on the other hand, parses the entire script.  So the whole script is read in.  It is parsed from beginning to end.  It is parsed once.  Again, expanded part shells, expanded parts every time it hits a line.  PowerShell reads holding once.  It parses it once.  It builds, in version one, it built an expression tree that is an executable representation  of the script.  In PowerShell v2, it builds a proper abstract syntax tree that has most of the elements  of the syntax are captured in the tree.  And that tree is used for execution.  And so if the user types the same command, with an ampersand in the beginning because  we need the call operator, it parses to call and variable command.  So you have a node in the tree that says this is a variable expression.  So you have these expressions in the tree.  And the variable has not been expanded at this point.  It won't be expanded until runtime.  So then you take that representation and you execute it.  And it goes through that there.  And it recursively walks through the thing and says, OK, this is my command.  These are my arguments.  OK, let me see if I have to evaluate the arguments.  Oh, yes, this is a variable argument.  I'll evaluate that argument and substitute the string.  And the string will become the argument all the way through the rest of the arguments.  So this produces a bunch of things, a bunch of significant differences from a traditional  shell, not the call operator is one.  One of the ones that's kind of annoying is that it restricts what aliases can do.  So aliases, again, it's all compile ones.  So the aliases get expanded at runtime, not at compile time.  Because you'd never be able to change the aliases in your compile script if you got  expanded at compile time.  And the other thing is it can only resolve to a command name.  And you can't have fragments of grammar in there.  And the example with the expanded part shell, I could have had something that said ls-l could  have been the value in the variable command.  And then it would actually parse into two arguments.  So you're actually doing kind of an eval on the resulting string.  But we needed to do this.  We needed to have this different approach because we're dealing with objects everywhere.  And we'll see that in a second.  Splatting.  This was designed to solve, or was added to solve commands calling commands problem.  In a lot of places, you have a command that wants to then call a sub-command with the  same parameters that were passed to it.  And so, Splatting does that.  You get your PS bound parameters variable, and you do app PS bound parameters to the  called the child command.  Splatting comes from Ruby.  It's another language, as I know.  I got it from Ruby.  Just to clear that up, because it matters to me for some reason.  We use the, but Ruby uses the asterisk to indicate a splatted variable.  We use the at symbol because the asterisk is a glob, right?  If I wanted to have, if I wanted to, I'm doing my command, my command space, asterisk,  foo, well, that's a glob expression.  If I wanted to pass that variable, we see at something or other.  So we needed to use a different sigil character to distinguish it from globbing expressions.  Script Lux and the call operator.  I've talked about this a little bit.  We needed the call operator because we're not an expanded part shell.  I need the call operator to be able to do an indirect call through an expression.  So, and this includes things like commands with spaces that have parans around it, because  that's an expression.  The script Lux grew in a strange way.  There was a gentleman on the MMC team who came to me and said, well, we're going to  want to use PowerShell, but we need to have this idea of isolated fragments of code.  Is there a way you can do that?  I thought, well, we don't have a way to do that now, but I've heard of this thing that  might work.  And so this seems like a job for a Lambda function.  And so we introduced script Lux using per end notation, block notation.  The notations are much more common now at the time they were sort of notoriously known  for being part of hard to use languages.  It was a strange exotic thing that nobody understood.  That would be a bad thing to do to our less than technical users.  So we wanted it to be more amenable.  And so they became script Lux rather than Lambda.  And take a look at an example with where reach or for.  It looks pretty natural.  Most users are not aware that they're using a script block there or that is anything strange.  So script Lux are pervasive throughout the language runtime.  Every piece of code gets compiled into a script block first.  When you run a script, it gets built into a script block and then evaluated.  And really, a script is nothing more than a script block on disk.  You could read it into memory, compile it, create script block, and then evaluate it.  They also allow things like delegates, which open up a whole bunch of areas of coding that  we weren't able to do before.  So we can do informed coding in PowerShell.  You can write Goobys in PowerShell.  You can do XAML coding in PowerShell.  You can even do stuff with the DOM and HTML if you really want to.  So that opens up a whole range of possibilities that wouldn't have been available otherwise.  Yes?  Yeah.  Yeah.  So the work command was very complicated and then, well, it was very complicated and  then they got reduced down to here's a script block just executed.  And then it got complicated again for the simple where.  Which is funny.  Be careful what you wish for.  You might get it.  Yeah.  And so we had, again, this function example where, I mean, then there's a parameterized  function because script blocks take parameters.  And so you can do the fully equivalent of a function definition with parameters and everything  by assigning a script block into a variable or into a, well, into the function drive.  So there's a bunch of language design questions that we had to deal with.  First off, PowerShell is an expression-oriented language.  Again, does it ring a bell with anybody?  Total silence.  So expression-oriented language is everything returns a value, which is what happens in PowerShell.  So even something like a foreach statement returns a value.  The if statement returns a value, which is our argument against having the conditional  operators.  One difference from a lot of expression-oriented languages, though, is that it returns everything.  Every statement, every line in a block of code returns a value.  In a lot of languages, like Lisp, it's typically the value of a collection of code is the value  of the last statement in that collection of code.  PowerShell is not.  It's everything.  The original version of the interpreter actually worked the way I said the Lisp interpreter  worked.  It only returned the value of the last statement.  But that's not very shell-like.  So PowerShell's right to streams.  Streams get aggregated from every line within a function or a script.  Lexical or dynamic scoping?  This is another one.  Ring a bell?  No.  Lexical scoping is most common in programming languages these days.  Dynamic scoping, the original Lisp interpreters were dynamically scoped.  Basically what it means is when you're looking for a variable, you look in your scope, your  media scope, and if it's not there, then you look in your color scope, and if it's not there,  then you look in the color scope all the way up until you get to global scope.  And this is basically how PowerShell works.  It makes a bunch of things simple.  You can pass a whole bunch of context through scoped variables rather than having to have  a lot of parameters.  But it also makes things fragile that you can have false coupling between functions because  the variable has leaked from the wrong place.  So good for small programming and small, not great for programming in the large.  We had discussions with some of the architects in the CLR team, and they were like, oh, you're  going to get it if you go with dynamic parameters.  But it has largely worked.  And one of the things that we did in V2 with modules is that we introduced this idea of  module scope so that you don't leak into the functions in the module.  Could the option character be dash or slash?  That was easy.  We're competing against the Unix shell, not command.exe, so we'll go with the dash.  How to handle switches.  Switches are parameters that are either present or absent.  Easy to deal with in an expanded parse because if your variable is empty, if your variable  is not empty and contains a switch parameter, then the switch parameter is there.  And it's on.  And if the variable is empty, then the switch parameter is not there and it's off.  Again, that doesn't work in a fully parsed environment.  And so we needed a different mechanism.  We were doing, addressing this, we thought, well, they'll just be Booleans.  But people wanted to use Booleans for a whole bunch of other things.  It seemed like kind of a large space to co-opt.  And so we introduced this new class switch parameter that has this characteristic of  being present or absent and doesn't take an argument until it does.  Because, again, command is calling commands.  You do want to be able to pass through an argument to a script block or through a command  through a switch parameter.  Adding in PSBoundPurr and there's Matek, a little bit less difficult.  But it's still a case where, in simple cases, where you want to be able to pass it through.  And you'd think this would be obscure enough that people wouldn't use it by accident.  But I've seen lots of people, well, lots of stuff online where people are putting in  switch parameters, colon, dollar, false.  I have no idea where that's coming from.  More language questions.  So expand in part shells.  They only deal with strings.  All the arguments to a command are strings on the command line.  And so it's up to the shell to figure out what it is.  It's up to the individual command to figure out what that string is for, whether it's  a number or a string or a file name.  But it means that everything just kind of, like, automagically works because the commands  know what they want to do with strings.  They have their own built-in parsers.  In our environment, we don't want people writing all these custom parsers.  We want to be able to pass real objects.  And so the primary reason for having the fully parsed shell is that because the variables  are not expanded into text, they're passed in as objects, then we can get objects passed  through to commandlets rather than expanding into, some people thought this is what we  were doing at first, is that we were expanding into this big-ass XML string and then turning  it back into an object inside, which would have been horrendously slow.  But we just passed in the object itself.  But then we still had this idea that we want everything to just kind of work.  You shouldn't have to worry about doing a lot of explicit conversions.  How do we get the user experience with a type-based shell to be similar to that of a string-based  shell?  And the answer to that was super aggressive type conversions.  So the runtime, when you're passing stuff around, the runtime makes a lot of implicit  guesses at what you meant.  There is a constraint on that.  One of the things that we considered doing was having an automatically-chain conversion,  and then we realized that that was a bad idea because everything can reverse the string  eventually.  And so we'd be right back where we started from.  And so we don't do implicit chaining, but we can do explicit chaining.  When we're doing this, we use an algorithm for searching for the converter between the  source type and the target type.  And we use an algorithm called the type distance algorithm so that in ambiguous cases, we  compute a distance between the target types, and whichever one is closest to the target  is the type conversion that we use.  I think that has been tweaked a little bit over time, but I think we're still using the  same basic type conversion algorithm.  So that's how generic type conversion work.  The parameter binder does not do this.  So this is one of the things that we, in version one, said, well, we can ship it this way and  we won't change it later.  And 10 years later, we haven't changed it.  So I don't think it's ever going to do.  But anyway, binding in the parameter binder is done in two passes.  The first pass goes through and tries to do an exact match on each of the parameters.  So if the types are the type of the argument and the type of the parameter are the same,  then it can simply bind the parameter.  If that doesn't succeed, then it does a second pass where it looks for a type conversion.  So there's a type conversion for the explicit target type, the source type to the target  type, then it can succeed and proceed with a bind.  So everything is done twice.  If you ever wonder why it's slow, well, that's one of the reasons why it's slow is that it  does this extremely complicated process using bitmaps or bitfields to determine the degree  of closeness of the match and so on.  It makes parameter binding very heavyweight.  But hey, we said we were not going to worry about CPU cycles and stuff back in the beginning.  So this was designed to give the best user experience at the cost of significant reserves  of resources.  We'd like to fix that.  We have some ideas.  One of the things that we don't do right now in the parameter binder is we don't do, so  the DLR runtime does type caching or method caching.  Once it figures out how to do something, then it caches that.  So the next time through, it says, if I'm still doing the same thing, I can use this  mechanism and I've cached.  We don't do that for the parameter binder today.  We could probably do that.  So we do the expensive computation once and then cache the result.  That would be fun.  I'd like to work on that.  Anybody else want to work on that?  What am I doing?  I have, oh, there's just no time at this point.  So finally, we're near the end of what I was going to talk about today.  So we have, we've built everything.  It's all spiffy and cool.  We have all these great ideas.  Now we need to ship.  Shipping is fun.  Because PowerShell started as an independent project inside Microsoft, it had its own build  system.  It had its own everything.  But later on, we shifted to be part of Windows.  Windows has its own build system.  So we switched from our build system to their build system and they have a bunch of rules  about how stuff looks.  So we changed our stuff to look like their stuff.  And then there's the Longhorn Reset and the world explodes.  We were a brand new.NET component and we were kicked in someplace in the anatomy out  of Windows.  Crap.  Now what?  Okay.  So we're drifting aimlessly and Longhorn's the exchange team, our best friends, our new  best friends.  They were investigating an automation solution for managing exchange servers.  And PowerShell pretty much matched all of their requirements just by simultaneous invention,  which was very cool.  So this was awesome.  We have a new development partner, a great development partner because they gave us a  lot of feedback, very constructive.  And we have a new shipping vehicle.  We can ship as an exchange shell.  There's some impact on the product.  First the fact that the Longhorn Reset kind of gave us a lot of time to work on things.  And so PowerShell became much, much more refined than it might have been.  So that was good.  On the somewhat less good side was there was a focus on the exchange team features first.  And that meant that some of the things that are applicable in general shells didn't get  that much attention.  Native command support is an example of that.  So Native command processor was barely there.  I barely had time to get anything working at the time frame.  And it still sucks.  And I know that Sergey Vorbev did a bunch of work to get to slightly unsuck it for PowerShell  6.  But it still needs a bunch more work.  It still has parameter binding issues.  The pipe doesn't work the way it should.  You are sort of just barely able to launch notepad.  And that wasn't bad for exchange, but it was bad for a general purpose shell.  And that's something I would really like to make work better.  Shipping is fun.  Three.  So we finally get approval to ship as a Windows feature, not as an exchange feature.  So now we're Windows PowerShell, not Exchange PowerShell, or Exchange Shell, or Exchange  Monad or whatever.  But we have to pass a bunch of Windows quality gates.  So there are versioning requirements.  There's a couple of ways you can go about doing versioning.  You can have loose versioning, also known as DLL help.  Or you can have rigid versioning, which is what the.NET team did with the GAC.  And that is completely brittle.  I mean, you'll never be wrong, but you might be useless.  So as Jeffrey says, this versioning is not a problem.  Problems can be solved.  This can be only mitigated.  Part of it had to do with how, the way we dealt with plugins.  There was a no plugins requirement, because IE was, well, IE was IE.  And it had lots of plugins, and it hung a lot.  I mean, it got to the point where there were fewer crashes.  There were more hangs than crashes in coming back from telemetry.  And a lot of the hangs were in IE, not in the code itself, but in the plugins that  people were writing.  So there was a big push not to have new plugin architectures.  And DevDev did a big effort to come up with, and I'll just describe it, a very resilient  plugin architecture.  It was also complicated and heavyweight.  And so we looked at it, but it would have been wildly inappropriate for us.  It was designed for the level of a component, not a commandlet.  The PowerShell is all about plugins.  So we had to do something to satisfy this requirement.  So our first attempt was mini-shells.  So there wouldn't be one PowerShell.  There would have been a PowerShell, and there would have been an AD PowerShell, and there  would have been an Exchange PowerShell, and there would have been everything.  Because what you would have had to do is you would take the libraries for PowerShell, and  you'd take your commandlets and some tooling, and out of this you would spit out a custom  shell that included your commandlets hard linked in.  And so the only way to call from commandlet to commandlet would be from shell to shell.  And there is a feature in PowerShell.  If you're within PowerShell and you type PowerShell, open brace, like a script lock,  closed brace, then that will be evaluated as an expression in the child shell.  That was actually added to make mini-shells work tractably well.  I think it's the only part of mini-shells that are left.  All the tooling and everything else got cut ages ago.  Exchange looked at this and said, no.  Are you guys idiots?  This is not going to work for us.  We need to be able to extend our shell with some kind of plug-in mechanism.  So once again we return to the halls of Windows Core Architecture team, supplicants on vended  knee begging for a solution, and since MMC already shipped with Windows and it had snap-ins,  we thought, okay, this is shipping and seems to be okay, so can we do this?  And I think there was a diverse set of opinions on MMC snap-in mechanism that are probably  not publicly consumable, but they ultimately agree that we could do this tap-in mechanism.  And so we went off to do the snap-in mechanism where you have to register the DLL and all  that kind of stuff.  And this still has dregs in the code.  You can see it under system management automation, single shell, and there's another thing,  single shell utilities.  And these are left over from, I think, the reason it's called single shell is because  we had mini-shells, which was mini-shells was mini-shells and single shell was one shell  with plug-ins.  So we got approval for snap-ins.  It was not optimal, but we could ship.  That was one of the first things we went after in PowerShell v2.  Yes, the language reviews, because PowerShell is the only language product that ships outside  of DevDiv.  The developer division was a little concerned about this because at the time everybody was  well.  We were kind of siloed and everybody was protecting their products and making sure that we weren't  confusing the user with the wrong thing and all the normal stuff that businesses do.  It used to do anyway.  So we had to have a review with the CLR and the relatively newly formed DLR team.  One of the big issues was the architectural approach to passing objects through the pipeline.  At the time DevDiv was producing link.  Link works by composition of enumerators and it pulls the objects through their pipeline.  On the other hand, the pipeline processor pushes the objects one by one through the pipeline.  So we're a push model.  They're a pull model.  There was some idea that we should have one model.  Fortunately, Anders Helzberg was part of the review and he pretty quickly figured out  what we were doing and he understood why we were doing it.  And interestingly enough, some significant number of years later, DevDiv has the reactive  framework and the reactive framework works by pushing objects through a pipeline.  So we felt slightly vindicated.  And then we're stuck in the corporate layer.  Oops.  Somebody noticed that you could build software with PowerShell.  Oh, my.  So the MS build was new at that time.  It was just coming out.  Microsoft invested a lot of energy into building these new tools.  Virtual Studio was notoriously weak on build automation.  So it was a tool for automating builds.  So mechanisms unknown.  Someone saw a slide talking about how PowerShell could be used as part of a software construction  tool chain.  This was, I think, actually being used.  The slide deck was used to justify the existence of PowerShell.  Look at all the places you can use it.  So of course, it tried on somebody's toes.  And it's not like that idea is unique.  I mean, the shelves have always been part of the programmers workbench on Unix.  The programmers workbench is a set of utilities, make, and so forth, or building software.  So it's not unexpected that PowerShell should be able to do this.  But we had a review.  And it was a review.  In the end, it actually, we sort of clarified the relative roles of things by kind of like  MS builders like Ant and PowerShell is like Shea and Nmake is like Make.  Once people understood the relative roles of the tools, we got through that little hurdle  without dying.  And then finally, we're through all the hurdles, we're through the gates, we've had our reviews.  Oh, yeah, I forgot the first Monad virus thing.  It was the first Vista virus where some kid in somewhere, this guy wrote self-replicating  code.  He wrote it in every language.  He liked using this to explore new languages.  And so he wrote the same self-replicating virus that he wrote in C Sharp or VB or Unix  Shell, he wrote in PowerShell just to see if he could do it.  And somehow this got picked up by the security community in probably like on a Sunday morning  after a big party when nobody's thinking clearly.  And it got promoted in a big article.  And this just started a huge fire store.  So again, had to calm everyone down, publish the appropriate discussions, explaining what  was going on.  And again, another shipping hurdle was passed.  But blood pressure, in that last period, blood pressure was going up and down like a yoyo.  So finally we're at the end.  One teeny thing left we have to fix.  Everything has decided that we will rename Monad, which everybody loved, to Windows PowerShell,  which was not as well received.  And it had to be Windows PowerShell.  It couldn't be PowerShell because there were other things that used PowerShell.  If we wanted to get a trademark, it had to be Windows PowerShell.  So we had to change it into code.  We had to change it everywhere.  Everywhere where the physical, visible, but also a lot of places in the code where things  might be visible through reflection.  And so with a bunch of scripts and a whole lot of scanning later, everything got cleaned  up.  Well, almost everything.  You will still see in the code, like the fact that the PS object, the core of our type  system, is stored in a file called MSH object.  You'll see, you'll still see bits and pieces of that.  So anyway, shipping is fine.  And now we're ready to ship.  Yay!  And finally, we have shipped PowerShell 1, shipped in 2006, to great acclaim and huge  size of relief.  And we've been going steady ever since.  And this is all the time I have, based on, oh boy, I'm way over.  So again, I encourage you, based on what you've heard today, if there are things that I haven't  answered or things that you'd like more information on, please let me know.  I should have had all my stuff in the back.  This is a long deck.  Oh, yes, above me.  So my email is here.  There's there.  Come on.  Stay.  Okay.  Yes.  Yes, I did it.  My email, my Twitter handle, let me know.  And stuff that you think I should cover, things stuff that you think I should cover in more  detail, stuff I should cover less detail, stuff I should emit completely, please let  me know.  Anyway, so thank you.
A_Historical_Architectural_Tour_of_PowerShell_0_47.png A_Historical_Architectural_Tour_of_PowerShell_48_95.png A_Historical_Architectural_Tour_of_PowerShell_96_143.png A_Historical_Architectural_Tour_of_PowerShell_144_227.png A_Historical_Architectural_Tour_of_PowerShell_228_311.png A_Historical_Architectural_Tour_of_PowerShell_312_1691.png A_Historical_Architectural_Tour_of_PowerShell_1692_5039.png A_Historical_Architectural_Tour_of_PowerShell_5040_7043.png A_Historical_Architectural_Tour_of_PowerShell_7044_11375.png A_Historical_Architectural_Tour_of_PowerShell_11376_14927.png A_Historical_Architectural_Tour_of_PowerShell_14928_17255.png A_Historical_Architectural_Tour_of_PowerShell_17256_17975.png A_Historical_Architectural_Tour_of_PowerShell_17976_18047.png A_Historical_Architectural_Tour_of_PowerShell_18048_18179.png A_Historical_Architectural_Tour_of_PowerShell_18180_18239.png A_Historical_Architectural_Tour_of_PowerShell_18240_18299.png A_Historical_Architectural_Tour_of_PowerShell_18300_18995.png A_Historical_Architectural_Tour_of_PowerShell_18996_19043.png A_Historical_Architectural_Tour_of_PowerShell_19044_19319.png A_Historical_Architectural_Tour_of_PowerShell_19320_22475.png A_Historical_Architectural_Tour_of_PowerShell_22476_24827.png A_Historical_Architectural_Tour_of_PowerShell_24828_26879.png A_Historical_Architectural_Tour_of_PowerShell_26880_29267.png A_Historical_Architectural_Tour_of_PowerShell_29268_32087.png A_Historical_Architectural_Tour_of_PowerShell_32088_34391.png A_Historical_Architectural_Tour_of_PowerShell_34392_38003.png A_Historical_Architectural_Tour_of_PowerShell_38004_40739.png A_Historical_Architectural_Tour_of_PowerShell_40740_42659.png A_Historical_Architectural_Tour_of_PowerShell_42660_42803.png A_Historical_Architectural_Tour_of_PowerShell_42804_44339.png A_Historical_Architectural_Tour_of_PowerShell_44340_49487.png A_Historical_Architectural_Tour_of_PowerShell_49488_51887.png A_Historical_Architectural_Tour_of_PowerShell_51888_52127.png A_Historical_Architectural_Tour_of_PowerShell_52128_53387.png A_Historical_Architectural_Tour_of_PowerShell_53388_57383.png A_Historical_Architectural_Tour_of_PowerShell_57384_58775.png A_Historical_Architectural_Tour_of_PowerShell_58776_59147.png A_Historical_Architectural_Tour_of_PowerShell_59148_64211.png A_Historical_Architectural_Tour_of_PowerShell_64212_64283.png A_Historical_Architectural_Tour_of_PowerShell_64284_64499.png A_Historical_Architectural_Tour_of_PowerShell_64500_64547.png A_Historical_Architectural_Tour_of_PowerShell_64548_70439.png A_Historical_Architectural_Tour_of_PowerShell_70440_74243.png A_Historical_Architectural_Tour_of_PowerShell_74244_77759.png A_Historical_Architectural_Tour_of_PowerShell_77760_77927.png A_Historical_Architectural_Tour_of_PowerShell_77928_78191.png A_Historical_Architectural_Tour_of_PowerShell_78192_79151.png A_Historical_Architectural_Tour_of_PowerShell_79152_81875.png A_Historical_Architectural_Tour_of_PowerShell_81876_85607.png A_Historical_Architectural_Tour_of_PowerShell_85608_88415.png A_Historical_Architectural_Tour_of_PowerShell_88416_88907.png A_Historical_Architectural_Tour_of_PowerShell_88908_92159.png A_Historical_Architectural_Tour_of_PowerShell_92160_93647.png A_Historical_Architectural_Tour_of_PowerShell_93648_96983.png A_Historical_Architectural_Tour_of_PowerShell_96984_97091.png A_Historical_Architectural_Tour_of_PowerShell_97092_97463.png A_Historical_Architectural_Tour_of_PowerShell_97464_98003.png A_Historical_Architectural_Tour_of_PowerShell_98004_98051.png A_Historical_Architectural_Tour_of_PowerShell_98052_98099.png A_Historical_Architectural_Tour_of_PowerShell_98100_98147.png A_Historical_Architectural_Tour_of_PowerShell_98148_98195.png A_Historical_Architectural_Tour_of_PowerShell_98196_98243.png A_Historical_Architectural_Tour_of_PowerShell_98244_98351.png A_Historical_Architectural_Tour_of_PowerShell_98352_98411.png A_Historical_Architectural_Tour_of_PowerShell_98412_98459.png A_Historical_Architectural_Tour_of_PowerShell_98460_98543.png A_Historical_Architectural_Tour_of_PowerShell_98544_98591.png A_Historical_Architectural_Tour_of_PowerShell_98592_98639.png A_Historical_Architectural_Tour_of_PowerShell_98640_98687.png A_Historical_Architectural_Tour_of_PowerShell_98688_99119.png A_Historical_Architectural_Tour_of_PowerShell_99120_99215.png A_Historical_Architectural_Tour_of_PowerShell_99216_99371.png A_Historical_Architectural_Tour_of_PowerShell_99372_99431.png A_Historical_Architectural_Tour_of_PowerShell_99432_99479.png A_Historical_Architectural_Tour_of_PowerShell_99480_99527.png A_Historical_Architectural_Tour_of_PowerShell_99528_99575.png A_Historical_Architectural_Tour_of_PowerShell_99576_99617.png