No more 'grunt watch': Modern build workflows with Broccoli
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Liss, Jo
Broccoli is the newest JavaScript-based build tool everyone's talking about. It helps you concatenate your scripts, compile your Sass files, and package up your ES6 modules—all while offering a concise declarative syntax, and blazing fast rebuilds. Building on that foundation, ember-cli is a project to bring an opinionated, Broccoli-based build stack to Ember developers, tuned with their specific needs in mind. Learn how to use Broccoli and ember-cli to define modern build workflows for your Ember applications, harnessing preprocessors such as Sass and CoffeeScript without losing sub-second rebuild times.
 I'm really excited to start using Broccoli. As soon as I can figure out how to spell Broccoli with my MPM install, I will be sure to check it out. Go ahead, Joe. Thank you, Tom. Thank you. Is the mic on? Yes. Awesome. So my talk is going to be about Broccoli. I'm Joe. I wrote Broccoli. And before I start off sort of going into the technical details, I want to give you a bit of background about what motivated Broccoli and how it fits into the larger ecosystem. So I started thinking about this at last year's Amber conference. And what was bothering me at the time was that we had no way to share code in the front-end browser JavaScript ecosystem. We could share code with Ruby or with MPM for node-based JavaScript. But for front-end JavaScript, all we were doing was like, here's a big JavaScript file. Please drop it into your vendor directory. And that just didn't seem really great. And as I started talking to people about this, I realized pretty quickly that the missing piece was actually not package management. We do have a package manager that is pretty good. And I think it's architecturally doing the right thing. And that's Bower. So what Bower will do is when you have an app with dependencies, it's going to load all your dependencies from the Internet and load their dependencies recursively. And then it's going to drop all of those into a big directory. And so it just dumps them into the file system and it's up to you to do something with those files. Now, with something like Ruby, it's pretty easy to, say, require some file. And then Ruby is just going to search through all those directories and things work. But for front-end JavaScript applications, we have the problem that we actually need a tool on top of this that ties all those files together and concatenates them and bundles them into one tidy app that we can deliver into the browser. And so I realized that this is the missing, this is the thing that is missing for us to have a great front-end package ecosystem. And the way we've been doing this is grant watch. We have the Rails asset pipeline, and it's not that bad actually, but of course it's tied to Rails. And so if you're building an app with JavaScript, the most common thing seems to be to make a grant file and you put a lot of, you define a lot of tasks that compile coffee script and handle bar templates and then you concatenate everything and then you minify and then you also have some sets and probably some more things. And it becomes complex very quickly, right? If you've seen grant files of real world amber apps, they generally look scary because you have to manage all these compiled files. So that is one problem. And the other problem is let's say you have a grant-based app and you work on it with a small team for a couple months. And as time passes, the build gets slower and slower. And so after a while, after a couple months, you edit a file in the editor and you reload the browser and it takes 10 seconds to reload because the rebuild is so slow. And that is clearly not acceptable, right? This is the thing that you're doing 100 times a day. This is your core feedback loop as a developer. If you have a delay there, it's going to make you massively more unproductive. So the goal of Broccoli is to have an architecture that allows us to not have 10 seconds rebuilds but be faster. So this talk has three parts. The first part is I'm going to show you how to write a Broccoli build definition in practice. The second part is I want to sketch out the Broccoli plugin API in very rough terms. And the third part is I want to talk a bit about how Broccoli integrates with other tools. And I'm not going to talk about the motivation behind some of the design choices in Broccoli. I wrote a blog post that I published about a month ago and I'm going to link it at the end and that has all the rationales for the design choices in excruciating detail. So let's talk about build definitions. By the way, there are some changes that I have planned for the Broccoli build definition syntax. They are purely cosmetic, but what I'm going to be showing you is the new syntax that is coming up soon in the next one or two weeks. If you want to see the old syntax, there is a Broccoli sample app on GitHub that has the current syntax, but I figured I'm going to show you the new stuff that is coming up. All right, let me switch into the editor. So this is a Brocfile.js build definition file. I'm going to start with a simple case where you don't do anything. We have a string here, lib, that refers to the lib directory. We're just going to export that string. When you run Broccoli, it is going to serve all the files in the lib directory without doing anything to them. If you run Broccoli build, it is going to copy those files into a new directory. So there are a lot of this JavaScript files. The first thing we want to do is concatenate those files into a single JavaScript bundle. There are many ways to do this. I'm going to show you the simplest way, which is just to use the Broccoli concat plugin. I'm omitting the require calls, but you have a require header at the top right where you pull all this stuff from NPM. So we have a concat plugin that takes that lib tree and you give it a bunch of input files using a glob pattern and you specify an output file. And the concat plugin is going to return a new tree object. I'm going to call it app.js here. And that tree object, it represents a tree of files, but it really contains just one directory, assets. And that directory contains one file, app.js. And we're going to export that app.js tree. And so if you now run Broccoli serve, it is going to serve one single file at the URL slash assets slash app.js. And that's your concatenated JavaScript output. Let's add coffee script support. So what we would like to do is take all the files in the lib directory, all the JavaScript files, they stay in place, but all the.coffee files are going to be replaced with compiled.js files. So we're going to use the Broccoli coffee plugin and call it on the lib tree. And it's going to return a new tree object that represents all the files in the lib directory except with.coffee files replaced with.js files. So then we have a new tree and we pass that new tree into this concatenator. So as you can see, we don't have to manage any kind of temporary directories to hold our build output. We can just chain things together. And the next thing we're going to do is minify. So we run this app.js tree through the Aglify JavaScript filter, through the Broccoli Aglify plugin. And we're only going to do that in production. Broccoli core doesn't have a notion of a build environment, but what we tend to do is just use an environment variable like this. So it's up to you in your Broccoli file, in your Broccoli file to have a notion of a build environment. So in production, you're going to filter this app.js tree through the Aglify filter. Let's add some sass next. So we take the styles directory and we pass it into the Broccoli sass plugin and we tell it to take the app.scss file. And that file is going to have import statements that import other files, which in turn import other files. And so sass is going to generate this huge blob of CSS. And that is going to go into the slash s at slash app.css file. And we store that in a new tree. Now, in order to export this, at the moment we're just exporting the JavaScript. So in order to export this, what we're going to do is call the Broccoli merge trees plugin and merge this app.js and app.css trees into one combined tree. Just copy them on top of each other. So now when we run Broccoli serve, it is serving two files at different URLs, one app.js and one app.css file. And finally, we're going to add an index.html file just to boot up our application. And we're going to place that into the public directory. So we would like to also merge the public directory into the output. So now all the files in public are going to show up in the Broccoli output as well without any changes. And that is a fairly simple example of a Brock file. So I hope you're equipped, if you want to start writing your own Broccoli build definitions to do this now to have an idea of what goes where. Thank you. Thank you. We have a part two where I talk about plugins, but my screen isn't playing along. Yes. Okay. So Broccoli's core plugin API is very, very simple. It only knows one type of object and that is a tree, right? So at first you would sort of intuit that a build tool should have a notion of a plugin that takes a tree and returns a new tree, but it turned out to be unnecessary and so the only primitive that Broccoli knows is a tree. A tree is represented by a string pointing to a directory or an object. And that object is going to have two functions and this is the kind of object that a plugin will generally return. And all the important work happens in the read function. The dot read, the read function on the tree just returns a directory and that directory contains the tree of files. Now every time we rebuild we are going to call the, Broccoli is going to call the dot read method on the tree. So it's going to call dot read repeatedly as files change, as it rebuilds it's going to call dot read. And then at the end when you shut down the server it's going to call the cleanup method to give the plugin a chance to delete any temporary directories. So as you can see the API is tiny. It doesn't really tell any story about how you should build stuff. But this is something that we kind of learned from the, from Grunt. The people working on Grunt they really impressed upon me that having a lot of API surface for a build tool is really painful because what happens is you have this, this ecosystem of plugins and you can never change the API without breaking everything basically. So everything you have in the API is going to have to stay there forever. So I tried really hard to keep the Broccoli API minimal and move all the functionality that could be moved out into external helper packages. And so plugins are generally going to use one of these helper packages as base classes to implement their functionality. One such package is the Broccoli Transform package. It is for the case where you have one input tree going into a plugin and one output tree coming out of the plugin. So now when you want to implement a plugin using the transform base class, you don't have to do the read thing and cleanup thing and you don't have to deal with temporary directories. You just have to implement this one function, sourcedir and destdir. And you get to read files from the sourcedir and then write any output files into the destination directory. And then on top of Broccoli Transform we have another package called Broccoli Filter and that is actually a really important package to implement plugins because it implements the case, the very common case where you have one-to-one relationships between input and output files. So Broccoli Coffee is an example of that. Every file either gets passed through, unmodified, or the.coffee files get compiled and get turned into.js files. And when you implement a filter plugin, this is the only thing that you have to implement, a process string method that takes a string, so for example the coffee script file contents, and that returns a new string, the compiled output. And the only thing you have to tell it is that it takes.coffee input files and that it will emit.js output files. And then the Broccoli Filter base class is going to handle all the file handling for you. And importantly, Broccoli Filter has a cache. The reason why that is important is that every time you rebuild the plugin is going to be called and it has to rerun everything. And so if we didn't cache, we would be recompiling all our coffee script files every time and it would be incredibly slow. So Broccoli Filter is what makes it fast. When Broccoli rebuilds, the filter base class is going to check if a file's last modified time has changed. And if it hasn't changed, it is just going to reuse the output from last time. So that's really important to make it fast. Unfortunately, Broccoli Filter is limited to one-to-one cases because that is the only way you can really reliably cache files. So it doesn't work for SAS. SAS, for example, a SAS file has import statements, so it can depend on other files and those files can depend on other files. So we need to rebuild the SAS file whenever one of those dependent files change as well. So implementing caching for more complex compilers like SAS is a bit non-trivial. There are several ways. One way is to move it into the plugin and I think that is probably the fundamentally the best way. I did this with a compiler that I wrote to concatenate ES6 module JavaScript files and it just brings its own cache. Another way is to just ignore the problem. If the compiler is extremely fast, then we can just rerun it every time. So that's what I actually did for SAS. The Broccoli SAS plugin is based on the Lib SAS library which is implemented in C. It is blazingly fast. So for now, for Broccoli SAS, I'm just ignoring the problem. And then for the Rails, then there's the third way and that's what the Rails Asset Pipeline does. You kind of parse out, you use a regex on the SAS file to figure out what it depends on and then you recursively build this dependency tree to figure out when you need to rebuild the SAS file. But clearly that's kind of a hack. We may have to do it in some cases. But I think we want to avoid it. So this is not, I don't think we have really converged on a great way to do this yet in the compiler ecosystem yet and I think we're going to have to talk to the authors of compilers like SAS and LES and what have you as well to really understand where we should be putting this caching and how we can better, how the compilers can help us implement this more easily. Now, part three, I want to talk about integration. Broccoli is a small library and I don't think of it as a tool as much as I think of it as a library. It does have a command line interface so you can type Broccoli serve and Broccoli build into the command line. But it's really more of a proof of concept and in reality you're going to want to proxy stuff to the API and you're going to want to handle URLs in certain ways. So there are all kinds of tricky things that you want to do in practice that are definitely not going to be covered by the command line tool. And so I think we're going to have to work a bit more on getting Broccoli integrated into other tools and having Broccoli provide a good API that is still kind of missing. Broccoli loves Grunt. There is actually a Grunt Broccoli plugin for Grunt. Grunt is a good task runner. You should be using it for stuff like deploying or generating stuff or running the test suite. Those are all things that should be living in your Grunt file. And just the core build loop is what Broccoli should be solving. So it's not that Broccoli is trying to replace Grunt, it's just that we shouldn't be using Grunt to build our applications. Grunt is a task runner, not a build tool. So the Grunt Broccoli plugin for Grunt is very simple at the moment and I hope that we can provide more hooks into the rebuilt lifecycle to make it better in the future. And I'm hoping that we also can integrate with backends. And the reason is that when you have a backend like Rails plus the asset pipeline, there's a lot of smart things that you can do by default because you can make a lot of assumptions about how the URLs work. So that would make it easier to, that will make it easier to come up with the default stack because we can make more assumptions about what things go where. In particular, speaking of default stacks, Steph has been working, doing a lot of work and other people as well, I believe, on the Ember CLI tool which is going to be, which is Broccoli based and which is going to provide a default tool stack for new Ember applications so that you can type Ember new to do MVC and you can make a new to-do list. So I'm going to be here for another two weeks in Portland and I would be very happy to pair with people on integrating Broccoli into other tools and maybe coming up with solving some of the problems that we've been having with implementing plugins the right way. So I would be happy to, please get in touch. This is my Twitter and my email address and as I promised earlier, the lengthy blog post about Broccoli's architecture is there if you want to read up. So I think we have a couple minutes for Q&A. Who has the mic? Rico? Excellent. Who has a question? Yes, over there. Hi, I'm Matt Newton. I work at Apple and we have a problem right now, well it's not a problem, we solved it with Grunt. We're building multiple Ember apps with a shared code base and I wonder if that's something that, does Broccoli have a user story for this? So for sharing, what you're talking about is basically sharing libraries and components between Ember apps. So the story for that is you would probably use something like Bower to pull in libraries into your app and there is a Broccoli Bower plugin that automatically picks up all your Bower directories. So yes, if you want to see an example of that, there is a Broccoli sample app on GitHub that uses Bower libraries. Do you happen to know, I don't know if anyone else knows, I haven't used Bower extensively, but can you use private Bower modules like your own registry, sort of like NPM? I don't know. And it pulls from Git or something like that? Any more questions? In the back, black T-shirt. Hi, I'm Dave Duchain from Absolute Software. No booze please, but I have to ask if, what the state is with regards to Windows support? Sorry, can you repeat that? I know, I know. Oh, I'm sorry. How well does it work on Windows? Broccoli, that is. Sorry, I couldn't hear the question. How well does Broccoli work on Windows? On Windows. At the moment, there is a problem with Node, Windows file I.O. where it has weird race conditions. So if you write a lot of files or delete a lot of files very quickly, and Node is going to throw randomly weird errors. So the Windows support is basically blocking on that as far as I know. So someone is going to have to fix Node. One question? Yes. Hey, my name is Joe Fiorini and I really like the new API for the Brock file. The one question I had is, would you think about letting arrays, be representation of trees like strings are? So instead of having to do merge trees in your Brock file, you can just export an array and not have to worry about merging them. So that's what, that's the original syntax where we just export an array, but it doesn't have the overriding semantics well defined. Like it has to either override or not override. So I moved it into an external plugin and then you can configure whether you want to override or not. Yes. Jake Bixby. I have an idea of a larger application where you want to maybe break it up into different modules. Maybe using trees to have separate build targets. So like you basically have your core app and then different routes break those into different build targets and then somehow with like requires or some sort of way to pull those in. But that's different. But as far as like the using broccoli for different build targets, is that fairly easy to do, do you think? Yes. You could generate multiple trees and then there is a plugin to move files into subdirectories. So you could have one tree in one sub directory, another tree in another sub directory. And then at the end you just merge all of these trees together to output five different subdirectories. So that is definitely a use case. Yeah. In the back. What about fingerprinting? Sorry, can you say it again? What about JavaScript and CSS and asset file fingerprinting for cache busting? Fingerprinting. I don't want to implement it on the CLI. So one of us is going to have to figure something out. Maybe if I was staying longer in Portland we could just pair on it and do it together. You'll have to convince Luke to let me stay a little bit longer. I have completely punted on writing a fingerprinting plugin. And the reason is fingerprinting files was very easy of course. You just renamed them. But the hard part is we write in URLs. And so it seems that we would want to have, let's say, script references in HTML file. And URL references in CSS files pointing to images. You want to rewrite all of these. And there is no general way I think to do URL rewriting reliably. So this really needs to live in each preprocessor. So that makes it a lot harder to do it right. I agree by the way. Since we're having a discussion about this. Do you foresee, this is what we've been talking about at this table. Do you foresee just a separate tool that's specialized for fingerprinting that lives as a sibling to broccoli? Or do you see it as something that's more integrated with broccoli? I think it would be integrated. So again, it's probably not a single plugin, but we need support in, let's say, SAS to rewrite URLs and a, let's say, EJS HTML preprocessor to generate URLs. So we need custom URL support, URL rewriting support in a bunch of places. That makes it hard. So I'm definitely willing to pair on that, but it's hard. It's surprisingly hard. Thanks for the quick discussion. I appreciate that. Everyone's patient. That sounds like a very good interim solution while we come up with a good solution. Toran, we should chat afterwards. Yeah, maybe we can turn that into a broccoli plugin and just to have it as a hack in the meantime. Yeah. One more question, maybe. Yes. Mostly I just was waiting so we'd have to run across the room again. I'm curious. I think it's great that we're building this, that we're going to integrate it with a templating for Ember itself. Is there any plans to integrate it with something that we could write custom templates like Yeoman or something? Sorry, you're talking about generating files? I was scaffolding. Okay. I'm jumping. I think it would be hard to hear on the thing. No, I'm sorry. It's probably horrible with the echo up there. So is there any plan to make a broccoli plugin for Yeoman? That's the simple question, I guess. I don't know. So scaffolding in general obviously shouldn't live in broccoli. I'm not sure how we would integrate with existing scaffolding tools. My impression is that perhaps a scaffolding tool would generate a Brock file as one of the things that it generates so that you have a starting point. All right. Thank you, everyone. Thank you.
 I'm really excited to start using Broccoli. As soon as I can figure out how to spell Broccoli  with my MPM install, I will be sure to check it out. Go ahead, Joe.  Thank you, Tom. Thank you. Is the mic on? Yes. Awesome. So my talk is going to be about  Broccoli. I'm Joe. I wrote Broccoli. And before I start off sort of going into the technical  details, I want to give you a bit of background about what motivated Broccoli and how it fits  into the larger ecosystem. So I started thinking about this at last year's  Amber conference. And what was bothering me at the time was that we had no way to share  code in the front-end browser JavaScript ecosystem. We could share code with Ruby or with MPM  for node-based JavaScript. But for front-end JavaScript, all we were doing was like, here's  a big JavaScript file. Please drop it into your vendor directory. And that just didn't  seem really great. And as I started talking to people about this, I realized pretty quickly  that the missing piece was actually not package management. We do have a package manager that  is pretty good. And I think it's architecturally doing the right thing. And that's Bower. So  what Bower will do is when you have an app with dependencies, it's going to load all  your dependencies from the Internet and load their dependencies recursively. And then it's  going to drop all of those into a big directory. And so it just dumps them into the file system  and it's up to you to do something with those files. Now, with something like Ruby, it's  pretty easy to, say, require some file. And then Ruby is just going to search through  all those directories and things work. But for front-end JavaScript applications, we  have the problem that we actually need a tool on top of this that ties all those files together  and concatenates them and bundles them into one tidy app that we can deliver into the  browser. And so I realized that this is the missing, this is the thing that is missing  for us to have a great front-end package ecosystem. And the way we've been doing this is grant  watch. We have the Rails asset pipeline, and it's not that bad actually, but of course  it's tied to Rails. And so if you're building an app with JavaScript, the most common thing  seems to be to make a grant file and you put a lot of, you define a lot of tasks that compile  coffee script and handle bar templates and then you concatenate everything and then you  minify and then you also have some sets and probably some more things. And it becomes  complex very quickly, right? If you've seen grant files of real world amber apps, they  generally look scary because you have to manage all these compiled files. So that is one problem.  And the other problem is let's say you have a grant-based app and you work on it with  a small team for a couple months. And as time passes, the build gets slower and slower.  And so after a while, after a couple months, you edit a file in the editor and you reload  the browser and it takes 10 seconds to reload because the rebuild is so slow. And that is  clearly not acceptable, right? This is the thing that you're doing 100 times a day. This  is your core feedback loop as a developer. If you have a delay there, it's going to  make you massively more unproductive. So the goal of Broccoli is to have an architecture  that allows us to not have 10 seconds rebuilds but be faster. So this talk has three parts.  The first part is I'm going to show you how to write a Broccoli build definition in practice.  The second part is I want to sketch out the Broccoli plugin API in very rough terms. And  the third part is I want to talk a bit about how Broccoli integrates with other tools.  And I'm not going to talk about the motivation behind some of the design choices in Broccoli.  I wrote a blog post that I published about a month ago and I'm going to link it at the  end and that has all the rationales for the design choices in excruciating detail. So  let's talk about build definitions. By the way, there are some changes that I have planned  for the Broccoli build definition syntax. They are purely cosmetic, but what I'm going  to be showing you is the new syntax that is coming up soon in the next one or two weeks.  If you want to see the old syntax, there is a Broccoli sample app on GitHub that has the  current syntax, but I figured I'm going to show you the new stuff that is coming up.  All right, let me switch into the editor. So this is a Brocfile.js build definition  file. I'm going to start with a simple case where you don't do anything. We have a string  here, lib, that refers to the lib directory. We're just going to export that string. When  you run Broccoli, it is going to serve all the files in the lib directory without doing  anything to them. If you run Broccoli build, it is going to copy those files into a new  directory. So there are a lot of this JavaScript files. The first thing we want to do is concatenate  those files into a single JavaScript bundle. There are many ways to do this. I'm going  to show you the simplest way, which is just to use the Broccoli concat plugin. I'm omitting  the require calls, but you have a require header at the top right where you pull all  this stuff from NPM. So we have a concat plugin that takes that lib tree and you give it a  bunch of input files using a glob pattern and you specify an output file. And the concat  plugin is going to return a new tree object. I'm going to call it app.js here. And that  tree object, it represents a tree of files, but it really contains just one directory,  assets. And that directory contains one file, app.js. And we're going to export that app.js  tree. And so if you now run Broccoli serve, it is going to serve one single file at the  URL slash assets slash app.js. And that's your concatenated JavaScript output. Let's  add coffee script support. So what we would like to do is take all the files in the lib  directory, all the JavaScript files, they stay in place, but all the.coffee files  are going to be replaced with compiled.js files. So we're going to use the Broccoli  coffee plugin and call it on the lib tree. And it's going to return a new tree object  that represents all the files in the lib directory except with.coffee files replaced with.js  files. So then we have a new tree and we pass that new tree into this concatenator. So as  you can see, we don't have to manage any kind of temporary directories to hold our build  output. We can just chain things together. And the next thing we're going to do is minify.  So we run this app.js tree through the Aglify JavaScript filter, through the Broccoli Aglify  plugin. And we're only going to do that in production. Broccoli core doesn't have a notion  of a build environment, but what we tend to do is just use an environment variable like  this. So it's up to you in your Broccoli file, in your Broccoli file to have a notion of  a build environment. So in production, you're going to filter this app.js tree through the  Aglify filter. Let's add some sass next. So we take the styles directory and we pass  it into the Broccoli sass plugin and we tell it to take the app.scss file. And that file  is going to have import statements that import other files, which in turn import other files.  And so sass is going to generate this huge blob of CSS. And that is going to go into  the slash s at slash app.css file. And we store that in a new tree. Now, in order to  export this, at the moment we're just exporting the JavaScript. So in order to export this,  what we're going to do is call the Broccoli merge trees plugin and merge this app.js and  app.css trees into one combined tree. Just copy them on top of each other. So now when  we run Broccoli serve, it is serving two files at different URLs, one app.js and one app.css  file. And finally, we're going to add an index.html file just to boot up our application. And  we're going to place that into the public directory. So we would like to also merge  the public directory into the output. So now all the files in public are going to show  up in the Broccoli output as well without any changes. And that is a fairly simple example  of a Brock file. So I hope you're equipped, if you want to start writing your own Broccoli  build definitions to do this now to have an idea of what goes where. Thank you. Thank you.  We have a part two where I talk about plugins, but my screen isn't playing along. Yes. Okay.  So Broccoli's core plugin API is very, very simple. It only knows one type of object and  that is a tree, right? So at first you would sort of intuit that a build tool should have  a notion of a plugin that takes a tree and returns a new tree, but it turned out to be  unnecessary and so the only primitive that Broccoli knows is a tree. A tree is represented  by a string pointing to a directory or an object. And that object is going to have two  functions and this is the kind of object that a plugin will generally return. And all the  important work happens in the read function. The dot read, the read function on the tree  just returns a directory and that directory contains the tree of files. Now every time  we rebuild we are going to call the, Broccoli is going to call the dot read method on the  tree. So it's going to call dot read repeatedly as files change, as it rebuilds it's going  to call dot read. And then at the end when you shut down the server it's going to call  the cleanup method to give the plugin a chance to delete any temporary directories. So as  you can see the API is tiny. It doesn't really tell any story about how you should build  stuff. But this is something that we kind of learned from the, from Grunt. The people  working on Grunt they really impressed upon me that having a lot of API surface for a  build tool is really painful because what happens is you have this, this ecosystem of  plugins and you can never change the API without breaking everything basically. So everything  you have in the API is going to have to stay there forever. So I tried really hard to keep  the Broccoli API minimal and move all the functionality that could be moved out into  external helper packages. And so plugins are generally going to use one of these helper  packages as base classes to implement their functionality. One such package is the Broccoli  Transform package. It is for the case where you have one input tree going into a plugin  and one output tree coming out of the plugin. So now when you want to implement a plugin  using the transform base class, you don't have to do the read thing and cleanup thing  and you don't have to deal with temporary directories. You just have to implement this  one function, sourcedir and destdir. And you get to read files from the sourcedir and then  write any output files into the destination directory. And then on top of Broccoli Transform  we have another package called Broccoli Filter and that is actually a really important package  to implement plugins because it implements the case, the very common case where you have  one-to-one relationships between input and output files. So Broccoli Coffee is an example  of that. Every file either gets passed through, unmodified, or the.coffee files get compiled  and get turned into.js files. And when you implement a filter plugin, this is the only  thing that you have to implement, a process string method that takes a string, so for  example the coffee script file contents, and that returns a new string, the compiled output.  And the only thing you have to tell it is that it takes.coffee input files and that  it will emit.js output files. And then the Broccoli Filter base class is going to handle  all the file handling for you. And importantly, Broccoli Filter has a cache. The reason why  that is important is that every time you rebuild the plugin is going to be called and it has  to rerun everything. And so if we didn't cache, we would be recompiling all our coffee script  files every time and it would be incredibly slow. So Broccoli Filter is what makes it  fast. When Broccoli rebuilds, the filter base class is going to check if a file's last modified  time has changed. And if it hasn't changed, it is just going to reuse the output from  last time. So that's really important to make it fast. Unfortunately, Broccoli Filter is  limited to one-to-one cases because that is the only way you can really reliably cache  files. So it doesn't work for SAS. SAS, for example, a SAS file has import statements,  so it can depend on other files and those files can depend on other files. So we need  to rebuild the SAS file whenever one of those dependent files change as well. So implementing  caching for more complex compilers like SAS is a bit non-trivial. There are several ways.  One way is to move it into the plugin and I think that is probably the fundamentally  the best way. I did this with a compiler that I wrote to concatenate ES6 module JavaScript  files and it just brings its own cache. Another way is to just ignore the problem. If the  compiler is extremely fast, then we can just rerun it every time. So that's what I actually  did for SAS. The Broccoli SAS plugin is based on the Lib SAS library which is implemented  in C. It is blazingly fast. So for now, for Broccoli SAS, I'm just ignoring the problem.  And then for the Rails, then there's the third way and that's what the Rails Asset Pipeline  does. You kind of parse out, you use a regex on the SAS file to figure out what it depends  on and then you recursively build this dependency tree to figure out when you need to rebuild  the SAS file. But clearly that's kind of a hack. We may have to do it in some cases.  But I think we want to avoid it. So this is not, I don't think we have really converged  on a great way to do this yet in the compiler ecosystem yet and I think we're going to have  to talk to the authors of compilers like SAS and LES and what have you as well to really  understand where we should be putting this caching and how we can better, how the compilers  can help us implement this more easily. Now, part three, I want to talk about integration.  Broccoli is a small library and I don't think of it as a tool as much as I think of it as  a library. It does have a command line interface so you can type Broccoli serve and Broccoli  build into the command line. But it's really more of a proof of concept and in reality  you're going to want to proxy stuff to the API and you're going to want to handle URLs  in certain ways. So there are all kinds of tricky things that you want to do in practice  that are definitely not going to be covered by the command line tool. And so I think we're  going to have to work a bit more on getting Broccoli integrated into other tools and having  Broccoli provide a good API that is still kind of missing. Broccoli loves Grunt. There  is actually a Grunt Broccoli plugin for Grunt. Grunt is a good task runner. You should be  using it for stuff like deploying or generating stuff or running the test suite. Those are  all things that should be living in your Grunt file. And just the core build loop is what  Broccoli should be solving. So it's not that Broccoli is trying to replace Grunt, it's  just that we shouldn't be using Grunt to build our applications. Grunt is a task runner,  not a build tool. So the Grunt Broccoli plugin for Grunt is very simple at the moment and  I hope that we can provide more hooks into the rebuilt lifecycle to make it better in  the future. And I'm hoping that we also can integrate with backends. And the reason is  that when you have a backend like Rails plus the asset pipeline, there's a lot of smart  things that you can do by default because you can make a lot of assumptions about how  the URLs work. So that would make it easier to, that will make it easier to come up with  the default stack because we can make more assumptions about what things go where. In  particular, speaking of default stacks, Steph has been working, doing a lot of work and  other people as well, I believe, on the Ember CLI tool which is going to be, which is Broccoli  based and which is going to provide a default tool stack for new Ember applications so that  you can type Ember new to do MVC and you can make a new to-do list. So I'm going to be  here for another two weeks in Portland and I would be very happy to pair with people  on integrating Broccoli into other tools and maybe coming up with solving some of the problems  that we've been having with implementing plugins the right way. So I would be happy to, please  get in touch. This is my Twitter and my email address and as I promised earlier, the lengthy  blog post about Broccoli's architecture is there if you want to read up. So I think we  have a couple minutes for Q&A. Who has the mic? Rico? Excellent. Who has a question?  Yes, over there. Hi, I'm Matt Newton. I work at Apple and we have a problem right now,  well it's not a problem, we solved it with Grunt. We're building multiple Ember apps  with a shared code base and I wonder if that's something that, does Broccoli have a user  story for this? So for sharing, what you're talking about is basically sharing libraries  and components between Ember apps. So the story for that is you would probably use something  like Bower to pull in libraries into your app and there is a Broccoli Bower plugin that  automatically picks up all your Bower directories. So yes, if you want to see an example of that,  there is a Broccoli sample app on GitHub that uses Bower libraries.  Do you happen to know, I don't know if anyone else knows, I haven't used Bower extensively,  but can you use private Bower modules like your own registry, sort of like NPM? I don't  know. And it pulls from Git or something like that?  Any more questions? In the back, black T-shirt.  Hi, I'm Dave Duchain from Absolute Software. No booze please, but I have to ask if, what  the state is with regards to Windows support?  Sorry, can you repeat that?  I know, I know. Oh, I'm sorry. How well does it work on Windows? Broccoli, that is.  Sorry, I couldn't hear the question. How well does Broccoli work on Windows?  On Windows. At the moment, there is a problem with Node, Windows file I.O. where it has  weird race conditions. So if you write a lot of files or delete a lot of files very quickly,  and Node is going to throw randomly weird errors. So the Windows support is basically  blocking on that as far as I know. So someone is going to have to fix Node.  One question? Yes. Hey, my name is Joe Fiorini and I really like  the new API for the Brock file. The one question I had is, would you think about letting arrays,  be representation of trees like strings are? So instead of having to do merge trees in  your Brock file, you can just export an array and not have to worry about merging them.  So that's what, that's the original syntax where we just export an array, but it doesn't  have the overriding semantics well defined. Like it has to either override or not override.  So I moved it into an external plugin and then you can configure whether you want to  override or not. Yes.  Jake Bixby. I have an idea of a larger application where you want to maybe break it up into different  modules. Maybe using trees to have separate build targets. So like you basically have  your core app and then different routes break those into different build targets and then  somehow with like requires or some sort of way to pull those in. But that's different.  But as far as like the using broccoli for different build targets, is that fairly easy  to do, do you think? Yes. You could generate multiple trees and then there is a plugin  to move files into subdirectories. So you could have one tree in one sub directory,  another tree in another sub directory. And then at the end you just merge all of these  trees together to output five different subdirectories. So that is definitely a use case. Yeah.  In the back.  What about fingerprinting? Sorry, can you say it again?  What about JavaScript and CSS and asset file fingerprinting for cache busting?  Fingerprinting. I don't want to implement it on the CLI. So one of us is going to have  to figure something out. Maybe if I was staying longer in Portland we could just pair on it  and do it together. You'll have to convince Luke to let me stay a little bit longer.  I have completely punted on writing a fingerprinting plugin. And the reason is fingerprinting files  was very easy of course. You just renamed them. But the hard part is we write in URLs.  And so it seems that we would want to have, let's say, script references in HTML file.  And URL references in CSS files pointing to images. You want to rewrite all of these.  And there is no general way I think to do URL rewriting reliably. So this really needs  to live in each preprocessor. So that makes it a lot harder to do it right.  I agree by the way. Since we're having a discussion about this.  Do you foresee, this is what we've been talking about at this table. Do you foresee just a  separate tool that's specialized for fingerprinting that lives as a sibling to broccoli? Or do  you see it as something that's more integrated with broccoli?  I think it would be integrated. So again, it's probably not a single plugin, but we  need support in, let's say, SAS to rewrite URLs and a, let's say, EJS HTML preprocessor  to generate URLs. So we need custom URL support, URL rewriting support in a bunch of places.  That makes it hard. So I'm definitely willing to pair on that, but it's hard. It's surprisingly  hard.  Thanks for the quick discussion. I appreciate that. Everyone's patient.  That sounds like a very good interim solution while we come up with a good solution. Toran,  we should chat afterwards.  Yeah, maybe we can turn that into a broccoli plugin and just to have it as a hack in the  meantime. Yeah.  One more question, maybe.  Yes.  Mostly I just was waiting so we'd have to run across the room again. I'm curious. I think  it's great that we're building this, that we're going to integrate it with a templating  for Ember itself. Is there any plans to integrate it with something that we could write custom  templates like Yeoman or something?  Sorry, you're talking about generating files?  I was scaffolding. Okay.  I'm jumping.  I think it would be hard to hear on the thing.  No, I'm sorry. It's probably horrible with the echo up there. So is there any plan  to make a broccoli plugin for Yeoman? That's the simple question, I guess.  I don't know. So scaffolding in general obviously shouldn't live in broccoli. I'm not sure how  we would integrate with existing scaffolding tools. My impression is that perhaps a scaffolding  tool would generate a Brock file as one of the things that it generates so that you have  a starting point.  All right. Thank you, everyone.  Thank you.
Modern_build_workflows_with_Broccoli_0_275.png Modern_build_workflows_with_Broccoli_276_383.png Modern_build_workflows_with_Broccoli_384_1199.png Modern_build_workflows_with_Broccoli_1200_1283.png Modern_build_workflows_with_Broccoli_1284_2759.png Modern_build_workflows_with_Broccoli_2760_2807.png Modern_build_workflows_with_Broccoli_2808_2879.png Modern_build_workflows_with_Broccoli_2880_4691.png Modern_build_workflows_with_Broccoli_4692_5291.png Modern_build_workflows_with_Broccoli_5292_6119.png Modern_build_workflows_with_Broccoli_6120_6263.png Modern_build_workflows_with_Broccoli_6264_6767.png Modern_build_workflows_with_Broccoli_6768_7403.png Modern_build_workflows_with_Broccoli_7404_7499.png Modern_build_workflows_with_Broccoli_7500_8807.png Modern_build_workflows_with_Broccoli_8808_8891.png Modern_build_workflows_with_Broccoli_8892_9563.png Modern_build_workflows_with_Broccoli_9564_9815.png Modern_build_workflows_with_Broccoli_9816_10883.png Modern_build_workflows_with_Broccoli_10884_15539.png Modern_build_workflows_with_Broccoli_15540_15647.png Modern_build_workflows_with_Broccoli_15648_15911.png Modern_build_workflows_with_Broccoli_15912_17687.png Modern_build_workflows_with_Broccoli_17688_18419.png Modern_build_workflows_with_Broccoli_18420_19115.png Modern_build_workflows_with_Broccoli_19116_19523.png Modern_build_workflows_with_Broccoli_19524_19919.png Modern_build_workflows_with_Broccoli_19920_20363.png Modern_build_workflows_with_Broccoli_20364_21659.png Modern_build_workflows_with_Broccoli_21660_22271.png Modern_build_workflows_with_Broccoli_22272_22751.png Modern_build_workflows_with_Broccoli_22752_23483.png Modern_build_workflows_with_Broccoli_23484_24467.png Modern_build_workflows_with_Broccoli_24468_25919.png Modern_build_workflows_with_Broccoli_25920_29435.png Modern_build_workflows_with_Broccoli_29436_29567.png Modern_build_workflows_with_Broccoli_29568_30803.png Modern_build_workflows_with_Broccoli_30804_30911.png Modern_build_workflows_with_Broccoli_30912_32183.png Modern_build_workflows_with_Broccoli_32184_33779.png Modern_build_workflows_with_Broccoli_33780_34307.png Modern_build_workflows_with_Broccoli_34308_48851.png Modern_build_workflows_with_Broccoli_48852_48995.png Modern_build_workflows_with_Broccoli_48996_49115.png Modern_build_workflows_with_Broccoli_49116_49175.png Modern_build_workflows_with_Broccoli_49176_49223.png Modern_build_workflows_with_Broccoli_49224_49439.png Modern_build_workflows_with_Broccoli_49440_49637.png