CFA - Config Files API
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Reidinger, Josef
YaST has been trying to find a solution to work with configuration files in a way that is easy and reusable, while ensuring the consistency of the resulting configuration. The response is Config Files API (CFA), a generic framework to work with configuration files in Ruby. Although currently is only used in the yast2-bootloader module, CFA will become one of the key components of YaST in the near future. Its design and foundation look beyond YaST, making it a useful resource in any environment needing programatic and semantic management of configuration files. The talk will provide an overall overview of CFA's architecture and down to earth examples on how CFA can be used and extended.
 Great. So welcome to presentation about CFA, which means config files API. It's an API that mainly used for some editing of configuration file. By editing, I mean fine small changes in this file that do not break existing stuff. So it's not something like a sound or CF engine or such stuff that owns whole file and modifies just deploy new version. It's API for the small changes like you expect, for example, from YAST, that it doesn't break your existing stuff, doesn't break your own comments and such stuff. So what's the content? I will explain why we create a new API, why old one is not enough. Then I explain design of this API, how it works together. And then I show some real time or real life examples of usage because on Tumbleweed it's already used. So I will show some code that you use in Tumbleweed. So why? It's obvious reason to many face palms when you use or see old code. This usually begins motivation. If the old code works somehow and you can somehow use it, then you do it. But if it starts to be very annoying, then you have motivation to do it better. So currently I will mostly talk about YAST because YAST even now still use some old API for this changes. It's called SCR. I even don't know what it actually means. It's very old acronym. And what's currently problem? It's not much reusable. It can be used only in YAST and even if some projects have some motivation to reuse it, in the end they just give up because it's too tight with the rest of YAST. It requires a lot of libraries from YAST. Also last point is the reason that it is used for communication YCP which is now very legacy and almost that language. It's also monolithic. So if you want to use just part of it, like its ability for parsing files and actually you don't want to read it or write it from current system, then you have a problem. Also it's completely written by YAST team and maintained by them. So every piece is done in-house, which is always not so good. We would like to share our workflow and also share maintenance of some parts with other interested teams. Yeah, its API is very strange and it's related that its API is confusing. And we see it again and again in YAST when someone new comes to the team. The SCR is the most scary part of YAST. It's hard to explain it for newcomers. It's hard to use it for them. And it's quite hard to also explain how it works because it has a lot of abstractions that are not needed. That makes sense 15 years ago when it started this project, but now no longer makes sense. And it's hard to explain something that doesn't make sense to anyone. So what's a requirement for a new API that we hope will replace SCR and will be easier to use? So of course the first stuff is easy to use. If you have something that's hard to use, no one will use it or try to find some shortcuts how to avoid complex stuff. It has to be modular that if in future something useful appears in open source community, we would like to replace for example some stuff that doesn't work well with something that's already existing or will exist in future and works better. So we would like to have replaceable parts. And also we would like to have our own parts that can be used at other places. So some modules written for CFA, we would like to reuse elsewhere also. Yeah. Of course requirements now is easy to test because what's not tested is usually broken. And of course it doesn't mean that if you have 100% test coverage that it's not broken but there's a very small chance it really decrease the number of bugs. Of course it should be object oriented because SCR is from YCP times so it's functional based but we would like to use object because currently it's in Ruby so we would like object API. And of course the most important part is friendly to newcomers. So if someone would like to start with YAST or would like to contribute to YAST then don't have a strange part in its code which is hard to understand. So now let's look on design of CFA. It basically has three components, the basic one is read a writer, it has a single responsibility, translate string to some target storage or string or whatever it is, just write string or read that string from target. Then you have parser, so this string is then somehow interpret to some tree or vice versa, you have modified three and want to translate it back to string that can be read or write. And then there is a model which is something like high level API for given configuration file, usually model have some actions for it, some operations and that operations work on a parsed tree. And that small line between a reader and model is dependence injection. So what basically user use is they have model and then can pass different reader and writer. So if you have, for example, file that's not on your system but somewhere over network you can write your own reader writer for that network file and pass it to model. So in the end the stuff was read and write from this reader. So now let me look closer to this. As I said, the writer is very simple interface. It allows read and write just two methods. Both methods get path as its argument. The path is relative to root. So if you want to read your, for example, grab to default file, you just pass I want read file that's path is ETC default grab. It works only with plain strings. So no formatting, no stripping of this string or any other operation. It's really just get me string. And examples we already have is common file in Ruby, which means read from this path, write to this path. Then we have memory file, which is very useful for testing. That string is hold only in memory. And you construct it in code. How a file should look like and then work on it and then compare if memory looks after some modification as it should look like. And then in YAST we create also our own specific that's not part of CFA, a target file that's used during installation that they recognize if you need to write to target system. That's where we install stuff. Of course, if someone else needs his own, the writer, for example, reading, writing over SSH, then just create your own class that implements these two methods. And it's enough. The second part is parser. It translates between string and parse. Each tree is parser specific. I'm also considering create some generic parse tree, but in the end I found it's too much abstraction and it can lose some features of given parser. So currently it's specific parse tree. It's quite low level tree. For example, in AUGAS it holds all commands. It doesn't know any relations. It just is positions and can recognize some basic syntax. And as I said, examples is AUGAS, which we use mainly for parsing and serialization. And another possible one is line parser that is for simple config files that have each line one option. So it's very, very simple parser. And of course, you can use much more. It's just currently used ones. And as you can see, the hardest part, which is parsing, we currently delegate to AUGAS that already have its own lenses that specify the syntax of trees. And there's many available lenses. So we don't spend time writing open parser for new files. Yeah, and model is high level API. It allows higher operations like enable something, some operation, or remove something. It's coupled with parser. So it knows it's parser and it's usually depend on it. So there is no option to switch to different parser for given model. But you can pass your own reader, as I mentioned before. Also models currently allows to switch globally reader. So if you have more models, you can in one place switch that all models use different reader, like a network one or installation one. Also models ensure for consistency. So if there is a config file that have, for example, two colliding options, then the model should ensure that it's properly used. The model is something for target user, so as I mentioned, it should be high level and easy to use for them. And examples are some graph models, which I will show in a few minutes. So now, more interesting part is examples. So let me show. Open it. So currently this is a plugin for CFA for Grap2. So the idea is that each software have its own plugin that handles all its files. Currently for Grap2 we have four configuration files, the views. And for example, the easy one is device map, which is some, it's mapping between kernel devices and Grap2 devices. And how it looks like, it's a model. It has its own parser and it use August. So you just write I want August and use this specific lens for it. It has its own path. So it's also constant where this file is. And as you can see, for example, we ensure consistency. Grap2 or basically, yeah, font. So let me enlarge the font. Okay, I hope how we can enlarge this one. Okay. So as you can see in SAFE, we ensure that we use at maximum eight devices because it's some limitation of Grap. It doesn't allow more. It's caused by some hardware limitations. So there is some consistency check. There is some helpers like get me Grap device for given kernel device. Get me system device for given Grap device. So it's really quite high level API that hides some details like some ordering and such stuff. It allows editing, removing, and also allows you to give some Grap devices, which as you can see filter out comments. So you just get the real stuff. And this is a simple model. For a more complex one, it's even more significant how it can make work easier. It's default one. It's usually the most interesting config file for Grap2. So there is a set of simple attributes, which basically mean that it creates a Ruby Assessor called default for key Grap default. And that's a common string once. So it's just defined. There is such options and you can read it. You can write it. And then comes a more tricky ones like kernel parameters. And kernel parameters have some data, but it looks like string kernel parameters. But in fact, it's more complex because it's a command line stuff, which is its own internal logic. And for example, you are interested if your default kernel get given parameter. So the model define its own parameters and then you can quote, like it's in documentation, there is parameter quiet. And if you get through, it means there is such parameter. If you get through, it means there is no such parameter. If you get straight, it means there is a single instance of this parameter with given value. And to be even more interesting, kernel allows more specify a parameter multiple time. So if there is, for example, if you use two serial consoles, then you can ask for it and you will get it. So this really makes working with configuration file much, much easier for target user. Because you have some helpers that ensure everything is properly voted, properly read it, placed properly. Also another example is if you have configuration option that have just few possible values. Of course, you can let user to add anything they want, but there is a risk that the graph doesn't recognize such option. So good model should mention what's possible values. And then you can, if you set it, then again validate that it used the proper one. Okay, and now let's check how it actually works. So now I show you a YAST code that use this device map. And also shows that there is, of course, much more logic above CFA. Because for example, in YAST, the idea is that we use UDF devices. And we want to use it in device map. But CFA itself doesn't know anything about UDF. So it's again layer above CFA. So if you want to check if device map contains given disk, you need to check that it contain given disk translated to UDF device. But if you do some proposing, just do some filling ordering, you can just ask, give me all this with HD prefix sorted by BIOS order inside stuff. And now let me show a bit about CFA itself. This model has one interesting feature I would like to show you is how it set value. Which is, sorry. Yeah. So how it set value is that basically it tries to modify if value already exists. And if in configuration file this value is not yet defined, then it try to uncomment it. Because usually what happens in default configuration files, it, they do have some comments. And below is, if you uncommented this line, it happened this. So we basically replace this commented out option and use it. And only as the last option is that we add this new line to the end of a file. So it try to behave quite smartly when it try to modify something. Yeah. And here is example invitation of memory file. As you see, it's very simple. Simple stuff. Just read write and hold it into internal memory as content. Okay. So this is example how it looks like. And do you have any questions regarding this API or its possible usage? Okay. It looks like not. So thanks for your attention and enjoy this evening.
 Great.  So welcome to presentation about CFA, which means config files API.  It's an API that mainly used for some editing of configuration file.  By editing, I mean fine small changes in this file that do not break existing stuff.  So it's not something like a sound or CF engine or such stuff that owns whole file and modifies  just deploy new version.  It's API for the small changes like you expect, for example, from YAST, that it doesn't break  your existing stuff, doesn't break your own comments and such stuff.  So what's the content?  I will explain why we create a new API, why old one is not enough.  Then I explain design of this API, how it works together.  And then I show some real time or real life examples of usage because on Tumbleweed it's  already used.  So I will show some code that you use in Tumbleweed.  So why?  It's obvious reason to many face palms when you use or see old code.  This usually begins motivation.  If the old code works somehow and you can somehow use it, then you do it.  But if it starts to be very annoying, then you have motivation to do it better.  So currently I will mostly talk about YAST because YAST even now still use some old  API for this changes.  It's called SCR.  I even don't know what it actually means.  It's very old acronym.  And what's currently problem?  It's not much reusable.  It can be used only in YAST and even if some projects have some motivation to reuse it,  in the end they just give up because it's too tight with the rest of YAST.  It requires a lot of libraries from YAST.  Also last point is the reason that it is used for communication YCP which is now very legacy  and almost that language.  It's also monolithic.  So if you want to use just part of it, like its ability for parsing files and actually  you don't want to read it or write it from current system, then you have a problem.  Also it's completely written by YAST team and maintained by them.  So every piece is done in-house, which is always not so good.  We would like to share our workflow and also share maintenance of some parts with other  interested teams.  Yeah, its API is very strange and it's related that its API is confusing.  And we see it again and again in YAST when someone new comes to the team.  The SCR is the most scary part of YAST.  It's hard to explain it for newcomers.  It's hard to use it for them.  And it's quite hard to also explain how it works because it has a lot of abstractions  that are not needed.  That makes sense 15 years ago when it started this project, but now no longer makes sense.  And it's hard to explain something that doesn't make sense to anyone.  So what's a requirement for a new API that we hope will replace SCR and will be easier  to use?  So of course the first stuff is easy to use.  If you have something that's hard to use, no one will use it or try to find some shortcuts  how to avoid complex stuff.  It has to be modular that if in future something useful appears in open source community, we  would like to replace for example some stuff that doesn't work well with something that's  already existing or will exist in future and works better.  So we would like to have replaceable parts.  And also we would like to have our own parts that can be used at other places.  So some modules written for CFA, we would like to reuse elsewhere also.  Yeah.  Of course requirements now is easy to test because what's not tested is usually broken.  And of course it doesn't mean that if you have 100% test coverage that it's not broken  but there's a very small chance it really decrease the number of bugs.  Of course it should be object oriented because SCR is from YCP times so it's functional based  but we would like to use object because currently it's in Ruby so we would like object API.  And of course the most important part is friendly to newcomers.  So if someone would like to start with YAST or would like to contribute to YAST then don't  have a strange part in its code which is hard to understand.  So now let's look on design of CFA.  It basically has three components, the basic one is read a writer, it has a single responsibility,  translate string to some target storage or string or whatever it is, just write string  or read that string from target.  Then you have parser, so this string is then somehow interpret to some tree or vice versa,  you have modified three and want to translate it back to string that can be read or write.  And then there is a model which is something like high level API for given configuration  file, usually model have some actions for it, some operations and that operations work  on a parsed tree.  And that small line between a reader and model is dependence injection.  So what basically user use is they have model and then can pass different reader and writer.  So if you have, for example, file that's not on your system but somewhere over network  you can write your own reader writer for that network file and pass it to model.  So in the end the stuff was read and write from this reader.  So now let me look closer to this.  As I said, the writer is very simple interface.  It allows read and write just two methods.  Both methods get path as its argument.  The path is relative to root.  So if you want to read your, for example, grab to default file, you just pass I want  read file that's path is ETC default grab.  It works only with plain strings.  So no formatting, no stripping of this string or any other operation.  It's really just get me string.  And examples we already have is common file in Ruby, which means read from this path,  write to this path.  Then we have memory file, which is very useful for testing.  That string is hold only in memory.  And you construct it in code.  How a file should look like and then work on it and then compare if memory looks after  some modification as it should look like.  And then in YAST we create also our own specific that's not part of CFA, a target file that's  used during installation that they recognize if you need to write to target system.  That's where we install stuff.  Of course, if someone else needs his own, the writer, for example, reading, writing  over SSH, then just create your own class that implements these two methods.  And it's enough.  The second part is parser.  It translates between string and parse.  Each tree is parser specific.  I'm also considering create some generic parse tree, but in the end I found it's too much  abstraction and it can lose some features of given parser.  So currently it's specific parse tree.  It's quite low level tree.  For example, in AUGAS it holds all commands.  It doesn't know any relations.  It just is positions and can recognize some basic syntax.  And as I said, examples is AUGAS, which we use mainly for parsing and serialization.  And another possible one is line parser that is for simple config files that have each  line one option.  So it's very, very simple parser.  And of course, you can use much more.  It's just currently used ones.  And as you can see, the hardest part, which is parsing, we currently delegate to AUGAS  that already have its own lenses that specify the syntax of trees.  And there's many available lenses.  So we don't spend time writing open parser for new files.  Yeah, and model is high level API.  It allows higher operations like enable something, some operation, or remove something.  It's coupled with parser.  So it knows it's parser and it's usually depend on it.  So there is no option to switch to different parser for given model.  But you can pass your own reader, as I mentioned before.  Also models currently allows to switch globally reader.  So if you have more models, you can in one place switch that all models use different  reader, like a network one or installation one.  Also models ensure for consistency.  So if there is a config file that have, for example, two colliding options, then the model  should ensure that it's properly used.  The model is something for target user, so as I mentioned, it should be high level and  easy to use for them.  And examples are some graph models, which I will show in a few minutes.  So now, more interesting part is examples.  So let me show.  Open it.  So currently this is a plugin for CFA for Grap2.  So the idea is that each software have its own plugin that handles all its files.  Currently for Grap2 we have four configuration files, the views.  And for example, the easy one is device map, which is some, it's mapping between kernel  devices and Grap2 devices.  And how it looks like, it's a model.  It has its own parser and it use August.  So you just write I want August and use this specific lens for it.  It has its own path.  So it's also constant where this file is.  And as you can see, for example, we ensure consistency.  Grap2 or basically, yeah, font.  So let me enlarge the font.  Okay, I hope how we can enlarge this one.  Okay.  So as you can see in SAFE, we ensure that we use at maximum eight devices because it's  some limitation of Grap.  It doesn't allow more.  It's caused by some hardware limitations.  So there is some consistency check.  There is some helpers like get me Grap device for given kernel device.  Get me system device for given Grap device.  So it's really quite high level API that hides some details like some ordering and such stuff.  It allows editing, removing, and also allows you to give some Grap devices, which as you  can see filter out comments.  So you just get the real stuff.  And this is a simple model.  For a more complex one, it's even more significant how it can make work easier.  It's default one.  It's usually the most interesting config file for Grap2.  So there is a set of simple attributes, which basically mean that it creates a Ruby Assessor  called default for key Grap default.  And that's a common string once.  So it's just defined.  There is such options and you can read it.  You can write it.  And then comes a more tricky ones like kernel parameters.  And kernel parameters have some data, but it looks like string kernel parameters.  But in fact, it's more complex because it's a command line stuff, which is its own internal  logic.  And for example, you are interested if your default kernel get given parameter.  So the model define its own parameters and then you can quote, like it's in documentation,  there is parameter quiet.  And if you get through, it means there is such parameter.  If you get through, it means there is no such parameter.  If you get straight, it means there is a single instance of this parameter with given value.  And to be even more interesting, kernel allows more specify a parameter multiple time.  So if there is, for example, if you use two serial consoles, then you can ask for it and  you will get it.  So this really makes working with configuration file much, much easier for target user.  Because you have some helpers that ensure everything is properly voted, properly read  it, placed properly.  Also another example is if you have configuration option that have just few possible values.  Of course, you can let user to add anything they want, but there is a risk that the graph  doesn't recognize such option.  So good model should mention what's possible values.  And then you can, if you set it, then again validate that it used the proper one.  Okay, and now let's check how it actually works.  So now I show you a YAST code that use this device map.  And also shows that there is, of course, much more logic above CFA.  Because for example, in YAST, the idea is that we use UDF devices.  And we want to use it in device map.  But CFA itself doesn't know anything about UDF.  So it's again layer above CFA.  So if you want to check if device map contains given disk, you need to check that it contain  given disk translated to UDF device.  But if you do some proposing, just do some filling ordering, you can just ask, give me  all this with HD prefix sorted by BIOS order inside stuff.  And now let me show a bit about CFA itself.  This model has one interesting feature I would like to show you is how it set value.  Which is, sorry.  Yeah.  So how it set value is that basically it tries to modify if value already exists.  And if in configuration file this value is not yet defined, then it try to uncomment  it.  Because usually what happens in default configuration files, it, they do have some comments.  And below is, if you uncommented this line, it happened this.  So we basically replace this commented out option and use it.  And only as the last option is that we add this new line to the end of a file.  So it try to behave quite smartly when it try to modify something.  Yeah.  And here is example invitation of memory file.  As you see, it's very simple.  Simple stuff.  Just read write and hold it into internal memory as content.  Okay.  So this is example how it looks like.  And do you have any questions regarding this API or its possible usage?  Okay.  It looks like not.  So thanks for your attention and enjoy this evening.
Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_0_191.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_192_347.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_348_23699.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_23700_23867.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_23868_24059.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_24060_24119.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_24120_24443.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_24444_25163.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_25164_25787.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_25788_25871.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_25872_28127.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_28128_28187.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_28188_28343.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_28344_28415.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_28416_28631.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_28632_29195.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_29196_29483.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_29484_29843.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_29844_29891.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_29892_29975.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_29976_30071.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_30072_30251.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_30252_31871.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_31872_32015.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_32016_33107.png Josef_Reidinger__Config_Files_API_O_aJdPsiIfs_33108_33231.png