FreeIPA and cross-distribution packaging experience
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Bokovoy, Alexander
The talk is going to reflect on the effort Fedora development community has done to allow complex solutions like FreeIPA to be tested continuously and to ensure a working solution at any release time. FreeIPA is one of projects that would benefit from a tighter collaboration between distributions and we would also like to discuss how its cross-distribution support could be improved to provide a more consistent behavior to our users across multiple distributions. FreeIPA is an identity management solution for POSIX environments. It is often characterized as an 'Active Directory for Linux systems' which, while not exactly right description, helps to visualize a level of complexity FreeIPA has to deal with. FreeIPA as a solution is built on a number of existing and proven technologies implemented as a free and open source software. As result, a FreeIPA deployment has to deal with coordination between a lot of packages. In Fedora, for example, a fully-functioning FreeIPA deployment requires to install several hundred binary packages, including but not limited to SSSD, MIT Kerberos, 389-ds LDAP server, Samba, Apache, OpenSSH, database libraries, Java components for Dogtag Certificate Authority server, and many Python libraries. It is not a surprise that coordinating these components often leads to complications in an operating system distribution release management. Packages need to be at right versions at right time with knowledge about that not always shared across multiple groups of developers and maintainers. Regressions need to be tracked and fixed. Security updates and cross-platform protocol compatibility issues are adding their own flavor.
 Thank you everyone who came for this talk. I'm going to talk about a bit. Experience that we had that free API project with packaging our stuff for multiple distributions, but primarily for Fedora. For those who are curious, those ships are the ships that come every year to Helsinki harbor for like 300 something years to sell the fish on the fish market once a year in September for a week. And they have this race that was raised from last year basically all in mist. That's why they all graze. Some of these ships, they actually have red sales there, which a bit of interesting part for me because I have actually two red heads at home, but I'm not wearing it all the time. So I work at Red Hat for the identity and security and do a lot of stuff upstream. So I'm working on free APAs. You could have guests from this talk and I also work on Samba for last 15 something years. And I'll contribute to Fedora whenever I can primarily around the areas that we get affected with for the Fedora packaging. But for example, I'm a sort of a life support for ZZ-Leap package, which is kind of dead upstream wise. But it's needed to build documentation for MIT Kerberos. And if we remove it from Fedora, we remove MIT Kerberos from Fedora. So sometimes your path to packaging can be absolutely unpredictable. So if we look at the IPA, it's actually quite a huge thing by itself, but it also a system that packages a lot of or combines a lot of other software together. So I'll start with the framework. It's mostly Python, but it also has a bunch of C plugins and components. So this is kind of code-based statistics that you can see. We have around 300,000 lines of Python code, around 50,000, 60,000 lines of C if you take it, give or take the comments into account. There's some JavaScript stuff, pretty sizable for the web UI. Some of these things, they actually get calculated wrongly, but that's how all these systems work that take the statistics on it. It's a web application that runs under, what WSGA and Apache has tight integration with the Mods.js API, which is the kind of provides you single sign-on through Kerberos. It has a lot of Python code in the installers and it has a lot of plugins that implement certain things. This is basically to make Kerberos and LDAP working on Linux. And when it started in 2008, 2007, the idea was to bring back this idea of nice manageable services on POSIX compatible or Unix platforms like Microsoft did when they basically took the Unix technologies, combined them in a nice package and made Active Directory at 10 years before that. So we're still doing this work and people keep us named as Active Directory for Linux, which is not exactly that. We do much more than Active Directory does, but so this is kind of a lot of comparisons on it. We do a lot of stuff around the MIT Kerberos to provide the single sign-on things and we have many features in MIT Kerberos driven by our needs. So for example, the two-factor authentication, the whole process of implementing it from RFC acceptance to driving to support was driven by the needs of free IP approach. And then there's Kerberos proxy implementation, which allows you to have indirect access to your KDCs so you don't put them on the Internet, even though it was created to be put into this kind of environment in 1980s. Microsoft invented a protocol that allows you to know this over HTTPS, so there wasn't any implementation until we started working on it. Now there are four or five different implementations, including those that are combined with VPN services, so kind of spurred up a lot of effort here and there and got to the nice things. So the other part, pretty important one, is the LDAP server, which serves as a backend to all these services. We use what used to be called Netscape Directory Server, then iPlanet Server, then when Red Hat bought it, it was called what it was, Fedora Directory Server, FDS, right, and then it was renamed to 389DS. 389 is the port for LDAP, so you see, but we have more than a dozen plugins on top of it that implement specifics of the behavioral things. This presentation, it's uploaded and it has links to other presentations, so it's kind of a web of rabbit holes that you might get into if you get bored. The other part is that we handle the certificates and reissues of the certificates automatically so we get a lot of stuff and this is another part of Netscape Heritage that Red Hat obtained. It's the DocTag Certificate Authority, which is upstream for Red Hat Certificates Services product, but also available as a part of free IPA. So we kind of cut down the number of features that we automate for typical users. Our HTS does a bit more, we just do what we want, but we share the same code base. And it's based on Java, so there's a lot of packages, and if you look at the review, the dependency is like 350 plus packages altogether. So we are what one of my colleagues says, essentially we are the rail, Red Hat Enterprise Linux. If anything is broken in rail, we most likely will find it because we depend on a lot of that stuff. And then on the client side is the another component of this ecosystem is SSSD, which is, I think these days, available almost everywhere on all distributions, that has a special provider mode that talks to free IPA or IPA and has support for most of its features. Not all of them, but it's kind of growing. It's also growing pain if you don't have those. So we also integrate with DNS, making it easy to dynamically provision hosts, enable, create the zones and records and zones. And so on we do DNS sec automatically and enable a lot of things. It's a bit fragile. Some of the pieces are a bit harder to maintain things because we touch a lot of inside of bind. Bind doesn't have really a good, pluggable API. So we have to play Frankenstein style operations to get it working. Surprisingly, it works pretty well for most use cases. And this binding the build up is the backend to look up the data in LDAP. So we have interesting case ones where some guys took the binding the build up, compiled it with bind on salaries and use that to load the zones that they had. They had something like 30,000 zones with sizable amount of host per zones to compare how this would work with like a typical flat file-based thing. And apparently this scaled much better the environment. So the difference was like hundreds of times in the performance. Then we integrate with Samba to get the Active Directory things and extensively use Python bindings to provide this integration because we are Python stuff there inside. So from some real world examples GNOME runs its account system. So account.gnome.org actually runs free IPA web interface for all GNOME participants. Fedora project infrastructure has integration within itself with IPA. It doesn't show you this thing externally, but you're required to use Kerberos authentication to some services and that's exactly where we are. And then Red Hat IT runs a modernization project since like 2015, so 2016 they actually started to deploy it. Since last year at FOSDOM there was a report on that one, it's linked from here. On the distributions level so we have more or less full support for both server and client side from Fedora, REL, CentOS and Outlinux. The full server support is a bit interesting thing and I will go into details just in a few minutes. Because IPA is not a single server, you could run it as a single server but you actually can scale horizontally by replicating all this data and you can split them in sites, put in different locations and so on. It becomes quite harder to deploy this if you don't have all the infrastructure in place. So in many distributions we actually lack that infrastructure but in some we do have. So Outlinux and Fedora and REL based it once, they all base it with NSS and OpenSSL, Interim Server but in components that basically provision all this data bases for these crypto libraries so we can do that stuff. In other distributions we are a bit troubled. And on Debian for example, in derivatives the client support mostly there so it includes SSSD and components that need to provision the IPA client but on the server side for long time it was a bit of a bitter taste. On the other distributions it's even more complicated so Outlinux doesn't have any packaging for the server, they had packaging for the client that got broke. OpenSusie and Sles they don't have server support anymore because OpenSusie had packaged free APA like four years ago and then they removed it, whatever was the reason. And the client side SSSD is built without IPA ID provider so you can use it but you cannot use the functionality, extended functionality there. So what this kind of mostly complete means, we work with a lot of upstream projects and drive changes there as we see bits and pieces. So one of these is like SSSD is our sister project so there's pretty good collaboration on what needs to be added. But if you get a distribution that doesn't have good updates for the released version or they have forced it that you cannot really release a new version into the updates, you might end up having pretty old versions that don't support the features that you might expect be there. So in the support channels in free APA there's users mailing these, we get a lot of mails about situation on Ubuntu with the older versions of Ubuntu where people kept using something that was released like eight to ten years ago and then insist to have features that were released two, three years ago. And that happens all the time, also with CentOS 5, like people still force it to use that, I don't know who is forcing them, if you're one of them please wink. Then yeah, so there's a lot of features were added into SSSD over years. One of the latest ones that I'd like to highlight is the smart card authentication for the local, for the remote users, for users from Active Directory or from IPA and the integration for GNOME environment where you can provision the settings and how desktops looks like, what applications are there through definitions of rules in IPA to GNOME desktops, very easy workstations. A pretty neat one, SSSD delivers the part of it, also there was a talk last year about this with a demo. With Curve us we have a bit more complex situation because it's an implementation of a protocol that gets a lot of scrutiny in terms of what it was put there and any new changes they basically come through, a committee that defines RFC specifications that might take like five years easily like it took for a SPAC exchange that allows us to finally protect in a single exchange when you obtain the initial ticket, protect your credentials against the men in the middle attacks but also get the two factor authentication in one go. We got certain APIs and plugins extended in MIT Kerberos because we saw a need for that for enterprise environments, one of them is for example certificate mapping. It used to be that if you want to have smart card authentication to get your Kerberos credentials you have to have strict rules of what is the certificate authority that issued the certificate. It had to have special extensions in it which didn't really work outside of a lab. If you really have environment where something is supplied by let's say your government, like your ID card with the government, you need to trust certain certificate rules and so on and you have to have this sort of definition somewhere and for different groups of users for different machines you have to have this differently defined mappings between what is acceptable and what is not and as a result you get environment where let's say like the French government was talking about 5.5 million users on the matrix yesterday. If you get that kind of environment suppose that you have to put all these mappings into the Kerberos 5.con for every single machine. It doesn't scale. A lot of things doesn't scale this way. So we were talking with the upstreams for introducing interfaces that we can dynamically load plugins that actually do this sort of decision making in the code instead of looking into statically defined things. It took several years to get these interfaces, get them rolled out into the upstream releases then in the distribution so eventually it comes in into Fedora, REL, Debian and others and you could start using them. But nobody really other than upstream projects like IPA really tracks that all these things exist and if you try to package this kind of thing you have to know, you have to get this knowledge somehow. This automated translation method for example or I didn't write here for the Kerberos staff location of the KDC according to your locations so that you don't talk to KDC on the other continent because it's just the one that your DNS SRV record mentions. These kind of things they were needed to be written and created. In this coordination it goes beyond that. So all these projects they are fairly independent. Like Samba has all needs, MIT Kerberos has all needs. We have good relationship but it doesn't mean that we drive them. We use them, we contribute but we don't hurt them. It's impossible to hurt all of these fellows who are of strong opinion of what they are doing. You could coordinate but always the coordination might fall, just fell because somebody forgot to tell somebody else. And especially it's hard on the distribution level where you have, aside from the upstream thing you have maintainers and maintainers might not give any interest into your tasks. They are interested in their own stuff. So typical example Samba often updates its own libraries and these libraries they have phone requirements so back ports and all this stuff might break the versions of Samba that are in the back ports under all the release distributions. You could do all this stuff but it's a snowflake that turns into Blizzard quite easily. Well I'm from Finland so for me Blizzard is kind of a living creature. I escape at one when I came here, it's still there. But really dependency chain reaction for these changes limits us from, well us as upstream plus all maintainers on all distributions from being flexible here. So we rely as I said in MIT Kerberos we do code level dependencies and the thing is while Kerberos is a protocol kind of interoperable you cannot really do this interoperability on the single host. There's simply it's not designed that you can load compile against MIT Kerberos and load Heimdall implementation of the same library. It just might crash application might not, typically crashes and depending on how far in the process you are you might get spectacular crashes. So and also there are features that don't fully match. You get the things like two factor authentication which might differ between implementations. You get support for credential caches that exist in one implementation not the other. So for example Heimdall doesn't know anything about key ring, kernel key ring based credential caches. And for a long time MIT Kerberos didn't know about the KCM protocol for the remote credential cache management that Heimdall invented. So it was kind of a game where you really cannot combine things. You could interoperate on the network but not really on all things. On the Debian for example Samba is packaged with Heimdall. So it's compiled against Heimdall because Debian wants to provide Samba AD DC and we spent seven years Samba upstream to do the MIT Kerberos based Samba AD build. So it's extremely hard work and we package that in Fedora now so we don't need to have Heimdall in Fedora and we can build everything from one source but Debian still doesn't build MIT version and it's frankly it's not completed yet. There's some pieces that still missing, we still need to complete them and then switch off the Heimdall support in Samba because it's also quite costly to support two code bases. And yeah we do have things in IPA that extends Samba so you load this IPA Samba driver that queries LDAP database for all this account information and you can load that but because on Debian Samba is compiled against Heimdall you get symbols crash and you get structures that basically do not correspond what the code expects. You get quite good crashes and of course a lot of bug reports. Then the other part is Java. So dot tax is all Java, well it's all Java plus, Python plus, C and stuff but it moves slowly because the primary use of that code base is something that is heavily related so if you do Red Hat certificate server deployment you expect that this will be a certificate authority for some big organization which is somewhere in the bunker under the ground and so on so protected with probably less moving parts there so they don't really jump around and implement all the new features like JDK11 was released last year in September or so. So Debian moved on to get this thing and dot tax doesn't support it yet so we cannot really get dot tax now working in Debian because of certain things are not compatible there on particular on the TLS level because there was a lot of change in JDK11 around the TLS 1.3 implementation. But the other part is that a lot of operations they assume that you use crypto library called NSS which is part of Mozilla and Firefox things. Firefox people think that this library is within the Firefox right so it's not really a shared libraries but long time ago Red Hat did standardize on NSS for crypto operations. We thought that we will convert the wall to use NSS and it didn't happen so nowadays whoever uses what they want they use it and we try to get the other side of it which is certification for governmental use and of spread out for NSS at least and the open SSL but dot tax depends on NSS a lot and expects certain things to work there for PKAS 11 modules for hardware crypto modules and so on. And there was a time to get this bridge between mostly open SSL based environment and NSS based environment with a special library that used private functions from NSS that the NSS project didn't want to publicly expose so they forked part of the code and then NSS decided that they killed this code. Kind of interesting story and NSS PAM finally got to Debian right last year thanks to Tima Althanin who is a loan supporter of this whole infrastructure on Debian distributions and not in his primary employment time and then we got this JDK 11 break and everything else so that's one of the stories that kind of quite sounds quite bad for Debian users and then the other part is that we get a lot of to coordinate system services so you have to run MIT Kerberos, you have to run in some environment some but then all that and all these things together. We depend on system D for doing a lot of these things like circuit activation to factor authentication because that's sort of a thing that gets activated automatically from the Kerberos KDC when there is a request coming in and otherwise it's shut down and there is an abstraction layer for doing all these configuration file changes if needed for example for all the rail versions there's auth config and then there was a change for providing auth select in Fedora so we had to adopt. On the other distribution somebody has to write this code so there was attempt to write recently for Debian so we get a bit of movement for how PAM and NSS configs are, NSS which configs are written down there and Arc Linux for example they have a downstream patch that doesn't apply anymore they never submitted it upstream to no wife. We basically would like everyone to work upstream but if it requires changes in the distribution we are not at the position who have to do this work. So now I'm coming to the actual thing that happened in Fedora and it's pretty interesting thing because it's collaboration between multiple distributions while it's benefits in Fedora here and FreeAPA it came originally from SUSE. So we get on one side on upstream FreeAPA we have continuous integration system that's put in use for every single pull request. It's running VMs in open stack environment so runs them in parallel whatever we have kind of capacity there so that you get pull request you get some tasks every test kind of unit gets deploying a server, deploying replica, deploying clients, doing some operations that's specific to the test then tearing this apart then doing all the other things. In total there are like 55 test suites for each pull request and you can run maybe a 5-6 pull request in parallel at the same time at our current capacity and it's roughly two hours of the wall clock so you have to wait but you have to wait anyway in any of those cases. Then on the other side we have nightly runs so we take whatever is in the Gitmaster or specific branch we have this for several branches and run this against Fedora version that is released and roughly once a month we update the images so that we include something that was in updates by that time or when we know that there's some problem fixed. And in total it's about twice of those test suites so we get around 700, 800 individual tests executed. It's pretty large so if you kind of take them sequentially that's not really nightly. I tried to find a planet or a dwarf or asteroid that has the same rotational period like this 60 hours or so there's none in this system. So then I started looking around what it's exactly and it's parallelized it is around 8 hours so that goes into nightly. Unfortunately we have a team that spawns from Australia to Brazil the other way around and that means that we have people working almost all the time. So for us nightly means that we still waste resources that somebody could use at that point. So this is a typical diagram how parallel those tests are. The interesting part here is that there are few tests that run 2.5 hours long and they are pretty heavy ones so in the middle closer to the lower part there's this long big line that's a DNSSEC test that does a lot of things like enrolling, creating a zone with DNSSEC signing it up then trying to get the renewals for the signatures there going back in time going forth in time all these things. That's why it takes also a lot of time there. The shortest line second from bottom is actually a build so build is small. The real thing is huge. Deployment is about the same time as the build so you get like 5-6 minutes to deploy the full single system and then you start doing the actual tests. And then we get to Fedora so we do the integration at the update submission time. So with Fedora the concept is that you first build packages and then you combine an update request that goes against particular distribution version. And at that point when this update request is submitted you get a bunch of messages actually on Fedora message and bus that somebody listens to. At least I hope somebody. I get them on IOC from the Fedora board but there are other boards that are listening for this. And one of the boards is actually running OpenKA set of tests and OpenKA will be described by the next presentation so if you're interested you can look at that and OpenKA was created by OpenSuzi. And it's a nice thing that allows you to get a lot of interesting things with the distribution. It starts with a bare metal and you can program everything there. So we get a lot of things. Some of them I didn't write this here in detail but the tests we run are typical tests like deploy one instance of IPA server, one client, try to create some records there, some administrative tasks and then try to use that. And we have tasks that actually imitate the full use of it like full desktop experience where you have the server, you have a client that's enrolled to that server and then you get a human logging into that system through GDM, so the graphical environment, supposedly getting Kerberos ticket at this point. So you launch Firefox, you do single sign-on into the site, into IPA management console and then see what you do there. And OpenKA automates checking that things are done right. So you do the operations that are actually there. And we also do the upgrades. So the upgrades is an interesting thing because you can run a server on the previous released Fedora version, then upgrade the server, then enroll a client which is running on the older version, then upgrade the client and verify that it continues to work. So it's kind of the work that you really need to do for the release engineering. And we do it at every package integration. This is a demo. So for example, right now this shows you a kind of environment where you have a server standing up first, then the client standing up, then a client connecting to the server and rolling into the IPA deployment. And then you try to log in into that client. And let me show you, there's basically two minute clips. I hope they actually will work. Okay. This is all recorded automatically by OpenKA. So it's a bit of sped up, of course, of what happens. You cannot do that manually unless you're a super-typer guy. Okay. So this is Fedora 28. Yes. Called something rebooted, installing some other packages that needs to be there. This is a heavy jet to get the randomness in the VM in an easier way. And now it's deploying the server. Fedora has this roll key thing that's scripted and of away any details how the actual deployment happens. I think it's around 30, 40 seconds in this compressed video. So you'll get something. These are local network within the environment that OpenKA creates. So you don't get there. Yes, weird IPv6 because there's no IPv6 in that environment. So now script is finished, but it waits until the whole stuff sort of settles down because services need to start. And I hope, yeah, we still have like 40 seconds to actually complete this particular scenario. I think there will be like 10 seconds or so after widget starts moving because something happens behind, right? Not visible on the actual console, which is recorded here. Okay. Yeah. So now it collects all the logs and provides us all the logs, including the Linux stuff. And then you can look up and see it. So a lot of uploads, updates, and final rebooting it. Oh, this is actually a second one already, right? Yes. So this is a second demo. So the client, now we have machine. We now have a client that tries to enroll to that machine. Totem doesn't make it easy without like looking at what is happening there. So stopping the playlist from execution was my mistake. I didn't do that. So it started playing the second video, but that's exactly what we need. So we have a server. I think it's been started again, 10 to 20 seconds something happens in the background of what we don't see on the console. But after that, it should start a graphic session, log in into Firefox into cockpit console and through cockpit actually enroll into the system. Okay. So, joins the domain as administrator does something. Okay. Now it goes into IP console and creates users there, creates some other things. Yes. Quite fast. Adds them into the access rules and then tries to verify that you can actually do these operations as that user and you can log in and see your settings, change the password and all that stuff. Amazing. All without human intervention. When sometimes it's kind of great, finished, we can switch back to the actual presentation. So yes, we can catch non-trivial box with this. And last time I asked Adam Williamson, who is a Fedora QE guy and who implemented all these tests in OpenQE, how many bugs we caught in I think 18 months that this is in place in Fedora and he sent me a list of 33 bugs we caught. Some of them are pretty weird. You really have to look into details why they happen. A lot of bugs are as a Linux access denials. So we actually find out that you need to extend the policy and so on. But there's one thing with two bugs. There was a bug in MIT Kerberos that caused crash in multiple applications. The real cause was actually a bug in 389DS and the LDAP server stepping over several threads, they can step over each other's Kerberos credential caches. So wipe out tickets for each other and then bedlock in Kerberos library because they kind of use some resources that touches the same credentials cache. So while fixing this bug in both of them, there was a security fix in MIT Kerberos. So they had to do a CV package update and they fixed it and there was a mistake. The guy backed off the patch that fixed the bug for 389DS in Kerberos. So we had several times back and forth with the security update that was simply broken for IPA and that was noticed by the OpenQ test that we run. And 389DS broke another part of IPA by changing their internal way of how they create LDAP sub-trees. So it's now fixed. The fix is in Fedora 29 since Friday. But the story was like two or three months of interoperability things and it's still visible in the upstream nightly CAA because we did not update the image that we test there which we should do next week. And the Dyrtor server update from Friday actually shows you that all these tests are fine except the first one which has some minor AVCs in the logs. That's why it kind of highlighted there. So that's the great thing. So what's next? What we want to do? We want to integrate all this stuff at, if we look at the, how we would test, so we test what code goes into the release of IPA. But we also test what code goes into release of Fedora or Rail because it's the same everywhere. And we need to test the other way. When things are just added to the packages in Fedora, we want to do it at the pull request time. But Fedora CI infrastructure, the standard one, doesn't allow you to run multi-host tests. So we want to plug in into that and listen to the messages that are issued on the bus and run this in our PRCI upstream thing to get it there. Why don't you use the multi-host testing in OpenQA? We do use multi-test testing in OpenQA. Yeah, we do. It's just that at that point it's too late. We want to test when pull request is submitted to something like NSS library or something like Curl. I've got ideas for that. Yeah. Yeah. So this is the typical thing. It's just something we need to work on to get it done. On the other side, we look for contributions for other operating systems. Like I really would love to see Susie actually packaging for the APA. You're working on it? Yeah. And that's why I want your test to be done? Yeah. And we need to share all these tests because we really share the same stuff there. And get your nightly runs on OBS for anything you could do. But it's a bit more of work because like I say 300-something packages involved in all this thing and you need to coordinate. This is more than herding cats. It's like herding asteroids. Or comets. So questions if I have time left? Five minutes. Okay. When A will be active? A for audit inside IBM. Sorry? The A for audit. Oh, the audit part. Okay. The question is IPA stands for identity policy audit. When audit will be implemented, that's the question. We have a project that's called common logging that tries to collect logs from different places and coordinate them together. It started with the open shift needs because they have a lot of logs that are not related to the host systems. They are related to containers and they need to collect them. On the other side, so we rely on that when it will be ready for collecting logs. Again, we have a lot of replicas. So you have to collect data from the replicas and from the clients as well. The other side for regulated industries, we have session recording already in Fedora. It's the T-LOG project and SSSD has integration with this. It's basically replacing shell that your user runs automatically depending on the rules that I define at an IPA in a central way. And then that whatever is done in the shell is recorded in journal on the machine with a lot of metadata in JSON format and integrated with cockpit so that you can browse the changes and see what services started, what they produced when somebody did these operations. Last for them when we had the identity dev room, there was a talk about it. Well, if there are no more questions, thank you.
 Thank you everyone who came for this talk.  I'm going to talk about a bit.  Experience that we had that free API project with packaging our stuff for multiple distributions,  but primarily for Fedora.  For those who are curious, those ships are the ships that come every year to Helsinki  harbor for like 300 something years to sell the fish on the fish market once a year in  September for a week.  And they have this race that was raised from last year basically all in mist.  That's why they all graze.  Some of these ships, they actually have red sales there, which a bit of interesting part  for me because I have actually two red heads at home, but I'm not wearing it all the time.  So I work at Red Hat for the identity and security and do a lot of stuff upstream.  So I'm working on free APAs.  You could have guests from this talk and I also work on Samba for last 15 something years.  And I'll contribute to Fedora whenever I can primarily around the areas that we get affected  with for the Fedora packaging.  But for example, I'm a sort of a life support for ZZ-Leap package, which is kind of dead  upstream wise.  But it's needed to build documentation for MIT Kerberos.  And if we remove it from Fedora, we remove MIT Kerberos from Fedora.  So sometimes your path to packaging can be absolutely unpredictable.  So if we look at the IPA, it's actually quite a huge thing by itself, but it also a system  that packages a lot of or combines a lot of other software together.  So I'll start with the framework.  It's mostly Python, but it also has a bunch of C plugins and components.  So this is kind of code-based statistics that you can see.  We have around 300,000 lines of Python code, around 50,000, 60,000 lines of C if you take  it, give or take the comments into account.  There's some JavaScript stuff, pretty sizable for the web UI.  Some of these things, they actually get calculated wrongly, but that's how all these systems work  that take the statistics on it.  It's a web application that runs under, what WSGA and Apache has tight integration with  the Mods.js API, which is the kind of provides you single sign-on through Kerberos.  It has a lot of Python code in the installers and it has a lot of plugins that implement  certain things.  This is basically to make Kerberos and LDAP working on Linux.  And when it started in 2008, 2007, the idea was to bring back this idea of nice manageable  services on POSIX compatible or Unix platforms like Microsoft did when they basically took  the Unix technologies, combined them in a nice package and made Active Directory at  10 years before that.  So we're still doing this work and people keep us named as Active Directory for Linux,  which is not exactly that.  We do much more than Active Directory does, but so this is kind of a lot of comparisons  on it.  We do a lot of stuff around the MIT Kerberos to provide the single sign-on things and we  have many features in MIT Kerberos driven by our needs.  So for example, the two-factor authentication, the whole process of implementing it from  RFC acceptance to driving to support was driven by the needs of free IP approach.  And then there's Kerberos proxy implementation, which allows you to have indirect access to  your KDCs so you don't put them on the Internet, even though it was created to be put into  this kind of environment in 1980s.  Microsoft invented a protocol that allows you to know this over HTTPS, so there wasn't  any implementation until we started working on it.  Now there are four or five different implementations, including those that are combined with VPN  services, so kind of spurred up a lot of effort here and there and got to the nice things.  So the other part, pretty important one, is the LDAP server, which serves as a backend  to all these services.  We use what used to be called Netscape Directory Server, then iPlanet Server, then when Red Hat  bought it, it was called what it was, Fedora Directory Server, FDS, right, and then it  was renamed to 389DS.  389 is the port for LDAP, so you see, but we have more than a dozen plugins on top of  it that implement specifics of the behavioral things.  This presentation, it's uploaded and it has links to other presentations, so it's kind  of a web of rabbit holes that you might get into if you get bored.  The other part is that we handle the certificates and reissues of the certificates automatically  so we get a lot of stuff and this is another part of Netscape Heritage that Red Hat obtained.  It's the DocTag Certificate Authority, which is upstream for Red Hat Certificates Services  product, but also available as a part of free IPA.  So we kind of cut down the number of features that we automate for typical users.  Our HTS does a bit more, we just do what we want, but we share the same code base.  And it's based on Java, so there's a lot of packages, and if you look at the review,  the dependency is like 350 plus packages altogether.  So we are what one of my colleagues says, essentially we are the rail, Red Hat Enterprise  Linux.  If anything is broken in rail, we most likely will find it because we depend on a lot of  that stuff.  And then on the client side is the another component of this ecosystem is SSSD, which  is, I think these days, available almost everywhere on all distributions, that has a special provider  mode that talks to free IPA or IPA and has support for most of its features.  Not all of them, but it's kind of growing.  It's also growing pain if you don't have those.  So we also integrate with DNS, making it easy to dynamically provision hosts, enable, create  the zones and records and zones.  And so on we do DNS sec automatically and enable a lot of things.  It's a bit fragile.  Some of the pieces are a bit harder to maintain things because we touch a lot of inside of  bind.  Bind doesn't have really a good, pluggable API.  So we have to play Frankenstein style operations to get it working.  Surprisingly, it works pretty well for most use cases.  And this binding the build up is the backend to look up the data in LDAP.  So we have interesting case ones where some guys took the binding the build up, compiled  it with bind on salaries and use that to load the zones that they had.  They had something like 30,000 zones with sizable amount of host per zones to compare  how this would work with like a typical flat file-based thing.  And apparently this scaled much better the environment.  So the difference was like hundreds of times in the performance.  Then we integrate with Samba to get the Active Directory things and extensively use Python  bindings to provide this integration because we are Python stuff there inside.  So from some real world examples GNOME runs its account system.  So account.gnome.org actually runs free IPA web interface for all GNOME participants.  Fedora project infrastructure has integration within itself with IPA.  It doesn't show you this thing externally, but you're required to use Kerberos authentication  to some services and that's exactly where we are.  And then Red Hat IT runs a modernization project since like 2015, so 2016 they actually started  to deploy it.  Since last year at FOSDOM there was a report on that one, it's linked from here.  On the distributions level so we have more or less full support for both server and client  side from Fedora, REL, CentOS and Outlinux.  The full server support is a bit interesting thing and I will go into details just in a  few minutes.  Because IPA is not a single server, you could run it as a single server but you actually  can scale horizontally by replicating all this data and you can split them in sites,  put in different locations and so on.  It becomes quite harder to deploy this if you don't have all the infrastructure in place.  So in many distributions we actually lack that infrastructure but in some we do have.  So Outlinux and Fedora and REL based it once, they all base it with NSS and OpenSSL, Interim  Server but in components that basically provision all this data bases for these crypto libraries  so we can do that stuff.  In other distributions we are a bit troubled.  And on Debian for example, in derivatives the client support mostly there so it includes  SSSD and components that need to provision the IPA client but on the server side for  long time it was a bit of a bitter taste.  On the other distributions it's even more complicated so Outlinux doesn't have any packaging  for the server, they had packaging for the client that got broke.  OpenSusie and Sles they don't have server support anymore because OpenSusie had packaged  free APA like four years ago and then they removed it, whatever was the reason.  And the client side SSSD is built without IPA ID provider so you can use it but you  cannot use the functionality, extended functionality there.  So what this kind of mostly complete means, we work with a lot of upstream projects and  drive changes there as we see bits and pieces.  So one of these is like SSSD is our sister project so there's pretty good collaboration  on what needs to be added.  But if you get a distribution that doesn't have good updates for the released version  or they have forced it that you cannot really release a new version into the updates, you  might end up having pretty old versions that don't support the features that you might  expect be there.  So in the support channels in free APA there's users mailing these, we get a lot of mails  about situation on Ubuntu with the older versions of Ubuntu where people kept using  something that was released like eight to ten years ago and then insist to have features  that were released two, three years ago.  And that happens all the time, also with CentOS 5, like people still force it to use that,  I don't know who is forcing them, if you're one of them please wink.  Then yeah, so there's a lot of features were added into SSSD over years.  One of the latest ones that I'd like to highlight is the smart card authentication for the local,  for the remote users, for users from Active Directory or from IPA and the integration  for GNOME environment where you can provision the settings and how desktops looks like,  what applications are there through definitions of rules in IPA to GNOME desktops, very easy  workstations.  A pretty neat one, SSSD delivers the part of it, also there was a talk last year about  this with a demo.  With Curve us we have a bit more complex situation because it's an implementation of a protocol  that gets a lot of scrutiny in terms of what it was put there and any new changes they  basically come through, a committee that defines RFC specifications that might take like five  years easily like it took for a SPAC exchange that allows us to finally protect in a single  exchange when you obtain the initial ticket, protect your credentials against the men in  the middle attacks but also get the two factor authentication in one go.  We got certain APIs and plugins extended in MIT Kerberos because we saw a need for that  for enterprise environments, one of them is for example certificate mapping.  It used to be that if you want to have smart card authentication to get your Kerberos credentials  you have to have strict rules of what is the certificate authority that issued the certificate.  It had to have special extensions in it which didn't really work outside of a lab.  If you really have environment where something is supplied by let's say your government,  like your ID card with the government, you need to trust certain certificate rules and  so on and you have to have this sort of definition somewhere and for different groups of users  for different machines you have to have this differently defined mappings between what  is acceptable and what is not and as a result you get environment where let's say like the  French government was talking about 5.5 million users on the matrix yesterday.  If you get that kind of environment suppose that you have to put all these mappings into  the Kerberos 5.con for every single machine.  It doesn't scale.  A lot of things doesn't scale this way.  So we were talking with the upstreams for introducing interfaces that we can dynamically  load plugins that actually do this sort of decision making in the code instead of looking  into statically defined things.  It took several years to get these interfaces, get them rolled out into the upstream releases  then in the distribution so eventually it comes in into Fedora, REL, Debian and others  and you could start using them.  But nobody really other than upstream projects like IPA really tracks that all these things  exist and if you try to package this kind of thing you have to know, you have to get  this knowledge somehow.  This automated translation method for example or I didn't write here for the Kerberos staff  location of the KDC according to your locations so that you don't talk to KDC on the other  continent because it's just the one that your DNS SRV record mentions.  These kind of things they were needed to be written and created.  In this coordination it goes beyond that.  So all these projects they are fairly independent.  Like Samba has all needs, MIT Kerberos has all needs.  We have good relationship but it doesn't mean that we drive them.  We use them, we contribute but we don't hurt them.  It's impossible to hurt all of these fellows who are of strong opinion of what they are  doing.  You could coordinate but always the coordination might fall, just fell because somebody forgot  to tell somebody else.  And especially it's hard on the distribution level where you have, aside from the upstream  thing you have maintainers and maintainers might not give any interest into your tasks.  They are interested in their own stuff.  So typical example Samba often updates its own libraries and these libraries they have  phone requirements so back ports and all this stuff might break the versions of Samba  that are in the back ports under all the release distributions.  You could do all this stuff but it's a snowflake that turns into Blizzard quite easily.  Well I'm from Finland so for me Blizzard is kind of a living creature.  I escape at one when I came here, it's still there.  But really dependency chain reaction for these changes limits us from, well us as upstream  plus all maintainers on all distributions from being flexible here.  So we rely as I said in MIT Kerberos we do code level dependencies and the thing is while  Kerberos is a protocol kind of interoperable you cannot really do this interoperability  on the single host.  There's simply it's not designed that you can load compile against MIT Kerberos and  load Heimdall implementation of the same library.  It just might crash application might not, typically crashes and depending on how far  in the process you are you might get spectacular crashes.  So and also there are features that don't fully match.  You get the things like two factor authentication which might differ between implementations.  You get support for credential caches that exist in one implementation not the other.  So for example Heimdall doesn't know anything about key ring, kernel key ring based credential  caches.  And for a long time MIT Kerberos didn't know about the KCM protocol for the remote credential  cache management that Heimdall invented.  So it was kind of a game where you really cannot combine things.  You could interoperate on the network but not really on all things.  On the Debian for example Samba is packaged with Heimdall.  So it's compiled against Heimdall because Debian wants to provide Samba AD DC and we  spent seven years Samba upstream to do the MIT Kerberos based Samba AD build.  So it's extremely hard work and we package that in Fedora now so we don't need to have  Heimdall in Fedora and we can build everything from one source but Debian still doesn't build  MIT version and it's frankly it's not completed yet.  There's some pieces that still missing, we still need to complete them and then switch  off the Heimdall support in Samba because it's also quite costly to support two code  bases.  And yeah we do have things in IPA that extends Samba so you load this IPA Samba driver that  queries LDAP database for all this account information and you can load that but because  on Debian Samba is compiled against Heimdall you get symbols crash and you get structures  that basically do not correspond what the code expects.  You get quite good crashes and of course a lot of bug reports.  Then the other part is Java.  So dot tax is all Java, well it's all Java plus, Python plus, C and stuff but it moves  slowly because the primary use of that code base is something that is heavily related  so if you do Red Hat certificate server deployment you expect that this will be a certificate  authority for some big organization which is somewhere in the bunker under the ground  and so on so protected with probably less moving parts there so they don't really jump  around and implement all the new features like JDK11 was released last year in September  or so.  So Debian moved on to get this thing and dot tax doesn't support it yet so we cannot really  get dot tax now working in Debian because of certain things are not compatible there  on particular on the TLS level because there was a lot of change in JDK11 around the TLS  1.3 implementation.  But the other part is that a lot of operations they assume that you use crypto library called  NSS which is part of Mozilla and Firefox things.  Firefox people think that this library is within the Firefox right so it's not really  a shared libraries but long time ago Red Hat did standardize on NSS for crypto operations.  We thought that we will convert the wall to use NSS and it didn't happen so nowadays whoever  uses what they want they use it and we try to get the other side of it which is certification  for governmental use and of spread out for NSS at least and the open SSL but dot tax  depends on NSS a lot and expects certain things to work there for PKAS 11 modules for hardware  crypto modules and so on.  And there was a time to get this bridge between mostly open SSL based environment and NSS  based environment with a special library that used private functions from NSS that the NSS  project didn't want to publicly expose so they forked part of the code and then NSS decided  that they killed this code.  Kind of interesting story and NSS PAM finally got to Debian right last year thanks to Tima  Althanin who is a loan supporter of this whole infrastructure on Debian distributions and  not in his primary employment time and then we got this JDK 11 break and everything else  so that's one of the stories that kind of quite sounds quite bad for Debian users and  then the other part is that we get a lot of to coordinate system services so you have  to run MIT Kerberos, you have to run in some environment some but then all that and all  these things together.  We depend on system D for doing a lot of these things like circuit activation to factor authentication  because that's sort of a thing that gets activated automatically from the Kerberos KDC when there  is a request coming in and otherwise it's shut down and there is an abstraction layer  for doing all these configuration file changes if needed for example for all the rail versions  there's auth config and then there was a change for providing auth select in Fedora so we had  to adopt.  On the other distribution somebody has to write this code so there was attempt to write  recently for Debian so we get a bit of movement for how PAM and NSS configs are, NSS which  configs are written down there and Arc Linux for example they have a downstream patch that  doesn't apply anymore they never submitted it upstream to no wife.  We basically would like everyone to work upstream but if it requires changes in the distribution  we are not at the position who have to do this work.  So now I'm coming to the actual thing that happened in Fedora and it's pretty interesting  thing because it's collaboration between multiple distributions while it's benefits  in Fedora here and FreeAPA it came originally from SUSE.  So we get on one side on upstream FreeAPA we have continuous integration system that's  put in use for every single pull request.  It's running VMs in open stack environment so runs them in parallel whatever we have  kind of capacity there so that you get pull request you get some tasks every test kind  of unit gets deploying a server, deploying replica, deploying clients, doing some operations  that's specific to the test then tearing this apart then doing all the other things.  In total there are like 55 test suites for each pull request and you can run maybe a  5-6 pull request in parallel at the same time at our current capacity and it's roughly two  hours of the wall clock so you have to wait but you have to wait anyway in any of those  cases.  Then on the other side we have nightly runs so we take whatever is in the Gitmaster or  specific branch we have this for several branches and run this against Fedora version that is  released and roughly once a month we update the images so that we include something that  was in updates by that time or when we know that there's some problem fixed.  And in total it's about twice of those test suites so we get around 700, 800 individual  tests executed.  It's pretty large so if you kind of take them sequentially that's not really nightly.  I tried to find a planet or a dwarf or asteroid that has the same rotational period like this  60 hours or so there's none in this system.  So then I started looking around what it's exactly and it's parallelized it is around  8 hours so that goes into nightly.  Unfortunately we have a team that spawns from Australia to Brazil the other way around and  that means that we have people working almost all the time.  So for us nightly means that we still waste resources that somebody could use at that  point.  So this is a typical diagram how parallel those tests are.  The interesting part here is that there are few tests that run 2.5 hours long and they  are pretty heavy ones so in the middle closer to the lower part there's this long big line  that's a DNSSEC test that does a lot of things like enrolling, creating a zone with DNSSEC  signing it up then trying to get the renewals for the signatures there going back in time  going forth in time all these things.  That's why it takes also a lot of time there.  The shortest line second from bottom is actually a build so build is small.  The real thing is huge.  Deployment is about the same time as the build so you get like 5-6 minutes to deploy the  full single system and then you start doing the actual tests.  And then we get to Fedora so we do the integration at the update submission time.  So with Fedora the concept is that you first build packages and then you combine an update  request that goes against particular distribution version.  And at that point when this update request is submitted you get a bunch of messages actually  on Fedora message and bus that somebody listens to.  At least I hope somebody.  I get them on IOC from the Fedora board but there are other boards that are listening  for this.  And one of the boards is actually running OpenKA set of tests and OpenKA will be described  by the next presentation so if you're interested you can look at that and OpenKA was created  by OpenSuzi.  And it's a nice thing that allows you to get a lot of interesting things with the distribution.  It starts with a bare metal and you can program everything there.  So we get a lot of things.  Some of them I didn't write this here in detail but the tests we run are typical tests like  deploy one instance of IPA server, one client, try to create some records there, some administrative  tasks and then try to use that.  And we have tasks that actually imitate the full use of it like full desktop experience  where you have the server, you have a client that's enrolled to that server and then you  get a human logging into that system through GDM, so the graphical environment, supposedly  getting Kerberos ticket at this point.  So you launch Firefox, you do single sign-on into the site, into IPA management console  and then see what you do there.  And OpenKA automates checking that things are done right.  So you do the operations that are actually there.  And we also do the upgrades.  So the upgrades is an interesting thing because you can run a server on the previous released  Fedora version, then upgrade the server, then enroll a client which is running on the older  version, then upgrade the client and verify that it continues to work.  So it's kind of the work that you really need to do for the release engineering.  And we do it at every package integration.  This is a demo.  So for example, right now this shows you a kind of environment where you have a server  standing up first, then the client standing up, then a client connecting to the server  and rolling into the IPA deployment.  And then you try to log in into that client.  And let me show you, there's basically two minute clips.  I hope they actually will work.  Okay.  This is all recorded automatically by OpenKA.  So it's a bit of sped up, of course, of what happens.  You cannot do that manually unless you're a super-typer guy.  Okay.  So this is Fedora 28.  Yes.  Called something rebooted, installing some other packages that needs to be there.  This is a heavy jet to get the randomness in the VM in an easier way.  And now it's deploying the server.  Fedora has this roll key thing that's scripted and of away any details how the actual deployment  happens.  I think it's around 30, 40 seconds in this compressed video.  So you'll get something.  These are local network within the environment that OpenKA creates.  So you don't get there.  Yes, weird IPv6 because there's no IPv6 in that environment.  So now script is finished, but it waits until the whole stuff sort of settles down because  services need to start.  And I hope, yeah, we still have like 40 seconds to actually complete this particular scenario.  I think there will be like 10 seconds or so after widget starts moving because something  happens behind, right?  Not visible on the actual console, which is recorded here.  Okay.  Yeah.  So now it collects all the logs and provides us all the logs, including the Linux stuff.  And then you can look up and see it.  So a lot of uploads, updates, and final rebooting it.  Oh, this is actually a second one already, right?  Yes.  So this is a second demo.  So the client, now we have machine.  We now have a client that tries to enroll to that machine.  Totem doesn't make it easy without like looking at what is happening there.  So stopping the playlist from execution was my mistake.  I didn't do that.  So it started playing the second video, but that's exactly what we need.  So we have a server.  I think it's been started again, 10 to 20 seconds something happens in the background  of what we don't see on the console.  But after that, it should start a graphic session, log in into Firefox into cockpit  console and through cockpit actually enroll into the system.  Okay.  So, joins the domain as administrator does something.  Okay.  Now it goes into IP console and creates users there, creates some other things.  Yes.  Quite fast.  Adds them into the access rules and then tries to verify that you can actually do these  operations as that user and you can log in and see your settings, change the password  and all that stuff.  Amazing.  All without human intervention.  When sometimes it's kind of great, finished, we can switch back to the actual presentation.  So yes, we can catch non-trivial box with this.  And last time I asked Adam Williamson, who is a Fedora QE guy and who implemented all  these tests in OpenQE, how many bugs we caught in I think 18 months that this is in place  in Fedora and he sent me a list of 33 bugs we caught.  Some of them are pretty weird.  You really have to look into details why they happen.  A lot of bugs are as a Linux access denials.  So we actually find out that you need to extend the policy and so on.  But there's one thing with two bugs.  There was a bug in MIT Kerberos that caused crash in multiple applications.  The real cause was actually a bug in 389DS and the LDAP server stepping over several  threads, they can step over each other's Kerberos credential caches.  So wipe out tickets for each other and then bedlock in Kerberos library because they kind  of use some resources that touches the same credentials cache.  So while fixing this bug in both of them, there was a security fix in MIT Kerberos.  So they had to do a CV package update and they fixed it and there was a mistake.  The guy backed off the patch that fixed the bug for 389DS in Kerberos.  So we had several times back and forth with the security update that was simply broken  for IPA and that was noticed by the OpenQ test that we run.  And 389DS broke another part of IPA by changing their internal way of how they create LDAP  sub-trees.  So it's now fixed.  The fix is in Fedora 29 since Friday.  But the story was like two or three months of interoperability things and it's still  visible in the upstream nightly CAA because we did not update the image that we test there  which we should do next week.  And the Dyrtor server update from Friday actually shows you that all these tests are  fine except the first one which has some minor AVCs in the logs.  That's why it kind of highlighted there.  So that's the great thing.  So what's next?  What we want to do?  We want to integrate all this stuff at, if we look at the, how we would test, so we  test what code goes into the release of IPA.  But we also test what code goes into release of Fedora or Rail because it's the same everywhere.  And we need to test the other way.  When things are just added to the packages in Fedora, we want to do it at the pull request  time.  But Fedora CI infrastructure, the standard one, doesn't allow you to run multi-host tests.  So we want to plug in into that and listen to the messages that are issued on the bus  and run this in our PRCI upstream thing to get it there.  Why don't you use the multi-host testing in OpenQA?  We do use multi-test testing in OpenQA.  Yeah, we do.  It's just that at that point it's too late.  We want to test when pull request is submitted to something like NSS library or something  like Curl.  I've got ideas for that.  Yeah.  Yeah.  So this is the typical thing.  It's just something we need to work on to get it done.  On the other side, we look for contributions for other operating systems.  Like I really would love to see Susie actually packaging for the APA.  You're working on it?  Yeah.  And that's why I want your test to be done?  Yeah.  And we need to share all these tests because we really share the same stuff there.  And get your nightly runs on OBS for anything you could do.  But it's a bit more of work because like I say 300-something packages involved in all  this thing and you need to coordinate.  This is more than herding cats.  It's like herding asteroids.  Or comets.  So questions if I have time left?  Five minutes.  Okay.  When A will be active?  A for audit inside IBM.  Sorry?  The A for audit.  Oh, the audit part.  Okay.  The question is IPA stands for identity policy audit.  When audit will be implemented, that's the question.  We have a project that's called common logging that tries to collect logs from different  places and coordinate them together.  It started with the open shift needs because they have a lot of logs that are not related  to the host systems.  They are related to containers and they need to collect them.  On the other side, so we rely on that when it will be ready for collecting logs.  Again, we have a lot of replicas.  So you have to collect data from the replicas and from the clients as well.  The other side for regulated industries, we have session recording already in Fedora.  It's the T-LOG project and SSSD has integration with this.  It's basically replacing shell that your user runs automatically depending on the rules  that I define at an IPA in a central way.  And then that whatever is done in the shell is recorded in journal on the machine with  a lot of metadata in JSON format and integrated with cockpit so that you can browse the changes  and see what services started, what they produced when somebody did these operations.  Last for them when we had the identity dev room, there was a talk about it.  Well, if there are no more questions, thank you.
FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_0_167.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_168_227.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_228_1547.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_1548_3083.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_3084_5171.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_5172_8171.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_8172_42887.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_42888_43367.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_43368_44879.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_44880_47123.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_47124_48887.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_48888_53891.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_53892_54119.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54120_54167.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54168_54239.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54240_54287.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54288_54407.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54408_54503.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54504_54551.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54552_54623.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54624_54683.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54684_54755.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54756_54827.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54828_54875.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_54876_55571.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_55572_55619.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_55620_55667.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_55668_55715.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_55716_56843.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_56844_56927.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_56928_56975.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_56976_57023.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_57024_57095.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_57096_57155.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_57156_57215.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_57216_57287.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_57288_57359.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_57360_57431.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_57432_57491.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_57492_58859.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_58860_58931.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_58932_59075.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59076_59135.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59136_59183.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59184_59339.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59340_59399.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59400_59471.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59472_59519.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59520_59579.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59580_59627.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59628_59711.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59712_59771.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59772_59819.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59820_59867.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59868_59939.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_59940_60071.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_60072_64127.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_64128_64199.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_64200_67259.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_67260_70559.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_70560_70607.png FreeIPA_and_cross-distribution_packaging_experience_lKKipD14ROw_70608_70723.png