Using OpenAPI to Maximise Your Pulp 3 Experience
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kliban, Dennis
Pulp enables users to organize and distribute software. Now that Pulp 3.0 is generally available, it’s time to integrate it into your software delivery workflows. While the REST API is the primary integration point, it is the OpenAPI schema definition of that API that enables users to build client libraries in various languages. These clients simplify the integration with Pulp 3. This talk will provide a brief introduction to OpenAPI. This will be followed by a demonstration of how to use the Pulp’s OpenAPI schema to generate a Python client for Pulp’s REST API. The Python client will then be used to perform various workflows in Pulp 3.
 So, my name is Dennis Cleveland and I'm here today to talk to you about how you can use OpenAPI to maximize your Pulp 3 experience. I've been a Python developer since 2009, so over 10 years now. And even before I joined Red Hat, I've been a fan of open source software, so it's been great being able to work on Pulp full time. And my free time, I like to juggle. I always have six balls with me, so if you want to find me after the talk and throw around, I'm up for that. So let's begin by talking about what you can do with Pulp. You can use Pulp to mirror repositories that are out in the wild and make them available to your local clients. You can also use it to upload your custom software and make repositories of that. And then you can also combine both sources of software and create custom repositories for whatever workflows you need to have. Pulp 3.0 became GA earlier this year. We actually released 3.1 recently. And with the release of 3.0, we had multiple plugins that went GA. The RPM plugin, the container plugin, which used to be called the Docker plugin in Pulp 2.0, and the file plugin, which used to be part of the RPM plugin. It was the ISO content type there. The plugin API was a very big focus of Pulp 3.0. And as a result, we've been able to produce a lot more plugins. And even though they're still in beta, we have a cookbook, a Debian, a Maven, NPM, and a Python plugin out there. So let's talk about OpenAPI. OpenAPI is a programming language agnostic specification for describing REST APIs. When we first started development on Pulp 3, OpenAPI was still called Swagger. Over those last three years, the company behind it has made the specification open source, and it's now called OpenAPI. There is a pretty big community around OpenAPI, and there are many tools out there. Some of them allow you to take your OpenAPI schema and generate REST API documentation. Other tools let you generate client libraries and also even generating server side code. Today, I'm going to focus on the documentation and on the client libraries. And the tool that we're going to use is called OpenAPI Generator. So let's go to the browser and look at the documentation page that's available on every installation of Pulp 3. If you browse to slash pulp API v3 docs, you will see this nice page. At the very top of it, there's actually a download button, and it lets you download... Thanks. So this is the documentation page that needs to be zoomed out. And on this page, there's this download button that opens up the schema itself. The schema has definitions for every path that you have, and it's nothing too fancy. What I really like to look at is the docs that we are able to generate using this OpenAPI schema. For example, you want to learn how to use our uploads API. You can go down to uploads. You can see how to create an upload and do all the things you want to do with the REST API. So next, I want to show you how you can use the OpenAPI schema to generate a client that will be able to interact with your Pulp 3 instance. Because each Pulp instance will have a different set of plugins installed on it, I recommend generating a client for each plugin specifically. And so that same URL that we are using to look at the docs, you can append to it and say that you want to get the schema for the binding and for the plugin. And the plugin in this case is Pulp Core, which is the main library that provides the REST API for Pulp. The binding keyword there says to remove any sort of HTML that's there that we've introduced to make that web page a little bit better. And so let's go to the browser, not the browser, to the command line and run this back script that I have. The first thing that we're going to do is we're going to retrieve the OpenAPI schema with that command that I showed you a second ago. And so that's what happened just now. Then let's look at the configuration that you can provide to the OpenAPI generator. For the Python plugin. OpenAPI generator can generate clients in many languages. This demo is going to focus on Python. And we're going to use some of these settings to generate the client. So this is the command that I'm using to generate the client. I'm using Podman to run a container that contains inside of the OpenAPI generator CLI. This OpenAPI generated CLI will let us generate the client for, first we're going to do it for Pulp Core, then we're going to do it for Pulp File, and then we're going to do it for Pulp Container. So let's continue. So all of this generated the client for Pulp Core. Now we're going to do the same thing for Pulp File. And we're doing the same thing for Pulp Container. So for the next part of the demo, we're going to work with a Python script that I've written that takes advantage of the library. And so we import from all the clients, and there's a method for monitoring a task, for doing uploads and chunks, and a lot more. We're going to go through the file in the slide. So I'm going to start by talking about the configuration. No matter what language you're using for interacting with Pulp, the bindings that will get generated will need to be configured with a setting of savechars for past perium, and it needs to be set to a forward slash. The main ID that we use in Pulp 3 for referencing any of the objects contains forward flashes, and we do not want those forward flashes to get URL encoded. So this is what the setting tells it. So after we've configured the clients, we can go ahead and create a file repository inside of Pulp. So let's run this script. So we're first going to create a repository, and we're going to call it POOP, and this is a file repository. And so now we have Pulp. It doesn't have any content inside of it, but it does have a file repository called POOP. And we're aware of some repo out on the internet that has some files inside of it. So next, we will want to sync that repository. In order to sync the repository, we will need to create a file remote. The file remote, the most important parameter of it is the URL. And a file remote needs to have a URL that points to a Pulp manifest, which is a metadata for file repositories. And we will see it a little bit later. So we're going to create the file remote, and then we're going to actually tell the repository to sync from that remote. So that's what's going to happen here next. This created the file remote, and now we are syncing, and we've synced. And so now, after a sync has been performed, the repository in Pulp now has a repository version associated with it. And this repository version is actually an immutable set of content in Pulp. And every time you modify the content of a repository, you're actually creating a new repository version. And so now we have three files in Pulp, and we have a repository version that's showing that. Now, before you can make that repository version available to your clients, you will need to make a file publication. The file publication is created using the commands here, and the creation of the publication is an asynchronous task, and so you will need to monitor that task to make sure it finishes. And when it finishes, you can move on to the next step. So let's continue with that. All right, so now we have a file publication, and that publication is pointing to that repository version that we created earlier. And let's talk about the difference between publications and repository versions. Some content types don't have any additional metadata that needs to be created, and in those cases, a repository version is all that is necessary. In the case of the file repository, we do generate a metadata file, the Pulp manifest, at publication time, so the difference is the content, the publication contains the content plus the metadata. And so once we have this publication, we can make it available to the clients by creating a file distribution. The file distribution takes a parameter called base path, and this is the path where this publication will be available to your clients. So let's go back to here and continue with creating the distribution. We created the distribution, and so now we have a repository version, a publication, a distribution, and so your clients should be able to get access to this repository, and we're going to take a look at it in the browser, actually. Here, I'm pointing to the root of our content app on port 2416, though on some installations, you don't even have to specify the port if the reverse proxies configured correctly. And so what we see here now is that we have this repository available to the clients. And there is this pulp manifest file that we can save, and we can open, and it has some information about this repository. Pulp actually can use this file to discover the files in another file repository out there. So for the next part of the demo, I'm going to work with the container plugin. And the first thing we're going to do is we're going to create a container repository. So in pulp three, the repositories are all typed, so whenever you're creating a repository that's going to hold containers, you create a container repository. So we're going to continue the script. The script will create a container repository for us. And so now we have this repository. We still have those old files in there from that other repo, but for the sake of the slide, we're pretending that there is no other content here. So next, we're going to upload a file into pulp that we're going to bake into a container. So I have this file called Hello World, and we're going to put some custom text into it. We're going to save it, and then we're going to upload it to pulp. So now we have that file that we just uploaded, and it has some text inside of it. What we can do next is we can use that file to form this string that gives that file a path inside of a container that you're building. So then you can have a docker file that references this file. The file that you put into that artifact parameter will actually become available inside of the build environment when builda is being run to build the container. And so this is the docker file that we're going to upload to build the container. And at the very end, you can see that it runs a command to cat that file that we just were looking at. So let's go back to here, and we're going to execute this command to build the container. This takes a few seconds. Most of the layers are actually already cached on my machine, so it doesn't need to download them from the Internet, but it is building a new image with that new file inside of it. Whenever it finishes, we will be able to pull down the image and run it. Under the hood, the pulp worker is actually running builda to make the container. So now, after we've built the container, that's the repository that we had now has a repository version. Inside that repository version, there is a manifest and some blobs. And so we can now create a distribution, a container distribution, that's going to make that repository available to your clients. The main thing to notice here is the base path once again. The base path is what you will be able to use with Podman or Docker to reference the repository. So let's go ahead and create this container distribution, and it's been created now. And so now we have a distribution that's pointing to that repository version. And so let's go back to here and do a Podman pull localhost of FOSDEM2020. And so it's downloading the latest blob. And now we're going to run this container. And look at that. We've got our custom text coming in from that container. And that's how you can use OpenAPI to maximize your PULP3 experience. So I've uploaded these slides to the event page so you can download them. There are some links to the script. We have a YouTube channel, and I'll be happy to answer some questions right now. Any questions? Yes, it's OpenAPI 2.0. So the question was, when we first started, it was called OpenAPI with Still Swagger, which means that the OpenAPI specification that we're using is 2.0. And are there any plans to switch over to 3.0? Yes, I would love for us to move to 3.0, but I have no timelines for that. And 3.0 is a much more descriptive standard, so you can make all this even better. Yes? So when will the timelines when the other beta plugins come out of the beta stuff? The question is, when will the beta, the plugins in beta come out of beta? Hopefully soon. I don't have any timelines. I know that the Maven plugin has very little functionality in it, and I'm happy to release it as being generally available, but it will just have limited functionality. And I can only speak for that one, because I'm the sole developer on that one right now. It will be interesting in the Debian plugins. The Debian plugins should be available very soon. Any more questions? I love questions. If not, then I thank you very much for this. Thank you. Thank you.
 So, my name is Dennis Cleveland and I'm here today to talk to you about how you can use  OpenAPI to maximize your Pulp 3 experience.  I've been a Python developer since 2009, so over 10 years now.  And even before I joined Red Hat, I've been a fan of open source software, so it's been  great being able to work on Pulp full time.  And my free time, I like to juggle.  I always have six balls with me, so if you want to find me after the talk and throw around,  I'm up for that.  So let's begin by talking about what you can do with Pulp.  You can use Pulp to mirror repositories that are out in the wild and make them available  to your local clients.  You can also use it to upload your custom software and make repositories of that.  And then you can also combine both sources of software and create custom repositories  for whatever workflows you need to have.  Pulp 3.0 became GA earlier this year.  We actually released 3.1 recently.  And with the release of 3.0, we had multiple plugins that went GA.  The RPM plugin, the container plugin, which used to be called the Docker plugin in Pulp  2.0, and the file plugin, which used to be part of the RPM plugin.  It was the ISO content type there.  The plugin API was a very big focus of Pulp 3.0.  And as a result, we've been able to produce a lot more plugins.  And even though they're still in beta, we have a cookbook, a Debian, a Maven, NPM, and  a Python plugin out there.  So let's talk about OpenAPI.  OpenAPI is a programming language agnostic specification for describing REST APIs.  When we first started development on Pulp 3, OpenAPI was still called Swagger.  Over those last three years, the company behind it has made the specification open source,  and it's now called OpenAPI.  There is a pretty big community around OpenAPI, and there are many tools out there.  Some of them allow you to take your OpenAPI schema and generate REST API documentation.  Other tools let you generate client libraries and also even generating server side code.  Today, I'm going to focus on the documentation and on the client libraries.  And the tool that we're going to use is called OpenAPI Generator.  So let's go to the browser and look at the documentation page that's available on every  installation of Pulp 3.  If you browse to slash pulp API v3 docs, you will see this nice page.  At the very top of it, there's actually a download button, and it lets you download...  Thanks.  So this is the documentation page that needs to be zoomed out.  And on this page, there's this download button that opens up the schema itself.  The schema has definitions for every path that you have, and it's nothing too fancy.  What I really like to look at is the docs that we are able to generate using this OpenAPI  schema.  For example, you want to learn how to use our uploads API.  You can go down to uploads.  You can see how to create an upload and do all the things you want to do with the REST  API.  So next, I want to show you how you can use the OpenAPI schema to generate a client that  will be able to interact with your Pulp 3 instance.  Because each Pulp instance will have a different set of plugins installed on it, I recommend  generating a client for each plugin specifically.  And so that same URL that we are using to look at the docs, you can append to it and  say that you want to get the schema for the binding and for the plugin.  And the plugin in this case is Pulp Core, which is the main library that provides the  REST API for Pulp.  The binding keyword there says to remove any sort of HTML that's there that we've introduced  to make that web page a little bit better.  And so let's go to the browser, not the browser, to the command line and run this back script  that I have.  The first thing that we're going to do is we're going to retrieve the OpenAPI schema  with that command that I showed you a second ago.  And so that's what happened just now.  Then let's look at the configuration that you can provide to the OpenAPI generator.  For the Python plugin.  OpenAPI generator can generate clients in many languages.  This demo is going to focus on Python.  And we're going to use some of these settings to generate the client.  So this is the command that I'm using to generate the client.  I'm using Podman to run a container that contains inside of the OpenAPI generator CLI.  This OpenAPI generated CLI will let us generate the client for, first we're going to do it  for Pulp Core, then we're going to do it for Pulp File, and then we're going to do it for  Pulp Container.  So let's continue.  So all of this generated the client for Pulp Core.  Now we're going to do the same thing for Pulp File.  And we're doing the same thing for Pulp Container.  So for the next part of the demo, we're going to work with a Python script that I've written  that takes advantage of the library.  And so we import from all the clients, and there's a method for monitoring a task, for  doing uploads and chunks, and a lot more.  We're going to go through the file in the slide.  So I'm going to start by talking about the configuration.  No matter what language you're using for interacting with Pulp, the bindings that will get generated  will need to be configured with a setting of savechars for past perium, and it needs  to be set to a forward slash.  The main ID that we use in Pulp 3 for referencing any of the objects contains forward flashes,  and we do not want those forward flashes to get URL encoded.  So this is what the setting tells it.  So after we've configured the clients, we can go ahead and create a file repository  inside of Pulp.  So let's run this script.  So we're first going to create a repository, and we're going to call it POOP, and this  is a file repository.  And so now we have Pulp.  It doesn't have any content inside of it, but it does have a file repository called POOP.  And we're aware of some repo out on the internet that has some files inside of it.  So next, we will want to sync that repository.  In order to sync the repository, we will need to create a file remote.  The file remote, the most important parameter of it is the URL.  And a file remote needs to have a URL that points to a Pulp manifest, which is a metadata  for file repositories.  And we will see it a little bit later.  So we're going to create the file remote, and then we're going to actually tell the  repository to sync from that remote.  So that's what's going to happen here next.  This created the file remote, and now we are syncing, and we've synced.  And so now, after a sync has been performed, the repository in Pulp now has a repository  version associated with it.  And this repository version is actually an immutable set of content in Pulp.  And every time you modify the content of a repository, you're actually creating a new  repository version.  And so now we have three files in Pulp, and we have a repository version that's showing that.  Now, before you can make that repository version available to your clients, you will need to  make a file publication.  The file publication is created using the commands here, and the creation of the publication  is an asynchronous task, and so you will need to monitor that task to make sure it finishes.  And when it finishes, you can move on to the next step.  So let's continue with that.  All right, so now we have a file publication, and that publication is pointing to that repository  version that we created earlier.  And let's talk about the difference between publications and repository versions.  Some content types don't have any additional metadata that needs to be created, and in  those cases, a repository version is all that is necessary.  In the case of the file repository, we do generate a metadata file, the Pulp manifest, at  publication time, so the difference is the content, the publication contains the content  plus the metadata.  And so once we have this publication, we can make it available to the clients by creating  a file distribution.  The file distribution takes a parameter called base path, and this is the path where this  publication will be available to your clients.  So let's go back to here and continue with creating the distribution.  We created the distribution, and so now we have a repository version, a publication,  a distribution, and so your clients should be able to get access to this repository,  and we're going to take a look at it in the browser, actually.  Here, I'm pointing to the root of our content app on port 2416, though on some installations,  you don't even have to specify the port if the reverse proxies configured correctly.  And so what we see here now is that we have this repository available to the clients.  And there is this pulp manifest file that we can save, and we can open, and it has some  information about this repository.  Pulp actually can use this file to discover the files in another file repository out there.  So for the next part of the demo, I'm going to work with the container plugin.  And the first thing we're going to do is we're going to create a container repository.  So in pulp three, the repositories are all typed, so whenever you're creating a repository  that's going to hold containers, you create a container repository.  So we're going to continue the script.  The script will create a container repository for us.  And so now we have this repository.  We still have those old files in there from that other repo, but for the sake of the slide,  we're pretending that there is no other content here.  So next, we're going to upload a file into pulp that we're going to bake into a container.  So I have this file called Hello World, and we're going to put some custom text into it.  We're going to save it, and then we're going to upload it to pulp.  So now we have that file that we just uploaded, and it has some text inside of it.  What we can do next is we can use that file to form this string that gives that file a path inside of a container that you're building.  So then you can have a docker file that references this file.  The file that you put into that artifact parameter will actually become available inside of the build environment  when builda is being run to build the container.  And so this is the docker file that we're going to upload to build the container.  And at the very end, you can see that it runs a command to cat that file that we just were looking at.  So let's go back to here, and we're going to execute this command to build the container.  This takes a few seconds.  Most of the layers are actually already cached on my machine, so it doesn't need to download them from the Internet,  but it is building a new image with that new file inside of it.  Whenever it finishes, we will be able to pull down the image and run it.  Under the hood, the pulp worker is actually running builda to make the container.  So now, after we've built the container, that's the repository that we had now has a repository version.  Inside that repository version, there is a manifest and some blobs.  And so we can now create a distribution, a container distribution,  that's going to make that repository available to your clients.  The main thing to notice here is the base path once again.  The base path is what you will be able to use with Podman or Docker to reference the repository.  So let's go ahead and create this container distribution, and it's been created now.  And so now we have a distribution that's pointing to that repository version.  And so let's go back to here and do a Podman pull localhost of FOSDEM2020.  And so it's downloading the latest blob.  And now we're going to run this container.  And look at that. We've got our custom text coming in from that container.  And that's how you can use OpenAPI to maximize your PULP3 experience.  So I've uploaded these slides to the event page so you can download them.  There are some links to the script.  We have a YouTube channel, and I'll be happy to answer some questions right now.  Any questions?  Yes, it's OpenAPI 2.0.  So the question was, when we first started, it was called OpenAPI with Still Swagger,  which means that the OpenAPI specification that we're using is 2.0.  And are there any plans to switch over to 3.0?  Yes, I would love for us to move to 3.0, but I have no timelines for that.  And 3.0 is a much more descriptive standard, so you can make all this even better.  Yes?  So when will the timelines when the other beta plugins come out of the beta stuff?  The question is, when will the beta, the plugins in beta come out of beta?  Hopefully soon.  I don't have any timelines.  I know that the Maven plugin has very little functionality in it,  and I'm happy to release it as being generally available,  but it will just have limited functionality.  And I can only speak for that one, because I'm the sole developer on that one right now.  It will be interesting in the Debian plugins.  The Debian plugins should be available very soon.  Any more questions?  I love questions.  If not, then I thank you very much for this.  Thank you.  Thank you.
openapi_0_143.png openapi_144_1751.png openapi_1752_5579.png openapi_5580_6107.png openapi_6108_8315.png openapi_8316_10379.png openapi_10380_10727.png openapi_10728_10811.png openapi_10812_11075.png openapi_11076_11123.png openapi_11124_11183.png openapi_11184_11351.png openapi_11352_11699.png openapi_11700_11771.png openapi_11772_13367.png openapi_13368_15563.png openapi_15564_15803.png openapi_15804_15887.png openapi_15888_17243.png openapi_17244_17783.png openapi_17784_19175.png openapi_19176_19259.png openapi_19260_19763.png openapi_19764_20963.png openapi_20964_21707.png openapi_21708_22127.png openapi_22128_22883.png openapi_22884_24491.png openapi_24492_28127.png openapi_28128_31607.png openapi_31608_31855.png