Transparent data at rest encryption in PostgreSQL
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Sawada, Masahiko
Data encryption is one of the powerful methods to protect data from threats such as eavesdropping and theft. It is essential when storing confidential information such as credit card numbers, personal information of users, etc. Also, various data protection standards and regulations such as PCI DSS, GDPR, etc. require it. We have been researching how to make PostgreSQL more secure by implementing the built-in transparent data encryption for PostgreSQL. In this talk I will first introduce some studies about database encryption. Then, I'll provide some internal details of the implementing transparent data encryption and the performance evaluation result of the PoC based on our solution. It also covers the robust encryption key management, which is another important feature for data encryption. Various data protection standards and regulations such as PCI DSS, GDPR, etc. requires data encryption. There are methods to encrypt data stored in PostgreSQL, such as pgcrypto and file system level encryption using LUKS. However they don't satisfy the following properties of database encryption that are required by user and some security standards in practice: Transparent data encryption (TDE) Minimize performance impact Platform-independence We have been researching the built-in database encryption for PostgreSQL. Our solution consists of per-tablespace transparent data encryption while keeping minimize performance impact. We made a proof-of-concept code to evaluate our solution's usability and impact on the performance. In this talk, I will discuss about the principles of confidential information protection and database encryption methods. Then, I'll provide some internal details of built-in transparent data encryption of PostgreSQL and the evaluation results. It also covers the robust encryption key management, which is another important feature for data encryption.
 Hello, everyone. Thank you for coming to my talk. My name is Masako Sawada, and I work for NTT, and I came from Tokyo, Japan. In this session, I'd like to talk about transparent data encryption in PostgreSQL. This feature is kind of the new type of built-in NTT. I'm proposing this feature to PostgreSQL hackers, and the feature is under discussion right now. My talk focuses on the mainly PostgreSQL internals and my proposal of TDE. Before getting into my proposal deeply, let me explain about the basic database security and security threats. The database server is always in danger of many security threats, and there are many security attacks. For example, database security injection attacks and storage intercept and even your pink attacks between client and server, the many security threats. Encryption is the most famous security solution to protect the data stored in the database or server from being revealed. For example, when the disk theft or eavesdropping attacks. This data could exist on everywhere, for example, on disk, there is disk on disk, and on D-LAMS, and CPU cache, and even networks and application server. The security threats that we can protect completely depends on how and where we encrypt data. For example, if we encrypt on network, like for example, using SSL, we can protect the data from the package sniffing attack. But on the other hand, the data on disk is not encrypted. So if my just user logged in the user, logged in the operating system, and leads the file directory, so they can get the data. So the point here is that I think that we need to consider what security threats we need to protect. And then we can consider what the security solution or security future we need. And to protect the data from disk theft or leading the file directory, we can take the way of data at rest encryption. The data at rest encryption prevents the data visibility in the event of unauthorized access or disk theft. So that is we can protect the data from attacks by passing database access control layer. So for example, if the OS user leads the database file directory, or a Marsha's backup operator takes the backup and leads the data file directory, we can protect data from these attacks. However, because the data at rest encryption encrypts only data on the disk, the data is not encrypted while being used. So data on the shell buffer or data on the network is not encrypted. So that means it does not protect the attacks by Marsha's favorite user. For example, the post-result by user logged in database and do the search SQL command to read the database tables and they can get, because they have access privilege to that tables. Data at rest encryption doesn't protect data from these attacks. And data at rest encryption is often implemented as transparent data encryption, like TDE, because it encrypts data at very lower layer, like be close to the storage layer. So that can be implemented transparently to the user. So the user doesn't need to care about which database object is encrypted. And to use the data at rest encryption, so we also, there is another solution to provide the data at rest encryption. So that is one is the full disk encryption provided by operating system. For example, in Linux, we can use the encrypt. So what difference between the full disk encryption and in database transparent data encryption? I think the difference is there are two differences. The full disk encryption is completely platform dependent. I see, yeah, I'm sure there is another solution that is not, that is platform independent. But usually if we use Linux, we need to know about the encrypt. But if we use Windows, we need to another solution. And full disk encryption doesn't protect data from logging or users. So even if we use the full disk encryption, the logging or user can read the file and get unencrypted plain text from plain text of data. But if full-suggest has it in database transparent data encryption, it will be platform independent and we can protect the data from logging or users. It's very powerful. And another solution to provide transparent data encryption is to use PGCrypt. PGCrypt is actually a set of functions providing a set of cryptographic functions. It's very convenient and I like that. But it also requires the application code or SQL so that they call the PGCrypt functions. It's sometimes going to be a problem because, for example, when database migration, Japanese customers often ask me that they want to migrate the database from Oracle database to the Postgres. The Oracle database has its transparent data encryption, but Postgres does not have. So that means the customer needs to modify a lot of SQLs or application code so that they can, they call the PGCrypt functions. And the database triggers and the database views helps that, but there is a performance problem because using the PGCrypt, the database data needs to be decrypted every time it's accessed. I will show the performance result later. So in response to many of our clients or in Japan and the Postgres users in Japan, I'm trying to implement the built-in data at the rest of the encryption to the Postgres core. So my proposal is part-table space encryption. The user can create the encryption in every table space and then the user can create any database object like tables or database or indexes on that encryption in every table space. And this database object is automatically encrypted at rest. And the point of this proposal is it provides fine-grained encryption object controls. So the user can choose which object is encrypted or which object is not encrypted which can suppress the performance overhead for the table that don't need to be encrypted. And in this proposal, if we encrypt a table, it's also encrypted its indexes and its toss table and wall. Also, it encrypts other database objects such as system catalogues and temporary files. I will talk about, I will explain about system catalogues temporary files and encryption later. And the proposal is under the discussion on PGSH hackers proposed by me and Moom, a work for entity. And if you are interested in this feature, please join to that discussion. And firstly, I'd like to explain about where and when we encrypt data. This diagram roughly shows the Postgres basic IOR architecture. Postgres has its share buffer to share the table or index data among all Postgres processes. And operating system like Linux has its page cache. And then we have disk. And the background processes these Postgres, these pages from the share buffer and modify them. So, makes the buffers dirty. And the Postgres busy background writer periodically writes the dirty pages out of the current page cache. And dirty pages are eventually flushed to the disk. It's done by, for example, check pointer or current thread. Now our proposal is to encrypt the data between Postgres share buffer and OS page cache. So, each time, every time we write to the share buffer data to the page cache, we encrypt it. And when every time we load data to share buffer, we decrypt it. If we configure the correct size or good size of share buffer, it's less encryption and decryption. Because, for example, if the database fits in the share buffer, the buffer replacement doesn't happen many times. So, it's very good performance. And also, this method can protect the data from picking the file on disk. Because the data on the disk is encrypted. So, even if the malicious user logged in OS and read the database file directly, they cannot database data. But it's possibly repeated encryption and decryption of same data if the database size doesn't fit the share buffer. This is a performance variation result. The left side graph shows the database size fit in the share buffer, but right side doesn't fit. And the blue line shows the normal postgres. And the yellow line is postgres with TDE. I made a POC patch. And the red line shows the PGcrypt. As you can see on this result, in this evaluation, there is no big performance overhead even if we use TDE. And even if the database size does not fit the share buffer, it depends on the workload and share buffer size and database size. But in my evaluation, there was at most several percent performance degradation. And next is how to encrypt... Yes, please. Do you know if it was larger than the frame? Database size is less than share buffer, but the higher... So it doesn't happen any disk I own. Any questions? So next is how to encrypt. The two encrypt database data, I'm going to use the AES, Alba and Sainter standard, which is the symmetric key algorithm. And I'm going to use AES 256. The AES has several key lengths, but 256 is the longest key length of AES. And AES is a block cipher mode, and each block size is 16 bytes. And like PgCript, if we use the OpenSSL to encrypt data by AES, it's very powerful because OpenSSL can use ASNI. ASNI is a special CPU instruction set to encrypt and decrypt by AES very speedy. So that brings us to a very good performance. And we also need to consider which block cipher mode operation should be used. The CBC and XDS are often used for disk encryption. I'm going to use XDS for data encryption. Next is encryption key management. Tutiyaki hierarchy is a very unproferred solution for faster key rotation. Tutiyaki hierarchy uses two types of keys, master key and table space key, or data key. And we have one master key for database cluster, and we have table space keys per table space. And the master key must be stored outside of the database. And the master key is used to encrypt the table space keys, which is stored inside of the database. And the table space key is used to encrypt and decrypt the actual database data. Tutiyaki hierarchy shows its value when the key rotation. The key rotation is very important because, for example, the sum security standard like PsiDSS requires the periodically, periodically key rotation. So when we do the key rotation, what do we need to do in Tutiyaki hierarchy is to rotate the master key and re-encrypt the table space key with new master key. So that means we don't need to access, we don't need to modify the database data. And the table space keys are very small. Key size is 32 bytes if we use AES to 56. So key rotation will complete within seconds. It's very fast, rather than we re-encrypt the database file. And Oracle database and Mashiqer database provides a TDE, and they also use this hierarchy, Tutiyaki hierarchies. Key management is very important. We store table space keys inside of the database, but we must store the master key outside of the database. To store the master key, we can use key management systems. Key management systems or key management servers are dedicated software or services to manage encryption key very robustly. And there are several key management systems or key management servers provided by some companies, like, for example, AWS KMS or JMR key secure and Oracle Keyboard. Each solution provides, or each solution supports, different protocols. The most famous protocol for key management is KMIP. But some systems support only HTTPS or PKCS. And the key management system provides robustly key management. So it's very good. So users can leave the key management to them. If we seamlessly integrate Postgres with key management system, it's very powerful, because the master key is managed by the KMS robustly, and deviators don't need to care about the key management or key lifecycle. So that's why I propose to integrate the Postgres with key management systems. This diagram shows the left design after we have a key manager module in Postgres Core. Because each key management system supports different protocols, I think the probable architecture or extensive architecture works here in this case. And the Postgres has buffer manager and storage manager and maybe buffer manager and storage manager. And maybe we will have encryption module. And the key manager module is the new module I R did. And the key manager manages the two types of keys, the table space keys and the master keys. And also it manages the key management plug-in. So let me explain briefly the startup sequence of key management modules. When Postmaster startup, the key manager gets the master key from the external services via key management API, like get key API. And it store the master key into the shared memory. And after that, every Postgres server could access to the encrypted tables, need to read the encrypted table space key that's stored on the disk. And they decrypt the encrypted table space keys with a master key. And table space key is stored into local memory. And every process, when the process needs to access or decrypt or encrypt the table, they can use the table space key stored into that store into local memory. In this architecture, we cache the keys, especially for the we cache the master key in the shared buffer. The caching the master key on the shared buffer could be a risk of key leakage. For example, when memory dump, the memory dump could have master key data. Not good, but we can prevent that by using the mAdvice, mAdv.donDump plug, so that the memory address that stores the master key is not included in the memory dump. It's a very good solution. And caching the master key on the shared memory also could be a risk of key leakage when swap out. So when operating system swap out the shared memory to the swap area, the data could be written to the disk. It also can be a risk. So to prevent that, I simply can use MLOC to prevent the data from being swapped out to the swap area. And after the master key is cached into the shared buffer, so back-end process or every possible server process gets the encrypted table-specific keys from the disk and decrypts all of them using the master key. And next is wall encryption. The wall also could have user-sensitive data or the table data, so we need to encrypt wall. I think there are two possible approaches to encrypt wall. The first one is encrypt each wall blocks. The second is encrypt each wall record. In the first approach, if we encrypt the wall blocks, we insert wall record into wall buffer and then encrypt and write to the disk when writing to the disk. In this approach, it could require to encrypt wall data every commit time. So it's relatively not good, but the encryption can be performed by a synchronously, by a writeer, a writeer process. And in the approach number two, if we encrypt the power record, we encrypt the data when inserting into the wall buffer and when we're writing to the wall to the disk, we just write. In this approach, back when it's safe, always need to encrypt the wall when inserting into the wall buffer. But it doesn't encrypt the wall that is not pertaining to encrypted tables. In terms of security, it would be more secure if we use the same encryption key for a wall encryption as that used for a table. So that is if we encrypt the table with table-specific key A, we also encrypt the wall using the same key. It's more secure. So given that we use the same encryption key, only approach number two seems to be possible. Because when writing to the disk, writing the wall buffer to the disk, we cannot find out what data exists in the wall buffer exactly. So I'm going to use approach number two, so encrypt the wall record, but I also compared them in terms of performance between number one, the first approach and the second approach. I benchmarked some performance tests using several postgres configurations. And in this performance evaluation, I located the PGR directory to the template face in order to avoid the eye orbital neck. And for the graph, the y-axis shows the ratio of the execution time of normal postgres to the execution time of encrypted, enabled postgres. And the four blue bars in this graph from left side, the most left side is the normal postgres. And the next one is the postgres with encryption and it encrypts the wall block, each wall block. Next is the postgres with encryption and it encrypts the power wall record, each wall record. And the next two bars shows the employees the wall record encryption, but it's only half of wall is encrypted or one-fifth wall is encrypted. And the workload is in such heavy workload, each transaction insert a few records and commit. And as you can see on this result, the performance degradation in this evaluation, performance evaluation, there was at most 7% performance degradation. And the rest encryption has a good performance. This performance degradation might be big, but it's not common use case because I located the PGR directory on template face. So if I locate the PGR directory on the HDD hard disk, there is no big performance overhead. Even if the postgres encrypts the tables, data and the walls, there is no big performance overhead. So the conclusion, this means the encryption overhead is very small compared to this guy or we're writing. So conclusion here is that we can ignore the wall encryption overhead in the normal use case, very common case. Regardless of these approaches or wall power record encryption or power or broken encryption. But in the future, if we write the wall to the very fast storage, like for example, past memory, we might need to consider the wall encryption overhead. And in wall record encryption, we also need to consider what part of the wall we should encrypt. The wall format has been changed at postgres 9.5 or 9.6. And this is the example of the wall of the heap update when wall label is logical. Wall record always starts with the header part, header data, which consists of a wall X log record, which is a fixed size header, and could have some header data. And block data follows that, which could have full page image or tuple itself. And finally main data follows. That could have little operation, for example, Excel heap update or Excel heap insert or Excel heap delete. And the main data could have tuple data. So what I want to say is the user data could exist on the block data and the main data. So we need to encrypt at least two part. To encrypt the wall, I think there are some strategies, some approaches to approaches. So choice one is encrypt all of the wall records. But this approach is not good because I think it requires a big change of wall format and a big change of the code around writing to the wall. So I don't think it's a good idea. So the choice two, second choice is to encrypt only block data and main data, like this. It also needs to add a flag to the Excel record, which indicates that, hey, this wall record is encrypted. At first glance, this approach is very straightforward and looks good. However, we need to, we should consider the front end programs that could read the walls. For example, PG rewind and PG wall dump. These utility commands read not only the header data, but also the main data. So if we take this approach, we need to change these front end programs so that they can obtain all encryption keys from the external server. So that could be very hard part. So the choice three is, third choice is to move the first of the main data to after the header data, like this. The front end program does not read the full page image or the type of data itself. So if we move the first of main data to the header, just after the header data, it's very good. And we encrypt the remaining data. At least the front end programs like PG rewind and PG wall dump doesn't need to encrypt, doesn't need to decrypt var in this format. But I'm not sure that it's possible to rearrange the var format right now. Maybe there is historical reason we have main data at the very end of the var. And the next stage is temporary file encryption. Temporary files also could have user sensitive data because temporary file here means intermediate query result like sorting, while sorting, or logical decoding could also spill out the intermediate decoding data to the disk. So we need to encrypt. But temporary file is written by a single set of buffer. So we need to encrypt the individually temporary files. To encrypt temporary files, I think we can use something like disposable key to encrypt temporary data rather than using table space keys. Because the temporary file can have multiple table data, so we cannot use one table space key to the temporary files. A disposable key is generated randomly by background process before use, and it's leaves only during the process of time. It's not program because the temporary files need to be read the particular background. So we don't need to share the encryption key for temporary files among the other background. But I think one problem at the code level is there is no uniform interface to writing temporary files. So for example, the table slot which is used while sorting and logical replication, each module is called the write system call individually. So if we need to encrypt temporary files, we need to modify each modules, every modules. The patch is proposed by, I think Antonin from Cybertech proposed to uniform interface to write the temporary file, which is very good. And the last thing is encrypt system catalog encryption. The system catalog also could have user sensitive data. The one very good example is PG statistics. PG statistics has histogram columns and most conviles columns that could have a table data. And in addition to that, I think the PG statistics EXT and PG-PLOCK, PG-CRAS, also have user sensitive data. For example, PG-PLOCK has source code of a procedure, and sometimes the table name or attribute name also could be very sensitive data in some cases. So I think all system catalogs must be encrypted. And in this proposal, we can create the encrypted enable table space, and we can now create the table space on a particular table space. That means if we create the new database on encrypted enable table space, then all system catalogs are automatically encrypted. Yes, please. So there will be a new table space that just holds catalogs in your database? Yes. But when you create new database, we copy the template database. So you and I suppose if you modify that, please. Yes, yes, yes. Okay. This is a conclusion. In my proposal, it's encrypt partable space and buffer level and transparent data address encryption. It's less performance overhead, and it's also encrypt the wall and system catalogs and temporary files. And it's an employee that 2-tier key hierarchy, 2-tier key architecture for very faster key rotation. It's a very powerful solution. I also propose to integrate Postgres with key management systems. It's provided the more flexible and robust key management, which is often required by some security standard. And this is the kind of status of the TDE on Postgres development community. There are two different proposals of this area. Antonin Hoshikawa from Cybertech also proposed the cluster-wide data address encryption, which is under the development. And I think this is also a good future and very simple architecture. And the next is the I&Moon provides the partable space or partable data address encryption. This approach is trying to provide data address encryption, but the unit of encryption is different. The first approach is encrypt whole cluster. My second approach is more provide more fine-grained control. So I think it's very good if we have more discussion on hackers and pick the good things from each proposal. Yes, please. I guess it's not yet. Yeah, that's true. The system catalog encryption is very hard part. I think it's not a problem even if the system catalog is encrypted or this system catalog is not encrypted because we also have encrypted tables or not encrypted tables. I wonder if there are some strange circular problems if you have a special tablespace for the system catalogs, but in order to look it up, you need a hidden space catalog. So there's a bit of a circular problem. I wonder if you need to have a special tablespace which isn't in the catalogs or it is in the catalogs but you don't need to put it up. We have a default tablespace. Do you also have a default security tablespace? Yeah. I guess that makes logical sense. There's certain areas like being like being like a security efficient. Some of that is time to get from now to do some more models. So now you're saying like you need two methods of that. Yeah, I think it's the hard part could be global system catalogs like PGA tablespace or PGA database. That system catalogs are created at the unit-to-day-bit time and we cannot move the tablespace. So if we want to support that, I think we need to make the unit-to-day-bit to get the master key or encryption key. I think that could be a very hard part. You know what I'm asking? Is there a reason that the first one, I mean you guys do a lot of work on the second one which if you can get it to work sounds like it's better. Is there a reason that the first one does what we need? Yeah, I think that can more less performance overhead and the part of the space encryption is more less performance overhead. So you want to make the performance of the end-of-show. Yes. I think the first thing actually specifies the fee when you're stopping the instance. You don't have things like key rotation or whatever. It's very simple to do. I'm just saying that's what it does right now. What it was doing when I was looking at it. When you want to control the key rotation or whatever, the database, you need to do something more like the second one. I think that our table space is part of the thing that you control. The end-of-show key is table space. Because table spaces are more like a solution to physical data. I don't know the whole story. We are trying to, in general, we are trying to prevent people from using table space because it means all types of different backups and stuff like that. I think that's what we're trying to do. The end-of-show database has two types of transparent data encryption. One is table space and one is power column level. Table space encryption maybe works similar to this. Is there a reason not to add another logical grouping of options? We have replications. This group of tables will be replicated. Why don't we do something like that for encryption? It would be logical and it would be properly played under the table space. I see that businesses do not write up the abstraction. On those analysis, it seems horrible. Table space is one big file. Whereas for us, table space is a directory. It seems to make sense if you've got one big file to control whether each file is encrypted or not. It makes sense on this. It makes a little bit less sense on this. The global shared stuff where the abstraction is very simple because you could just define it as a table space problem. Even in the catalogs itself, that may not be that difficult. We have so much stuff that is shared across databases. You said that the encryption happens when we have writing stuff of the e-mail offers. How does this relate to the block-ups? Do you imagine storage engines for the table-ups that don't actually use shared bumpers? I'm thinking to provide the storage manager layer. Even if we have the heap, if it uses shared buffer, the data is automatically encrypted. You showed the wall. One of the options was that the metadata is not encrypted. In order for a replica server to decode this, it needs the key. You need to have the table-based OID in the bin. Where is the key? The key is stored into the print text file after it is encrypted. For example, the pg-based backup includes table-based keys. The master server and the standard server need to connect the same key manager system to get the same master key. Key rotation is not done inside the database. When you are decoding this stuff, at some point in the wall, it can be taken. How is the guy that is decoding this? The replication element. I don't think that is a big issue. Getting the wall encrypted in a way that comes with other tools like feature relikes, which are like standard tools running when the device is running or like a wall-damp or whatever, I think that is the trickier part. The moment that we rotate the key, it is kind of blurry. It doesn't happen in a certain LXN. We don't know when exactly the changes are. I think we need to have a CKS number for key generation like that. It should be written to the control files or check point records. When key rotation, we also need to write the wall that we rotated. CKS number is incremented and customized. Do you actually write the ID somewhere in the wall? When you are encrypting a buffer, do you say this block is encrypted with the 775? We marked buffer. This buffer is encrypted. Which key is used in the buffer? The encryption key is not changed. The table space key is not changed unless it is dropped. Even when the key rotation is not changed, the decrypted table space key is the same. Just masochist rotate. Any questions? Is it already time? Thank you.
 Hello, everyone. Thank you for coming to my talk. My name is Masako Sawada, and I work  for NTT, and I came from Tokyo, Japan. In this session, I'd like to talk about transparent  data encryption in PostgreSQL. This feature is kind of the new type of built-in NTT.  I'm proposing this feature to PostgreSQL hackers, and the feature is under discussion right now.  My talk focuses on the mainly PostgreSQL internals and my proposal of TDE.  Before getting into my proposal deeply, let me explain about the basic database security and security threats.  The database server is always in danger of many security threats, and there are many security attacks.  For example, database security injection attacks and storage intercept and even your pink attacks  between client and server, the many security threats.  Encryption is the most famous security solution to protect the data stored in the database  or server from being revealed. For example, when the disk theft or eavesdropping attacks.  This data could exist on everywhere, for example, on disk, there is disk on disk, and on D-LAMS,  and CPU cache, and even networks and application server.  The security threats that we can protect completely depends on how and where we encrypt data.  For example, if we encrypt on network, like for example, using SSL, we can protect the data from the package sniffing attack.  But on the other hand, the data on disk is not encrypted.  So if my just user logged in the user, logged in the operating system, and leads the file directory,  so they can get the data. So the point here is that I think that we need to consider what security threats we need to protect.  And then we can consider what the security solution or security future we need.  And to protect the data from disk theft or leading the file directory, we can take the way of data at rest encryption.  The data at rest encryption prevents the data visibility in the event of unauthorized access or disk theft.  So that is we can protect the data from attacks by passing database access control layer.  So for example, if the OS user leads the database file directory, or a Marsha's backup operator takes the backup and leads the data file directory,  we can protect data from these attacks.  However, because the data at rest encryption encrypts only data on the disk, the data is not encrypted while being used.  So data on the shell buffer or data on the network is not encrypted.  So that means it does not protect the attacks by Marsha's favorite user.  For example, the post-result by user logged in database and do the search SQL command to read the database tables and they can get,  because they have access privilege to that tables.  Data at rest encryption doesn't protect data from these attacks.  And data at rest encryption is often implemented as transparent data encryption, like TDE,  because it encrypts data at very lower layer, like be close to the storage layer.  So that can be implemented transparently to the user.  So the user doesn't need to care about which database object is encrypted.  And to use the data at rest encryption, so we also, there is another solution to provide the data at rest encryption.  So that is one is the full disk encryption provided by operating system.  For example, in Linux, we can use the encrypt.  So what difference between the full disk encryption and in database transparent data encryption?  I think the difference is there are two differences.  The full disk encryption is completely platform dependent.  I see, yeah, I'm sure there is another solution that is not, that is platform independent.  But usually if we use Linux, we need to know about the encrypt.  But if we use Windows, we need to another solution.  And full disk encryption doesn't protect data from logging or users.  So even if we use the full disk encryption, the logging or user can read the file and get unencrypted plain text  from plain text of data.  But if full-suggest has it in database transparent data encryption, it will be platform independent  and we can protect the data from logging or users.  It's very powerful.  And another solution to provide transparent data encryption is to use PGCrypt.  PGCrypt is actually a set of functions providing a set of cryptographic functions.  It's very convenient and I like that.  But it also requires the application code or SQL so that they call the PGCrypt functions.  It's sometimes going to be a problem because, for example, when database migration,  Japanese customers often ask me that they want to migrate the database from Oracle database to the Postgres.  The Oracle database has its transparent data encryption, but Postgres does not have.  So that means the customer needs to modify a lot of SQLs or application code  so that they can, they call the PGCrypt functions.  And the database triggers and the database views helps that, but there is a performance problem  because using the PGCrypt, the database data needs to be decrypted every time it's accessed.  I will show the performance result later.  So in response to many of our clients or in Japan and the Postgres users in Japan,  I'm trying to implement the built-in data at the rest of the encryption to the Postgres core.  So my proposal is part-table space encryption.  The user can create the encryption in every table space and then the user can create any database object  like tables or database or indexes on that encryption in every table space.  And this database object is automatically encrypted at rest.  And the point of this proposal is it provides fine-grained encryption object controls.  So the user can choose which object is encrypted or which object is not encrypted  which can suppress the performance overhead for the table that don't need to be encrypted.  And in this proposal, if we encrypt a table, it's also encrypted its indexes and its toss table and wall.  Also, it encrypts other database objects such as system catalogues and temporary files.  I will talk about, I will explain about system catalogues temporary files and encryption later.  And the proposal is under the discussion on PGSH hackers proposed by me and Moom, a work for entity.  And if you are interested in this feature, please join to that discussion.  And firstly, I'd like to explain about where and when we encrypt data.  This diagram roughly shows the Postgres basic IOR architecture.  Postgres has its share buffer to share the table or index data among all Postgres processes.  And operating system like Linux has its page cache.  And then we have disk.  And the background processes these Postgres, these pages from the share buffer and modify them.  So, makes the buffers dirty.  And the Postgres busy background writer periodically writes the dirty pages out of the current page cache.  And dirty pages are eventually flushed to the disk. It's done by, for example, check pointer or current thread.  Now our proposal is to encrypt the data between Postgres share buffer and OS page cache.  So, each time, every time we write to the share buffer data to the page cache, we encrypt it.  And when every time we load data to share buffer, we decrypt it.  If we configure the correct size or good size of share buffer, it's less encryption and decryption.  Because, for example, if the database fits in the share buffer, the buffer replacement doesn't happen many times.  So, it's very good performance.  And also, this method can protect the data from picking the file on disk.  Because the data on the disk is encrypted.  So, even if the malicious user logged in OS and read the database file directly, they cannot database data.  But it's possibly repeated encryption and decryption of same data if the database size doesn't fit the share buffer.  This is a performance variation result.  The left side graph shows the database size fit in the share buffer, but right side doesn't fit.  And the blue line shows the normal postgres.  And the yellow line is postgres with TDE.  I made a POC patch.  And the red line shows the PGcrypt.  As you can see on this result, in this evaluation, there is no big performance overhead even if we use TDE.  And even if the database size does not fit the share buffer, it depends on the workload and share buffer size and database size.  But in my evaluation, there was at most several percent performance degradation.  And next is how to encrypt...  Yes, please.  Do you know if it was larger than the frame?  Database size is less than share buffer, but the higher...  So it doesn't happen any disk I own.  Any questions?  So next is how to encrypt.  The two encrypt database data, I'm going to use the AES, Alba and Sainter standard, which is the symmetric key algorithm.  And I'm going to use AES 256.  The AES has several key lengths, but 256 is the longest key length of AES.  And AES is a block cipher mode, and each block size is 16 bytes.  And like PgCript, if we use the OpenSSL to encrypt data by AES, it's very powerful because OpenSSL can use ASNI.  ASNI is a special CPU instruction set to encrypt and decrypt by AES very speedy.  So that brings us to a very good performance.  And we also need to consider which block cipher mode operation should be used.  The CBC and XDS are often used for disk encryption.  I'm going to use XDS for data encryption.  Next is encryption key management.  Tutiyaki hierarchy is a very unproferred solution for faster key rotation.  Tutiyaki hierarchy uses two types of keys, master key and table space key, or data key.  And we have one master key for database cluster, and we have table space keys per table space.  And the master key must be stored outside of the database.  And the master key is used to encrypt the table space keys, which is stored inside of the database.  And the table space key is used to encrypt and decrypt the actual database data.  Tutiyaki hierarchy shows its value when the key rotation.  The key rotation is very important because, for example, the sum security standard like PsiDSS  requires the periodically, periodically key rotation.  So when we do the key rotation, what do we need to do  in Tutiyaki hierarchy is to rotate the master key and re-encrypt the table space key with new master key.  So that means we don't need to access, we don't need to modify the database data.  And the table space keys are very small. Key size is 32 bytes if we use AES to 56.  So key rotation will complete within seconds. It's very fast, rather than we re-encrypt the database file.  And Oracle database and Mashiqer database provides a TDE, and they also use this hierarchy, Tutiyaki hierarchies.  Key management is very important. We store table space keys inside of the database,  but we must store the master key outside of the database.  To store the master key, we can use key management systems.  Key management systems or key management servers are dedicated software or services to manage encryption key very robustly.  And there are several key management systems or key management servers provided by some companies,  like, for example, AWS KMS or JMR key secure and Oracle Keyboard.  Each solution provides, or each solution supports, different protocols.  The most famous protocol for key management is KMIP.  But some systems support only HTTPS or PKCS.  And the key management system provides robustly key management.  So it's very good. So users can leave the key management to them.  If we seamlessly integrate Postgres with key management system, it's very powerful,  because the master key is managed by the KMS robustly, and deviators don't need to care about the key management or key lifecycle.  So that's why I propose to integrate the Postgres with key management systems.  This diagram shows the left design after we have a key manager module in Postgres Core.  Because each key management system supports different protocols,  I think the probable architecture or extensive architecture works here in this case.  And the Postgres has buffer manager and storage manager and maybe buffer manager and storage manager.  And maybe we will have encryption module.  And the key manager module is the new module I R did.  And the key manager manages the two types of keys, the table space keys and the master keys.  And also it manages the key management plug-in.  So let me explain briefly the startup sequence of key management modules.  When Postmaster startup, the key manager gets the master key from the external services via key management API,  like get key API.  And it store the master key into the shared memory.  And after that, every Postgres server could access to the encrypted tables,  need to read the encrypted table space key that's stored on the disk.  And they decrypt the encrypted table space keys with a master key.  And table space key is stored into local memory.  And every process, when the process needs to access or decrypt or encrypt the table,  they can use the table space key stored into that store into local memory.  In this architecture, we cache the keys, especially for the we cache the master key in the shared buffer.  The caching the master key on the shared buffer could be a risk of key leakage.  For example, when memory dump, the memory dump could have master key data.  Not good, but we can prevent that by using the mAdvice, mAdv.donDump plug,  so that the memory address that stores the master key is not included in the memory dump.  It's a very good solution.  And caching the master key on the shared memory also could be a risk of key leakage when swap out.  So when operating system swap out the shared memory to the swap area,  the data could be written to the disk. It also can be a risk.  So to prevent that, I simply can use MLOC to prevent the data from being swapped out to the swap area.  And after the master key is cached into the shared buffer,  so back-end process or every possible server process gets the encrypted table-specific keys from the disk  and decrypts all of them using the master key.  And next is wall encryption.  The wall also could have user-sensitive data or the table data, so we need to encrypt wall.  I think there are two possible approaches to encrypt wall.  The first one is encrypt each wall blocks.  The second is encrypt each wall record.  In the first approach, if we encrypt the wall blocks, we insert wall record into wall buffer  and then encrypt and write to the disk when writing to the disk.  In this approach, it could require to encrypt wall data every commit time.  So it's relatively not good, but the encryption can be performed by a synchronously, by a writeer, a writeer process.  And in the approach number two, if we encrypt the power record, we encrypt the data when inserting into the wall buffer  and when we're writing to the wall to the disk, we just write.  In this approach, back when it's safe, always need to encrypt the wall when inserting into the wall buffer.  But it doesn't encrypt the wall that is not pertaining to encrypted tables.  In terms of security, it would be more secure if we use the same encryption key for a wall encryption as that used for a table.  So that is if we encrypt the table with table-specific key A, we also encrypt the wall using the same key.  It's more secure.  So given that we use the same encryption key, only approach number two seems to be possible.  Because when writing to the disk, writing the wall buffer to the disk, we cannot find out what data exists in the wall buffer exactly.  So I'm going to use approach number two, so encrypt the wall record, but I also compared them in terms of performance between number one, the first approach and the second approach.  I benchmarked some performance tests using several postgres configurations.  And in this performance evaluation, I located the PGR directory to the template face in order to avoid the eye orbital neck.  And for the graph, the y-axis shows the ratio of the execution time of normal postgres to the execution time of encrypted, enabled postgres.  And the four blue bars in this graph from left side, the most left side is the normal postgres.  And the next one is the postgres with encryption and it encrypts the wall block, each wall block.  Next is the postgres with encryption and it encrypts the power wall record, each wall record.  And the next two bars shows the employees the wall record encryption, but it's only half of wall is encrypted or one-fifth wall is encrypted.  And the workload is in such heavy workload, each transaction insert a few records and commit.  And as you can see on this result, the performance degradation in this evaluation, performance evaluation, there was at most 7% performance degradation.  And the rest encryption has a good performance.  This performance degradation might be big, but it's not common use case because I located the PGR directory on template face.  So if I locate the PGR directory on the HDD hard disk, there is no big performance overhead.  Even if the postgres encrypts the tables, data and the walls, there is no big performance overhead.  So the conclusion, this means the encryption overhead is very small compared to this guy or we're writing.  So conclusion here is that we can ignore the wall encryption overhead in the normal use case, very common case.  Regardless of these approaches or wall power record encryption or power or broken encryption.  But in the future, if we write the wall to the very fast storage, like for example, past memory, we might need to consider the wall encryption overhead.  And in wall record encryption, we also need to consider what part of the wall we should encrypt.  The wall format has been changed at postgres 9.5 or 9.6.  And this is the example of the wall of the heap update when wall label is logical.  Wall record always starts with the header part, header data, which consists of a wall X log record, which is a fixed size header, and could have some header data.  And block data follows that, which could have full page image or tuple itself. And finally main data follows.  That could have little operation, for example, Excel heap update or Excel heap insert or Excel heap delete.  And the main data could have tuple data. So what I want to say is the user data could exist on the block data and the main data.  So we need to encrypt at least two part.  To encrypt the wall, I think there are some strategies, some approaches to approaches.  So choice one is encrypt all of the wall records.  But this approach is not good because I think it requires a big change of wall format and a big change of the code around writing to the wall.  So I don't think it's a good idea.  So the choice two, second choice is to encrypt only block data and main data, like this.  It also needs to add a flag to the Excel record, which indicates that, hey, this wall record is encrypted.  At first glance, this approach is very straightforward and looks good.  However, we need to, we should consider the front end programs that could read the walls.  For example, PG rewind and PG wall dump.  These utility commands read not only the header data, but also the main data.  So if we take this approach, we need to change these front end programs so that they can obtain all encryption keys from the external server.  So that could be very hard part.  So the choice three is, third choice is to move the first of the main data to after the header data, like this.  The front end program does not read the full page image or the type of data itself.  So if we move the first of main data to the header, just after the header data, it's very good.  And we encrypt the remaining data.  At least the front end programs like PG rewind and PG wall dump doesn't need to encrypt, doesn't need to decrypt var in this format.  But I'm not sure that it's possible to rearrange the var format right now.  Maybe there is historical reason we have main data at the very end of the var.  And the next stage is temporary file encryption.  Temporary files also could have user sensitive data because temporary file here means intermediate query result like sorting, while sorting,  or logical decoding could also spill out the intermediate decoding data to the disk.  So we need to encrypt.  But temporary file is written by a single set of buffer.  So we need to encrypt the individually temporary files.  To encrypt temporary files, I think we can use something like disposable key to encrypt temporary data rather than using table space keys.  Because the temporary file can have multiple table data, so we cannot use one table space key to the temporary files.  A disposable key is generated randomly by background process before use, and it's leaves only during the process of time.  It's not program because the temporary files need to be read the particular background.  So we don't need to share the encryption key for temporary files among the other background.  But I think one problem at the code level is there is no uniform interface to writing temporary files.  So for example, the table slot which is used while sorting and logical replication,  each module is called the write system call individually.  So if we need to encrypt temporary files, we need to modify each modules, every modules.  The patch is proposed by, I think Antonin from Cybertech proposed to uniform interface to write the temporary file, which is very good.  And the last thing is encrypt system catalog encryption.  The system catalog also could have user sensitive data.  The one very good example is PG statistics.  PG statistics has histogram columns and most conviles columns that could have a table data.  And in addition to that, I think the PG statistics EXT and PG-PLOCK, PG-CRAS, also have user sensitive data.  For example, PG-PLOCK has source code of a procedure, and sometimes the table name or attribute name also could be very sensitive data in some cases.  So I think all system catalogs must be encrypted.  And in this proposal, we can create the encrypted enable table space, and we can now create the table space on a particular table space.  That means if we create the new database on encrypted enable table space, then all system catalogs are automatically encrypted.  Yes, please.  So there will be a new table space that just holds catalogs in your database?  Yes.  But when you create new database, we copy the template database. So you and I suppose if you modify that, please.  Yes, yes, yes.  Okay. This is a conclusion.  In my proposal, it's encrypt partable space and buffer level and transparent data address encryption.  It's less performance overhead, and it's also encrypt the wall and system catalogs and temporary files.  And it's an employee that 2-tier key hierarchy, 2-tier key architecture for very faster key rotation.  It's a very powerful solution.  I also propose to integrate Postgres with key management systems. It's provided the more flexible and robust key management, which is often required by some security standard.  And this is the kind of status of the TDE on Postgres development community. There are two different proposals of this area.  Antonin Hoshikawa from Cybertech also proposed the cluster-wide data address encryption, which is under the development.  And I think this is also a good future and very simple architecture.  And the next is the I&Moon provides the partable space or partable data address encryption.  This approach is trying to provide data address encryption, but the unit of encryption is different.  The first approach is encrypt whole cluster.  My second approach is more provide more fine-grained control.  So I think it's very good if we have more discussion on hackers and pick the good things from each proposal.  Yes, please.  I guess it's not yet.  Yeah, that's true. The system catalog encryption is very hard part.  I think it's not a problem even if the system catalog is encrypted or this system catalog is not encrypted because we also have encrypted tables or not encrypted tables.  I wonder if there are some strange circular problems if you have a special tablespace for the system catalogs, but in order to look it up, you need a hidden space catalog.  So there's a bit of a circular problem. I wonder if you need to have a special tablespace which isn't in the catalogs or it is in the catalogs but you don't need to put it up.  We have a default tablespace. Do you also have a default security tablespace?  Yeah.  I guess that makes logical sense.  There's certain areas like being like being like a security efficient. Some of that is time to get from now to do some more models.  So now you're saying like you need two methods of that.  Yeah, I think it's the hard part could be global system catalogs like PGA tablespace or PGA database. That system catalogs are created at the unit-to-day-bit time and we cannot move the tablespace.  So if we want to support that, I think we need to make the unit-to-day-bit to get the master key or encryption key. I think that could be a very hard part.  You know what I'm asking? Is there a reason that the first one, I mean you guys do a lot of work on the second one which if you can get it to work sounds like it's better. Is there a reason that the first one does what we need?  Yeah, I think that can more less performance overhead and the part of the space encryption is more less performance overhead.  So you want to make the performance of the end-of-show.  Yes.  I think the first thing actually specifies the fee when you're stopping the instance. You don't have things like key rotation or whatever. It's very simple to do.  I'm just saying that's what it does right now. What it was doing when I was looking at it.  When you want to control the key rotation or whatever, the database, you need to do something more like the second one.  I think that our table space is part of the thing that you control. The end-of-show key is table space. Because table spaces are more like a solution to physical data.  I don't know the whole story. We are trying to, in general, we are trying to prevent people from using table space because it means all types of different backups and stuff like that.  I think that's what we're trying to do.  The end-of-show database has two types of transparent data encryption. One is table space and one is power column level. Table space encryption maybe works similar to this.  Is there a reason not to add another logical grouping of options? We have replications. This group of tables will be replicated. Why don't we do something like that for encryption?  It would be logical and it would be properly played under the table space. I see that businesses do not write up the abstraction.  On those analysis, it seems horrible. Table space is one big file. Whereas for us, table space is a directory. It seems to make sense if you've got one big file to control whether each file is encrypted or not.  It makes sense on this. It makes a little bit less sense on this.  The global shared stuff where the abstraction is very simple because you could just define it as a table space problem. Even in the catalogs itself, that may not be that difficult.  We have so much stuff that is shared across databases.  You said that the encryption happens when we have writing stuff of the e-mail offers. How does this relate to the block-ups?  Do you imagine storage engines for the table-ups that don't actually use shared bumpers?  I'm thinking to provide the storage manager layer. Even if we have the heap, if it uses shared buffer, the data is automatically encrypted.  You showed the wall. One of the options was that the metadata is not encrypted.  In order for a replica server to decode this, it needs the key. You need to have the table-based OID in the bin. Where is the key?  The key is stored into the print text file after it is encrypted. For example, the pg-based backup includes table-based keys. The master server and the standard server need to connect the same key manager system to get the same master key.  Key rotation is not done inside the database.  When you are decoding this stuff, at some point in the wall, it can be taken. How is the guy that is decoding this?  The replication element.  I don't think that is a big issue. Getting the wall encrypted in a way that comes with other tools like feature relikes, which are like standard tools running when the device is running or like a wall-damp or whatever, I think that is the trickier part.  The moment that we rotate the key, it is kind of blurry. It doesn't happen in a certain LXN. We don't know when exactly the changes are.  I think we need to have a CKS number for key generation like that. It should be written to the control files or check point records.  When key rotation, we also need to write the wall that we rotated. CKS number is incremented and customized.  Do you actually write the ID somewhere in the wall? When you are encrypting a buffer, do you say this block is encrypted with the 775?  We marked buffer. This buffer is encrypted.  Which key is used in the buffer?  The encryption key is not changed. The table space key is not changed unless it is dropped.  Even when the key rotation is not changed, the decrypted table space key is the same.  Just masochist rotate.  Any questions? Is it already time? Thank you.
Transparent_Data_at_Rest_-_Masahiko_Sawada_0_47.png Transparent_Data_at_Rest_-_Masahiko_Sawada_48_131.png Transparent_Data_at_Rest_-_Masahiko_Sawada_132_1343.png Transparent_Data_at_Rest_-_Masahiko_Sawada_1344_2195.png Transparent_Data_at_Rest_-_Masahiko_Sawada_2196_4895.png Transparent_Data_at_Rest_-_Masahiko_Sawada_4896_7799.png Transparent_Data_at_Rest_-_Masahiko_Sawada_7800_10331.png Transparent_Data_at_Rest_-_Masahiko_Sawada_10332_10379.png Transparent_Data_at_Rest_-_Masahiko_Sawada_10380_10463.png Transparent_Data_at_Rest_-_Masahiko_Sawada_10464_10571.png Transparent_Data_at_Rest_-_Masahiko_Sawada_10572_10763.png Transparent_Data_at_Rest_-_Masahiko_Sawada_10764_10871.png Transparent_Data_at_Rest_-_Masahiko_Sawada_10872_10955.png Transparent_Data_at_Rest_-_Masahiko_Sawada_10956_11063.png Transparent_Data_at_Rest_-_Masahiko_Sawada_11064_11147.png Transparent_Data_at_Rest_-_Masahiko_Sawada_11148_11255.png Transparent_Data_at_Rest_-_Masahiko_Sawada_11256_11447.png Transparent_Data_at_Rest_-_Masahiko_Sawada_11448_11555.png Transparent_Data_at_Rest_-_Masahiko_Sawada_11556_11639.png Transparent_Data_at_Rest_-_Masahiko_Sawada_11640_11747.png Transparent_Data_at_Rest_-_Masahiko_Sawada_11748_11831.png Transparent_Data_at_Rest_-_Masahiko_Sawada_11832_11939.png Transparent_Data_at_Rest_-_Masahiko_Sawada_11940_12131.png Transparent_Data_at_Rest_-_Masahiko_Sawada_12132_12239.png Transparent_Data_at_Rest_-_Masahiko_Sawada_12240_12323.png Transparent_Data_at_Rest_-_Masahiko_Sawada_12324_12431.png Transparent_Data_at_Rest_-_Masahiko_Sawada_12432_12515.png Transparent_Data_at_Rest_-_Masahiko_Sawada_12516_12599.png Transparent_Data_at_Rest_-_Masahiko_Sawada_12600_13007.png Transparent_Data_at_Rest_-_Masahiko_Sawada_13008_15395.png Transparent_Data_at_Rest_-_Masahiko_Sawada_15396_15443.png Transparent_Data_at_Rest_-_Masahiko_Sawada_15444_15563.png Transparent_Data_at_Rest_-_Masahiko_Sawada_15564_15743.png Transparent_Data_at_Rest_-_Masahiko_Sawada_15744_15851.png Transparent_Data_at_Rest_-_Masahiko_Sawada_15852_15935.png Transparent_Data_at_Rest_-_Masahiko_Sawada_15936_16019.png Transparent_Data_at_Rest_-_Masahiko_Sawada_16020_16139.png Transparent_Data_at_Rest_-_Masahiko_Sawada_16140_16211.png Transparent_Data_at_Rest_-_Masahiko_Sawada_16212_16583.png Transparent_Data_at_Rest_-_Masahiko_Sawada_16584_16823.png Transparent_Data_at_Rest_-_Masahiko_Sawada_16824_17135.png Transparent_Data_at_Rest_-_Masahiko_Sawada_17136_19343.png Transparent_Data_at_Rest_-_Masahiko_Sawada_19344_21179.png Transparent_Data_at_Rest_-_Masahiko_Sawada_21180_21323.png Transparent_Data_at_Rest_-_Masahiko_Sawada_21324_22439.png Transparent_Data_at_Rest_-_Masahiko_Sawada_22440_22571.png Transparent_Data_at_Rest_-_Masahiko_Sawada_22572_23735.png Transparent_Data_at_Rest_-_Masahiko_Sawada_23736_24035.png Transparent_Data_at_Rest_-_Masahiko_Sawada_24036_24119.png Transparent_Data_at_Rest_-_Masahiko_Sawada_24120_24227.png Transparent_Data_at_Rest_-_Masahiko_Sawada_24228_24311.png Transparent_Data_at_Rest_-_Masahiko_Sawada_24312_24623.png Transparent_Data_at_Rest_-_Masahiko_Sawada_24624_24671.png Transparent_Data_at_Rest_-_Masahiko_Sawada_24672_26747.png Transparent_Data_at_Rest_-_Masahiko_Sawada_26748_26891.png Transparent_Data_at_Rest_-_Masahiko_Sawada_26892_28103.png Transparent_Data_at_Rest_-_Masahiko_Sawada_28104_28163.png Transparent_Data_at_Rest_-_Masahiko_Sawada_28164_30671.png Transparent_Data_at_Rest_-_Masahiko_Sawada_30672_32675.png Transparent_Data_at_Rest_-_Masahiko_Sawada_32676_34439.png Transparent_Data_at_Rest_-_Masahiko_Sawada_34440_37415.png Transparent_Data_at_Rest_-_Masahiko_Sawada_37416_40103.png Transparent_Data_at_Rest_-_Masahiko_Sawada_40104_41735.png Transparent_Data_at_Rest_-_Masahiko_Sawada_41736_44999.png Transparent_Data_at_Rest_-_Masahiko_Sawada_45000_46835.png Transparent_Data_at_Rest_-_Masahiko_Sawada_46836_48875.png Transparent_Data_at_Rest_-_Masahiko_Sawada_48876_49883.png Transparent_Data_at_Rest_-_Masahiko_Sawada_49884_51623.png Transparent_Data_at_Rest_-_Masahiko_Sawada_51624_51851.png Transparent_Data_at_Rest_-_Masahiko_Sawada_51852_52511.png Transparent_Data_at_Rest_-_Masahiko_Sawada_52512_53183.png Transparent_Data_at_Rest_-_Masahiko_Sawada_53184_53267.png Transparent_Data_at_Rest_-_Masahiko_Sawada_53268_53315.png Transparent_Data_at_Rest_-_Masahiko_Sawada_53316_54287.png Transparent_Data_at_Rest_-_Masahiko_Sawada_54288_57095.png Transparent_Data_at_Rest_-_Masahiko_Sawada_57096_59807.png Transparent_Data_at_Rest_-_Masahiko_Sawada_59808_61163.png Transparent_Data_at_Rest_-_Masahiko_Sawada_61164_76763.png Transparent_Data_at_Rest_-_Masahiko_Sawada_76764_76895.png Transparent_Data_at_Rest_-_Masahiko_Sawada_76896_83243.png Transparent_Data_at_Rest_-_Masahiko_Sawada_83244_83291.png Transparent_Data_at_Rest_-_Masahiko_Sawada_83292_83405.png