SMB3 UNIX Extensions
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Allison, Jeremy
The SMB3 POSIX Extensions, a set of protocol extensions to allow for optimal Linux and Unix interoperability with NAS and Cloud file servers, have evolved over the past year, with test implementations in Samba and in the Linux kernel. These extensions address various compatibility problems for Linux and Unix clients (such as case sensitivity, locking, delete semantics and mode bits among others). This presentation will review the state of the protocol extensions, what was learned in the implementations in Samba and provide an opportunity for feedback and suggestions for future improvements to the POSIX extensions.
 Okay, please welcome Jeremy and another presentation about Samba. Thanks very much. So I'm going to try and speak up because there's no amplification. Can you hear me okay at the back? Yeah? Okay, cool. So my name is Jeremy Allison. I work for Google in the open source program office. But I really want to make it very clear there's my Samba.org email address on the presentation. And this is not a Google badge presentation, just in case there's any reporters hiding in the corners or something. This is not an official Google presentation. This is not what Google thinks. This is not what Google says, etc. Any Google lawyers in the room should run screaming for the hills because nobody's reviewed it or seen it. So, having said that, let's dig into SMB3 Unix Extensions. This is going to be a reasonably technical talk. It's not going to dive down as deeply as I could. If you want more details, please feel free to ask questions or ask some more detail at any point. Stick your hands up and I'm happy to be interrupted and take questions in the middle of the talk. So, first of all, let's go back one step and say, does everyone here in the room know what Samba is? Anyone doesn't know what Samba is? Okay, cool. And here, SMB3, is there anyone here who doesn't know what SMB3 is? Okay, so we have one person. That's always good to say. So, SMB3 is the third version of a protocol originally developed by IBM, taken on by Microsoft. And it is the protocol that Windows uses to talk to Windows to access files and directories. So, if you mount, you know, if you bring up your Windows Explorer and you attach to another server, you can see the files and directories, SMB is the mechanism that's allowing that to happen over a network. And SMB3 is the third version of that protocol that Microsoft have developed. And it's actually quite nice, I think. So, why extend SMB3? Why add Unix Extensions to it? Well, SMB3 is a Windows to Windows protocol, so it doesn't have much accommodation for specific things that Unix machines need. So, originally back in the SMB one days, various vendors wanted to add features to SMB. Now, SMB doesn't have a standards organization. It only has Microsoft, and what Microsoft does pretty much becomes the standard. So, it's evolved only to fit the needs of Windows to Windows. So, other vendors who needed to interoperate with Windows, sort of like HP and SCO before they went insane and started suing IBM of Linux, etc. They thought, well, we already have an SMB redirector in our kernel, you know, which is what allows us to mount the Windows systems. So, it wouldn't be so hard to extend this to talk to the SMB servers that we also have on our Unix systems. And, SAMBO was the most ubiquitous server out there and was the easiest to extend, because you could just type the code in. So, SCO, HP and a few of the vendors started proposing additions to SMB1 to make it work better between Unix to Unix. Because if you're aware, Windows to Windows has case insensitive path names, deleting filesworks differently, renaming filesworks differently, file locking is completely different, the semantics don't match. So, by extending SMB1, they could make the semantics match Unix to Unix much better, so that Unix applications would get the semantics that they expected when they mounted a remote file system. And, excuse me, I'm going to have a, is the Ganesha maintainer in the room? He was earlier. Oh, okay, if he's not here, I can get away with more with my NFS jokes. So, SMB1 Unix was actually better at Unix to Unix, posix to posix semantics than NFS. NFS has a boatload of hideous things in it, like silly renames, that basically hide the fact that it's a stateless protocol, at least NFS v3, and doesn't work very well posix to posix. Well, SMB1 with Unix extensions had better fidelity for Unix to Unix than NFS does. So, plus, you know, NFS sucks and it should just die. So, I like this slide, I made it because you can actually choose who you are depending on which protocol you like. You know, you could say that NFS is Kirk's screaming, Kaan! And SMB is saying, from hell's heart, I spit at thee. Or it could be the other way around. And the other thing I really love is, meanwhile in the cloud, we will add your biological and technological distinctiveness to our own, your culture will adopt to service us, resistance is futile. And no better summation of the cloud vendor, storage vendors, I could possibly imagine. So, yes, so Kaan and Kirk are having fun there while the Borg are out there waiting for us all. What happened with the SMB1 Unix extensions? Well, they became a hideous train wreck. Why was this? Partly my fault. Partly the fault of a now Microsoft employee, because I wouldn't say no to him. The issue was there was no real oversight on what got added. And it was, well, if it works, it's good. It doesn't have to do protocol design, it just has to work, which is, you know, you can see the point in that. So it was an enabler for some security problems. And I'm going to go into those in a lot more detail later on. But allowing server-side sim links, in other words, sim links that are created on the server-side file system by the client, that the server then follows is a recipe for security disaster. And I will explain why in a little while. So we just added anything we thought of. And we want to have a new input level for the POSIX semantics. We want to have new input levels for POSIX semantics, even if there was a Windows input level that was kind of similar. Yeah, let's just throw a boatload of more input levels in. So, like I say, it was just ad hoc thrown together. Having said that, it had some very useful effects. In chatting to the Microsoft engineers who implemented SMB2 transport encryption, they told me that they originally had terrible pushback from their internal security teams who said, no, no, don't do this at your file system layer. We've got IPsec to handle all security in the network layer. And as everyone knows, IPsec is widely adopted and easy to set up. So that was a joke for people who don't have to support IPsec. It's terrible and doesn't work very well. So they were able to add transport level encryption directly into SMB1 just by saying, well, look at those open source guys. They've got transport level encryption. We need it for feature parity. So that was actually quite useful. And the SMB3 transport level encryption is actually very well implemented and secure. So I'm happy about that. So monorail, monorail, who needs a monorail? Sorry, Simpsons. A clean slate. We can design something glowing, sliding into the future like the Springfield monorail. But one thing that we came away with, and one of the reasons this has taken a great deal of time, also because I'm kind of lazy and don't do things unless people are shouting at me, but I finally came down with a philosophy of utter minimalism, which was that rather than the train wreck that was SMB1, Unix Extensions, the SMB3 Unix Extensions would only add things that we could not do any other way. So don't reinvent anything that SMB3 already has. Just tweak and add in very small ways. Now, because the SMB3 protocol is actually quite nicely designed, that made this not too hard, actually. So what does this mean? Cut down on the extra stuff you need to add. Don't duplicate stuff. If you have an info level in Windows that returns create timestamps and whatever, don't add another POSIX level that does the same thing. You've already got that data, get it the Windows way. Allow semantics that aren't 100% POSIX, but are kind of close enough. And I'll explain a little more what that means in a later slide. Reuse the existing SMB3 features that Windows uses. So what does that mean? It means that we don't have to invent our own mechanism for encryption. Windows already has ACLs and they love them or hate them, and I certainly do hate them. They are the most widely deployed ACL mechanism on the planet. Everybody understands. Nobody understands Windows ACLs, but everybody uses Windows ACLs. That's probably the best way to put it. So they're very common and we should adopt the same. Microsoft already added some features to SMB3, not widely used info levels and such, that would make their NFS server work against a Windows back-in-file system. So when we have those mechanisms that Microsoft already invented, just reuse them. Don't try and invent our own. Don't try and say ours are better. Just say, hey, this is the Windows way of setting things. Let's use that. So what did Apple do? Let's contrast the Apple approach. Apple standardized on SMB2 and above for all of their file sharing needs. They replaced their proprietary AFP file sharing protocol. But they kind of did it in a SMB1 Unix extension similar way. In other words, I think they made a real mess of it. So what they did was they said, oh, we're just going to keep a regular SMB1 connection, and then at the point that you actually attach to the network share, the first operation for an Apple server is a special magic open. And when you send that special magic open with a create context, I'll get into what create context, in a minute, of AAPL, then the server knows I'm a Mac client, and therefore I will behave differently and do things, do different things. Now that's basically the same kind of magic action horror, send an info level that has no meaning other than to change the behavior of the server long term. And I'm trying to avoid that in SMB3 Unix extensions. So this magic action means the server must now respond differently to various different operations. And the effect of this is that poor Ralph from Cernet, who was the original AFP maintainer for the AFP server on Linux, who now maintains the Apple compatibility with Samba, Samba used to claim we were bug for bug compatible with Windows. Now we have to claim we're bug for bug compatible with Windows and Apple. So he has to duplicate the same bugs that the Mac server has in order to make the Mac's clients run. So this is a pain. So what do we end up with? How should it look? So the nice thing about SMB3 Negotiate is that you have this concept of negotiate concepts, context, sorry, which are basically arbitrary little pieces of data you can attach to the first packet. So we added a new one. I originally thought we could get away without adding this context. So you send an empty context saying, hey, I'm a client that is capable of talking POSIX SMB3 Unix extensions. And if the server then replies saying, yes, I also can speak SMB3 Unix extensions, then the client knows it's talking to a Unix capable server. So the nice thing about this is why is this there? Well, firstly, if the server doesn't respond with that, the client knows not to try any other POSIX or Unix extensions. The protocol spec says that they are non-mandatory. You can send as many of these contexts as you like, and the server only responds to the ones it understands, which is nice. And here's the kicker. Here's the reason why it's there. Theoretically, you don't need it because every single operation in SMB3 has to have a handle, and a handle has to have context, and those contexts could specify SMB3 Unix. So theoretically, I originally thought we can get away with simply doing context on the creates rather than on the initial negotiate. And then one of the Microsoft engineers in discussion on this pointed out that we really need a create context because if you start sending these Unix commands to the Windows server or to a server that you think might support them, and it doesn't respond or rather responds to the rest of the create, but not the Unix part, you don't know whether it doesn't understand what you're sending it and is ignoring it or whether it understands what you're sending it, but is choosing to reject it on this particular operation. And so that's what the negotiate context is for. If the server responds to the initial negotiate context, you know that the server is at least capable of doing the Unix extensions. And then when you send them in further operations, if it refuses to do them, you know that the server is making a policy decision. It's not just, you can tell the difference between I'm ignoring you or I don't understand you, which can be kind of important in protocols. And the other thing it allows us to do is if we mess this up and we need to rev the extension version number, which is possible, probably in fact, we have a way to say for the client to tell the server, oh, I want to speak, excuse me, Unix extensions version two, not version one or version three, et cetera. So the secret to making this work is essentially to leave alone all of the protocol that Windows to Windows speaks. So the first thing an SMB3 operation does is it says, here's a path name, I want to turn this into a handle, I will then do operations on this handle. This turning a path name into a handle is done by a create, if you're familiar with the Win32 API, that's the create file API. It's turning to an operation on the wire create. And to modify the create where you can say things like, I want to open an older snapshot, I want to do this, I want to do that, you add these contexts called create contexts. And so the way the Unix extensions are structured is such that adding the Unix, I want to do a Unix create, turns on Unix behavior, turns on Unix behavior with renames, deletes, locking, redraw it, the rest of it. For that handle only. So you can open a boatload of Windows handles, then you can open a Unix handle. Now, at least to SAMA, this made it easy for us, because we already have to handle the case where we're serving SMB1 Unix extensions and Windows clients simultaneously. So we already have the code that can differentiate between the two. So I was kind of a cheat in that it was already easy for us. So what does a create context look like? It actually, I mean, I was thinking about showing the picture of it and whatever, but it really, it's pointless, it's just a grid, it's just a 128 bit random number. I generated a write on this laptop and then stuck in the code. If the server responds with a reply saying, yes, I understood this, then you know that you've got Unix semantics. So the first thing you have to run into is case, case sensitivity. Windows to Windows, it's case in sensitive, Unix to Unix, it needs to be case sensitive. So this is the first thing, adding in that Unix create context tells the server, I want you to pass this path name case sensitively, as a proper Unix to Unix connection would. Now, it is only a request for the server to do so. It is not mandated in the spec. Why is that? You might say, well, Unix to Unix has to be case in sensitive, but that's not true. If you look at Linux, imagine that you are mounting a NTFS file system on a USB stick or a USB drive or whatever. Unix already has to cope with case in sensitive path name lookups. So asking for a Unix create context on a USB3 and getting something back doesn't 100% mean that you've got case sensitive semantics. It just means you asked the server for that and it did its best effort. If you really want to know, did you get case in sensitive? Can this share, can this file system give me case sensitive semantics? There's a separate info level with a bit in it that says, hey, my share can do case sensitivity or not. So it's a best effort request. So the rest of the changes that you need are remarkably small. You do need one info level. I tried to get away without it, went back and forth, but there is really one extra piece of info, a few extra pieces of information that you need to return from the Unix stat system call that you can't get by doing just, you would have to do multiple round trips from Windows info levels to get the same thing. And you still wouldn't get, you'd have to fetch ACLs for every single file, for instance, to get the mode bits. So it's a lot easier to have one very timely modified info level that will return all the information a POSIX stat needs. For setting the mode for doing Chimad, the easiest way to do that is you have to support ACLs anyway, is you do a set access control list call with a special set of ACL masks that say user group owner. Microsoft already designed this. They already did it for their NFS server. So just reuse it. We already did. POSIX ACLs? Anyone here uses POSIX ACLs on their Unix server? Tough. Windows ACLs, they're gone now. Windows ACLs only. Your POSIX ACLs will have to be mapped into the Windows ACLs best effort basis. As ZFS is becoming very, very common on FreeBSD and Linux systems, yes, I know about the licensing, people still use it. That supports NFSV4 ACLs, I think XFS does. The world is moving to Windows like ACLs. So you can set for SMB3 Unix to Unix, you can take the incoming Windows ACLs, turn them into NFSV4 ACLs on the wire. That code is already in the Linux kernel to support NFSV4 servers. So let's just reuse that. I'm tired of fighting Kristoff Helwig over the Windows ACLs piece. Let's just put the NFS ACLs in. Other thing you don't get? No UIDs and GRIDs. Everything comes back as Windows sits. Why is this? This was a big argument for many months over this. The reason for it ultimately is any SMB client has to cope with mapping Windows, SIDs, security IDs, UIDs and GRIDs basically, into Unix numbers, UIDs and GRIDs. The client already has to do this work, so I'm not going to make the server do it. What you will end up with, and we did end up with this in the SMB1 Unix extensions, is you ended up with being able to make POSIX calls to get UIDs and GRIDs back, and then you would make Windows calls to get SIDs back, and you would be exposing the mapping that the server was using between SIDs and UIDs. That mapping may have been different from the one that the client was using. And that lies the source of a million bugs and possibly security issues too. If you have one Windows SID that's mapped into one UID on the client, and a different UID on the server, that way lies madness and security errors. So, eliminate the problem, Windows SIDs on the wire, nothing else. Changing ownership is simple, you just do a set-ackle, nothing different there, exactly as Windows would do. So the file name handling is the largest change. How many people are familiar with Windows Streams names? Windows has this hideous concept which I will explain in one slide, why it is the worst idea in the world, but that will be the last slide of the presentation, because it has nothing to do with the rest of it really. I just want to show you why it's a terrible idea. So Windows has this concept of a special file name, you say, colon colon name, and what that does is anything before that is the actual file, so foo bar bars is the file, and then colon colon stream one stream two, they become different independently seekable data streams within the file. This is the world's worst idea. So for the Unix to Unix in SMB3, don't expose that, leave that in Windows hell. If you want a Windows stream, open a Windows handle without a Unix create context on it. Windows uses UCS2 encoding of Unicode, not UTF-8. It would, we are very tempted to say, oh, okay, if you get a Unix create context with, the path name must be in UTF-8 just to make our lives easier, we decided to drop that. It just complicates the path name processing, it means you have to handle path names coming in in two separate encodings, and it's just easy to say, no, stick with Windows encoding. Windows encoding is good enough to encode any Unicode in UCS2, so yeah, just leave that alone. Hard links are already supported by Windows, we don't need to add that. You get goodies like snapshots, quotas and stuff from TimeWarp tokens for free, and you get all the other wonderful things that SMB3 gives you that David talked about, actually, clustering, SMB direct leases, encryption, everything you want. So, I've got to speed up a little, I think. So, what's kind of ugly, what's close enough? Extended attributes. Windows extended attributes are, they predate any of the SMB3 works, so they are ANSI character set, case, incentive encoded. I didn't add a separate Unix extended attribute call, because EA's are, well, to be honest, they're usually ANSI encoded, case, incentive. It's very rare that you would have a UTF-A encoded case, sensitive, extended attribute. Maybe we'll run into some apps that need that, maybe not. For now, I'm going to leave it alone. The other thing is, EA's on Windows and also in Sandbox, and they're not valid on symbolic links, and the only issue is SE Linux uses this. But being as the most thing that, first thing that people do is usually turn off SE Linux, I'm not terribly worried about that. Maybe we'll have to revisit that. For weird things like FIFOs, block and character device files, Windows already has a way of encoding those, they needed it for the NFS server, just reuse it. So I'm doing that. And so the hardest thing I had to do was basically to say no to the feature creep that people kept wanting to put in this thing. So some people really wanted to add some crap into this protocol. So why is following SimLix on the server a bad idea? So there you've got an isolated root slash slave slash export, and then the client sends a relative path name, and while you're in the middle of processing that, if another client can come in and replace that after you've processed past there and replace it with a SimLink outside the XS exporter zone, you have a race condition and it's a security hole. This was actually the security hole that I talked about in the presentation yesterday that Jan Horn, the Google developer, found. But the reason that that's a problem is that the server transparently follows SimLix. If that was not a SimLink and the client could never create a SimLink, that problem just goes away. So I hate SimLinks. There've been a security disaster in Samba. SimLinks are kind of a useful holdover when you had to add a new disk and you wanted to move some stuff over and create a SimLink from where it used to be, etc. Great flexibility of scripts, security disaster. As soon as SMB1 could write those server-side SimLinks, and it was a terrible decision and it was my fault to store them as real SimLinks on the server file system, it's a horrible mess. It's convenient for NFS and SMB1 servers, and I absolutely want to forbid this in Unix Extension 3. So how are we going to do SimLinks? Windows already has a concept of repass points. Unfortunately, as in many things with Windows, there are an over-designed mess. You can have repass points on directories, on files, you can turn a file into a repass point, you can turn it back, there's all sorts of weird crap that you can do with it that I just don't want to deal with. So for SMB3 SimLinks, SMB3 Unix SimLinks, what I've decided to say is you can create a zero-length file and turn it into a repass point and store the SimLink there in a special extended attribute, but nothing else. There was a mining modification needed to the protocol. I've still got to run past Microsoft to make sure that we can return the SimLink error on traversal. And storing SimLinks on this way breaks the local file system. It means that local apps will not follow SimLinks created by SMB3 clients. I'm okay with that. I'm okay with that because it's secure. So one of the easy parts we've already got, working, all the code for locking reads, writes, rename, unlink, that already exists. We did that for the SMB1 in Unix Extensions. So with some tweaking, the code is remarkably small, a few thousand lines of code changes. You can map the SMB3 Unix create context onto an existing request Unix semantic in the SMB1 stack. And then when they get down below into the NTFS layer inside SAMBA, they kind of act the same way. I need to write down this behavior to make it explicit and write tests to make sure that everyone understands what the protocol should do and be. So what works now? There's a WikiSambar to Org link that explains what the details are on the wire protocol changes. There is a Wireshark extension, I think a Relian did, I think he's just left though, that actually will decode these extensions inside Wireshark. Here is my test branch which I keep rebasing as master changes. This will be outside of master for a while yet. I'm hoping in a few months I can get more of it merged in. And then the Linux kernel, Steve French is rapidly implementing this on the kernel side of things inside the Linux client. So the server side code is mostly complete in terms of functionality and features. So the real work gets, you know, the 90% of the 10% left is to be done now is basically it needs to be splitting, split up into micro commits so that it's an easy to follow upstream merge. Then I need lots of tests. I need to finish the client piece which I haven't done yet and I absolutely need to add lots and lots and lots and lots of tests. So this is going to take a while. The Linux client code already works, it's still evolving to match the server. This is the big missing piece, the libSMB client code, it needs it enhancing to negotiate and implement the SMB3 Unix extensions. So the APIs are mostly there from SMB1 and to be honest I could do with some help getting this finished. You know, I will do it, it will just take longer if it's just me. So how do we know we've won? Well SMB3 Unix to Unix replaces NFS. Yay! So Steve French, who's the Microsoft Sanctity member, he has some spectacular numbers using the Linux client code to the SAMBA server. He's really happy about that. We were very careful and collaborated with Microsoft. They are making no promises whatsoever, but we would really like other server vendors to implement these extensions and not make it too hard for them to do so. And if you think about why Microsoft might be interested, right now they have greater than 50% Linux hosts in the Azure cloud. They would really like those Linux hosts to be able to mount Azure file storage over SMB3. And the closer the POSIX they can make that and Unix they can make that, the more traction they think they're going to get in the Azure cloud, which is fair enough. There's another vendor, third party SMB3 vendor who's also interested and started to implement. So I'm hoping that this will eventually take off. And eventually once we've got it all written up we'll publish it as an addendum to the SMB3 spec. Because of legal issues Microsoft can probably never publish it as an official Microsoft doc, but they can link to it and say, you know, we've carved out the info level space for these guys. They basically make it as official as they can. That's what I'm hoping. Oh, okay, so here's my last slide and then we can go to questions. So here's going on about why alternate data streams are a terrible idea. You probably can't see that terribly well. Is there any Linux kernel file systems for people in the audience? But what this is, this was actually a slide provided to me by Ted Cho, who is a Linux kernel file system person. And it's his personal reason why alternate data streams, this ability to have multiple different streams inside a file, is a terrible idea. What it shows is the Windows Task Manager because Windows has alternate data streams. Running a list of processors and if anyone's familiar with Windows processors, they're the standard ones, you know, Spool SS, server host, you know, Explorer, you name it. Then at the bottom, there's a process running called myfile.txt. How's that happen? Well, what's actually running is a virus that was embedded inside myfile.txt in an alternate data stream with the name of, you know, myhappyvirus, bitcoinminor.exe or something. And guess what? Windows will execute those directly. So, alternate data streams only exist to gladden the hearts of CIA exfiltrators who genuinely use these to steal data from people, which we've seen from the tools that they got hacked and dumped. And virus writers, they're the only two people that I can see who have any use for alternate data streams. So please, please, please do not add this to the Linux kernel. And I don't want to expose the ability to access them over SMB3 Unix extensions. Windows is bad enough. So, does anyone have any questions and comments? You were very polite and quiet throughout the entire thing. Yes? No. With NFS, you can have a single mount for multiple users. So the question is, with NFS, you can have a single mount multiple users. Can you do that with SMB? You've been able to do that since SMB1. Someone, you go around and cuff him. Yes, it's every single connection. You basically do a new session setup, you create a new context ID, and there you go. Multiple user, same connection. Yes, any others? Yes? Given the recent Microsoft push for multi-tall SMB3, are any Unix extensions to help them, or is it usually the same? So the question is, Microsoft really is pushing multi-channel on SMB3. Do you need to make any Unix extension changes? No, we get it for free. It's just a speed-up feature. We already have multi-channel in SAMBA. That's one of the reasons that I think Steve may have implemented multi-channel in the Linux client, I'm not sure. But that's one of the reasons why we can get wonderful performance. Yes, question? Is there any chance of getting something like Linux iNotify over SMB? Well, good news for you, it turns out that Linux iNotify was based on the existing Windows notify, which has been in since you guessed it, SMB1. So yes, that's a... Cuff your friend for me, will you? So yes, that's another feature that has been in the protocol since timing memorial, and we will continue to enjoy the benefits of just by existing in the same space. Any other questions? Oh, cool. Well, I guess everyone wants to go and have some beer. Well, thank you very much. And be sure to write some feedback to Jeremy Stogger on Costa and Adark. Yes, please. Thank you. Thank you. Thank you. Thank you. Thank you. I'll just turn it on because we don't have a PA anyway. I'll give you time hints. Okay.
 Okay, please welcome Jeremy and another presentation about Samba.  Thanks very much.  So I'm going to try and speak up because there's no amplification.  Can you hear me okay at the back?  Yeah? Okay, cool.  So my name is Jeremy Allison.  I work for Google in the open source program office.  But I really want to make it very clear there's my Samba.org email address on the presentation.  And this is not a Google badge presentation, just in case there's any reporters hiding in the corners or something.  This is not an official Google presentation. This is not what Google thinks.  This is not what Google says, etc.  Any Google lawyers in the room should run screaming for the hills because nobody's reviewed it or seen it.  So, having said that, let's dig into SMB3 Unix Extensions.  This is going to be a reasonably technical talk.  It's not going to dive down as deeply as I could.  If you want more details, please feel free to ask questions or ask some more detail at any point.  Stick your hands up and I'm happy to be interrupted and take questions in the middle of the talk.  So, first of all, let's go back one step and say, does everyone here in the room know what Samba is?  Anyone doesn't know what Samba is?  Okay, cool.  And here, SMB3, is there anyone here who doesn't know what SMB3 is?  Okay, so we have one person. That's always good to say.  So, SMB3 is the third version of a protocol originally developed by IBM, taken on by Microsoft.  And it is the protocol that Windows uses to talk to Windows to access files and directories.  So, if you mount, you know, if you bring up your Windows Explorer and you attach to another server,  you can see the files and directories, SMB is the mechanism that's allowing that to happen over a network.  And SMB3 is the third version of that protocol that Microsoft have developed.  And it's actually quite nice, I think.  So, why extend SMB3? Why add Unix Extensions to it?  Well, SMB3 is a Windows to Windows protocol, so it doesn't have much accommodation for specific things that Unix machines need.  So, originally back in the SMB one days, various vendors wanted to add features to SMB.  Now, SMB doesn't have a standards organization.  It only has Microsoft, and what Microsoft does pretty much becomes the standard.  So, it's evolved only to fit the needs of Windows to Windows.  So, other vendors who needed to interoperate with Windows, sort of like HP and SCO before they went insane and started suing IBM of Linux, etc.  They thought, well, we already have an SMB redirector in our kernel, you know, which is what allows us to mount the Windows systems.  So, it wouldn't be so hard to extend this to talk to the SMB servers that we also have on our Unix systems.  And, SAMBO was the most ubiquitous server out there and was the easiest to extend, because you could just type the code in.  So, SCO, HP and a few of the vendors started proposing additions to SMB1 to make it work better between Unix to Unix.  Because if you're aware, Windows to Windows has case insensitive path names, deleting filesworks differently, renaming filesworks differently,  file locking is completely different, the semantics don't match.  So, by extending SMB1, they could make the semantics match Unix to Unix much better,  so that Unix applications would get the semantics that they expected when they mounted a remote file system.  And, excuse me, I'm going to have a, is the Ganesha maintainer in the room?  He was earlier.  Oh, okay, if he's not here, I can get away with more with my NFS jokes.  So, SMB1 Unix was actually better at Unix to Unix, posix to posix semantics than NFS.  NFS has a boatload of hideous things in it, like silly renames,  that basically hide the fact that it's a stateless protocol, at least NFS v3,  and doesn't work very well posix to posix.  Well, SMB1 with Unix extensions had better fidelity for Unix to Unix than NFS does.  So, plus, you know, NFS sucks and it should just die.  So, I like this slide, I made it because you can actually choose who you are depending on which protocol you like.  You know, you could say that NFS is Kirk's screaming,  Kaan!  And SMB is saying, from hell's heart, I spit at thee.  Or it could be the other way around.  And the other thing I really love is, meanwhile in the cloud,  we will add your biological and technological distinctiveness to our own,  your culture will adopt to service us, resistance is futile.  And no better summation of the cloud vendor, storage vendors, I could possibly imagine.  So, yes, so Kaan and Kirk are having fun there while the Borg are out there waiting for us all.  What happened with the SMB1 Unix extensions?  Well, they became a hideous train wreck.  Why was this?  Partly my fault.  Partly the fault of a now Microsoft employee, because I wouldn't say no to him.  The issue was there was no real oversight on what got added.  And it was, well, if it works, it's good.  It doesn't have to do protocol design, it just has to work, which is, you know, you can see the point in that.  So it was an enabler for some security problems.  And I'm going to go into those in a lot more detail later on.  But allowing server-side sim links, in other words, sim links that are created on the server-side file system by the client,  that the server then follows is a recipe for security disaster.  And I will explain why in a little while.  So we just added anything we thought of.  And we want to have a new input level for the POSIX semantics.  We want to have new input levels for POSIX semantics,  even if there was a Windows input level that was kind of similar.  Yeah, let's just throw a boatload of more input levels in.  So, like I say, it was just ad hoc thrown together.  Having said that, it had some very useful effects.  In chatting to the Microsoft engineers who implemented SMB2 transport encryption,  they told me that they originally had terrible pushback from their internal security teams who said,  no, no, don't do this at your file system layer.  We've got IPsec to handle all security in the network layer.  And as everyone knows, IPsec is widely adopted and easy to set up.  So that was a joke for people who don't have to support IPsec.  It's terrible and doesn't work very well.  So they were able to add transport level encryption directly into SMB1  just by saying, well, look at those open source guys.  They've got transport level encryption.  We need it for feature parity.  So that was actually quite useful.  And the SMB3 transport level encryption is actually very well implemented and secure.  So I'm happy about that.  So monorail, monorail, who needs a monorail?  Sorry, Simpsons.  A clean slate.  We can design something glowing, sliding into the future like the Springfield monorail.  But one thing that we came away with,  and one of the reasons this has taken a great deal of time,  also because I'm kind of lazy and don't do things unless people are shouting at me,  but I finally came down with a philosophy of utter minimalism,  which was that rather than the train wreck that was SMB1, Unix Extensions,  the SMB3 Unix Extensions would only add things that we could not do any other way.  So don't reinvent anything that SMB3 already has.  Just tweak and add in very small ways.  Now, because the SMB3 protocol is actually quite nicely designed,  that made this not too hard, actually.  So what does this mean?  Cut down on the extra stuff you need to add.  Don't duplicate stuff.  If you have an info level in Windows that returns create timestamps and whatever,  don't add another POSIX level that does the same thing.  You've already got that data, get it the Windows way.  Allow semantics that aren't 100% POSIX, but are kind of close enough.  And I'll explain a little more what that means in a later slide.  Reuse the existing SMB3 features that Windows uses.  So what does that mean?  It means that we don't have to invent our own mechanism for encryption.  Windows already has ACLs and they love them or hate them,  and I certainly do hate them.  They are the most widely deployed ACL mechanism on the planet.  Everybody understands.  Nobody understands Windows ACLs, but everybody uses Windows ACLs.  That's probably the best way to put it.  So they're very common and we should adopt the same.  Microsoft already added some features to SMB3, not widely used info levels and such,  that would make their NFS server work against a Windows back-in-file system.  So when we have those mechanisms that Microsoft already invented, just reuse them.  Don't try and invent our own.  Don't try and say ours are better.  Just say, hey, this is the Windows way of setting things.  Let's use that.  So what did Apple do?  Let's contrast the Apple approach.  Apple standardized on SMB2 and above for all of their file sharing needs.  They replaced their proprietary AFP file sharing protocol.  But they kind of did it in a SMB1 Unix extension similar way.  In other words, I think they made a real mess of it.  So what they did was they said, oh, we're just going to keep a regular SMB1 connection,  and then at the point that you actually attach to the network share,  the first operation for an Apple server is a special magic open.  And when you send that special magic open with a create context,  I'll get into what create context, in a minute, of AAPL,  then the server knows I'm a Mac client,  and therefore I will behave differently and do things, do different things.  Now that's basically the same kind of magic action horror,  send an info level that has no meaning other than to change the behavior of the server long term.  And I'm trying to avoid that in SMB3 Unix extensions.  So this magic action means the server must now respond differently to various different operations.  And the effect of this is that poor Ralph from Cernet,  who was the original AFP maintainer for the AFP server on Linux,  who now maintains the Apple compatibility with Samba,  Samba used to claim we were bug for bug compatible with Windows.  Now we have to claim we're bug for bug compatible with Windows and Apple.  So he has to duplicate the same bugs that the Mac server has in order to make the Mac's clients run.  So this is a pain.  So what do we end up with? How should it look?  So the nice thing about SMB3 Negotiate is that you have this concept of negotiate concepts,  context, sorry, which are basically arbitrary little pieces of data you can attach to the first packet.  So we added a new one.  I originally thought we could get away without adding this context.  So you send an empty context saying,  hey, I'm a client that is capable of talking POSIX SMB3 Unix extensions.  And if the server then replies saying, yes, I also can speak SMB3 Unix extensions,  then the client knows it's talking to a Unix capable server.  So the nice thing about this is why is this there?  Well, firstly, if the server doesn't respond with that,  the client knows not to try any other POSIX or Unix extensions.  The protocol spec says that they are non-mandatory.  You can send as many of these contexts as you like,  and the server only responds to the ones it understands, which is nice.  And here's the kicker.  Here's the reason why it's there.  Theoretically, you don't need it because every single operation in SMB3  has to have a handle, and a handle has to have context,  and those contexts could specify SMB3 Unix.  So theoretically, I originally thought we can get away with simply doing context  on the creates rather than on the initial negotiate.  And then one of the Microsoft engineers in discussion on this pointed out  that we really need a create context because if you start sending these Unix commands  to the Windows server or to a server that you think might support them,  and it doesn't respond or rather responds to the rest of the create,  but not the Unix part, you don't know whether it doesn't understand  what you're sending it and is ignoring it or whether it understands  what you're sending it, but is choosing to reject it on this particular operation.  And so that's what the negotiate context is for.  If the server responds to the initial negotiate context,  you know that the server is at least capable of doing the Unix extensions.  And then when you send them in further operations, if it refuses to do them,  you know that the server is making a policy decision.  It's not just, you can tell the difference between I'm ignoring you  or I don't understand you, which can be kind of important in protocols.  And the other thing it allows us to do is if we mess this up  and we need to rev the extension version number, which is possible,  probably in fact, we have a way to say for the client to tell the server,  oh, I want to speak, excuse me, Unix extensions version two,  not version one or version three, et cetera.  So the secret to making this work is essentially to leave alone  all of the protocol that Windows to Windows speaks.  So the first thing an SMB3 operation does is it says,  here's a path name, I want to turn this into a handle,  I will then do operations on this handle.  This turning a path name into a handle is done by a create,  if you're familiar with the Win32 API, that's the create file API.  It's turning to an operation on the wire create.  And to modify the create where you can say things like,  I want to open an older snapshot, I want to do this, I want to do that,  you add these contexts called create contexts.  And so the way the Unix extensions are structured is such that adding the Unix,  I want to do a Unix create, turns on Unix behavior,  turns on Unix behavior with renames, deletes, locking, redraw it, the rest of it.  For that handle only.  So you can open a boatload of Windows handles,  then you can open a Unix handle.  Now, at least to SAMA, this made it easy for us,  because we already have to handle the case where we're serving SMB1 Unix extensions  and Windows clients simultaneously.  So we already have the code that can differentiate between the two.  So I was kind of a cheat in that it was already easy for us.  So what does a create context look like?  It actually, I mean, I was thinking about showing the picture of it and whatever,  but it really, it's pointless, it's just a grid, it's just a 128 bit random number.  I generated a write on this laptop and then stuck in the code.  If the server responds with a reply saying, yes, I understood this,  then you know that you've got Unix semantics.  So the first thing you have to run into is case, case sensitivity.  Windows to Windows, it's case in sensitive, Unix to Unix, it needs to be case sensitive.  So this is the first thing, adding in that Unix create context tells the server,  I want you to pass this path name case sensitively,  as a proper Unix to Unix connection would.  Now, it is only a request for the server to do so.  It is not mandated in the spec.  Why is that?  You might say, well, Unix to Unix has to be case in sensitive, but that's not true.  If you look at Linux, imagine that you are mounting a NTFS file system on a USB stick  or a USB drive or whatever.  Unix already has to cope with case in sensitive path name lookups.  So asking for a Unix create context on a USB3 and getting something back  doesn't 100% mean that you've got case sensitive semantics.  It just means you asked the server for that and it did its best effort.  If you really want to know, did you get case in sensitive?  Can this share, can this file system give me case sensitive semantics?  There's a separate info level with a bit in it that says, hey,  my share can do case sensitivity or not.  So it's a best effort request.  So the rest of the changes that you need are remarkably small.  You do need one info level.  I tried to get away without it, went back and forth,  but there is really one extra piece of info,  a few extra pieces of information that you need to return from the Unix stat system call  that you can't get by doing just, you would have to do multiple round trips  from Windows info levels to get the same thing.  And you still wouldn't get, you'd have to fetch ACLs for every single file,  for instance, to get the mode bits.  So it's a lot easier to have one very timely modified info level  that will return all the information a POSIX stat needs.  For setting the mode for doing Chimad,  the easiest way to do that is you have to support ACLs anyway,  is you do a set access control list call with a special set of ACL masks  that say user group owner.  Microsoft already designed this.  They already did it for their NFS server.  So just reuse it.  We already did.  POSIX ACLs?  Anyone here uses POSIX ACLs on their Unix server?  Tough.  Windows ACLs, they're gone now.  Windows ACLs only.  Your POSIX ACLs will have to be mapped into the Windows ACLs best effort basis.  As ZFS is becoming very, very common on FreeBSD and Linux systems,  yes, I know about the licensing, people still use it.  That supports NFSV4 ACLs, I think XFS does.  The world is moving to Windows like ACLs.  So you can set for SMB3 Unix to Unix,  you can take the incoming Windows ACLs,  turn them into NFSV4 ACLs on the wire.  That code is already in the Linux kernel to support NFSV4 servers.  So let's just reuse that.  I'm tired of fighting Kristoff Helwig over the Windows ACLs piece.  Let's just put the NFS ACLs in.  Other thing you don't get?  No UIDs and GRIDs.  Everything comes back as Windows sits.  Why is this?  This was a big argument for many months over this.  The reason for it ultimately is any SMB client has to cope with mapping  Windows, SIDs, security IDs, UIDs and GRIDs basically,  into Unix numbers, UIDs and GRIDs.  The client already has to do this work,  so I'm not going to make the server do it.  What you will end up with, and we did end up with this in the SMB1 Unix extensions,  is you ended up with being able to make POSIX calls to get UIDs and GRIDs back,  and then you would make Windows calls to get SIDs back,  and you would be exposing the mapping that the server was using between SIDs and UIDs.  That mapping may have been different from the one that the client was using.  And that lies the source of a million bugs and possibly security issues too.  If you have one Windows SID that's mapped into one UID on the client,  and a different UID on the server, that way lies madness and security errors.  So, eliminate the problem, Windows SIDs on the wire, nothing else.  Changing ownership is simple, you just do a set-ackle, nothing different there,  exactly as Windows would do.  So the file name handling is the largest change.  How many people are familiar with Windows Streams names?  Windows has this hideous concept which I will explain in one slide,  why it is the worst idea in the world, but that will be the last slide of the presentation,  because it has nothing to do with the rest of it really.  I just want to show you why it's a terrible idea.  So Windows has this concept of a special file name, you say,  colon colon name, and what that does is anything before that is the actual file,  so foo bar bars is the file, and then colon colon stream one stream two,  they become different independently seekable data streams within the file.  This is the world's worst idea.  So for the Unix to Unix in SMB3, don't expose that, leave that in Windows hell.  If you want a Windows stream, open a Windows handle without a Unix create context on it.  Windows uses UCS2 encoding of Unicode, not UTF-8.  It would, we are very tempted to say,  oh, okay, if you get a Unix create context with,  the path name must be in UTF-8 just to make our lives easier,  we decided to drop that.  It just complicates the path name processing,  it means you have to handle path names coming in in two separate encodings,  and it's just easy to say, no, stick with Windows encoding.  Windows encoding is good enough to encode any Unicode in UCS2,  so yeah, just leave that alone.  Hard links are already supported by Windows, we don't need to add that.  You get goodies like snapshots, quotas and stuff from TimeWarp tokens for free,  and you get all the other wonderful things that SMB3 gives you that David talked about,  actually, clustering, SMB direct leases, encryption, everything you want.  So, I've got to speed up a little, I think.  So, what's kind of ugly, what's close enough?  Extended attributes.  Windows extended attributes are, they predate any of the SMB3 works,  so they are ANSI character set, case, incentive encoded.  I didn't add a separate Unix extended attribute call,  because EA's are, well, to be honest, they're usually ANSI encoded, case, incentive.  It's very rare that you would have a UTF-A encoded case, sensitive, extended attribute.  Maybe we'll run into some apps that need that, maybe not.  For now, I'm going to leave it alone.  The other thing is, EA's on Windows and also in Sandbox,  and they're not valid on symbolic links, and the only issue is SE Linux uses this.  But being as the most thing that, first thing that people do is usually turn off SE Linux,  I'm not terribly worried about that.  Maybe we'll have to revisit that.  For weird things like FIFOs, block and character device files,  Windows already has a way of encoding those, they needed it for the NFS server,  just reuse it.  So I'm doing that.  And so the hardest thing I had to do was basically to say no to the feature creep  that people kept wanting to put in this thing.  So some people really wanted to add some crap into this protocol.  So why is following SimLix on the server a bad idea?  So there you've got an isolated root slash slave slash export,  and then the client sends a relative path name,  and while you're in the middle of processing that,  if another client can come in and replace that after you've processed past there  and replace it with a SimLink outside the XS exporter zone,  you have a race condition and it's a security hole.  This was actually the security hole that I talked about in the presentation yesterday  that Jan Horn, the Google developer, found.  But the reason that that's a problem is that the server transparently follows SimLix.  If that was not a SimLink and the client could never create a SimLink,  that problem just goes away.  So I hate SimLinks.  There've been a security disaster in Samba.  SimLinks are kind of a useful holdover when you had to add a new disk  and you wanted to move some stuff over and create a SimLink from where it used to be, etc.  Great flexibility of scripts, security disaster.  As soon as SMB1 could write those server-side SimLinks,  and it was a terrible decision and it was my fault to store them as real SimLinks on the server file system,  it's a horrible mess.  It's convenient for NFS and SMB1 servers,  and I absolutely want to forbid this in Unix Extension 3.  So how are we going to do SimLinks?  Windows already has a concept of repass points.  Unfortunately, as in many things with Windows, there are an over-designed mess.  You can have repass points on directories, on files, you can turn a file into a repass point,  you can turn it back, there's all sorts of weird crap that you can do with it  that I just don't want to deal with.  So for SMB3 SimLinks, SMB3 Unix SimLinks,  what I've decided to say is you can create a zero-length file and turn it into a repass point  and store the SimLink there in a special extended attribute, but nothing else.  There was a mining modification needed to the protocol.  I've still got to run past Microsoft to make sure that we can return the SimLink error on traversal.  And storing SimLinks on this way breaks the local file system.  It means that local apps will not follow SimLinks created by SMB3 clients.  I'm okay with that. I'm okay with that because it's secure.  So one of the easy parts we've already got, working,  all the code for locking reads, writes, rename, unlink, that already exists.  We did that for the SMB1 in Unix Extensions.  So with some tweaking, the code is remarkably small, a few thousand lines of code changes.  You can map the SMB3 Unix create context onto an existing request Unix semantic in the SMB1 stack.  And then when they get down below into the NTFS layer inside SAMBA, they kind of act the same way.  I need to write down this behavior to make it explicit and write tests to make sure that everyone understands what the protocol should do and be.  So what works now?  There's a WikiSambar to Org link that explains what the details are on the wire protocol changes.  There is a Wireshark extension, I think a Relian did, I think he's just left though,  that actually will decode these extensions inside Wireshark.  Here is my test branch which I keep rebasing as master changes.  This will be outside of master for a while yet.  I'm hoping in a few months I can get more of it merged in.  And then the Linux kernel, Steve French is rapidly implementing this on the kernel side of things inside the Linux client.  So the server side code is mostly complete in terms of functionality and features.  So the real work gets, you know, the 90% of the 10% left is to be done now is basically it needs to be splitting,  split up into micro commits so that it's an easy to follow upstream merge.  Then I need lots of tests.  I need to finish the client piece which I haven't done yet and I absolutely need to add lots and lots and lots and lots of tests.  So this is going to take a while.  The Linux client code already works, it's still evolving to match the server.  This is the big missing piece, the libSMB client code, it needs it enhancing to negotiate and implement the SMB3 Unix extensions.  So the APIs are mostly there from SMB1 and to be honest I could do with some help getting this finished.  You know, I will do it, it will just take longer if it's just me.  So how do we know we've won?  Well SMB3 Unix to Unix replaces NFS.  Yay!  So Steve French, who's the Microsoft Sanctity member,  he has some spectacular numbers using the Linux client code to the SAMBA server.  He's really happy about that.  We were very careful and collaborated with Microsoft.  They are making no promises whatsoever, but we would really like other server vendors to implement these extensions  and not make it too hard for them to do so.  And if you think about why Microsoft might be interested, right now they have greater than 50% Linux hosts in the Azure cloud.  They would really like those Linux hosts to be able to mount Azure file storage over SMB3.  And the closer the POSIX they can make that and Unix they can make that,  the more traction they think they're going to get in the Azure cloud, which is fair enough.  There's another vendor, third party SMB3 vendor who's also interested and started to implement.  So I'm hoping that this will eventually take off.  And eventually once we've got it all written up we'll publish it as an addendum to the SMB3 spec.  Because of legal issues Microsoft can probably never publish it as an official Microsoft doc,  but they can link to it and say, you know, we've carved out the info level space for these guys.  They basically make it as official as they can. That's what I'm hoping.  Oh, okay, so here's my last slide and then we can go to questions.  So here's going on about why alternate data streams are a terrible idea.  You probably can't see that terribly well.  Is there any Linux kernel file systems for people in the audience?  But what this is, this was actually a slide provided to me by Ted Cho, who is a Linux kernel file system person.  And it's his personal reason why alternate data streams, this ability to have multiple different streams inside a file, is a terrible idea.  What it shows is the Windows Task Manager because Windows has alternate data streams.  Running a list of processors and if anyone's familiar with Windows processors, they're the standard ones, you know,  Spool SS, server host, you know, Explorer, you name it.  Then at the bottom, there's a process running called myfile.txt.  How's that happen?  Well, what's actually running is a virus that was embedded inside myfile.txt in an alternate data stream  with the name of, you know, myhappyvirus, bitcoinminor.exe or something.  And guess what? Windows will execute those directly.  So, alternate data streams only exist to gladden the hearts of CIA exfiltrators who genuinely use these to steal data from people,  which we've seen from the tools that they got hacked and dumped.  And virus writers, they're the only two people that I can see who have any use for alternate data streams.  So please, please, please do not add this to the Linux kernel.  And I don't want to expose the ability to access them over SMB3 Unix extensions.  Windows is bad enough.  So, does anyone have any questions and comments?  You were very polite and quiet throughout the entire thing.  Yes?  No.  With NFS, you can have a single mount for multiple users.  So the question is, with NFS, you can have a single mount multiple users.  Can you do that with SMB?  You've been able to do that since SMB1.  Someone, you go around and cuff him.  Yes, it's every single connection.  You basically do a new session setup, you create a new context ID, and there you go.  Multiple user, same connection.  Yes, any others?  Yes?  Given the recent Microsoft push for multi-tall SMB3, are any Unix extensions to help them, or is it usually the same?  So the question is, Microsoft really is pushing multi-channel on SMB3.  Do you need to make any Unix extension changes?  No, we get it for free.  It's just a speed-up feature.  We already have multi-channel in SAMBA.  That's one of the reasons that I think Steve may have implemented multi-channel in the Linux client, I'm not sure.  But that's one of the reasons why we can get wonderful performance.  Yes, question?  Is there any chance of getting something like Linux iNotify over SMB?  Well, good news for you, it turns out that Linux iNotify was based on the existing Windows notify, which has been in since you guessed it, SMB1.  So yes, that's a...  Cuff your friend for me, will you?  So yes, that's another feature that has been in the protocol since timing memorial, and we will continue to enjoy the benefits of just by existing in the same space.  Any other questions?  Oh, cool.  Well, I guess everyone wants to go and have some beer.  Well, thank you very much.  And be sure to write some feedback to Jeremy Stogger on Costa and Adark.  Yes, please.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  I'll just turn it on because we don't have a PA anyway.  I'll give you time hints.  Okay.
SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_0_179.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_180_2435.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_2436_2555.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_2556_3827.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_3828_7811.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_7812_9131.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_9132_12815.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_12816_16739.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_16740_19427.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_19428_24143.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_24144_26315.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_26316_29243.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_29244_34943.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_34944_37943.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_37944_40163.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_40164_41495.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_41496_42671.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_42672_44267.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_44268_45431.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_45432_46619.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_46620_48107.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_48108_50507.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_50508_53351.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_53352_56615.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_56616_56675.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_56676_56723.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_56724_56807.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_56808_56855.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_56856_56903.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_56904_56975.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_56976_57131.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_57132_57287.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_57288_57335.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_57336_57407.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_57408_57455.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_57456_57539.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_57540_57599.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_57600_57755.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_57756_57827.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_57828_57875.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_57876_57947.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_57948_58163.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_58164_58223.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_58224_59111.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_59112_59159.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_59160_59231.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_59232_59291.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_59292_59351.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_59352_60131.png SMB2_POSIX_Extensions_Where_we_are__what_remains_to_be_done__4ws33fE6m1g_60132_60244.png