Building a high throughput low-latency PCIe based SDR
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Chemeris, Alexander
Kostanbaev, Sergey
Software Defined Radios (SDRs) became a mainstream tool for wireless engineers and security researches and there are plenty of them available on the market. Most if not all SDRs in the affordable price range are using USB2/USB3 as a transport, because of implementation simplicity. While being so popular, USB has limited bandwidth, high latency and is not really suitable for embedded applications. PCIe/miniPCIe is the only widespread bus which is embedded friendly, low latency and high bandwidth at the same time. But implementing PCIe/miniPCIe is not for the faint of heart - you have to write your own FPGA code, write your own Linux kernel driver and ensure compatibility with different chipsets, each with its own quirks. In this talk we will look at the requirements for a high performance SDR like XTRX, how this leads to certain design decisions and share pitfalls and gotchas we encountered (and solved).
 Has anyone in here ever worked with LibUSB or PiUSB? Hands up. Okay. Who also thinks USB is a pain? Okay. Sergei and Alexander were here back in at the 26C3, that's a long time ago, I think it was back in Berlin. And back then they presented their first homemade or not homemade SDR, Soft Ready Fan Radio. This year they are back again and they want to show us how they implemented another one using an FPGA and to communicate with it they used PCI Express. So I think if you thought USB was a pain, let's see what they can tell us about PCI Express. A warm round of applause for Alexander and Sergei for building a high throughput low latency PCI based Soft Ready Fan Radio. Hi everyone. Good morning and welcome to the first day of the congress. So just a little bit background about what we've done previously and why we are doing what we are doing right now is that we started working with Soft Ready Fan Radius and by the way who knows what Soft Ready Fan Radio is? Okay. Perfect. And whoever actually uses Soft Ready Fan Radio is RTL-SDR or less people but still quiet a lot. Okay, good. I wonder whether anyone here uses more expensive radios like usurps, less people but few. Okay, good. So before 2008 I had no idea what Soft Ready Fan Radio is, was working with voice over IP, software person, etc. So in 2008 I heard about it on BTS, got introduced to Soft Ready Fan Radio and I wanted to make it really work. And that's what led us to today. In 2009 we had to develop a clock tamer, a hardware which allowed to use usurp1 to run GSM without problems. If anyone ever tried doing this without a good clock source, know what I'm talking about. And we presented this, it wasn't really NSDR, it was just a clock source. We presented this in 2009 in the 2063. Then we realized that using usurp1 is not really a good idea because we wanted to build a robust industrial grade base stations. So we started developing our own Soft Ready Fan Radio which we call UMPTRIX. And it was in 2011, our first base stations were deployed in 2013, but I always wanted to have something really small and really inexpensive. And back then it wasn't possible. My original idea in 2011, we were to build a PCI express card. And not PCI express card, but mini PCI card. If you remember, there were like a lot of Wi-Fi cards in mini PCI form factor. And I thought that would be really cool to have an SDR and mini PCI so I can plug this into my laptop or in some embedded PC and have a nice SDR equipment. But back then it just was not really possible because electronics were bigger and more power hungry and just didn't work that way. So we designed UMPTRIX to work over Gigabit Ethernet and it was about that size. So now we spend this year designing something which really brings me to what I wanted those years ago. So the XTRIX is a mini PCI express. Again, there was no PCI express back then. So now it's mini PCI express which is even smaller than mini PCI. And it's built to be embedded friendly so you can plug this into a single board computer and by the single board computer, if you have a laptop with a mini PCI express, you can plug this into your laptop and you have a really small software defined radio equipment. And we really want to make it inexpensive. That's why I was asking how many of you have ever worked with RTLSDR, how many of you have ever worked with usurps because the gap between them is pretty big. And we want to really bring the software defined radio to masses. Definitely won't be as cheap as RTLSDR but we try to make it as close as possible. And at the same time, so at the size of RTLSDR, the price will higher but hopefully it will be affordable to pretty much everyone, we really want to bring high performance into your hands. And by high performance, I mean this is a full transmits receive with two channels receive which is usually called 2x2 MIMO in the radio world. The goal was to bring it to 160 mega samples per second which can roughly give you like 120 megahertz of radio spectrum available. So what we were able to achieve is, again this is mini PCI express form factor. It has small Arctic 7, that's the smallest and most inexpensive FPGA which has ability to work with PCI express. It has LMS 7000 chip for RFIC, very high performance, very tightly embedded chip with even DSP blocks inside. It has even a GPS chip here. So it has, you can actually on the right upper side you can see GPS chip so you can actually synchronize your SDR to GPS for perfect clock stability. So you won't have any problems running any telecommunication systems like GS7 3G, 4G due to clock problems. And it also has interface for SIM cards so you can actually create a software defined radio modem and there are all the control projects to build one in a 4LT or the SRS B if you're interested, etc. So really, really highly packed one. And if you put this into perspective, that's how it all started in 2006 and that's what you have 10 years later. It's pretty impressive. So thanks. But I think it's actually up close to the whole industry who is working on shrinking the sizes because we just put stuff on the PCB. We are not building the silicon itself. Interesting thing is that we did the first approach. We said, let's pack everything. Let's do a very tight PCB design. We did an 8-layer PCB design. And when we send it to a fab to estimate the cost, it turned out it's 15,000 USD per piece. Well, in small volumes, obviously, but still a little bit too much. So we had to redesign this. And the first thing which we did is we still kept 8 layers because in our experience, number of layers nowadays have only minimal impact on the cost of the device. So like 6, 8 layers. The price difference is not so big. But we did complete rerouting and only kept 2D microvias and never used Buried Vias. So this made it much easier and much faster for the fab to manufacture it. And the price suddenly went 5, 6 times down. And in volume again, it will be significantly cheaper. And that's just for GeekPorn how PCB looks inside. So now let's go into real stuff. So PCI Express, why did we choose PCI Express? As it was said, USB is pain in the ass, you can't really use USB in industrial systems for whole variety of reasons, just unstable. So we did use Ethernet for many years successfully, but Ethernet has one problem. First of all, inexpensive Ethernet is only 1 gigabit and 1 gigabit does not offer you enough bandwidth to carry all the data you want, plus it's power hungry, etc. So PCI Express is really a good choice because it's low power, it has low latency, it has very high bandwidth, and it's available almost universally. When we started looking into this, we realized that even ARM boards, some of ARM boards have PCI, mini PCI Express slots, which was a big surprise for me, for example. So the problem is that unlike USB, you do need to write your own kernel driver for this, and there is no way around. And it is really hard to write this driver universally, so we are writing it obviously for Linux because we are working with embedded systems, but if we want to rewrite it for Windows or for MacOS, we will have to do a lot of rewriting. So we focus on what we want on Linux only right now. And now the hardest part, debugging is really non-trivial. Once more error, and your PC is completely hanged because you did something wrong. And you have to reboot it and restart it. That's like debugging kernel, but sometimes even harder. To make it worse, there is no really easy-to-use plug-and-play interface. If you want to restart normally when you develop a PCI Express card, when you want to restart, you have to restart your development machine. Again, not a nice way. It's really hard. So the first thing we did is we found that we can use Thunderbolt 3, which is just recently released, and it has the ability to work directly with PCI Express bus. So it basically has a mode in which it converts PCI Express into plug-and-play interface. So if you have a laptop which supports Thunderbolt 3, then you can use this to plug-and-play your plug-and-plug your device to make your development easier. There are always problems. There's no easy way. There is no documentation. Thunderbolt is not compatible with Thunderbolt. Thunderbolt 3 is not compatible with Thunderbolt 2. So we had to buy a special laptop with Thunderbolt 3 with special cables, like all this hard stuff. And if you really want to get documentation, you have to sign NDA and send a business plan to them so they can approve that your business makes sense. So we actually opted out, we decided not to go through this. What we did is we found that someone is actually making PCI Express with Thunderbolt 3 converters and selling them as DevBoards. And that was a big relief because it saved us lots of time, lots of money. Just ordered it from some Asian company. And yeah, this is how it looks like this converter. So you buy it, like with several pieces, you can plug in your PCI Express card there and you plug this into your laptop. And this is with X-Turics, or you plug into it. The only problem we found is that typically, UFI has a security control enabled so that any random Thunderbolt device can't hijack your PCI bus and can't get access to your kernel memory and do some bad stuff. Which is a good idea. The only problem is that it's not fully implemented in Linux. So under Windows, if you plug in a device which has no security features, which is not certified, that will politely ask you, like, do you really trust this device? Do you want to use it? And you say yes, under Linux, it just does not work. So we spent some time trying to figure out how to get around this. There are some patches from Intel which are not mainline and we were not able to actually get them work. So we just had to disable all this security measure in the laptop. So be aware that this is the case. And we suspect that happy users of Apple might not be able to do this because Apple don't have BIOS so you probably can't disable this feature. So probably a good incentive for someone to actually finish writing the driver. So now to the goal. So we want to achieve 160 mega-samples per second, 2x2 MIMO, which means to transmit, to receive channels at 12 bits, which is roughly 7.5 gigabit per second. So first result when we got this board from the fab, it didn't work. So the first interesting thing we realized is that, first of all, the FPGA has hardware blocks for talking to a PCI Express, which called GTP, which basically implement PCI Express serial physical layer. The thing is the numbering is reversed in PCI Express and in FPGA. And we didn't realize this. So we had to do very, very fine soldering to actually swap the lanes. You can see this very fine work there. We also found that one of the components was that bug, which is a well-known term for chips which at design stage are placed mirrored. So we mirrored, occasionally mirrored the pinout. So we had to solder it upside down. And if you can realize how small it is, you can also appreciate the work done. And what's funny, when I was looking at the dead bugs, I actually found a manual from NASA which describes how to properly solder dead bugs to get it approved. So this is the link. I think you can go there and enjoy. There's lots of fun stuff there. So after fixing all of this, our next attempt kind of works. So next stage is debugging the FPGA code which has to talk to PCI Express. And PCI Express has to talk to Linux kernel. Linux kernel has to talk to the driver. Driver has to talk to the user space. So peripherals are easy. So all the Yards, SPIs. We've got to work almost immediately. No problems with that. But DMA was real beast. So we spent lots of time trying to get DMA to work. And the problem is that with DMA, it's an FPGA, so you can't just place a breakpoint like you do in C or C++ or in other languages. It's a real-time hardware which is running on the fabric. So we had to, Sorgi was mainly developing this, had to write a lot of small test benches and test everything piece by piece. So all parts of the DMA code we had was routed into a small test bench which was emulating all the tricks. And as classics predicted, it took about five to ten times more than actually writing the code. So we really blew up our predicted timelines by doing this. But at the end, we've got really stable work. So some suggestions for anyone who will try to repeat this exercise is there is a logic analyzer built into Xilinx. And you can use it. It's nice. It's sometimes it's very helpful. But you can't debug transient bugs which are coming out when some weird conditions are coming up. So you have to implement some read back registers which shows important statistic, important data about how your system behaves. In our case, it's various counters on the DMA interface. So you can actually see, kind of see what's happening with your data. Is it received? Is it send? How much is send? How much is received? So for example, we can see when we saturate the bus or when actually there's an under run so the host is not providing data fast enough so we can at least understand whether it's a Hust problem or whether it's an FPGA problem on which part should we debug next. Because again, it's a very multi-layer problem. We started with FPGA, PCI express, kernel, driver, user space, and any part can fail. So you can't work blind like this. So again, the goal was to get 160 mega samples with the first name implementation, we got two mega samples per second, roughly 60 times slower. The problem is that software just wasn't keeping up and wasn't sending data fast enough. So there was like many things done but the most important parts is use real time priority if you want to get very stable results and well, big software bugs and one of the most important bugs we had was that DMA buffers were not freed in proper time immediately so they were busy for longer than they should be which introduced extra cycles and basically just reduced bandwidth. So at this point, let's talk a little bit about how to implement a high performance driver for Linux because if you want to get real performance, you have to start with the right design. So there are basically three approaches and the whole spectrum in between, like two approaches and the whole spectrum in between which you can refer to three. So the first approach is full kernel control in which case kernel driver not only is the transfer, it actually has all the logics of controlling your device and all the export Iokitl to the user space and that's the kind of a traditional way of writing drivers. For user space is completely abstracted from all the details. Well the problem is that this is probably the slowest way to do it. So the other way is what's called the zero copy interface. Your only control is held in the kernel and data is provided, raw data is provided to user space as is so you avoid memory copy which make it faster but still not fast enough if you really want to achieve maximum performance because you still have context switches between the kernel and the user space. So the most, the fastest approach possible is to have full user space implementation when kernel just expose everything and says now you do it yourself and you have no context switches like almost now and you can really optimize everything. So what are the problems with this? The pros I already mentioned, no switches between kernel and user space, it's very low latency because of this as well, it's very high bandwidth. But if you're not interested in getting the very high performance, the most performance and you just want to have some low bandwidth performance then you will have to add hacks because you can't get the notification from the kernel that resources available is more data available. So it also makes it vulnerable because if user space can access it then it can do whatever it want. So we at the end decided, so one more important thing, so how to actually to get the best performance out of the bus and this is, we have to say the best we want to pull your device or not to pull, get notified. So what is pulling? I guess everyone has a programmer understands it, so pulling is when you ask repeatedly are you ready, are you ready, are you ready and when it's ready you get the data immediately. So it's basically a busy loop of your, you're just constantly asking device what's happening and you need to dedicate a full core and thanks God we have multi-core CPUs nowadays so you can dedicate the full core to this pulling and you can just pull constantly. But again, if you don't need this high performance you just need to get something then you will be wasting a lot of CPU resources. So at the end we decided to do a combined architecture, it is possible to pull but there is also a chance to get notification from the kernel for applications which need low bandwidth but also require better CPU performance which I think is the best way if you are trying to target both worlds. So very quickly, so the architecture of the system, so we tried to make it very, very portable and flexible. So there is a kernel driver, yeah there is a kernel driver which talks to low level library which implements all this logic which we took out of the driver to control the PCI Express to work with DMA to provide all the details of the actual bus implementation. And then there is a high level library which talks to this low level library and also to libraries which implement control of actual peripherals and most importantly to the library which implements control over our FIC chip. So this way it is very modular, we can replace PCI Express with something else later, we might be able to port it to other operating systems and that is the goal. Another interesting issue is when you start writing a Linux kernel driver you very quickly realize that while an LDD which is a classic book for a Linux driver writing is good and it will give you a good insight, it is not actually up to date, it is more than 10 years old and there is all of the new interfaces which are not described there so you have to resort to reading the manuals and all the documentation in the kernel itself. At least you get the after-date information. So the decisions we made is to make everything easy. So we use TTI and TTY for GPS and so you can really attach a pretty much any application which talks to GPS, all of the existing applications can just work out of the books and we also wanted to be able to synchronize system clock to GPS so we got automatic log synchronization across multiple systems which is very important when we are deploying multiple many devices around the world. So we plan to do two interfaces, one is key PPS and another is DCD because DCD line on the UR exposed over TTY. Because again we found that there are two types of applications, one to support one API and others that support other API and there is no common thing so we have to support both. What's interesting is that as we described we want to have pull so we can get notifications from the kernel when data is available and we don't need to do real busy looping all the time. So after all this thought for optimizations we've got to like 10 mega samples per second, still very very far from what we want to achieve. Now there should have been a lot of explanations about PCI express but when we actually wrote everything we wanted to say we realized it's just like a full two hours talk just on PCI express so we are not going to give it here, I'll just give some highlights which are most interesting. If there is real interest we can set up a workshop at some of the later days and talk in more details about PCI express specifically. So the thing is there is no open source course for PCI express which are optimized for high performance real time applications. There is ZaliBoss which is understanding open source but they provide you a source if you pay them but it's very popular because it's very very easy to do but it's not giving you performance. If I remember correctly the best it can do is maybe like 50% bus saturation. So there is also Xilinx implementation but if you are using Xilinx implementation with XII bus that you are really locked in with XII bus with Xilinx and it's also not very efficient in terms of resources and if you remember we want to make this very very inexpensive so our goal is to be able to fit everything in the smallest RTX 7 FPGA and that's quite challenging with all the stuff in there and we just can't waste resources. So decision is to write your own PCI express implementation that's how it looks like, we are not going to discuss it right now. There were several iterations initially looked much simpler, turned out not to work well. Some interesting stuff about PCI express which we stumbled upon is that it was working really well on Atom which is our main development platform because we are doing a lot of embedded stuff. It worked really well when we tried to plug this into Core i7 just started hanging once in a while. So after like several days maybe debugging, Sergey found that there is a very interesting statement in the standard which says that value zero in byte count actually stands not for zero bytes but for 4096 bytes. That's a really cool optimization. So another thing is completion which is a term in PCI express basically for acknowledgement which also can carry some data back to your request and sometimes if you are not sending completion device just hangs. And what happens is that in this case due to some historical heritage of x86 it just starts returning you FFF. And if you have a register which says is your device okay and this register shows one to say the device is okay, guess what will happen? You will always be reading that your device is okay. So the suggestion is not to use one as the status for okay and use either zero or better like a two-bit sequence so you definitely sure that you are okay and not getting FFFs. So when you have a device which may fail at any of the layers you just got this new board, it's really hard, it's really hard to debug and there is a lot of memory corruption so we had a software bug and it was writing the main addresses incorrectly and we were wondering why we are not getting any data in our buffers and at the same time after several starts operating system just crashes. Well that's the reason why there is this UFI protection which prevents you from plugging devices like this into your computer because it was basically writing random data into random portions of your memory. So a lot of debugging, a lot of tests and test benches and we were able to find this. And another thing is if you initialize your driver incorrectly and that's what's happening again when you have plug and play device which you can plug and unplug then you may end up in a situation where you are trying to write into memory which is already freed by a purging system and used for something else. Very well known problem but also happens here. So why DMI is really hard is because it has this completion architecture for reading data. Writes are easy, you just send the data, you forget about it, it's fire and forget system. But for reading you really need to get your data back and the thing is it looks like this. I really hoped there would be some pointing device here. But basically on the top left you can see requests for read and on the right you can see completion transactions. So basically each transaction can be and most likely will be split into multiple transactions. So first of all you have to collect all these pieces and write them into proper parts of the memory. But that's not all. The thing is the latency between request and completion is really high. It's like 50 cycles. So if you have a single, only single transaction in fly you will get really bad performance. You do need to have multiple transactions in fly. And worst thing is that transactions can return data in a random order. So it's a much more complicated state machine than we expected originally. So when I said our architecture was much simpler originally, we did not have all of this and we had to realize this while implementing. So again here was a whole description of how exactly this works. But not this time. So now after all these optimizations we got 20 mega samples per second. Which is just six times lower than what we are aiming at. So now the next thing is PCI express lane scalability. So PCI express is a serial bus. So it has multiple lanes and they allow you to basically horizontally scale your bandwidth. One lane is like X, then two lanes is 2X, four lanes is 4X. So the more lanes you have, the more performance you are getting out of your bus. So the issue is that a typical mini PCI express, the mini PCI express standard only standardize one lane. And second lane is left as optional. So most motherboards don't support this. There are some, but not all of them. And we really wanted to get this done. So we designed a special converter board which allows you to plug your mini PCI express into a full-size PCI express and get two lanes working. And we're also planning to have a similar board which will have multiple slots so you will be able to get multiple X-tricks SDRs onto the same carrier board and plug this into the, say, PCI express 16X. And you will get really a lot of IQ data which then will be your problem, how to process. So with two A-axis, it's about twice performance, so we are getting 50 mega-samples per second. And that's the time to really cut the fat because the real sample size of LMS7 is 12 bits. And we are transmitting 16 because it's easier because CPU is working on 8, 16, 32. So we originally designed the driver to support 8-bit, 12-bit, and 16-bit to be able to do this scaling. And for the test, we said, okay, let's go from 16 to 8-bit. We'll lose some dynamic range, but who cares these days. Still stayed the same. Still 50 mega-samples per second no matter what we did. And that was a lot of interesting debugging going on. And we realized that we actually made another, not that I really mistake, we didn't really know this when we designed, but we should have used a higher voltage for this high-speed bus to get it to the full performance. And at 1.8, it was just degrading too fast, and the bus itself was not performing well. So our next prototype will be using higher voltage specifically for this bus. And this is kind of stuff which makes designing hardware for high-speed really hard because you have to care about the coherence of the parallel buses on your system. So at the same time, we do want to keep 1.8 volts for everything else as much as possible because another problem we are facing with this device is that by the standard mini PCI Express allows only 2.5 watts of power consumption no more. And that's your, we were very lucky that LMS7 has so good power consumption performance. We actually had some extra space to have FPGA and GPS and all this stuff, but we just can't let the power consumption go up. Our measurements on this device are short about 2.3 watts of power consumption, so we are at the limit at this point. So when we fixed the bus with a higher voltage, now it's a theoretical exercise because we haven't done this yet. That's plenty to happen in a couple of months. We should be able to get to this number which was just 1.2 times slower. And then the next thing will be to fix another issue which we made at the very beginning. We procured a wrong chip. Just one digit difference. You can see it's highlighted in red and green. And this chip supports only generation 1 PCI Express which is twice slower than generation 2 PCI Express. So again, hopefully we'll replace the chip and just get very simple doubling of the performance. Still, it will be slower than we wanted it to be, and here is what kind of practical versus theoretical numbers. Well, as every bus it has overhats. And one of the things which again we realized when we were implementing this is that even though the standard digit is a payload size of 4 kilobytes, actual implementations are different. For example, desktop computers like Core Intel Core or Intel Atom, they only have 128-byte payload. So there is much more overhead going on the bus to transfer data. And even theoretically you can only achieve 87% efficiency. And on Xeon we tested and found that they're using 256 payload size and this can give you like 92% efficiency on the bus. And this is before the overhead. So the real reality is even worse. An interesting thing which we also did not expect is that we originally were developing on Intel Atom and everything was working great. When we plugged this into a laptop like Core i7, multi-core, really powerful device, we didn't expect that it wouldn't work. Obviously, like Core i7 should work better than Atom. No, not always. The thing is we were plugging into a laptop which had a built-in video card which was sitting on the same PCI bus and probably manufacturer hard-quartered, higher, that's called the word priority. Yes, higher priority for the video card than for everything else in the system because you don't want your screen to flicker. And so when you move a window, you actually see there are late packets coming to your PCI device. We had to introduce a jitter buffer and add more FIFO into the device to smooth it out. And on the other hand, Xeon is performing really well. So it's very optimized. That said, we tested it with discrete card and outperforms everything by whooping 5.7%. That's what you get for the price. So this is actually the end of the presentation. So we still have not scheduled any workshop, but if there's any interest in actually seeing the device working or if you're interested in learning more about the PCI express in details, let us know. We'll schedule something in the next few days. That's it. And I think we can proceed with questions if there are any. Okay. Thank you very much. If you are leaving now, please try to leave quietly because we might have some questions and we want to hear them. And if you have questions, please line up right behind the microphones. And I think we'll just rate because we don't have anything from the signal angel. However, if you're watching on stream, you can hop into the channels and over social media to ask questions and they will be answered, hopefully. So that microphone. Yeah. What's the minimum and maximum frequency of the card? You mean our frequency? And now the minimum frequency you can sample at the most at LSTR devices can only sample at over 50 megahertz. Is there a similar limitation at your card? So if you're talking about a reference, it can go from almost zero, even though it works worse below 50 megahertz and all the way to 3.8 gigahertz, if I remember correctly. And in terms of the sample rate, right now it works from about two mega samples per second and all to about 50 right now. But again, we're planning to get it to this number as we quoted. Okay. The microphone over there. Thanks for your talk. Did you manage to put your Linux kernel driver to the mainline? Oh, not yet. I mean, it's not even like fully published. So I did not say in the beginning, sorry for this, that we only just manufactured the first prototype which we debugged heavily. So we are only planning to manufacture the second prototype with all these fixes and then we will release like the kernel driver and everything and we are I'm not sure we will maybe will try or maybe one try haven't decided yet. Thanks. Okay. And over here the whole other experience. Okay. Over there. Hey, looks like you went through some incredible amounts of pain to make this work. So I was wondering, aren't there any simulators at least for parts of the system for the PCIe bus for the DMA something? Any simulator so that you can actually first design the system there and debug it more easily, you know? Yes, there are available simulators but the problems so they are non free so you have to pay for them. So yeah. And we chose the hard way. Okay, thanks. Okay. We have a question from the signal angel. Yeah. Are the FPGA codes, Linux driver and library code and the design project fights public and if so, did they post them yet? I won't, did we will have posted them yet? Can't find them on x-tirex.io. Yeah. So they are not published yet. As I said, we haven't released them. So the drivers and libraries will definitely available FPGA code. We are considering it probably also will be available in open source but we will publish them together with the public announcement of the device. Okay. That microphone. Yes, did you see any signal integrity issues between the PCI bus or on this bus to the LMS chip, the micro chip I think that's doing the other. Did you try to measure signal integrity issues or because there were some reliability issues right? Yeah. We actually, so with PCI we never had issues if I remember correctly. No. It was just working. Actually the board is so small and there is small traces, there is no problem in signal integrity so it's actually save us. Yeah. And designing a small board is easier. With the LMS7 the problem is not the signal integrity in terms of difference in the length of the traces but rather the fact that the signal degrades over voltage, over speed in terms of voltage and drops below the detection level and all this stuff. We used some measurements. I actually wanted to add some pictures here but decided it's not going to be super interesting. Okay. One over there. Yeah, thanks for the talk. How much work would it be to convert the 2x2 SDR into an 8 input logic analyzer in terms of heart and software? So to have a really fast logic analyzer where you can record unlimited traces with? Logic analyzer. So basically it's just also an analog digital converter and you would largely want fast sampling and a large amount of memory to store the traces. Well, I just think it's not the best use for it. It's probably, I don't know. Maybe Sorgi has any ideas but I think it's just maybe easier to get a high speed ADC and replace the LIME chip with a high speed ADC to get what you want because the LIME chip has so many things there specifically for RF. Yeah, the main problem you cannot just sample original data. You should shift it over frequency so you cannot sample original signal and using it for something else except spectrum analyzing is hard. Okay. Thanks. Okay. Another question from the internet. Yes. Have you compared the sample rate of the ADC of the LIME's DR chip to the USRP ADC's and if so how does the lower sample rate affect the performance? So comparing low sample rate to high sample rate, we have not done many testing of, we haven't done much testing on the RF performance yet because we were so busy with all this stuff. So we are yet to see in terms of low bit rates versus sample rate versus high sample rate. Well high sample rate always gives you better performance but you also get higher power consumption. So I guess it's the question of what's more important for you. Okay. Over there. I guess there is no mixer bypass so you can't directly sample the signal. Is there a way to use the same antenna for send and receive yet? Well actually there is input for ADC. But it's not a bypass. It's a dedicated pin on the LMS chip and since we are very space constrained we didn't road them so you cannot actually bypass it. Okay in this in our specific hardware so in general so in the LMS chip there is a special again like to rephrase a special pin which allows you to drive your signal directly to ADC without all the mixers filters all this radio stuff just directly to ADC. So yes theoretically that's possible. We even thought about this but it doesn't fit with design. Okay and can I share antennas because I have an existing laptop with existing antennas so but I would use the same antenna for send and receive. Yeah so I mean that's depends on like what exactly do you want to do. If you want a TDD system then yes if you want an FDD system then you will have to put a small duplexer in there but yeah that's the idea so you can plug this into your laptop and use your existing antennas. That's one of the one of the ideas of how to use X-tricks. Yeah because those are all four connectors. Yeah one thing which actually forgot to mention is like I kind of mentioned in the slides is that any other SDRs which are based on Ethernet or on the USB can't work with a CSMA wireless systems and the most famous CSMA system is Wi-Fi. So turns out that because of the latency between your operating system and your radio on USB you just can't react fast enough for Wi-Fi to work because probably you know that in Wi-Fi you carry your sense and if you sense that the spectrum is free you start transmitting. Doesn't make sense when you have huge latency because you know that the spectrum was free back then. So with X-tricks you actually can work with CSMA systems like Wi-Fi. So again it makes it possible to have a fully software implementation of Wi-Fi in your laptop. It obviously won't work like as good as your commercial Wi-Fi because you will have to do a lot of processing on your CPU but for some purposes like experimentation for example for wireless labs and R&D labs that's really valuable. Okay over there. Okay what PCB design package that you use? All two. And I'd be interested in the PCI Express workshop. Would be really great if you do this one. So just again? Would be really great if you do the PCI Express workshop. PCI Express workshop. Okay thank you. Okay I think we have one more question from the microphones and that's you. Okay great talk and again I would appreciate the PCI workshop if it ever happens. What are the synchronization options between multiple cards? Can you synchronize the ADC clock and can you synchronize the presumably digitally created IF? Yes. Unfortunately just IF synchronization is not possible because LIME chip doesn't expose a low frequency but we can synchronize digitally so we have special 1PPS signal synchronization, we have lines for clock synchronization and other stuff we can do in software so LIME chip has a fast correction register. So when you measure, it has a fast difference so you can compensate it on different boards. So tune to a station along the way and then rotate the phase until it aligns. Yeah. Thank you. Yeah a little tricky but possible. So that's one of our plans for future because we do want to see like 128 by 128 MIMO at home. Okay we have another question from the internet? No I actually have two questions. The first one is what is the expected price after a prototype stage and the second one is can you tell us more about this setup you had for debugging the PCIe issues? So you could repeat the second question? It's setup the bug. More about the setup you had for debugging the PCIe issues. Second question I think it's most about our next workshop because it's a more complicated setup so we mostly remove everything about it in our current presentation. Yeah but in general in terms of hardware setup that was our hardware setup so we bought this PCIe express to Thunderbolt 3. We bought a laptop which supports Thunderbolt 3 and that's how we were debugging it so we don't need like a full-fledged PCIe we don't have to restart it all the time. So in terms of price we don't have the fixed price yet so what I can say right now is that we are targeting no more than your like Blade.RF or Hacker.RF devices and probably even cheaper for sound versions. We are out of time so thank you again Sergei and Alexander.
 Has anyone in here ever worked with LibUSB or PiUSB?  Hands up.  Okay.  Who also thinks USB is a pain?  Okay.  Sergei and Alexander were here back in at the 26C3, that's a long time ago, I think  it was back in Berlin.  And back then they presented their first homemade or not homemade SDR, Soft Ready Fan Radio.  This year they are back again and they want to show us how they implemented another one  using an FPGA and to communicate with it they used PCI Express.  So I think if you thought USB was a pain, let's see what they can tell us about PCI  Express.  A warm round of applause for Alexander and Sergei for building a high throughput low  latency PCI based Soft Ready Fan Radio.  Hi everyone.  Good morning and welcome to the first day of the congress.  So just a little bit background about what we've done previously and why we are doing  what we are doing right now is that we started working with Soft Ready Fan Radius and by the  way who knows what Soft Ready Fan Radio is?  Okay.  Perfect.  And whoever actually uses Soft Ready Fan Radio is RTL-SDR or less people but still quiet  a lot.  Okay, good.  I wonder whether anyone here uses more expensive radios like usurps, less people but few.  Okay, good.  So before 2008 I had no idea what Soft Ready Fan Radio is, was working with voice over IP,  software person, etc.  So in 2008 I heard about it on BTS, got introduced to Soft Ready Fan Radio and I wanted to make  it really work.  And that's what led us to today.  In 2009 we had to develop a clock tamer, a hardware which allowed to use usurp1 to run  GSM without problems.  If anyone ever tried doing this without a good clock source, know what I'm talking about.  And we presented this, it wasn't really NSDR, it was just a clock source.  We presented this in 2009 in the 2063.  Then we realized that using usurp1 is not really a good idea because we wanted to build  a robust industrial grade base stations.  So we started developing our own Soft Ready Fan Radio which we call UMPTRIX.  And it was in 2011, our first base stations were deployed in 2013, but I always wanted  to have something really small and really inexpensive.  And back then it wasn't possible.  My original idea in 2011, we were to build a PCI express card.  And not PCI express card, but mini PCI card.  If you remember, there were like a lot of Wi-Fi cards in mini PCI form factor.  And I thought that would be really cool to have an SDR and mini PCI so I can plug this  into my laptop or in some embedded PC and have a nice SDR equipment.  But back then it just was not really possible because electronics were bigger and more power  hungry and just didn't work that way.  So we designed UMPTRIX to work over Gigabit Ethernet and it was about that size.  So now we spend this year designing something which really brings me to what I wanted those  years ago.  So the XTRIX is a mini PCI express.  Again, there was no PCI express back then.  So now it's mini PCI express which is even smaller than mini PCI.  And it's built to be embedded friendly so you can plug this into a single board computer  and by the single board computer, if you have a laptop with a mini PCI express, you can  plug this into your laptop and you have a really small software defined radio equipment.  And we really want to make it inexpensive.  That's why I was asking how many of you have ever worked with RTLSDR, how many of you have  ever worked with usurps because the gap between them is pretty big.  And we want to really bring the software defined radio to masses.  Definitely won't be as cheap as RTLSDR but we try to make it as close as possible.  And at the same time, so at the size of RTLSDR, the price will higher but hopefully it will  be affordable to pretty much everyone, we really want to bring high performance into  your hands.  And by high performance, I mean this is a full transmits receive with two channels receive  which is usually called 2x2 MIMO in the radio world.  The goal was to bring it to 160 mega samples per second which can roughly give you like  120 megahertz of radio spectrum available.  So what we were able to achieve is, again this is mini PCI express form factor.  It has small Arctic 7, that's the smallest and most inexpensive FPGA which has ability  to work with PCI express.  It has LMS 7000 chip for RFIC, very high performance, very tightly embedded chip with  even DSP blocks inside.  It has even a GPS chip here.  So it has, you can actually on the right upper side you can see GPS chip so you can actually  synchronize your SDR to GPS for perfect clock stability.  So you won't have any problems running any telecommunication systems like GS7 3G, 4G  due to clock problems.  And it also has interface for SIM cards so you can actually create a software defined  radio modem and there are all the control projects to build one in a 4LT or the SRS  B if you're interested, etc.  So really, really highly packed one.  And if you put this into perspective, that's how it all started in 2006 and that's what  you have 10 years later.  It's pretty impressive.  So thanks.  But I think it's actually up close to the whole industry who is working on shrinking  the sizes because we just put stuff on the PCB.  We are not building the silicon itself.  Interesting thing is that we did the first approach.  We said, let's pack everything.  Let's do a very tight PCB design.  We did an 8-layer PCB design.  And when we send it to a fab to estimate the cost, it turned out it's 15,000 USD per  piece.  Well, in small volumes, obviously, but still a little bit too much.  So we had to redesign this.  And the first thing which we did is we still kept 8 layers because in our experience, number  of layers nowadays have only minimal impact on the cost of the device.  So like 6, 8 layers.  The price difference is not so big.  But we did complete rerouting and only kept 2D microvias and never used Buried Vias.  So this made it much easier and much faster for the fab to manufacture it.  And the price suddenly went 5, 6 times down.  And in volume again, it will be significantly cheaper.  And that's just for GeekPorn how PCB looks inside.  So now let's go into real stuff.  So PCI Express, why did we choose PCI Express?  As it was said, USB is pain in the ass, you can't really use USB in industrial systems  for whole variety of reasons, just unstable.  So we did use Ethernet for many years successfully, but Ethernet has one problem.  First of all, inexpensive Ethernet is only 1 gigabit and 1 gigabit does not offer you  enough bandwidth to carry all the data you want, plus it's power hungry, etc.  So PCI Express is really a good choice because it's low power, it has low latency, it has  very high bandwidth, and it's available almost universally.  When we started looking into this, we realized that even ARM boards, some of ARM boards have  PCI, mini PCI Express slots, which was a big surprise for me, for example.  So the problem is that unlike USB, you do need to write your own kernel driver for this,  and there is no way around.  And it is really hard to write this driver universally, so we are writing it obviously  for Linux because we are working with embedded systems, but if we want to rewrite it for  Windows or for MacOS, we will have to do a lot of rewriting.  So we focus on what we want on Linux only right now.  And now the hardest part, debugging is really non-trivial.  Once more error, and your PC is completely hanged because you did something wrong.  And you have to reboot it and restart it.  That's like debugging kernel, but sometimes even harder.  To make it worse, there is no really easy-to-use plug-and-play interface.  If you want to restart normally when you develop a PCI Express card, when you want to restart,  you have to restart your development machine.  Again, not a nice way.  It's really hard.  So the first thing we did is we found that we can use Thunderbolt 3, which is just recently  released, and it has the ability to work directly with PCI Express bus.  So it basically has a mode in which it converts PCI Express into plug-and-play interface.  So if you have a laptop which supports Thunderbolt 3, then you can use this to plug-and-play  your plug-and-plug your device to make your development easier.  There are always problems.  There's no easy way.  There is no documentation.  Thunderbolt is not compatible with Thunderbolt.  Thunderbolt 3 is not compatible with Thunderbolt 2.  So we had to buy a special laptop with Thunderbolt 3 with special cables, like all this hard  stuff.  And if you really want to get documentation, you have to sign NDA and send a business plan  to them so they can approve that your business makes sense.  So we actually opted out, we decided not to go through this.  What we did is we found that someone is actually making PCI Express with Thunderbolt 3 converters  and selling them as DevBoards.  And that was a big relief because it saved us lots of time, lots of money.  Just ordered it from some Asian company.  And yeah, this is how it looks like this converter.  So you buy it, like with several pieces, you can plug in your PCI Express card there and  you plug this into your laptop.  And this is with X-Turics, or you plug into it.  The only problem we found is that typically, UFI has a security control enabled so that  any random Thunderbolt device can't hijack your PCI bus and can't get access to your  kernel memory and do some bad stuff.  Which is a good idea.  The only problem is that it's not fully implemented in Linux.  So under Windows, if you plug in a device which has no security features, which is not  certified, that will politely ask you, like, do you really trust this device?  Do you want to use it?  And you say yes, under Linux, it just does not work.  So we spent some time trying to figure out how to get around this.  There are some patches from Intel which are not mainline and we were not able to actually  get them work.  So we just had to disable all this security measure in the laptop.  So be aware that this is the case.  And we suspect that happy users of Apple might not be able to do this because Apple don't  have BIOS so you probably can't disable this feature.  So probably a good incentive for someone to actually finish writing the driver.  So now to the goal.  So we want to achieve 160 mega-samples per second, 2x2 MIMO, which means to transmit,  to receive channels at 12 bits, which is roughly 7.5 gigabit per second.  So first result when we got this board from the fab, it didn't work.  So the first interesting thing we realized is that, first of all, the FPGA has hardware  blocks for talking to a PCI Express, which called GTP, which basically implement PCI  Express serial physical layer.  The thing is the numbering is reversed in PCI Express and in FPGA.  And we didn't realize this.  So we had to do very, very fine soldering to actually swap the lanes.  You can see this very fine work there.  We also found that one of the components was that bug, which is a well-known term for chips  which at design stage are placed mirrored.  So we mirrored, occasionally mirrored the pinout.  So we had to solder it upside down.  And if you can realize how small it is, you can also appreciate the work done.  And what's funny, when I was looking at the dead bugs, I actually found a manual from  NASA which describes how to properly solder dead bugs to get it approved.  So this is the link.  I think you can go there and enjoy.  There's lots of fun stuff there.  So after fixing all of this, our next attempt kind of works.  So next stage is debugging the FPGA code which has to talk to PCI Express.  And PCI Express has to talk to Linux kernel.  Linux kernel has to talk to the driver.  Driver has to talk to the user space.  So peripherals are easy.  So all the Yards, SPIs.  We've got to work almost immediately.  No problems with that.  But DMA was real beast.  So we spent lots of time trying to get DMA to work.  And the problem is that with DMA, it's an FPGA, so you can't just place a breakpoint like  you do in C or C++ or in other languages.  It's a real-time hardware which is running on the fabric.  So we had to, Sorgi was mainly developing this, had to write a lot of small test benches  and test everything piece by piece.  So all parts of the DMA code we had was routed into a small test bench which was emulating  all the tricks.  And as classics predicted, it took about five to ten times more than actually writing the  code.  So we really blew up our predicted timelines by doing this.  But at the end, we've got really stable work.  So some suggestions for anyone who will try to repeat this exercise is there is a logic  analyzer built into Xilinx.  And you can use it.  It's nice.  It's sometimes it's very helpful.  But you can't debug transient bugs which are coming out when some weird conditions are  coming up.  So you have to implement some read back registers which shows important statistic, important  data about how your system behaves.  In our case, it's various counters on the DMA interface.  So you can actually see, kind of see what's happening with your data.  Is it received?  Is it send?  How much is send?  How much is received?  So for example, we can see when we saturate the bus or when actually there's an under  run so the host is not providing data fast enough so we can at least understand whether  it's a Hust problem or whether it's an FPGA problem on which part should we debug next.  Because again, it's a very multi-layer problem.  We started with FPGA, PCI express, kernel, driver, user space, and any part can fail.  So you can't work blind like this.  So again, the goal was to get 160 mega samples with the first name implementation, we got  two mega samples per second, roughly 60 times slower.  The problem is that software just wasn't keeping up and wasn't sending data fast enough.  So there was like many things done but the most important parts is use real time priority  if you want to get very stable results and well, big software bugs and one of the most  important bugs we had was that DMA buffers were not freed in proper time immediately  so they were busy for longer than they should be which introduced extra cycles and basically  just reduced bandwidth.  So at this point, let's talk a little bit about how to implement a high performance  driver for Linux because if you want to get real performance, you have to start with the  right design.  So there are basically three approaches and the whole spectrum in between, like two approaches  and the whole spectrum in between which you can refer to three.  So the first approach is full kernel control in which case kernel driver not only is the  transfer, it actually has all the logics of controlling your device and all the export  Iokitl to the user space and that's the kind of a traditional way of writing drivers.  For user space is completely abstracted from all the details.  Well the problem is that this is probably the slowest way to do it.  So the other way is what's called the zero copy interface.  Your only control is held in the kernel and data is provided, raw data is provided to  user space as is so you avoid memory copy which make it faster but still not fast enough  if you really want to achieve maximum performance because you still have context switches between  the kernel and the user space.  So the most, the fastest approach possible is to have full user space implementation  when kernel just expose everything and says now you do it yourself and you have no context  switches like almost now and you can really optimize everything.  So what are the problems with this?  The pros I already mentioned, no switches between kernel and user space, it's very low  latency because of this as well, it's very high bandwidth.  But if you're not interested in getting the very high performance, the most performance  and you just want to have some low bandwidth performance then you will have to add hacks  because you can't get the notification from the kernel that resources available is more  data available.  So it also makes it vulnerable because if user space can access it then it can do whatever  it want.  So we at the end decided, so one more important thing, so how to actually to get the best  performance out of the bus and this is, we have to say the best we want to pull your  device or not to pull, get notified.  So what is pulling?  I guess everyone has a programmer understands it, so pulling is when you ask repeatedly  are you ready, are you ready, are you ready and when it's ready you get the data immediately.  So it's basically a busy loop of your, you're just constantly asking device what's happening  and you need to dedicate a full core and thanks God we have multi-core CPUs nowadays so you  can dedicate the full core to this pulling and you can just pull constantly.  But again, if you don't need this high performance you just need to get something then you will  be wasting a lot of CPU resources.  So at the end we decided to do a combined architecture, it is possible to pull but there  is also a chance to get notification from the kernel for applications which need low  bandwidth but also require better CPU performance which I think is the best way if you are trying  to target both worlds.  So very quickly, so the architecture of the system, so we tried to make it very, very  portable and flexible.  So there is a kernel driver, yeah there is a kernel driver which talks to low level  library which implements all this logic which we took out of the driver to control the PCI  Express to work with DMA to provide all the details of the actual bus implementation.  And then there is a high level library which talks to this low level library and also to  libraries which implement control of actual peripherals and most importantly to the library  which implements control over our FIC chip.  So this way it is very modular, we can replace PCI Express with something else later, we  might be able to port it to other operating systems and that is the goal.  Another interesting issue is when you start writing a Linux kernel driver you very quickly  realize that while an LDD which is a classic book for a Linux driver writing is good and  it will give you a good insight, it is not actually up to date, it is more than 10 years  old and there is all of the new interfaces which are not described there so you have  to resort to reading the manuals and all the documentation in the kernel itself.  At least you get the after-date information.  So the decisions we made is to make everything easy.  So we use TTI and TTY for GPS and so you can really attach a pretty much any application  which talks to GPS, all of the existing applications can just work out of the books and we also  wanted to be able to synchronize system clock to GPS so we got automatic log synchronization  across multiple systems which is very important when we are deploying multiple many devices  around the world.  So we plan to do two interfaces, one is key PPS and another is DCD because DCD line on  the UR exposed over TTY.  Because again we found that there are two types of applications, one to support one  API and others that support other API and there is no common thing so we have to support  both.  What's interesting is that as we described we want to have pull so we can get notifications  from the kernel when data is available and we don't need to do real busy looping all  the time.  So after all this thought for optimizations we've got to like 10 mega samples per second,  still very very far from what we want to achieve.  Now there should have been a lot of explanations about PCI express but when we actually wrote  everything we wanted to say we realized it's just like a full two hours talk just on PCI  express so we are not going to give it here, I'll just give some highlights which are most  interesting.  If there is real interest we can set up a workshop at some of the later days and talk  in more details about PCI express specifically.  So the thing is there is no open source course for PCI express which are optimized for high  performance real time applications.  There is ZaliBoss which is understanding open source but they provide you a source if you  pay them but it's very popular because it's very very easy to do but it's not giving you  performance.  If I remember correctly the best it can do is maybe like 50% bus saturation.  So there is also Xilinx implementation but if you are using Xilinx implementation with  XII bus that you are really locked in with XII bus with Xilinx and it's also not very  efficient in terms of resources and if you remember we want to make this very very inexpensive  so our goal is to be able to fit everything in the smallest RTX 7 FPGA and that's quite  challenging with all the stuff in there and we just can't waste resources.  So decision is to write your own PCI express implementation that's how it looks like, we  are not going to discuss it right now.  There were several iterations initially looked much simpler, turned out not to work well.  Some interesting stuff about PCI express which we stumbled upon is that it was working really  well on Atom which is our main development platform because we are doing a lot of embedded  stuff.  It worked really well when we tried to plug this into Core i7 just started hanging once  in a while.  So after like several days maybe debugging, Sergey found that there is a very interesting  statement in the standard which says that value zero in byte count actually stands not  for zero bytes but for 4096 bytes.  That's a really cool optimization.  So another thing is completion which is a term in PCI express basically for acknowledgement  which also can carry some data back to your request and sometimes if you are not sending  completion device just hangs.  And what happens is that in this case due to some historical heritage of x86 it just starts  returning you FFF.  And if you have a register which says is your device okay and this register shows one to  say the device is okay, guess what will happen?  You will always be reading that your device is okay.  So the suggestion is not to use one as the status for okay and use either zero or better  like a two-bit sequence so you definitely sure that you are okay and not getting FFFs.  So when you have a device which may fail at any of the layers you just got this new board,  it's really hard, it's really hard to debug and there is a lot of memory corruption so  we had a software bug and it was writing the main addresses incorrectly and we were wondering  why we are not getting any data in our buffers and at the same time after several starts  operating system just crashes.  Well that's the reason why there is this UFI protection which prevents you from plugging  devices like this into your computer because it was basically writing random data into  random portions of your memory.  So a lot of debugging, a lot of tests and test benches and we were able to find this.  And another thing is if you initialize your driver incorrectly and that's what's happening  again when you have plug and play device which you can plug and unplug then you may end up  in a situation where you are trying to write into memory which is already freed by a purging  system and used for something else.  Very well known problem but also happens here.  So why DMI is really hard is because it has this completion architecture for reading data.  Writes are easy, you just send the data, you forget about it, it's fire and forget system.  But for reading you really need to get your data back and the thing is it looks like this.  I really hoped there would be some pointing device here.  But basically on the top left you can see requests for read and on the right you can  see completion transactions.  So basically each transaction can be and most likely will be split into multiple transactions.  So first of all you have to collect all these pieces and write them into proper parts of  the memory.  But that's not all.  The thing is the latency between request and completion is really high.  It's like 50 cycles.  So if you have a single, only single transaction in fly you will get really bad performance.  You do need to have multiple transactions in fly.  And worst thing is that transactions can return data in a random order.  So it's a much more complicated state machine than we expected originally.  So when I said our architecture was much simpler originally, we did not have all of this and  we had to realize this while implementing.  So again here was a whole description of how exactly this works.  But not this time.  So now after all these optimizations we got 20 mega samples per second.  Which is just six times lower than what we are aiming at.  So now the next thing is PCI express lane scalability.  So PCI express is a serial bus.  So it has multiple lanes and they allow you to basically horizontally scale your bandwidth.  One lane is like X, then two lanes is 2X, four lanes is 4X.  So the more lanes you have, the more performance you are getting out of your bus.  So the issue is that a typical mini PCI express, the mini PCI express standard only standardize  one lane.  And second lane is left as optional.  So most motherboards don't support this.  There are some, but not all of them.  And we really wanted to get this done.  So we designed a special converter board which allows you to plug your mini PCI express into  a full-size PCI express and get two lanes working.  And we're also planning to have a similar board which will have multiple slots so you  will be able to get multiple X-tricks SDRs onto the same carrier board and plug this  into the, say, PCI express 16X.  And you will get really a lot of IQ data which then will be your problem, how to process.  So with two A-axis, it's about twice performance, so we are getting 50 mega-samples per second.  And that's the time to really cut the fat because the real sample size of LMS7 is 12  bits.  And we are transmitting 16 because it's easier because CPU is working on 8, 16, 32.  So we originally designed the driver to support 8-bit, 12-bit, and 16-bit to be able to do  this scaling.  And for the test, we said, okay, let's go from 16 to 8-bit.  We'll lose some dynamic range, but who cares these days.  Still stayed the same.  Still 50 mega-samples per second no matter what we did.  And that was a lot of interesting debugging going on.  And we realized that we actually made another, not that I really mistake, we didn't really  know this when we designed, but we should have used a higher voltage for this high-speed  bus to get it to the full performance.  And at 1.8, it was just degrading too fast, and the bus itself was not performing well.  So our next prototype will be using higher voltage specifically for this bus.  And this is kind of stuff which makes designing hardware for high-speed really hard because  you have to care about the coherence of the parallel buses on your system.  So at the same time, we do want to keep 1.8 volts for everything else as much as possible  because another problem we are facing with this device is that by the standard mini PCI  Express allows only 2.5 watts of power consumption no more.  And that's your, we were very lucky that LMS7 has so good power consumption performance.  We actually had some extra space to have FPGA and GPS and all this stuff, but we just can't  let the power consumption go up.  Our measurements on this device are short about 2.3 watts of power consumption, so we  are at the limit at this point.  So when we fixed the bus with a higher voltage, now it's a theoretical exercise because we  haven't done this yet.  That's plenty to happen in a couple of months.  We should be able to get to this number which was just 1.2 times slower.  And then the next thing will be to fix another issue which we made at the very beginning.  We procured a wrong chip.  Just one digit difference.  You can see it's highlighted in red and green.  And this chip supports only generation 1 PCI Express which is twice slower than generation  2 PCI Express.  So again, hopefully we'll replace the chip and just get very simple doubling of the performance.  Still, it will be slower than we wanted it to be, and here is what kind of practical  versus theoretical numbers.  Well, as every bus it has overhats.  And one of the things which again we realized when we were implementing this is that even  though the standard digit is a payload size of 4 kilobytes, actual implementations are  different.  For example, desktop computers like Core Intel Core or Intel Atom, they only have 128-byte  payload.  So there is much more overhead going on the bus to transfer data.  And even theoretically you can only achieve 87% efficiency.  And on Xeon we tested and found that they're using 256 payload size and this can give you  like 92% efficiency on the bus.  And this is before the overhead.  So the real reality is even worse.  An interesting thing which we also did not expect is that we originally were developing  on Intel Atom and everything was working great.  When we plugged this into a laptop like Core i7, multi-core, really powerful device, we  didn't expect that it wouldn't work.  Obviously, like Core i7 should work better than Atom.  No, not always.  The thing is we were plugging into a laptop which had a built-in video card which was  sitting on the same PCI bus and probably manufacturer hard-quartered, higher, that's  called the word priority.  Yes, higher priority for the video card than for everything else in the system because  you don't want your screen to flicker.  And so when you move a window, you actually see there are late packets coming to your  PCI device.  We had to introduce a jitter buffer and add more FIFO into the device to smooth it out.  And on the other hand, Xeon is performing really well.  So it's very optimized.  That said, we tested it with discrete card and outperforms everything by whooping 5.7%.  That's what you get for the price.  So this is actually the end of the presentation.  So we still have not scheduled any workshop, but if there's any interest in actually seeing  the device working or if you're interested in learning more about the PCI express in  details, let us know.  We'll schedule something in the next few days.  That's it.  And I think we can proceed with questions if there are any.  Okay.  Thank you very much.  If you are leaving now, please try to leave quietly because we might have some questions  and we want to hear them.  And if you have questions, please line up right behind the microphones.  And I think we'll just rate because we don't have anything from the signal angel.  However, if you're watching on stream, you can hop into the channels and over social  media to ask questions and they will be answered, hopefully.  So that microphone.  Yeah.  What's the minimum and maximum frequency of the card?  You mean our frequency?  And now the minimum frequency you can sample at the most at LSTR devices can only sample  at over 50 megahertz.  Is there a similar limitation at your card?  So if you're talking about a reference, it can go from almost zero, even though it works  worse below 50 megahertz and all the way to 3.8 gigahertz, if I remember correctly.  And in terms of the sample rate, right now it works from about two mega samples per second  and all to about 50 right now.  But again, we're planning to get it to this number as we quoted.  Okay.  The microphone over there.  Thanks for your talk.  Did you manage to put your Linux kernel driver to the mainline?  Oh, not yet.  I mean, it's not even like fully published.  So I did not say in the beginning, sorry for this, that we only just manufactured the first  prototype which we debugged heavily.  So we are only planning to manufacture the second prototype with all these fixes and  then we will release like the kernel driver and everything and we are I'm not sure we  will maybe will try or maybe one try haven't decided yet.  Thanks.  Okay.  And over here the whole other experience.  Okay.  Over there.  Hey, looks like you went through some incredible amounts of pain to make this work.  So I was wondering, aren't there any simulators at least for parts of the system for the PCIe  bus for the DMA something?  Any simulator so that you can actually first design the system there and debug it more  easily, you know?  Yes, there are available simulators but the problems so they are non free so you have  to pay for them.  So yeah.  And we chose the hard way.  Okay, thanks.  Okay.  We have a question from the signal angel.  Yeah.  Are the FPGA codes, Linux driver and library code and the design project fights public  and if so, did they post them yet?  I won't, did we will have posted them yet?  Can't find them on x-tirex.io.  Yeah.  So they are not published yet.  As I said, we haven't released them.  So the drivers and libraries will definitely available FPGA code.  We are considering it probably also will be available in open source but we will publish  them together with the public announcement of the device.  Okay.  That microphone.  Yes, did you see any signal integrity issues between the PCI bus or on this bus to the  LMS chip, the micro chip I think that's doing the other.  Did you try to measure signal integrity issues or because there were some reliability issues  right?  Yeah.  We actually, so with PCI we never had issues if I remember correctly.  No.  It was just working.  Actually the board is so small and there is small traces, there is no problem in signal  integrity so it's actually save us.  Yeah.  And designing a small board is easier.  With the LMS7 the problem is not the signal integrity in terms of difference in the length  of the traces but rather the fact that the signal degrades over voltage, over speed in  terms of voltage and drops below the detection level and all this stuff.  We used some measurements.  I actually wanted to add some pictures here but decided it's not going to be super interesting.  Okay.  One over there.  Yeah, thanks for the talk.  How much work would it be to convert the 2x2 SDR into an 8 input logic analyzer in terms  of heart and software?  So to have a really fast logic analyzer where you can record unlimited traces with?  Logic analyzer.  So basically it's just also an analog digital converter and you would largely want fast  sampling and a large amount of memory to store the traces.  Well, I just think it's not the best use for it.  It's probably, I don't know.  Maybe Sorgi has any ideas but I think it's just maybe easier to get a high speed ADC  and replace the LIME chip with a high speed ADC to get what you want because the LIME  chip has so many things there specifically for RF.  Yeah, the main problem you cannot just sample original data.  You should shift it over frequency so you cannot sample original signal and using it  for something else except spectrum analyzing is hard.  Okay.  Thanks.  Okay.  Another question from the internet.  Yes.  Have you compared the sample rate of the ADC of the LIME's DR chip to the USRP ADC's and  if so how does the lower sample rate affect the performance?  So comparing low sample rate to high sample rate, we have not done many testing of, we  haven't done much testing on the RF performance yet because we were so busy with all this  stuff.  So we are yet to see in terms of low bit rates versus sample rate versus high sample rate.  Well high sample rate always gives you better performance but you also get higher power  consumption.  So I guess it's the question of what's more important for you.  Okay.  Over there.  I guess there is no mixer bypass so you can't directly sample the signal.  Is there a way to use the same antenna for send and receive yet?  Well actually there is input for ADC.  But it's not a bypass.  It's a dedicated pin on the LMS chip and since we are very space constrained we didn't  road them so you cannot actually bypass it.  Okay in this in our specific hardware so in general so in the LMS chip there is a special  again like to rephrase a special pin which allows you to drive your signal directly to  ADC without all the mixers filters all this radio stuff just directly to ADC.  So yes theoretically that's possible.  We even thought about this but it doesn't fit with design.  Okay and can I share antennas because I have an existing laptop with existing antennas  so but I would use the same antenna for send and receive.  Yeah so I mean that's depends on like what exactly do you want to do.  If you want a TDD system then yes if you want an FDD system then you will have to put a  small duplexer in there but yeah that's the idea so you can plug this into your laptop  and use your existing antennas.  That's one of the one of the ideas of how to use X-tricks.  Yeah because those are all four connectors.  Yeah one thing which actually forgot to mention is like I kind of mentioned in the slides  is that any other SDRs which are based on Ethernet or on the USB can't work with a CSMA  wireless systems and the most famous CSMA system is Wi-Fi.  So turns out that because of the latency between your operating system and your radio  on USB you just can't react fast enough for Wi-Fi to work because probably you know that  in Wi-Fi you carry your sense and if you sense that the spectrum is free you start transmitting.  Doesn't make sense when you have huge latency because you know that the spectrum was free  back then.  So with X-tricks you actually can work with CSMA systems like Wi-Fi.  So again it makes it possible to have a fully software implementation of Wi-Fi in your laptop.  It obviously won't work like as good as your commercial Wi-Fi because you will have to  do a lot of processing on your CPU but for some purposes like experimentation for example  for wireless labs and R&D labs that's really valuable.  Okay over there.  Okay what PCB design package that you use?  All two.  And I'd be interested in the PCI Express workshop.  Would be really great if you do this one.  So just again?  Would be really great if you do the PCI Express workshop.  PCI Express workshop.  Okay thank you.  Okay I think we have one more question from the microphones and that's you.  Okay great talk and again I would appreciate the PCI workshop if it ever happens.  What are the synchronization options between multiple cards?  Can you synchronize the ADC clock and can you synchronize the presumably digitally  created IF?  Yes.  Unfortunately just IF synchronization is not possible because LIME chip doesn't expose  a low frequency but we can synchronize digitally so we have special 1PPS signal synchronization,  we have lines for clock synchronization and other stuff we can do in software so LIME  chip has a fast correction register.  So when you measure, it has a fast difference so you can compensate it on different boards.  So tune to a station along the way and then rotate the phase until it aligns.  Yeah.  Thank you.  Yeah a little tricky but possible.  So that's one of our plans for future because we do want to see like 128 by 128 MIMO at  home.  Okay we have another question from the internet?  No I actually have two questions.  The first one is what is the expected price after a prototype stage and the second one  is can you tell us more about this setup you had for debugging the PCIe issues?  So you could repeat the second question?  It's setup the bug.  More about the setup you had for debugging the PCIe issues.  Second question I think it's most about our next workshop because it's a more complicated  setup so we mostly remove everything about it in our current presentation.  Yeah but in general in terms of hardware setup that was our hardware setup so we bought this  PCIe express to Thunderbolt 3.  We bought a laptop which supports Thunderbolt 3 and that's how we were debugging it so we  don't need like a full-fledged PCIe we don't have to restart it all the time.  So in terms of price we don't have the fixed price yet so what I can say right now is that  we are targeting no more than your like Blade.RF or Hacker.RF devices and probably even cheaper  for sound versions.  We are out of time so thank you again Sergei and Alexander.
33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_0_719.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_720_1091.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_1092_1331.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_1332_1559.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_1560_2099.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_2100_2159.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_2160_3479.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_3480_3539.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_3540_5027.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_5028_5471.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_5472_7295.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_7296_9743.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_9744_11279.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_11280_13415.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_13416_13847.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_13848_14771.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_14772_15251.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_15252_15407.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_15408_17399.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_17400_17627.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_17628_18851.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_18852_20843.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_20844_21311.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_21312_23039.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_23040_23219.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_23220_26759.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_26760_27035.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_27036_27407.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_27408_27575.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_27576_28871.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_28872_31727.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_31728_37211.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_37212_41639.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_41640_41771.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_41772_43043.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_43044_44099.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_44100_44399.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_44400_44615.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_44616_47411.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_47412_52787.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_52788_52907.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_52908_53111.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_53112_56639.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_56640_57419.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_57420_58691.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_58692_59495.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_59496_60803.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_60804_63791.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_63792_65255.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_65256_66419.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_66420_67211.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_67212_68351.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_68352_68759.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_68760_70703.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_70704_72671.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_72672_72935.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_72936_73259.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_73260_74051.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_74052_74123.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_74124_74951.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_74952_76907.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_76908_85811.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_85812_85883.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_85884_86267.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_86268_86351.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_86352_88379.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_88380_89483.png 33c3-8338-building_a_high_throughput_low-latency_pcie_based_sdr_89484_91351.png