Dissecting modern (3G/4G) cellular modems
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Welte, Harald (LaForge)
Freyther, Holger Hans Peter
Let's have a detailed look at some modern 3G/4G cellular modems and see what we can find out about their internals using undocumented debug interfaces and software or hardware based hacking techniques.
 So, the next talk is titled, Dyssecting Modern 3G 4G Cellular Modems. This is by Harold Welt and Holger Freiter. That was totally mispronounced. We saw in the previous presentations on Smart Cities that there's a lot of IoT that doesn't need to communicate. And while there's ZigBee and LoraWan and other protocols, most likely they're going to fall back on what is ready there, tried and proven, and those are the 3G 4G modems. I don't really need to introduce our speakers today. They're well known for years and years here at the Congress. So I'm just going to pass it right over to them and have a great talk. So today we're going to talk about cellular modems just to differentiate it. It's not about baseband or baseband exploitation. It's really about a GSM module or 3G or 4G module. Our talk is going to be structured in a couple of phases. First of all, our motivation. Why are we looking into it? Where are we coming from? The second part will lead to the history. What have we done before in terms of modems? Looking at how we picked the modems, we actually looked at then something that we actually didn't expect when looking at it. Also then looking at the firmware upgrade mechanism if there's one, how it works, what has been done, and we will finish with our recommendations and wishes. First of all, we're implementing GSM specifications for more than a decade now. It started with humble beginnings of sending AT commands to modems on mobile Linux devices to actually working on a free software smartphone at OpenMoco and then working on OpenBSC and Osmo-com to implement radio area network software and core network software. It has been eight years since LaForge presented about how our Zen modern smartphone hardware looks like. It's seven years since we worked on Osmo-com BB to run our own baseband software on our commercial grade hardware. Finally we have worked with M2M devices and have built M2M devices ourselves using 2G modems. From this point we started to explore how which kind of device would we use in modern embedded devices or M2M or IoT devices these days for our implementation of 3G and 4G network software. If you send messages over the air and you don't get a response, it's always difficult like did you encode it correctly, was it sent? We looked into having a device that allows us to get logging to see if the message arrived or throws an hour, even able to extract traces from it. What's also important for us and got us into Osmo-com and OpenBSC is to build tools to allow others to understand how cellular technology works. While TCP IP might be well known to many of us, how the IP is actually transmitted back to the core net and routed to the internet is not that clear and we want to make it more visible and having technology and tools helps for it. For a brief moment picture yourself trying to build a classic M2M device. You might pick a modem because it's already certified and easy to use but you need to run some application code on it. The traditional approach would be to get a microcontroller or a bigger processor and connect these two devices using USB or serial. It means you need to have a bigger PCB, more power consumption. It would be nice if you can run application software on the modem itself already. One of the driving factors for it is to reduce the PCB space, to have a lower power consumption and to save on the bill of material to have fewer components. Back then we found something called OpenAT. It's done by Sierra Wireless and mostly it allows you to write C software which is then compiled with GCC and uploaded to the modem and you can start running it. It will be loaded into the real time operating system. It runs as a normal process. There's no MMU, no privilege separation. If your application crashes, the entire modem will crash. To ease debugging, Sierra Wireless has a nice tool to get output and send AT commands and logging debugging. The problem with this approach is that if you build an application and make it stable, you know so much about this OpenAT stuff that you're mostly locked into this API. Your architecture and software is following what an OpenAT application will look like. Then you spend years developing an application and suddenly there's no pass for 4G. You're locked in and even your 2G modems will be discontinued. It's a nice platform to get started but it's kind of a dead end. This brings us to our modern requirements. What does a good modem look like? One is we still want to be able to run our own code in it and not like some Python script or some limited Java but like our real C application with access to the device and no artificial control. We don't want to be locked in by a single modem vendor. It might be okay to use a specific ship set but we don't want to be forced to follow whatever the modem supplier wants us to do. For debugging purposes, we want to be able to get lock messages to see what the modem is doing, see if it's throwing away stuff, debug output, be able to control it. For 3G and 4G development, we want to be able to see the radio messages. You might know a tool called X-Goldmon. It's been written by 2BS Engel. It allows you to get tracing information from an infinite baseband but it has some limitations so we want something like X-Goldmon but for GPRS, Edge, UMTS and LTE. The modem market or general, the cellular market is kind of dominated by Qualcomm and it's kind of set by itself but it also means that if you pick a modem most likely it runs or it's based on a Qualcomm ship set and Qualcomm is very close to what we want from a modem because they expose something called the DIAC protocol. It's also used in many different Qualcomm products from DVBH to GSM to FAM2 cells. The first time I personally heard about it was at the 28C3 by a talk from Guillaume that looked into the baseband Qualcomm stack and this DIAC protocol, the framing is very simple. It's like classic HDLC with a start and an end marker with a comment bit, some payload and a checksum and it's used for events like if your modem is switching a network, you get an event. You can enable logging and then you get a lot of textual output but also for comments and response. You can send a comment to read the memory address and to get the value from this memory address back. Literally there are thousands of different messages that can be sent or received and in terms of free software implementations like the modem manager or GSM parser only use a fraction of these available messages. It means this DIAC protocol is something you want to have direct access to because it allows you to see what the modem is doing. It brings us to the point of selecting a device that is exposing DIAC and in the past the option icon might have been a very good device to use because it's exposing DIAC on USB out of the box but it's kind of old. It's using old Qualcomm software. It's limited to 2G and 3G so we look into something more modern. One of the devices we found is from a Chinese modem manufacturer called CrackTel. It's a UC20, it exposes DIAC out of the box. It's even documented in their hardware interface that it's a diagnostic interface but sadly it doesn't support LTE. If you look at a modern device it makes sense to not go for 2G and 3G only but also have an option to go for 4G which brings us to the EC20. It looks like the UC20 but it has LTE so that sounded quite nice. For building a product it can be soldered to your BCP but for development purposes they also offer it in a mini PCI express form factor so you can just plug it into one of your devices that might already have mini PCI express. We picked the EC20 as a module we want to look at it and once we started to look it has like the Qualcomm MDM9615 chipsets which surprisingly is also used in the iPhone 5 but beyond that there's not a lot of documentation of what it has which brings us to the unexpected surprise. After I got the modem I used the AT comment interface just to play with it to see if my application could do what it wants and was hanging like after a day and I got involved with the supply of the modem and they gave me a firmware update which was a zip file and I unpacked it and the file names looked awfully like a linux system which is why would my modem have a linux system in it but maybe I'm just mistaking it and then they have a flash tool and it looks like the flash tool of Android was fast but maybe it's coincidences or it was just convenient to use it but actually other people have already seen that like the MDM9615 runs linux on it like at the Defcon 26 by Mickey Schalkotov so apparently it runs linux on your modem and the question is why would it even run linux because Qualcomm is known to have put an IP stack into the modem, IP stack, zip, why would I stop doing it? It didn't really make sense so we started to look at it and also there was no information that linux actually runs on this device which means no written offer. I used the Rmines GPL tools to unpack the flash file system, it really looked like linux and I started to look at some of the binaries that are not standard but are quite Qualcomm specific and you see funny strings that look like AT commands but like AT plus linux or Q linux command, what would it do? And at this point we started to explore both technically so what is this platform really looking like but also from a legal point of view like can we please get these source codes, can you please put a written offer in it and at this point I hand over to LeForge. So as a lucky coincidence I have been doing some GPL enforcement in the past so well okay but first we started to look at the hardware and if you are doing hardware anyway professionally and you have all the tools and the process and your partners for assembly and so on and so on why not just do a couple of boards to help you with the task at hand. So many of the mini PCI express modules that you can find for cellular modems they have additional signals on undocumented pins of the mini PCI express connector like PCM audio or even URs and so on and well of course if you just put the modem in a regular PC main board or an embedded device those signals they don't terminate anywhere they are just not used on the slot side and soldering wires to the pitch of a mini PCI express board is not very convenient so I created what we call a breakout board and you can see a picture of that it's an open hardware project, schematics and everything design files have been published so you have this connector on the right hand side which exposes all these extra signals so you can actually easily access the various undocumented signals. So the EC20 solder module documents there are a debug UR pins which are not the normal UR pins on which you speak AT commands but it's additional UR but it seems like not all modules have it enabled so we bought modules from three different suppliers they all have different firmware versions and different configurations and some of them have it enabled some not but those that had it enabled have it at 1.8 volts but how can I say in their wisdom the designers decided not to expose the 1.8 volt voltage rail on any of the other pins of the modem so you have external voltages of 3.6 volts 3.3 volts and so on but no 1.8 volts and since in previous projects whenever I needed to attach to a UR of an embedded device I built another level shifter for a specific voltage I already had built for 2.8, 2.5, 2.3 volts before I decided okay let's do a board that is a multi voltage UR so you can actually select the voltage of your UR with a small rotary switch and connect to serial ports of various devices and that's what's here is also another open hardware project at Osmo.com yeah and then you attach to the serial port and you get a Linux login prompt and interestingly if you look at the firmware update file that Holger just mentioned it contains of course an ETC password file and there is only a desh hash and if you do a little bit of password cracking you see it's OE Linux 123 is the root password of the device. So by now we were 250% sure that there is no there is Linux in the device and yeah this is by the way how the full setup looks like you can see I sold the small additional 3 pin header to get access to the UR on this module and this way we could start to explore it further so okay brings us to the topic of well where can we find the source code the module didn't ship with a written offer it didn't ship with a license text it didn't include source code it didn't even mention that Linux or other GPI license software was used so while digging around the official supplier of the manufacturer didn't release anything or didn't provide any such information we were looking a bit around because I mean it's likely the other devices do the same and we found that actually Qualcomm is publishing a complete open embedded build system for building Linux for those modem chip sets so this includes the open embedded meta layers the kernel the boot loader and various other bits and pieces and it's almost completely undocumented there basically some Git refos and you can peek around them but there's not really a documentation how to use it and well if you look at the entry level website of this open source project from Qualcomm which is at code Aurora well there's like literally hundreds of branches and tags and you don't really know what to use for what and they say well this is one example version that you can build and then somebody I think years ago posted well I tried your instructions but well it doesn't compile I'm facing some issues and of course nobody ever responds because well you know why would they ever respond but anyway it's public and on that website you can find it and you can can start building so we started building that of course this we also ran into the issues that didn't build but anyway in the end we managed to build some of the code and meanwhile we were talking to the manufacturer of the modem and asking them for the complete and corresponding source code and then what they send is the complete and corresponding source code to the firmware update tool that you run on your host PC which we didn't ask for and it was not gp l license we still receive the source code okay nice it's good at that point we could understand how the firmware update protocol works towards the modem and then while you ask again for the complete and corresponding source code and say oh we never been by the way all the typos and grammar issues are not introduced by us so we never been in legal dispute and we always make sure to understand that intellectual property rights ahead of using technology belonging to a third party well clearly they did not in the linux case and then they sent us this nice little letter this is an excerpt from it it's like oh we always respect the importance of intellectual property rights and laws and we actively engage with known essential intellectual property right owners apparently the copyright owners of linux and other free software are not essential and so on in order to be compliant with the rights so then you ask again and you ask again so you see basically we always ask the same question it's like oh we appreciate the efforts this was to the lawyer of the net filter ip table project they missed the fact that it's multiple tables and was singular but anyway well then your client me doesn't have the right to empower copyright it's like oh okay that's new to me and they claimed that I had transferred my copyright to the free software foundation which I never did and which nobody I think or at least it's highly unusual if you work on linux kernel code to do so so I did not have copyright okay anyway and call com all sorry sorry my mistake quick tail always respects intellectual property rights of course okay so still no source code then we ask again and ask again well then thank you for your detailed explanations da da da we will provide a tar ball and then we're always willing to achieve tpr compliance and so on and so on and then another month or so passes and then I finally well we do some legal enforcement we do some warning notices and so on and then yeah we are not perfect and we cannot construct a perfect website which I never requested we just wanted a source code and then well you get some source code and it doesn't build and then say well there is a header file missing and by coincidence it's a header file that I wrote I don't know if they did this on intention I mean how many header files did I write and if you know IP tables there is a match for the differentiated services code point for the dcp field in the IP header and that has a header file which is like eight lines of boilerplate definitions and this was missing and as a no we don't have this file and Qualcomm also doesn't have this file and Qualcomm never provided this file to us I mean it's in the public repositories that Qualcomm hosts on code Aurora and the kernel doesn't build without those files and then by the way we will not discuss compiling issues by email anymore okay then some more time I get more and more then you get individual header files and individual emails and you put those header files in your kernel source tree and then you see that the scripts in the kernel have missing executable bits but certain C files suddenly have executable bits and then another header file is missing and so on and so on but by now we have received various source code tar balls they interestingly contain not only the gpll gpll code but also other code with like these these type or patchy type licenses where they wouldn't have to release it which is good and they did this intentionally and I think it's a very nice sign of them that they don't release only what they have to but they release more for the EC20 it's still not license compliant there is no busybox source code included for example not that busybooks would ever have done any gpll enforcement in the past and well but that's not my primary concern and I think it's it's it's getting there and it's workable and you can use the source code that they release interestingly there's other motor manufacturers like Sierra Wireless which are also building modems on such Qualcomm systems and they release not only the source code but extensive documentation so this is a small excerpt from a screenshot where you can actually oh they describe how to build it with open embedded to describe how to use fastboot to install the firmware on the module and so on and so on however they try as good as they are in the open source side they try to lure customers into a proprietary framework like they did with open at in the past and that well again would result in vendor lock in so it's not really recommended or I think a smart move to to go ahead that way and with that I'm going to hand back to Holger before returning later. We're going to very briefly look at the hardware it's a Qualcomm MDM ship that I've already mentioned also in iPhones and it turns out maybe in your future car at least right now it's a modems from QuarkTel and Sierra Wireless but sadly from a free software point of view like it runs Linux it talks to the hardware but there is absolutely no documentation about the hardware in the internet like even on other websites not nothing because it brings us to the hardware overview so we know there's a ARM processor inside probably hexagon they're connected somehow nothing so that's very frustrating to have spent many years on free software to see Linux winning even getting into the modem devices but no hardware documentation being available. Not even a block diagram. You have to repeat it. Not even a block diagram. Yeah, so nothing. Let's look at the software part and explore the system from a software point of view. LaForge has shown how to get a serial console on it but not every modem has it enabled and we didn't look at what it takes to enable it and also soldering is not that nice so after exploring the root file or the unpacked root file system we saw it runs an Android debug bridge so if you have used Android so ADB shell should give you a shell and nicely we have already seen the AT plus Q Linux comments to execute something on the device and we found a script that is reconfiguring the Android USB gadget to actually put ADB in it as well so let's try it you execute the AT command and then suddenly the serial on your host Linux don't work anymore because the QC serial kernel module is written it's matching a device based on the number of interfaces and if you edit ADB to it you suddenly don't have four interfaces but five interfaces then your driver cannot identify the device so we had to first hack it a bit and LaForge has made a more clean patch to actually get it but after this small experience we have ADB shell on it it works on any module. And the ADB shell is root of course I mean so you don't even need to root the device it's like you get the root shell immediately. So there's no lockdown as well it's root there's no SE Linux and just a very nice and open Linux system. To build it it's a bit odd that it has a boot loader that seems to be proprietary then it has the Android boot loader and Android Linux kernel the Android debug bridge we have mentioned but surprisingly it's not using the rest of the Android system but it has a GNU lip see with busybox userland system 5 in it so it's a very classic open embedded build and it's actively developed and maintained by Qualcomm but they make so many releases you actually don't know which one to pick it's a bit of a zoom. Yeah and then you start to look a bit at the Linux kernel that is released and luckily and interestingly and to my pleasant surprise there's no binary only kernel modules so everything in the kernel is released in source code nevertheless of course it's a lot of source code so if you look at a little bit at the number of lines a diff shows up between the closest mainline version and the kernel that's used in those modules you end up with 1.5 to 1.9 million lines of diff I mean this is not actual code lines this is counting all the lines of a diff output including the context but still it gives you an idea about the size of the differences compared to mainline and of course you expect on those kernels that there's you know all the CPU specific stuff and lots of driver code and so on and so on but then if you look at it in more detail and as a disclaimer I haven't looked at Qualcomm Android Linux kernels during the past 10 years or so well to say eight years no six years whatever a long time and I know there's a lot of code in there but I didn't expect all the different things that I found in there I mean they have their own shared memory based logging infrastructure and shared memory is not shared memory with the modem processor it's only shared in the Linux system you have an inter processor communication logging process IPC is not inter process communication like you would know it's inter processor communication and you have something which I completely flabbergasted me it's called remote spin locks I mean you know spin lock you basically you have a mutually exclusive exclusion mechanisms are only one thread or one one CPU can enter a critical part of the code on your multi processor Linux system but then here you can actually also block the modem processor the hexagon on the other side from entering a particular section what could possibly go wrong if you hold if you keep the real-time operating system in busy waiting but okay yeah then you look at the source code and I've actually since I haven't looked at Qualcomm kernel source code for quite some time I was expecting well this has been in all these Linux Android phones that have Qualcomm chipsets and lots of since it's open source plenty of people must have analyzed it and there's certainly some documentation some high-level overview about all these individual subsystems how to glue together how this works and I could just look this documentation or look at that documentation but interestingly it doesn't exist so I had to start to write that documentation it's now a lot of information in our wiki and some of the interesting parts that you find is in there is the shared memory device which is the core of all the interaction between different CPU cores you have inter processor communications you have remote network you have a BAM that's also a BAM to BAM is the bus access manager and we have IPA this is not your favorite beverage but the internet packet accelerator and it there's some diagnostics forwarding and so on and if you look at these subsystems I draw a graph you know if you normally know how rare that is I draw a picture and it to symbolize you what's happening in this modem so if you look at this picture you will see basically the large square at the top is the application processor the bottom small part is the modem processor about which we don't know much of course but then on the left side due to the availability of source code we can see what's happening so you have the shared memory device and then you have channels implemented by the shared memory device and individual different subsystem binding to those channels as two for a T commands which attached to the serial function gadget of the USB gadget code in Linux so basically the important part is to see that the USB you speak you don't speak to the modem actually to the modem process to the basement processor but you speak USB to the Linux gadget inside the Linux ARM core on those devices and then that Linux ARM core forwards or handles different interfaces on your USB configuration in different ways and you have this small box symbolizing the user space and you can see how the different paths go it's quite interesting if you look at the serial ports you have a serial port for GPS and you have a serial port for AT commands and you think well okay these are both serial port devices they must be handled quite similarly but no the GPS part is actually handled here over here it goes into user space it's bridged it goes into a virtual serial port here again and it goes into the serial gadgets whereas the AT commands go straight here inside the kernel and never end up in user space so I don't know why but it's quite sophisticated to say. If we look at the diagnostic subsystem which is particularly of interest to us and now you can see I didn't draw graphs anymore I just wrote a little bit of dotty. We have the modem DSP on the left hand side we have the then we have the Linux kernel where we have the shared memory device on this a DIAC forwarding module in the kernel binds on that we have a connection to the diagnostic function gadget of the USB gadget driver and that is goes to your host so if you talk the DIAC protocol to the modem it actually goes this way through Linux through the shared memory device in the modem DSP and what's even more interesting is that there is a diagnostics character device on Linux called DEF DIAC which well for example to QMOS to your other processes they basically attach to this diagnostics device and all the logging that you find in those Linux user space processes they don't use this log they don't use an Android logging framework but they log through the Qualcomm diagnostic subsystem and you get the log messages of those processes through this kernel device over to the function device over to USB into the host so if you manage to figure out which logging flags and so on to enable you get the log output of those Linux user land processes over DIAC well very Qualcomm like not so surprising but still sort of unusual in the Linux world if you look at the networking the QMI which controls your basically the modem which network you attach to whether you activate PDP context your QOS parameters and so on you might have used QMI CLI or other tools on your Linux laptop to talk to such modems and in this specific case of those Linux based Qualcomm modems well you again have the modem DSP goes through the shared memory device talks to the RMNet device the USB gadget USB to host and you have your host PC somewhere on the right hand side over there so this is basically the path your QMI takes but then you have also QMI in the user land on the modem itself which is what's presented here and they have what's called the QMux D the QMI multiplexer demon which then offers Unix domain sockets to various different client programs so basically all these user space programs by using this Unix domain socket they can talk QMI to the modem as well and all of them can basically you know configure get status reports and so on all the different parts and all the different services on the modem which is interesting and of course something that we want to have coming from our initial motivation we want to run our own applications in there and we want to talk to this QMux D and talk QMI in there so we created a couple of tools to help the analysis on the one hand side we use the open embedded that Qualcomm released to build a matching OPKG and the packages for tools that you need like so-called S-Trace, LSOF and so on for some exploration we also have written a couple of C programs for testing basically for accessing the QMI from code inside the modem and that's successful and then we have a couple of indeed those still link to the proprietary libraries that are provided in the modem and then we have started with some entirely free open source programs like the QMux D wrapper which is an LD preload library so you can trace this QMux D communication and there is ongoing work for Lib QMI G-Lib transport for this QMux D which then would enable you to run a program that's linked against the free software Lib QMI G-Lib inside the modem so you can develop it like you run it on a laptop but you can run it transparently inside the modem after cross compilation. There's also a tool which we call Osmo QC-DIYAC which is basically a host tool for obtaining these DIYAC based logs from the modem so you can run this on your laptop attach it to the modem and then you get all kinds of traces not only the interface traces but also QMI protocol traces which we then again decode using Lib QMI so you get a textual representation. There's ongoing work here to basically move all of that into wiresharks so you get the full decode of that in wireshark but that's not yet there. So what kind of user space programs do we find? Okay ADBD we know what that does. We have an ATFWD daemon. Well AT forwarding daemon well what does it do? It implements those things like an AT plus Q Linux CMD and other AT commands so basically a user space program on Linux can register like a callback within the modem to forward certain AT commands into the Linux user space where you can then implement them so you can basically implement custom AT commands in user space programs. There is all kinds of other software which we haven't really figured out yet a lot what they do. There's one monster called the QC map connection manager which basically allows you to run a Linux based Wi-Fi access point with LTE backhaul excuse me for the typo. So basically you have to attach a Wi-Fi chip to the SDIO interface of your modem module and then you have a full personal access point device that has an LTE backhaul. And then the Wi-Fi you like the parameters for example the key and the SSID and the channel and so on you configure all of that through what? AT commands of course. If you ever wanted to look at software that receives AT commands and then generates textual config files for WPA and for host APD you can look at this code. I prefer not to. So we have the QuackTail bridge which is a very simple device. It reads from one device and it writes it to another device. And apparently this is such a complex task that you need to schedule a different process and I think it has three threads or something obscure like that. So okay well. Which brings us to the funny bits and pieces that you find in those modems. Well the first thing is AT plus Q Linux command. We spoke about that. You can run arbitrary shell commands as root in a read write rule file system in there. So basically you can do anything. I mean you can send an RM dash RF slash and then well it's gone. It's dead Jim. So we also have commands to switch into fast boot mode so you can update the firmware. You have a special AT command to print a DMSC and you have also all kinds of other AT commands. And when you do a strings, just a strings call on those executables it looks like shell scripts in many cases. And one of the most funny things I found in this modem is how many processes and threads does it take to reboot a system. It's apparently a very complex question. How do you reboot your system? Apparently this was the easiest method they could find. So there's one process, the reboot dyag app which registers a dyag command with command code 0x29 with the dyag infrastructure and it spawns a thread which executes another executable called QMI simple real test with an input FIFO. And then it calls system echo modem reset to write modem reset into the FIFO of the input of the process it just spawned which then causes this to send a QMI message to the modem which will reboot the baseband processor. And then of course you clean up. You remove that temporary file because slash TMP is not a tempfist but it's read write file system. And then it goes on to write the string reboot into slash def slash rebooter def using F write not using echo this time. This is a C program, not a shell script. So apparently they discovered that and used F write. And then we have a reboot demon, a second process again with two or three threads which reads this rebooter device. And then actually they published the source code so this is the actual source code. So you read from this def rebooter device then there's a nice comment documented what it does. You do a string compare. You have the first printf going for reboot. You have the second printf initiating reboot and then you call system on the reboot executable. So this is apparently the most simple method. So if you ever were wondering how we would reboot your Linux system, this is the new reference. Then we have C programs that look like shell scripts. So this is an actual condensed output of strings on the QuackTill demon. You see echo in sysfs files. You see copying files. Even with semicolon in there. You see echo into the duty cycle of some pulse with modulation. No idea what that does. And then they even grab for the process and they kill processes with the most obscure things. And they even don't use reader but they do like ls and then parse the output of that rather than use open-dir reader and the usual calls you would do to get a list of files. It's quite amusing. Which brings us to the topic of firmware updates. But you can, I actually missed this example, sorry I have to make a quick interruption here. All these machine to machine modems they typically have an LED and the blinking rhythm of the LED indicates to you whether it's registered to the network, whether the data connection is open, whether it's searching for networks, that's all different blinking patterns of the LED. And how do you implement this on this modem? Well, you run a user space demon that calls system echo 1 to the dev GPIO which controls the LED all the time, you know, to blink the LED. It's not like the kernel wouldn't have infrastructure for LED blinking patterns and so on and so on. But okay, so you have a demon that does nothing else but basically toggling your GPIO by spawning shell processes using the system syscall. Okay, with that I hand over to Olga. Now the question is do you expect anything after the topic of firmware upgrade or is it going to be an empty slide? And the answer is they know that they have to offer firmware upgrades through over the air updates making it as small as possible. And actually it's something that Qualcomm is preparing for modem vendors. So it's based on the Android from around 4.0 recovery gets, not using Android myself. I haven't looked at how it worked before. Many of you might be more familiar with it, but it's mostly a zip file and it contains delta updates and surprisingly they are somehow hashed to a SHA-1 and then the SHA-1 is being signed with a private key and the result is being put into a comment of the zip file. That's probably pretty standard but looked a bit odd and it's nice that they prepare probably secure firmware upgrades with like minimal delta functionally. So what has QuackTel done to this code? They still use it. That's quite nice. But they have removed or actually not really removed but they're not using the RSA code to verify a signature. And instead of using the standard Android way to patch the systems, they use a proprietary component from a company that used to be called RedBand but now it's Harman and probably soon Samsung. This RedBand component is nothing that QuackTel has written but it's a commercial product and it's used in the QuackTel UC20 module as well and also in other automotive projects we have seen RedBand updates being used. So at this point I started to look into how does this update format look like and instead of presenting a very complex form of the format I have this slice. So other people like Matthew Solnik at I think Black Hat presented something called attacks on OMA device management. These can be remotely triggered but the update mechanism used here cannot be remotely triggered. The modem needs to be asked to start an update so that's already a bit more secure but I started to look at the HexThumbs of a specific delta update and with a lot of help from Ditas Spa we actually managed to understand how the updates binary looks like and we have created a small tool to take an existing update and put it into smaller parts and also be able to create our own defiles. The format itself has many different pointers and offsets so in the example you might already see the offsets here and compression size. So it starts with a common header and then after the header you have a LZMA compressed table of contents or however you want to call it and outside in the header you have an offset into the decompressed version of this table of contents where file updates start and when you start creating your own file you might get the offset wrong and the update binary just crashes with your malformed update file so it's not very robust code, a very complicated file format and nothing is cryptographically signed. So when you use strings on the binary you see the word signature but it only refers to a CSE32 checksum. The next part like now we understand how the update format works. We can create our own update files. The question is how do they end up on the device and it's something that is implemented in the AT forward that Harold or LaForge has mentioned and if you grab for some specific strings like W gets, the QC map connection manager or photo or update dot zip you already kind of guess how this application works. So you issue an AT command with a URL for updating it, in turn it will disable your normal IP connection that you have might established on your host and opens our PDP context on the device itself using the QC map connection manager then it will spawn W get to download the file then it will use system to move it to the right directory. It will remember what it wants to do with this file and it will reboot into the recovery partition and system and at this point the update zip file will just be applied and the system will reboot once again. Again without any cryptographically signature or checks. So if you manage to hijack the update process you can install any binary on the device or on the modem or anywhere else on the system as you want. But instead of just seeing how bad it is we want to say like what do we expect them to do and I'm handing over to Harold again. Yeah so rather than saying oh this is all bad, it's all unlocked and it's insecure and so on, well it's fun for us of course because that's what we wanted right. We wanted a modem device where we could do basically whatever we want to and where we don't have to break sophisticated security mechanisms that are designed to keep the user or the customer, the owner of the product out. So yes there are security issues and those security issues must be fixed but we need security mechanisms that work without locking out the user or the owner of the device of course. So this is our public call to the manufacturers if you fix those issues keep in mind that the openness of the platform is interesting for all kinds of legitimate use cases and while you want to protect against malicious attackers you of course still want to enable the actual owners of the device and the users of those devices to use the flexibility they provide because there's a lot of, yeah. So what's the status today in Outlook? Well we have just opened the Wiki on the last slide there are all the links so on the Osmo com project we now have a Wiki for QuackTel Qualcomm LE modems where all the information that I gathered from reading the thousands and thousands of lines of source code is in there. We have released the debug tools presented in this talk in a Git repository in source code. The hardware boards are released as open hardware and available and what's unfortunately still ongoing is the libQMI Glib integration which is to some extent to the fact that I've never written anything against Glib before or anything inside a program that uses Glib. It's like all this infrastructure I'm usually more low level than that. And while we hope to grow this documentation and we kindly invite all of you with an interest in understanding those platforms better to help us out. You don't actually need to necessarily reverse engineer and disassemble things. It's just read the source code, understand what it does and play a bit with the device. We are planning an open embedded package feed so we can actually easily install additional, everyone can easily install additional packages on those modems. There's plenty of flash, I think it's like 20, 30 megabytes of free flash so you can install quite a number of additional packages in there. And our aim is to have free software only user land on this Cortex A5 CPU so to do away with all those proprietary processes that run in user space and the libraries and run the open source kernel and have basically the libQMI Glib integration and all other bits needed to run our own standard Linux user land code in there and have custom images that we can run on modems in all kinds of use cases. Okay. Now before we go for Q&A in a minute, there's an unrelated announcement that we would like to present here. The OsmoCom project has gained support for running your own 3G 3.5G network during the last year. Not sure who has noticed that. Unfortunately, it suffers a bit of a lack of contributions. So we want to motivate people to contribute more. And we have just started an Accelerate 3.5G program which provides 53 3.5G femtosells to people who can convince us that they would contribute something reasonable to our project. So these femtosells are already supported by the OsmoCom code. So using the femtosell and the OsmoCom code, you can run your own 3.5G network. And we're giving those away for free. So if you're interested in any of that, please submit a proposal until the end of January and then you will hopefully receive your free femtosell until the end of February. Which brings us to questions. Yes. It's Q&A time. We have a total of eight microphones here. One, two, three, four, five, six, seven, eight. I can do math. Please step up to the microphones to ask your question. Meanwhile, we have from the signals angel, we have a question. Yes. The internet wants to know if there will be in quotes next open moco. Not from us. No, we're not working on a mobile phone. We're looking at modems, not at mobile phones at the moment. So microphone two. So just to clarify, does it mean that Linux runs on the iPhone 5? We don't know. We don't know. We'd be happy to hear from you if you can find it. But I think you can run the trip with something else on this Cortex processor. But we don't know. Signals angel? Yeah. The internet wants to know why there are no mini PCI express slash M2 to USB 3 adapters because they think LTE is capable of at least 300 megabits and USB 2 could be a bottleneck. Well, I'm not really in the business of manufacturing or selling adapters. So I mean, yes, we did this open hardware device out of a specific need, but you would have to ask the hardware manufacturers that I'm sorry. Can I ask the people who are leaving to leave quietly, please? Can you buy the entire EC20 with the breakout board and the UR as a single kit somewhere? Well, you can contact us, but it's not really something that we have prepared for. But yes, it's certainly an option. But I mean, we're not here to sell you anything. We're here to invite you to help us learn about modems. Thanks. And with the Android debug shell, you actually don't really need the serial. So you can refresh the device using USB, even if you break it, it can be refreshed. And with ADB shell, you have your root shell. So that works. And with ATQ Linux command, you can actually start a lock-in on the NMEA console as well. So even if ADB doesn't work, you can get a lock-in on it. So it's very easy to get started without serial. Signals Angel? Hello? Okay. Do you have, have you tried to get a source code from other manufacturers? Well, we know at this point of, I think, three different manufacturers that use this MDM9615 plus Linux combination from QuakeTill. We have just described how it went. Sierra has published all this already by themselves in a very good way. So there's no need to ask them. It's out there. You can just download it and it's documented. And the third one is an old end-of-life Huawei module where we have also asked, but this is still ongoing. They are now, it's interesting, lead it from a Chinese supplier. You get the excuse, oh, there is Christmas coming up. I was like, okay, so many Christians in China, but okay. I think I'm quite sure if I ask in January, oh, there is Chinese New Year coming up. But we'll see about that. Microphone 4. You mentioned the Qualcomm chip used in the iPhone 5. Does this mean, well, is it likely that the iPhone application processor talks AT commands to this Qualcomm chip? Is this still a state of the art? No, it's not state of the art. I don't think it's happening. I mean, also on these modern modules, the AT command is there for legacy purposes. You have the QMI exported over USB and normally your motor manager or whatever you use, or phone or whatever infrastructure talks QMI to those devices. I'm going to go back to the signals again. The question is, what is the total size of the flash or the root of S on there and how much RAM has this thing got? I don't know. I don't remember the flash size. Was RAM 32 megabytes? Yeah, this 32 megabytes of RAM on this, the successor, the EC25 has 128 megs of RAM. But a flash size I also don't remember right now. But check out the wiki. We have boot logs and all kinds of like D-Mask outputs and so on all on the wiki. So I'm quite sure it's somewhere in there. Microphone 4. You said that you tried to check Legato Linux from Sierra. Wouldn't it be possible to completely build your own kernel there and your applications? Yes, it's certainly possible. But then of course, if you do this, you buy a lot of things with your money that you don't use in the end. And you support a vendor that tries to lock people into this proprietary framework. So it's possible. But do you have to use a framework? I don't think so. No, you don't have to. But I mean, they developing this and part of what you pay for is this framework. And that's why their products are more expensive. Microphone 2. Yes, do you have some ballpark figures about the power consumption in the lowest power quiz and current mode? No, I'm quite sure it's going to be high. I think Holger has some comment on that. So when you look at this device, you use S-Trace against the Q-Max-D and it's like mostly waking up every couple of milliseconds. So it's not power efficient. It's a C code, it's really, really, really annoying to see processes that run all the time. And from the previous slide, this echo mem to CISFS, that's actually the advanced power management. So you can trigger an AT command to have the device sleep and it writes echo mem into it to go to sleep. So it can be tuned and with a free software user space, it can probably be better than what it is right now. Signals Angel again? Yeah. How accessible is the baseband? Like memory and like EMI and stuff? We haven't really investigated this in detail, but we also haven't seen any signature verification and so on there. So I think it's completely open. So the DSP firmware is in three separate partitions and there's no RSA signature and it's the end of these partitions. So it seems that you can modify it if they have locked down like the poke command. That's something we haven't tried or looked at it. Microphone one. The modules readily available in the LGA package rather than the mini PCI dev board, like if people want to start to use these in open hardware. They should be available again. Yes, they were temporarily not available due to the TPI enforcement, but I think now they should be available again. What are the quarter of the costs of those modules? I think it's like 47 something euros around that region or maybe no, it was more than 50, but anyway, somewhere in that region. So one more question from the internet. And the internet wants to know if you can capture layer 2 network stuff with this. Yes, yes, you can. So I think that's the end of our Q&A session. Please join me in thanking LeForge and Holger on this fantastic talk about things that I really didn't want to know about 3G modems.
 So, the next talk is titled, Dyssecting Modern 3G 4G Cellular Modems.  This is by Harold Welt and Holger Freiter.  That was totally mispronounced.  We saw in the previous presentations on Smart Cities that there's a lot of IoT that doesn't  need to communicate.  And while there's ZigBee and LoraWan and other protocols, most likely they're going  to fall back on what is ready there, tried and proven, and those are the 3G 4G modems.  I don't really need to introduce our speakers today.  They're well known for years and years here at the Congress.  So I'm just going to pass it right over to them and have a great talk.  So today we're going to talk about cellular modems just to differentiate it.  It's not about baseband or baseband exploitation.  It's really about a GSM module or 3G or 4G module.  Our talk is going to be structured in a couple of phases.  First of all, our motivation.  Why are we looking into it?  Where are we coming from?  The second part will lead to the history.  What have we done before in terms of modems?  Looking at how we picked the modems, we actually looked at then something that we actually  didn't expect when looking at it.  Also then looking at the firmware upgrade mechanism if there's one, how it works, what  has been done, and we will finish with our recommendations and wishes.  First of all, we're implementing GSM specifications for more than a decade now.  It started with humble beginnings of sending AT commands to modems on mobile Linux devices  to actually working on a free software smartphone at OpenMoco and then working on OpenBSC and  Osmo-com to implement radio area network software and core network software.  It has been eight years since LaForge presented about how our Zen modern smartphone hardware  looks like.  It's seven years since we worked on Osmo-com BB to run our own baseband software on our  commercial grade hardware.  Finally we have worked with M2M devices and have built M2M devices ourselves using 2G  modems.  From this point we started to explore how which kind of device would we use in modern  embedded devices or M2M or IoT devices these days for our implementation of 3G and 4G  network software.  If you send messages over the air and you don't get a response, it's always difficult  like did you encode it correctly, was it sent?  We looked into having a device that allows us to get logging to see if the message arrived  or throws an hour, even able to extract traces from it.  What's also important for us and got us into Osmo-com and OpenBSC is to build tools to  allow others to understand how cellular technology works.  While TCP IP might be well known to many of us, how the IP is actually transmitted back  to the core net and routed to the internet is not that clear and we want to make it more  visible and having technology and tools helps for it.  For a brief moment picture yourself trying to build a classic M2M device.  You might pick a modem because it's already certified and easy to use but you need to  run some application code on it.  The traditional approach would be to get a microcontroller or a bigger processor and  connect these two devices using USB or serial.  It means you need to have a bigger PCB, more power consumption.  It would be nice if you can run application software on the modem itself already.  One of the driving factors for it is to reduce the PCB space, to have a lower power consumption  and to save on the bill of material to have fewer components.  Back then we found something called OpenAT.  It's done by Sierra Wireless and mostly it allows you to write C software which is then  compiled with GCC and uploaded to the modem and you can start running it.  It will be loaded into the real time operating system.  It runs as a normal process.  There's no MMU, no privilege separation.  If your application crashes, the entire modem will crash.  To ease debugging, Sierra Wireless has a nice tool to get output and send AT commands and  logging debugging.  The problem with this approach is that if you build an application and make it stable,  you know so much about this OpenAT stuff that you're mostly locked into this API.  Your architecture and software is following what an OpenAT application will look like.  Then you spend years developing an application and suddenly there's no pass for 4G.  You're locked in and even your 2G modems will be discontinued.  It's a nice platform to get started but it's kind of a dead end.  This brings us to our modern requirements.  What does a good modem look like?  One is we still want to be able to run our own code in it and not like some Python script  or some limited Java but like our real C application with access to the device and no artificial  control.  We don't want to be locked in by a single modem vendor.  It might be okay to use a specific ship set but we don't want to be forced to follow whatever  the modem supplier wants us to do.  For debugging purposes, we want to be able to get lock messages to see what the modem  is doing, see if it's throwing away stuff, debug output, be able to control it.  For 3G and 4G development, we want to be able to see the radio messages.  You might know a tool called X-Goldmon.  It's been written by 2BS Engel.  It allows you to get tracing information from an infinite baseband but it has some limitations  so we want something like X-Goldmon but for GPRS, Edge, UMTS and LTE.  The modem market or general, the cellular market is kind of dominated by Qualcomm and  it's kind of set by itself but it also means that if you pick a modem most likely it runs  or it's based on a Qualcomm ship set and Qualcomm is very close to what we want from  a modem because they expose something called the DIAC protocol.  It's also used in many different Qualcomm products from DVBH to GSM to FAM2 cells.  The first time I personally heard about it was at the 28C3 by a talk from Guillaume that  looked into the baseband Qualcomm stack and this DIAC protocol, the framing is very simple.  It's like classic HDLC with a start and an end marker with a comment bit, some payload  and a checksum and it's used for events like if your modem is switching a network, you  get an event.  You can enable logging and then you get a lot of textual output but also for comments  and response.  You can send a comment to read the memory address and to get the value from this memory  address back.  Literally there are thousands of different messages that can be sent or received and  in terms of free software implementations like the modem manager or GSM parser only use  a fraction of these available messages.  It means this DIAC protocol is something you want to have direct access to because it allows  you to see what the modem is doing.  It brings us to the point of selecting a device that is exposing DIAC and in the past the  option icon might have been a very good device to use because it's exposing DIAC on USB  out of the box but it's kind of old.  It's using old Qualcomm software.  It's limited to 2G and 3G so we look into something more modern.  One of the devices we found is from a Chinese modem manufacturer called CrackTel.  It's a UC20, it exposes DIAC out of the box.  It's even documented in their hardware interface that it's a diagnostic interface but sadly  it doesn't support LTE.  If you look at a modern device it makes sense to not go for 2G and 3G only but also have  an option to go for 4G which brings us to the EC20.  It looks like the UC20 but it has LTE so that sounded quite nice.  For building a product it can be soldered to your BCP but for development purposes they  also offer it in a mini PCI express form factor so you can just plug it into one of your devices  that might already have mini PCI express.  We picked the EC20 as a module we want to look at it and once we started to look it has  like the Qualcomm MDM9615 chipsets which surprisingly is also used in the iPhone 5 but beyond that  there's not a lot of documentation of what it has which brings us to the unexpected surprise.  After I got the modem I used the AT comment interface just to play with it to see if my  application could do what it wants and was hanging like after a day and I got involved  with the supply of the modem and they gave me a firmware update which was a zip file  and I unpacked it and the file names looked awfully like a linux system which is why would  my modem have a linux system in it but maybe I'm just mistaking it and then they have a  flash tool and it looks like the flash tool of Android was fast but maybe it's coincidences  or it was just convenient to use it but actually other people have already seen that like the  MDM9615 runs linux on it like at the Defcon 26 by Mickey Schalkotov so apparently it runs  linux on your modem and the question is why would it even run linux because Qualcomm is  known to have put an IP stack into the modem, IP stack, zip, why would I stop doing it?  It didn't really make sense so we started to look at it and also there was no information  that linux actually runs on this device which means no written offer.  I used the Rmines GPL tools to unpack the flash file system, it really looked like linux  and I started to look at some of the binaries that are not standard but are quite Qualcomm  specific and you see funny strings that look like AT commands but like AT plus linux or  Q linux command, what would it do?  And at this point we started to explore both technically so what is this platform really  looking like but also from a legal point of view like can we please get these source  codes, can you please put a written offer in it and at this point I hand over to LeForge.  So as a lucky coincidence I have been doing some GPL enforcement in the past so well okay  but first we started to look at the hardware and if you are doing hardware anyway professionally  and you have all the tools and the process and your partners for assembly and so on and  so on why not just do a couple of boards to help you with the task at hand.  So many of the mini PCI express modules that you can find for cellular modems they have  additional signals on undocumented pins of the mini PCI express connector like PCM audio  or even URs and so on and well of course if you just put the modem in a regular PC main  board or an embedded device those signals they don't terminate anywhere they are just  not used on the slot side and soldering wires to the pitch of a mini PCI express board is  not very convenient so I created what we call a breakout board and you can see a picture  of that it's an open hardware project, schematics and everything design files have been published  so you have this connector on the right hand side which exposes all these extra signals  so you can actually easily access the various undocumented signals.  So the EC20 solder module documents there are a debug UR pins which are not the normal  UR pins on which you speak AT commands but it's additional UR but it seems like not  all modules have it enabled so we bought modules from three different suppliers they all have  different firmware versions and different configurations and some of them have it enabled  some not but those that had it enabled have it at 1.8 volts but how can I say in their  wisdom the designers decided not to expose the 1.8 volt voltage rail on any of the other  pins of the modem so you have external voltages of 3.6 volts 3.3 volts and so on but no 1.8  volts and since in previous projects whenever I needed to attach to a UR of an embedded  device I built another level shifter for a specific voltage I already had built for 2.8,  2.5, 2.3 volts before I decided okay let's do a board that is a multi voltage UR so you  can actually select the voltage of your UR with a small rotary switch and connect to  serial ports of various devices and that's what's here is also another open hardware  project at Osmo.com yeah and then you attach to the serial port and you get a Linux login  prompt and interestingly if you look at the firmware update file that Holger just mentioned  it contains of course an ETC password file and there is only a desh hash and if you do  a little bit of password cracking you see it's OE Linux 123 is the root password of  the device. So by now we were 250% sure that there is no there is Linux in the device and  yeah this is by the way how the full setup looks like you can see I sold the small additional  3 pin header to get access to the UR on this module and this way we could start to explore  it further so okay brings us to the topic of well where can we find the source code the  module didn't ship with a written offer it didn't ship with a license text it didn't  include source code it didn't even mention that Linux or other GPI license software  was used so while digging around the official supplier of the manufacturer didn't release  anything or didn't provide any such information we were looking a bit around because I mean  it's likely the other devices do the same and we found that actually Qualcomm is publishing  a complete open embedded build system for building Linux for those modem chip sets so  this includes the open embedded meta layers the kernel the boot loader and various other  bits and pieces and it's almost completely undocumented there basically some Git refos  and you can peek around them but there's not really a documentation how to use it and well  if you look at the entry level website of this open source project from Qualcomm which  is at code Aurora well there's like literally hundreds of branches and tags and you don't  really know what to use for what and they say well this is one example version that  you can build and then somebody I think years ago posted well I tried your instructions  but well it doesn't compile I'm facing some issues and of course nobody ever responds  because well you know why would they ever respond but anyway it's public and on that  website you can find it and you can can start building so we started building that of course  this we also ran into the issues that didn't build but anyway in the end we managed to  build some of the code and meanwhile we were talking to the manufacturer of the modem and  asking them for the complete and corresponding source code and then what they send is the  complete and corresponding source code to the firmware update tool that you run on your  host PC which we didn't ask for and it was not gp l license we still receive the source  code okay nice it's good at that point we could understand how the firmware update protocol  works towards the modem and then while you ask again for the complete and corresponding  source code and say oh we never been by the way all the typos and grammar issues are not  introduced by us so we never been in legal dispute and we always make sure to understand  that intellectual property rights ahead of using technology belonging to a third party  well clearly they did not in the linux case and then they sent us this nice little letter  this is an excerpt from it it's like oh we always respect the importance of intellectual  property rights and laws and we actively engage with known essential intellectual property  right owners apparently the copyright owners of linux and other free software are not essential  and so on in order to be compliant with the rights so then you ask again and you ask again  so you see basically we always ask the same question it's like oh we appreciate the efforts  this was to the lawyer of the net filter ip table project they missed the fact that it's  multiple tables and was singular but anyway well then your client me doesn't have the  right to empower copyright it's like oh okay that's new to me and they claimed that I had  transferred my copyright to the free software foundation which I never did and which nobody  I think or at least it's highly unusual if you work on linux kernel code to do so so  I did not have copyright okay anyway and call com all sorry sorry my mistake quick tail  always respects intellectual property rights of course okay so still no source code then  we ask again and ask again well then thank you for your detailed explanations da da da  we will provide a tar ball and then we're always willing to achieve tpr compliance and  so on and so on and then another month or so passes and then I finally well we do some  legal enforcement we do some warning notices and so on and then yeah we are not perfect  and we cannot construct a perfect website which I never requested we just wanted a source  code and then well you get some source code and it doesn't build and then say well there  is a header file missing and by coincidence it's a header file that I wrote I don't know  if they did this on intention I mean how many header files did I write and if you know IP  tables there is a match for the differentiated services code point for the dcp field in the  IP header and that has a header file which is like eight lines of boilerplate definitions  and this was missing and as a no we don't have this file and Qualcomm also doesn't have  this file and Qualcomm never provided this file to us I mean it's in the public repositories  that Qualcomm hosts on code Aurora and the kernel doesn't build without those files and  then by the way we will not discuss compiling issues by email anymore okay then some more  time I get more and more then you get individual header files and individual emails and you  put those header files in your kernel source tree and then you see that the scripts in  the kernel have missing executable bits but certain C files suddenly have executable bits  and then another header file is missing and so on and so on but by now we have received  various source code tar balls they interestingly contain not only the gpll gpll code but also  other code with like these these type or patchy type licenses where they wouldn't have to  release it which is good and they did this intentionally and I think it's a very nice  sign of them that they don't release only what they have to but they release more for  the EC20 it's still not license compliant there is no busybox source code included for  example not that busybooks would ever have done any gpll enforcement in the past and  well but that's not my primary concern and I think it's it's it's getting there and  it's workable and you can use the source code that they release interestingly there's other  motor manufacturers like Sierra Wireless which are also building modems on such Qualcomm  systems and they release not only the source code but extensive documentation so this  is a small excerpt from a screenshot where you can actually oh they describe how to build  it with open embedded to describe how to use fastboot to install the firmware on the module  and so on and so on however they try as good as they are in the open source side they try  to lure customers into a proprietary framework like they did with open at in the past and  that well again would result in vendor lock in so it's not really recommended or I think  a smart move to to go ahead that way and with that I'm going to hand back to Holger before  returning later.  We're going to very briefly look at the hardware it's a Qualcomm MDM ship that I've already  mentioned also in iPhones and it turns out maybe in your future car at least right now  it's a modems from QuarkTel and Sierra Wireless but sadly from a free software point of view  like it runs Linux it talks to the hardware but there is absolutely no documentation about  the hardware in the internet like even on other websites not nothing because it brings  us to the hardware overview so we know there's a ARM processor inside probably hexagon they're  connected somehow nothing so that's very frustrating to have spent many years on free software  to see Linux winning even getting into the modem devices but no hardware documentation  being available.  Not even a block diagram.  You have to repeat it.  Not even a block diagram.  Yeah, so nothing.  Let's look at the software part and explore the system from a software point of view.  LaForge has shown how to get a serial console on it but not every modem has it enabled and  we didn't look at what it takes to enable it and also soldering is not that nice so after  exploring the root file or the unpacked root file system we saw it runs an Android debug  bridge so if you have used Android so ADB shell should give you a shell and nicely we have  already seen the AT plus Q Linux comments to execute something on the device and we  found a script that is reconfiguring the Android USB gadget to actually put ADB in it as well  so let's try it you execute the AT command and then suddenly the serial on your host  Linux don't work anymore because the QC serial kernel module is written it's matching a device  based on the number of interfaces and if you edit ADB to it you suddenly don't have four  interfaces but five interfaces then your driver cannot identify the device so we had to first  hack it a bit and LaForge has made a more clean patch to actually get it but after this small  experience we have ADB shell on it it works on any module.  And the ADB shell is root of course I mean so you don't even need to root the device  it's like you get the root shell immediately.  So there's no lockdown as well it's root there's no SE Linux and just a very nice and open  Linux system.  To build it it's a bit odd that it has a boot loader that seems to be proprietary then it  has the Android boot loader and Android Linux kernel the Android debug bridge we have mentioned  but surprisingly it's not using the rest of the Android system but it has a GNU lip see  with busybox userland system 5 in it so it's a very classic open embedded build and it's  actively developed and maintained by Qualcomm but they make so many releases you actually  don't know which one to pick it's a bit of a zoom.  Yeah and then you start to look a bit at the Linux kernel that is released and luckily  and interestingly and to my pleasant surprise there's no binary only kernel modules so everything  in the kernel is released in source code nevertheless of course it's a lot of source code so if  you look at a little bit at the number of lines a diff shows up between the closest mainline  version and the kernel that's used in those modules you end up with 1.5 to 1.9 million  lines of diff I mean this is not actual code lines this is counting all the lines of a  diff output including the context but still it gives you an idea about the size of the  differences compared to mainline and of course you expect on those kernels that there's you  know all the CPU specific stuff and lots of driver code and so on and so on but then if  you look at it in more detail and as a disclaimer I haven't looked at Qualcomm Android Linux  kernels during the past 10 years or so well to say eight years no six years whatever a  long time and I know there's a lot of code in there but I didn't expect all the different  things that I found in there I mean they have their own shared memory based logging infrastructure  and shared memory is not shared memory with the modem processor it's only shared in the  Linux system you have an inter processor communication logging process IPC is not inter  process communication like you would know it's inter processor communication and you  have something which I completely flabbergasted me it's called remote spin locks I mean you  know spin lock you basically you have a mutually exclusive exclusion mechanisms are only one  thread or one one CPU can enter a critical part of the code on your multi processor  Linux system but then here you can actually also block the modem processor the hexagon  on the other side from entering a particular section what could possibly go wrong if you  hold if you keep the real-time operating system in busy waiting but okay yeah then you look  at the source code and I've actually since I haven't looked at Qualcomm kernel source  code for quite some time I was expecting well this has been in all these Linux Android phones  that have Qualcomm chipsets and lots of since it's open source plenty of people must have  analyzed it and there's certainly some documentation some high-level overview about all these individual  subsystems how to glue together how this works and I could just look this documentation or  look at that documentation but interestingly it doesn't exist so I had to start to write  that documentation it's now a lot of information in our wiki and some of the interesting parts  that you find is in there is the shared memory device which is the core of all the interaction  between different CPU cores you have inter processor communications you have remote  network you have a BAM that's also a BAM to BAM is the bus access manager and we have  IPA this is not your favorite beverage but the internet packet accelerator and it there's  some diagnostics forwarding and so on and if you look at these subsystems I draw a graph  you know if you normally know how rare that is I draw a picture and it to symbolize you  what's happening in this modem so if you look at this picture you will see basically the  large square at the top is the application processor the bottom small part is the modem  processor about which we don't know much of course but then on the left side due to the  availability of source code we can see what's happening so you have the shared memory device  and then you have channels implemented by the shared memory device and individual different  subsystem binding to those channels as two for a T commands which attached to the serial  function gadget of the USB gadget code in Linux so basically the important part is to  see that the USB you speak you don't speak to the modem actually to the modem process  to the basement processor but you speak USB to the Linux gadget inside the Linux ARM core  on those devices and then that Linux ARM core forwards or handles different interfaces  on your USB configuration in different ways and you have this small box symbolizing the  user space and you can see how the different paths go it's quite interesting if you look  at the serial ports you have a serial port for GPS and you have a serial port for AT  commands and you think well okay these are both serial port devices they must be handled  quite similarly but no the GPS part is actually handled here over here it goes into user space  it's bridged it goes into a virtual serial port here again and it goes into the serial  gadgets whereas the AT commands go straight here inside the kernel and never end up in  user space so I don't know why but it's quite sophisticated to say.  If we look at the diagnostic subsystem which is particularly of interest to us and now  you can see I didn't draw graphs anymore I just wrote a little bit of dotty.  We have the modem DSP on the left hand side we have the then we have the Linux kernel  where we have the shared memory device on this a DIAC forwarding module in the kernel  binds on that we have a connection to the diagnostic function gadget of the USB gadget  driver and that is goes to your host so if you talk the DIAC protocol to the modem it  actually goes this way through Linux through the shared memory device in the modem DSP  and what's even more interesting is that there is a diagnostics character device on  Linux called DEF DIAC which well for example to QMOS to your other processes they basically  attach to this diagnostics device and all the logging that you find in those Linux user  space processes they don't use this log they don't use an Android logging framework but  they log through the Qualcomm diagnostic subsystem and you get the log messages of those processes  through this kernel device over to the function device over to USB into the host so if you  manage to figure out which logging flags and so on to enable you get the log output of  those Linux user land processes over DIAC well very Qualcomm like not so surprising  but still sort of unusual in the Linux world if you look at the networking the QMI which  controls your basically the modem which network you attach to whether you activate PDP context  your QOS parameters and so on you might have used QMI CLI or other tools on your Linux  laptop to talk to such modems and in this specific case of those Linux based Qualcomm  modems well you again have the modem DSP goes through the shared memory device talks to the  RMNet device the USB gadget USB to host and you have your host PC somewhere on the right  hand side over there so this is basically the path your QMI takes but then you have also  QMI in the user land on the modem itself which is what's presented here and they have what's  called the QMux D the QMI multiplexer demon which then offers Unix domain sockets to various  different client programs so basically all these user space programs by using this Unix  domain socket they can talk QMI to the modem as well and all of them can basically you  know configure get status reports and so on all the different parts and all the different  services on the modem which is interesting and of course something that we want to have  coming from our initial motivation we want to run our own applications in there and we  want to talk to this QMux D and talk QMI in there so we created a couple of tools to help  the analysis on the one hand side we use the open embedded that Qualcomm released to build  a matching OPKG and the packages for tools that you need like so-called S-Trace, LSOF  and so on for some exploration we also have written a couple of C programs for testing  basically for accessing the QMI from code inside the modem and that's successful and  then we have a couple of indeed those still link to the proprietary libraries that are  provided in the modem and then we have started with some entirely free open source programs  like the QMux D wrapper which is an LD preload library so you can trace this QMux D communication  and there is ongoing work for Lib QMI G-Lib transport for this QMux D which then would  enable you to run a program that's linked against the free software Lib QMI G-Lib inside  the modem so you can develop it like you run it on a laptop but you can run it transparently  inside the modem after cross compilation.  There's also a tool which we call Osmo QC-DIYAC which is basically a host tool for obtaining  these DIYAC based logs from the modem so you can run this on your laptop attach it to the  modem and then you get all kinds of traces not only the interface traces but also QMI  protocol traces which we then again decode using Lib QMI so you get a textual representation.  There's ongoing work here to basically move all of that into wiresharks so you get the  full decode of that in wireshark but that's not yet there.  So what kind of user space programs do we find?  Okay ADBD we know what that does.  We have an ATFWD daemon.  Well AT forwarding daemon well what does it do?  It implements those things like an AT plus Q Linux CMD and other AT commands so basically  a user space program on Linux can register like a callback within the modem to forward  certain AT commands into the Linux user space where you can then implement them so you can  basically implement custom AT commands in user space programs.  There is all kinds of other software which we haven't really figured out yet a lot what  they do.  There's one monster called the QC map connection manager which basically allows you to run  a Linux based Wi-Fi access point with LTE backhaul excuse me for the typo.  So basically you have to attach a Wi-Fi chip to the SDIO interface of your modem module  and then you have a full personal access point device that has an LTE backhaul.  And then the Wi-Fi you like the parameters for example the key and the SSID and the channel  and so on you configure all of that through what?  AT commands of course.  If you ever wanted to look at software that receives AT commands and then generates textual  config files for WPA and for host APD you can look at this code.  I prefer not to.  So we have the QuackTail bridge which is a very simple device.  It reads from one device and it writes it to another device.  And apparently this is such a complex task that you need to schedule a different process  and I think it has three threads or something obscure like that.  So okay well.  Which brings us to the funny bits and pieces that you find in those modems.  Well the first thing is AT plus Q Linux command.  We spoke about that.  You can run arbitrary shell commands as root in a read write rule file system in there.  So basically you can do anything.  I mean you can send an RM dash RF slash and then well it's gone.  It's dead Jim.  So we also have commands to switch into fast boot mode so you can update the firmware.  You have a special AT command to print a DMSC and you have also all kinds of other AT commands.  And when you do a strings, just a strings call on those executables it looks like shell  scripts in many cases.  And one of the most funny things I found in this modem is how many processes and threads  does it take to reboot a system.  It's apparently a very complex question.  How do you reboot your system?  Apparently this was the easiest method they could find.  So there's one process, the reboot dyag app which registers a dyag command with command  code 0x29 with the dyag infrastructure and it spawns a thread which executes another executable  called QMI simple real test with an input FIFO.  And then it calls system echo modem reset to write modem reset into the FIFO of the input  of the process it just spawned which then causes this to send a QMI message to the modem  which will reboot the baseband processor.  And then of course you clean up.  You remove that temporary file because slash TMP is not a tempfist but it's read write  file system.  And then it goes on to write the string reboot into slash def slash rebooter def using F  write not using echo this time.  This is a C program, not a shell script.  So apparently they discovered that and used F write.  And then we have a reboot demon, a second process again with two or three threads which  reads this rebooter device.  And then actually they published the source code so this is the actual source code.  So you read from this def rebooter device then there's a nice comment documented what  it does.  You do a string compare.  You have the first printf going for reboot.  You have the second printf initiating reboot and then you call system on the reboot executable.  So this is apparently the most simple method.  So if you ever were wondering how we would reboot your Linux system, this is the new  reference.  Then we have C programs that look like shell scripts.  So this is an actual condensed output of strings on the QuackTill demon.  You see echo in sysfs files.  You see copying files.  Even with semicolon in there.  You see echo into the duty cycle of some pulse with modulation.  No idea what that does.  And then they even grab for the process and they kill processes with the most obscure things.  And they even don't use reader but they do like ls and then parse the output of that  rather than use open-dir reader and the usual calls you would do to get a list of files.  It's quite amusing.  Which brings us to the topic of firmware updates.  But you can, I actually missed this example, sorry I have to make a quick interruption  here.  All these machine to machine modems they typically have an LED and the blinking rhythm of the  LED indicates to you whether it's registered to the network, whether the data connection  is open, whether it's searching for networks, that's all different blinking patterns of  the LED.  And how do you implement this on this modem?  Well, you run a user space demon that calls system echo 1 to the dev GPIO which controls  the LED all the time, you know, to blink the LED.  It's not like the kernel wouldn't have infrastructure for LED blinking patterns and so on and so  on.  But okay, so you have a demon that does nothing else but basically toggling your GPIO by spawning  shell processes using the system syscall.  Okay, with that I hand over to Olga.  Now the question is do you expect anything after the topic of firmware upgrade or is it  going to be an empty slide?  And the answer is they know that they have to offer firmware upgrades through over the  air updates making it as small as possible.  And actually it's something that Qualcomm is preparing for modem vendors.  So it's based on the Android from around 4.0 recovery gets, not using Android myself.  I haven't looked at how it worked before.  Many of you might be more familiar with it, but it's mostly a zip file and it contains  delta updates and surprisingly they are somehow hashed to a SHA-1 and then the SHA-1 is being  signed with a private key and the result is being put into a comment of the zip file.  That's probably pretty standard but looked a bit odd and it's nice that they prepare  probably secure firmware upgrades with like minimal delta functionally.  So what has QuackTel done to this code?  They still use it.  That's quite nice.  But they have removed or actually not really removed but they're not using the RSA code  to verify a signature.  And instead of using the standard Android way to patch the systems, they use a proprietary  component from a company that used to be called RedBand but now it's Harman and probably  soon Samsung.  This RedBand component is nothing that QuackTel has written but it's a commercial product  and it's used in the QuackTel UC20 module as well and also in other automotive projects  we have seen RedBand updates being used.  So at this point I started to look into how does this update format look like and instead  of presenting a very complex form of the format I have this slice.  So other people like Matthew Solnik at I think Black Hat presented something called attacks  on OMA device management.  These can be remotely triggered but the update mechanism used here cannot be remotely triggered.  The modem needs to be asked to start an update so that's already a bit more secure but I  started to look at the HexThumbs of a specific delta update and with a lot of help from Ditas  Spa we actually managed to understand how the updates binary looks like and we have  created a small tool to take an existing update and put it into smaller parts and also be  able to create our own defiles.  The format itself has many different pointers and offsets so in the example you might already  see the offsets here and compression size.  So it starts with a common header and then after the header you have a LZMA compressed  table of contents or however you want to call it and outside in the header you have an offset  into the decompressed version of this table of contents where file updates start and when  you start creating your own file you might get the offset wrong and the update binary  just crashes with your malformed update file so it's not very robust code, a very complicated  file format and nothing is cryptographically signed.  So when you use strings on the binary you see the word signature but it only refers  to a CSE32 checksum.  The next part like now we understand how the update format works.  We can create our own update files.  The question is how do they end up on the device and it's something that is implemented  in the AT forward that Harold or LaForge has mentioned and if you grab for some specific  strings like W gets, the QC map connection manager or photo or update dot zip you already  kind of guess how this application works.  So you issue an AT command with a URL for updating it, in turn it will disable your  normal IP connection that you have might established on your host and opens our PDP context on  the device itself using the QC map connection manager then it will spawn W get to download  the file then it will use system to move it to the right directory.  It will remember what it wants to do with this file and it will reboot into the recovery  partition and system and at this point the update zip file will just be applied and the  system will reboot once again.  Again without any cryptographically signature or checks.  So if you manage to hijack the update process you can install any binary on the device or  on the modem or anywhere else on the system as you want.  But instead of just seeing how bad it is we want to say like what do we expect them to  do and I'm handing over to Harold again.  Yeah so rather than saying oh this is all bad, it's all unlocked and it's insecure and  so on, well it's fun for us of course because that's what we wanted right.  We wanted a modem device where we could do basically whatever we want to and where we  don't have to break sophisticated security mechanisms that are designed to keep the user  or the customer, the owner of the product out.  So yes there are security issues and those security issues must be fixed but we need  security mechanisms that work without locking out the user or the owner of the device of  course.  So this is our public call to the manufacturers if you fix those issues keep in mind that  the openness of the platform is interesting for all kinds of legitimate use cases and  while you want to protect against malicious attackers you of course still want to enable  the actual owners of the device and the users of those devices to use the flexibility they  provide because there's a lot of, yeah.  So what's the status today in Outlook?  Well we have just opened the Wiki on the last slide there are all the links so on the Osmo  com project we now have a Wiki for QuackTel Qualcomm LE modems where all the information  that I gathered from reading the thousands and thousands of lines of source code is in  there.  We have released the debug tools presented in this talk in a Git repository in source  code.  The hardware boards are released as open hardware and available and what's unfortunately still  ongoing is the libQMI Glib integration which is to some extent to the fact that I've never  written anything against Glib before or anything inside a program that uses Glib.  It's like all this infrastructure I'm usually more low level than that.  And while we hope to grow this documentation and we kindly invite all of you with an interest  in understanding those platforms better to help us out.  You don't actually need to necessarily reverse engineer and disassemble things.  It's just read the source code, understand what it does and play a bit with the device.  We are planning an open embedded package feed so we can actually easily install additional,  everyone can easily install additional packages on those modems.  There's plenty of flash, I think it's like 20, 30 megabytes of free flash so you can  install quite a number of additional packages in there.  And our aim is to have free software only user land on this Cortex A5 CPU so to do away  with all those proprietary processes that run in user space and the libraries and run  the open source kernel and have basically the libQMI Glib integration and all other  bits needed to run our own standard Linux user land code in there and have custom images  that we can run on modems in all kinds of use cases.  Okay.  Now before we go for Q&A in a minute, there's an unrelated announcement that we would like  to present here.  The OsmoCom project has gained support for running your own 3G 3.5G network during the  last year.  Not sure who has noticed that.  Unfortunately, it suffers a bit of a lack of contributions.  So we want to motivate people to contribute more.  And we have just started an Accelerate 3.5G program which provides 53 3.5G femtosells  to people who can convince us that they would contribute something reasonable to our project.  So these femtosells are already supported by the OsmoCom code.  So using the femtosell and the OsmoCom code, you can run your own 3.5G network.  And we're giving those away for free.  So if you're interested in any of that, please submit a proposal until the end of January  and then you will hopefully receive your free femtosell until the end of February.  Which brings us to questions.  Yes.  It's Q&A time.  We have a total of eight microphones here.  One, two, three, four, five, six, seven, eight.  I can do math.  Please step up to the microphones to ask your question.  Meanwhile, we have from the signals angel, we have a question.  Yes.  The internet wants to know if there will be in quotes next open moco.  Not from us.  No, we're not working on a mobile phone.  We're looking at modems, not at mobile phones at the moment.  So microphone two.  So just to clarify, does it mean that Linux runs on the iPhone 5?  We don't know.  We don't know.  We'd be happy to hear from you if you can find it.  But I think you can run the trip with something else on this Cortex processor.  But we don't know.  Signals angel?  Yeah.  The internet wants to know why there are no mini PCI express slash M2 to USB 3 adapters  because they think LTE is capable of at least 300 megabits and USB 2 could be a bottleneck.  Well, I'm not really in the business of manufacturing or selling adapters.  So I mean, yes, we did this open hardware device out of a specific need, but you would  have to ask the hardware manufacturers that I'm sorry.  Can I ask the people who are leaving to leave quietly, please?  Can you buy the entire EC20 with the breakout board and the UR as a single kit somewhere?  Well, you can contact us, but it's not really something that we have prepared for.  But yes, it's certainly an option.  But I mean, we're not here to sell you anything.  We're here to invite you to help us learn about modems.  Thanks.  And with the Android debug shell, you actually don't really need the serial.  So you can refresh the device using USB, even if you break it, it can be refreshed.  And with ADB shell, you have your root shell.  So that works.  And with ATQ Linux command, you can actually start a lock-in on the NMEA console as well.  So even if ADB doesn't work, you can get a lock-in on it.  So it's very easy to get started without serial.  Signals Angel?  Hello?  Okay.  Do you have, have you tried to get a source code from other manufacturers?  Well, we know at this point of, I think, three different manufacturers that use this MDM9615  plus Linux combination from QuakeTill.  We have just described how it went.  Sierra has published all this already by themselves in a very good way.  So there's no need to ask them.  It's out there.  You can just download it and it's documented.  And the third one is an old end-of-life Huawei module where we have also asked, but this  is still ongoing.  They are now, it's interesting, lead it from a Chinese supplier.  You get the excuse, oh, there is Christmas coming up.  I was like, okay, so many Christians in China, but okay.  I think I'm quite sure if I ask in January, oh, there is Chinese New Year coming up.  But we'll see about that.  Microphone 4.  You mentioned the Qualcomm chip used in the iPhone 5.  Does this mean, well, is it likely that the iPhone application processor talks AT commands  to this Qualcomm chip?  Is this still a state of the art?  No, it's not state of the art.  I don't think it's happening.  I mean, also on these modern modules, the AT command is there for legacy purposes.  You have the QMI exported over USB and normally your motor manager or whatever you use, or  phone or whatever infrastructure talks QMI to those devices.  I'm going to go back to the signals again.  The question is, what is the total size of the flash or the root of S on there and how  much RAM has this thing got?  I don't know.  I don't remember the flash size.  Was RAM 32 megabytes?  Yeah, this 32 megabytes of RAM on this, the successor, the EC25 has 128 megs of RAM.  But a flash size I also don't remember right now.  But check out the wiki.  We have boot logs and all kinds of like D-Mask outputs and so on all on the wiki.  So I'm quite sure it's somewhere in there.  Microphone 4.  You said that you tried to check Legato Linux from Sierra.  Wouldn't it be possible to completely build your own kernel there and your applications?  Yes, it's certainly possible.  But then of course, if you do this, you buy a lot of things with your money that you don't  use in the end.  And you support a vendor that tries to lock people into this proprietary framework.  So it's possible.  But do you have to use a framework?  I don't think so.  No, you don't have to.  But I mean, they developing this and part of what you pay for is this framework.  And that's why their products are more expensive.  Microphone 2.  Yes, do you have some ballpark figures about the power consumption in the lowest power  quiz and current mode?  No, I'm quite sure it's going to be high.  I think Holger has some comment on that.  So when you look at this device, you use S-Trace against the Q-Max-D and it's like mostly waking  up every couple of milliseconds.  So it's not power efficient.  It's a C code, it's really, really, really annoying to see processes that run all the  time.  And from the previous slide, this echo mem to CISFS, that's actually the advanced power  management.  So you can trigger an AT command to have the device sleep and it writes echo mem into it  to go to sleep.  So it can be tuned and with a free software user space, it can probably be better than  what it is right now.  Signals Angel again?  Yeah.  How accessible is the baseband?  Like memory and like EMI and stuff?  We haven't really investigated this in detail, but we also haven't seen any signature verification  and so on there.  So I think it's completely open.  So the DSP firmware is in three separate partitions and there's no RSA signature and  it's the end of these partitions.  So it seems that you can modify it if they have locked down like the poke command.  That's something we haven't tried or looked at it.  Microphone one.  The modules readily available in the LGA package rather than the mini PCI dev board, like if  people want to start to use these in open hardware.  They should be available again.  Yes, they were temporarily not available due to the TPI enforcement, but I think now they  should be available again.  What are the quarter of the costs of those modules?  I think it's like 47 something euros around that region or maybe no, it was more than  50, but anyway, somewhere in that region.  So one more question from the internet.  And the internet wants to know if you can capture layer 2 network stuff with this.  Yes, yes, you can.  So I think that's the end of our Q&A session.  Please join me in thanking LeForge and Holger on this fantastic talk about things that I  really didn't want to know about 3G modems.
33c3-8151-dissecting_modern_3g_4g_cellular_modems_0_419.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_420_647.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_648_731.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_732_1547.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_1548_1679.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_1680_1751.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_1752_3887.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_3888_3983.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_3984_4151.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_4152_4247.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_4248_8699.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_8700_15647.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_15648_15755.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_15756_18803.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_18804_22199.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_22200_22895.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_22896_23303.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_23304_30815.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_30816_33527.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_33528_33659.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_33660_33707.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_33708_33779.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_33780_37307.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_37308_49199.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_49200_49271.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_49272_60359.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_60360_60695.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_60696_68483.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_68484_68579.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_68580_69023.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_69024_74663.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_74664_74723.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_74724_74987.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_74988_75035.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_75036_75119.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_75120_75299.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_75300_75575.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_75576_75995.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_75996_76811.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_76812_76967.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_76968_77027.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_77028_78143.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_78144_78263.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_78264_78503.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_78504_78815.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_78816_79643.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_79644_80015.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_80016_80795.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_80796_81179.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_81180_81587.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_81588_81647.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_81648_81695.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_81696_81791.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_81792_81887.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_81888_83195.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_83196_83591.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_83592_84539.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_84540_84659.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_84660_84779.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_84780_85919.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_85920_86999.png 33c3-8151-dissecting_modern_3g_4g_cellular_modems_87000_87316.png