Beyond Virtual and Augmented Reality
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kunze, Kai
With recent development in capture technology, preserving one's’ daily experiences and one's’ knowledge becomes richer and more comprehensive. Furthermore, new recording technologies beyond simple audio/video recordings become available: 360° videos, tactile recorders and even odor recorders are becoming available. . The new recording technology and the massive amounts of data require new means for selecting, displaying and sharing experiences.
 I'm inviting you to a little imagination activity. Imagine that you can take all of your experiences that you have made at this Congress, take them home with you and share them with your friends and your family. Everything. The first time that you tried mate. The moment, endless moments of going through the labyrinth of this Congress hall and getting lost for the hundredth time, trying to enter the club downstairs and feeling like you can cut the smoke with a knife or actually being very surprised and seeing Snowden appear on stage. So all of these wonderful memories, imagine you can record them and you can take them home with you. Today, this kind of recording activity or possibility is actually possible to make. This technology that could theoretically record everything 24 seven. Kai Kunze will give us an overview of sensual and actual actuation technology and he will tell us what the future holds for AR and VR. He has brought with him some experience, some projects that are actually happening right now and will give us an overview of how we are already pushing the boundaries today of human sharing experience. Kai is a researcher in wearable computing in AR and VR and really cool. He's also a founding member of the Japanese superhuman sports society. Thanks, is this on? And with further ado, Kai Kunze, enjoy. Is the mic on? Yeah, okay. Thanks a lot for the great introduction and thanks a lot also for waking up so early in the morning to come and join the talk. It's always a special occasion for me to talk at the Chaos Communication Congress and I hope then for the next 30 minutes or so on, I won't be wasting your time. So the talk will be about kind of from superhuman sports to amplifying human senses and as the introduction said already, we are in the year zero of consumer available VR devices. So Oculus, the wife, and also the Sony VR system and the cheaper cardboard variants like Google cardboard and so on. And we are in the brink of getting consumer level AR. And you know, kind of already my grandparents have HoloLens. So yeah, in this case, I always like to include pictures of my grandparents because they are awesome. I always try out any type of technology I give them and they provide valuable feedback. However, I have to start with a big disclaimer and I will disappoint you, especially regarding the great introduction I got for the talk. Please don't trust me, especially about predictions. I was working 15 years now in the variable computing field and around 2004, I thought, you know, today we would run around like this. I see a lot of you wearing your head mounted displays and having your one handed keyboard. Nobody. Dammit. So I'm really, really bad with predictions. That was me at the time. One handed keyboard, Q-Big. This is a belt integrated computer from ETH. At the time, you have to remember we didn't have smartphones, so we had to carry our computing somewhere else and then, you know, kind of seriously a couple of years back, I also thought this would be a thing. So Google Glass and this type of stuff. So don't trust me about predictions and don't trust me on what's coming next. However, for this talk, you might wonder why did you get up so early in the morning then. What I will do today is I will talk a little bit about a hot topic in Japan right now and that's super human sports. The idea for super human sports is to use technology to enhance human abilities over, you know, kind of what we usually can do. And because, you know, we are in Japan, we of course founded a society. So since one and a half years, we have the super human sports society disclaimer as said in the introduction. I'm one of the society members. And also one of the, so originally the idea came from Inami sensei and Rekimoto sensei and Nakamura. I usually don't like name dropping, but especially if you're interested in human augmentation and human computer interaction, check out all the works from Inami and Rekimoto and also follow them closely because they are two of my personal heroes. And then, you know, kind of we have the society now. We're trying to expand it internationally, so since a couple of months we're working with TU Delft, the Sports Engineering Institute. And I don't want to bore you longer. However, I just want to highlight one person in the advisory board and that's Robert Riener because Robert kind of saved 2016 for me because, you know, it started all with Boy and so on and then Brexit and Trump, but Robert organized the Cybertlon in Zurich. I just wondered how many of you heard about the Cybertlon? Oh, okay, a couple. Whoa, nice. Has anybody been there? Oh, yeah, even great. Cool. My kind of audience. Awesome. So in this case, this was an international competition for disabled competitors using bionic assistive technology, so exoskeletons and so on. And you know, kind of that was just great and it's in the terms of what I want to talk to you about today. But what I want to talk to you about today is then super human sports and I kind of selected these type of topics, so augmenting our body, augmenting the playing field, augmenting training and cheering. And I will show you a couple of examples from researchers in the society. So first, going over to augmenting our body, there's a lot of work also mechanically that's not so interesting for us computer scientists. However, it looks cool, so I included it. One is a company called Skeletonics. They manufacture these exoskeletons. They are completely moved by muscle movement. So nothing IT here, but still cool. And then another thing is the bubble jumper. That's an idea that came out of a hackathon. We organized at KO and we started, you know, kind of with first sketches and so on. But the important part also for our hackathons is you have to demo. You have to implement. It should be playing. Oh, yeah. So you see the combination of a bubble ball and sky runner and to make a very safe and nice sport. Yeah, coexistence of safety and force. I'm not so sure about the safety. And now getting over this to something a little bit more serious. Spider vision. This is now work from Inami sensei. In this case, this is really about extending the human field of view. And it's very, very simple. They use the DK one and they have two cameras, one in front, one in the back, and they overlay the two video feeds on top of each other. Like you will see right here. And the interesting part is at the beginning it looks really terrible, but after two or three minutes using this, you can really separate the two images and you can even read or do other things. So you know, kind of then you can be environmentally aware. The only trouble is, and that's something you see here, because you're wearing a head mounted display still VR set. You know, there's an offset from the camera to your actual eyes. So it's very hard to interact with your environment. But that's kind of the initial work where you want to get at. The next work I want to show you is the synesthesia suit. It's a full body haptic feedback suit made by Misoguchi and Minami Sawa, two of my colleagues at KOMedia Design. And this was first done to promote rest. That's a VR game for the PlayStation. And what it gives you, you see it here on the right side, is it has a haptic feedback vibrators 24 on the whole body. So you can get a touch feeling on your whole body. And the idea Misoguchi sensei wants to do with this is he wants to give you simultaneously visual as well as auditory and haptic and touch feedback to put you in some kind of trans state. And if you try, you know, rest with this suit, it's quite an experience actually. So you get, you know, visual auditory and touch feedback at the same time. Then moving over to augmenting the sports field. Here I want to highlight a work from Rekimoto sensei, Aqua Cave. You see a water tank you can swim in, but the swimmer actually wears 3D shutter glasses and you have a 3D display around the water tank. So you can then really dive in the great barrier reef or you can do competitions with sharks and similar things. Then augmenting training. Here I will show a little bit of old work. Maybe some of you will know, but I just find it cool enough to still include it. And that's galvanic vestibular stimulation by Maeda and also in Ami sensei. And what they were doing in 2004 is they apply a tiny current behind your ear. So you have anode and cut out behind your ear. And what happens if you apply this tiny current is you're messing with your equilibrium, with your sense of equilibrium. So you will tend towards the anode if you do that. And then now the reason why I show this video is because this would be impossible to do for a researcher in Europe or US. You see a student wearing this device. You see the professor with a control unit. And you can steer the person just left and right. And I have to say I tried it, it's not safe to try out. We don't know about long term effects. But it's interesting and it's also user dependent. So depending on the user and depending on your skin, you might need more or less of this electricity. And it's an interesting thing to think about, especially for sports that require equilibrium or other things. And then another work, this is work from one of my students, Takuro. He was working on peripheral vision glasses. And his first idea was to just give you notifications in a way that other people don't realize. So I could look still into your eyes and you don't realize that I might get some notification on my peripheral vision. I can still keep eye contact. So far if you look at your watch, the other person will realize that you're looking there. So he started with this idea. And the boring thing is, yes, it works for notifications. So you can have up to eight different types of notifications. I don't know why the video stopped, but that's actually not so important. You can have eight different types of notifications. The other person cannot see that you're getting them. However, what he really wants to do, and here we get to the interesting part, is he wants to try to influence movement over your peripheral vision. And there's some other research, some interesting research from Furukawa et al. also, augmented human conference. They use some patterns on the floor to guide pedestrians, to influence the movements of pedestrians. And then Takuro wondered if it's possible using then also peripheral vision glasses. And this is the first test. This is partly unpublished, but I thought I want to share it with you. So in this case, what he's using is he's using some lines that move along this way. And he lets people walk. And guess what happens if you let the lines move faster? Yeah, I heard. So people will move slower. So walk slower. So it's already possible to maybe use it to influence speed. And the next thing he wants to try is also try to influence direction. So the next thing then is augmenting watch and cheer. So in this case, I want to show effective wear. That's glasses that can detect your facial expressions in a very cheap or simple way. That's Masai. And you see kind of the faces he makes. He wears his own self-made glasses. And how this works is he uses just photo-reflective sensors in the glasses frame. And he detects the distance to the skin. And with this, you can get user-dependent very well what type of expressions a person is doing. So you see then the, I like the training phase. So here you see the photo-reflective sensors and the distance measures between the sensors. And then kind of for the start, he has to then calibrate the sensors once. And here you see the principles. So for example, for smiling, kind of your cheeks move up. So it's easy to recognize that you're smiling and even laughter. So eight facial expressions are recognizable. And yeah, and I like just Masai making faces. That's the normalization phase, then the training phase, and then the recognition phase. That's user-dependent, but it's very cheap. And if you want to build this, especially smiling, it's easy. You just need one photo-reflector here above the cheekbone, and then one maybe for reference. And then it's easy to implement. So of course, kind of what we can do with this is we can put it into a VR headset, and then your character in VR can have the same facial expressions. You can think about remote participants in an event or so on, and then also showing the mood or so on of the remote participants. It's not playing. Yeah, but I think you get the idea. I don't know why the video stopped. And also more interesting then is you can also maybe use this for design. So if you want to design a very, very scary dungeon, you can kind of detect how people are reacting to some elements in your VR environment and then do the correct adjustments. And also you could wear it throughout the day and just get an overview over your daily activities so you know kind of how much smiling versus frowning are you doing. Also related to some of the activities you have. That's interesting, and that's in line with some work also from Rekimoto Sensei who did the happiness counter. So here the idea is he uses smile as an interaction modality. So in the morning when you wake up and your alarm clock rings, you don't have to push a button but you have to smile to your alarm clock. That's a little bit scary or creepy. And because we are with this, this next one is a little bit of a joke. You know kind of if you recognize that you don't smile enough, we can also make you smile. So this is now, Tuniji Smile is now activated over the electrodes. As I said, this is a little bit of a joke. I don't recommend anybody or I hope nobody will make these type of technologies in the future. But here the interesting part is students were playing with EMS and we recognized one of the main, most of the nerves to control our face actually come over from this side of the face. So what you can do is if you put electrodes on different parts here, you can actually activate different muscles. The students wanted to get smiling working but first when they tried it on me, my eye was blinking and other things. You can also do eye blink. And then, I mean this is a little bit on the joke side as I said before. However if you want to have references for more serious electric muscle stimulation work, I refer you to Max Pfeiffer and also Pedro Lopez. They also have a kit out, so two references to URLs if you want to build it your own or if you want to build your own EMS and also their source code for all of their papers is I think out there on Bitbucket and GitHub. And this is also just cool, you know, cruise control for pedestrians and also muscle plotters. So using your body as output for a computer. However, now getting back to the super human sports. So the idea also for the super human sports society is to work towards Olympic 2020 and we want to create actually an augmented game culture in Japan. So not only inventing new sports and using new sports to pursue amplifying human senses, but also creating this culture to make these sports. So we are organizing workshops. This is an example from Wycombe beginning of this year. So you know, kind of you have a lot of devices. You get together researchers, artists, designers, but also local people that are just interested in participating. We do one day of ideation and trying out different types of technologies and research outputs. So this is for example also check in one of the researchers. You see all of the four people, you know, you have a view of all four people. Anyways and then at the end trying to design these games together. So these are some more pictures from some of the workshops. And here the idea is first of all, as I said before, we want to create a proof of concept by making and playing sports and augmented sports. So not just ideation sketches or so on, but really demonstrations, demo or die. And then also getting fit ourselves so far it didn't really work for me, but maybe in future and then also creating this augmented game creators for the time being. And now you might wonder after these slides, what's the scientific impact? I mean it's a lot of fun to do this, but you know, why am I focusing on this? And if you heard my previous talk two years ago, might know that I'm interested in recognizing and improving cognitive activities, so concentration, attention, comprehension. And I focus a lot on eyewear computing. That's my old talk, a little bit of background about myself. I was working with Jin's Japanese glasses company on sensing glasses that have accelerometer and also some low level eye tracking in them. And now since actually December, I have some good news. I have support from the Japanese government, especially as JST, to sponsor Collective Open Eyewear platform. And I'm also looking for collaborators to work on this because I think this is crucial to start now. Because I don't want to end up in a situation that we have today with our mobile phones that we have just more or less, or that most people just have two providers, two big companies that tell you what you are able to run on your mobile phones. And I think eyewear for me, believe it or not, because my track record is not so good predicting the future. However, I think eyewear is the next stepping stone, it's the next thing, and I really want to have an open eyewear platform out there. So that's the background for myself. And I think for me, the super human sports is a great testing ground for these type of technologies and then really not only doing sensing, but also really trying to amplify human senses. In this case, I have to hurry a bit because I'm behind time and I want to still show a demo. However, here the idea is that sensors, digital sensors are in some parts already better than human sensors. So you can have higher frame rates. You can have broader spectrums, so you could see infrared and so on. However, what is lacking from our side is the interface to it. So I wonder really, can we create new and amplified sensors based on digital technologies? And just going with the digital camera system analogy, I would just want to give you a small example at the end of the talk now. And that's squint to zoom. So if I would be standing here and there's a sign in the back and I want to read the sign and I cannot do it with my normal eyesight, I will squint. And wouldn't it be nice if I could wear some kind of glasses that automatically if they recognize that I tried to read this and squint would just zoom it in. And I don't even realize that I'm using technology. And unfortunately, I wanted to show you a demo on the HoloLens, I have one here. However, I didn't manage to get this working on the HoloLens. And I'm happy that I have a student, George, who could set up a demo on a laptop that gives you a feeling how this works. So can you switch? Yeah. So in this case, this is unfortunately just on the laptop, but you can get an idea. In this case, I'm using a Tobi eye tracker to track my eyes. I have a camera and just a simple open CV to track my face. And then if I now go ahead and squint, it zooms in. I can show you also this on a document. So if I squint, it zooms in, if I don't, it stops. And that's actually a very, very simple type of interaction you could do. I hope that could show it to you on an AR system, but that's maybe something for next year. Here again, as I said, kind of the desktop eye tracker. And I can show you a little bit how it works. So this is just an open CV system. I think George uses the eyebrows and the relation to the nose and the size of my eyes. So if I squint down, this relation changes and it zooms in. And if I stop, it zooms out. So this should give you a small idea of what I'm trying to create or what I want to create. Things, technology that's easy to use, that you don't even realize that you're using, similar to the analog glasses and so on. And also not only for sensing, but hearing and other senses as well. So this brings me to the end of my talk. I just have some obligatory thank you slides. So these are all people I'm working with and that contributed to the research you've been seeing. And I have a special thank you slide for the students that actually did the work and especially George who made the demo you just saw, but also Masai for the effective wear and the rest of the crew. And now this is the end of my talk. So now I think there's a time for a couple of questions, remarks or violent descent. So thanks a lot for your attention. Wow. Really cool stuff. That was so cool. You have a chance to talk to Kai right now to ask him questions. There are four microphones in the room. So find yourself behind one of those microphones if you have a question. Also the internet is always able to ask questions if they are awake by now. Does the internet have anything? Yes, one question from the internet. Yes. Is that code for eye tracking available? So the which one? I can recommend you pupil labs. So the eye tracking code right now I'm just using Tobi IX. So that's kind of Tobi proprietary. They have an API. It's just a desktop eye tracker. However, if you want to have a mobile eye tracker, I would recommend you the pupil labs. That's an open source eye tracker. They just keep the latest revision of their hardware close source. And they have all of their code up on GitHub. So it's very, very easy to adjust. And they also have settings for VR headsets. So you can easily use it. You can just, you know, kind of use, build your two own cameras. That's easy. Use their code if you know kind of you have budget constraints. The Tobi IX is okay, but I don't like Tobi or SMI so much because they use close source APIs. And it's hard to get to the actual images because, you know, you want to have, for example, pupil dilation because that gives you information about stress and other things. And that's not so easy. Great. Actually, let's go to microphone number six, please. So I have just a quick question about how open you are to be able to collaborate with different universities from around the world to use your technology and research. I work in the effective sciences department in the University of Geneva. And we actually use a lot of motion tracking. This eye movement tracking would be super interesting for effective science research, especially as well our facial expressions. We use that a lot. So effective eye movement would also be interesting. So you've discussed how to use it for like opening up possibilities in the real world and actually enhancing everyone as a general population, but how open are you to using it for research as well? Yeah, very open. I mean, that's also the reason why I give the talk. I mean, now finally I get also a little bit of money to fund this from the Japan side. And I really want to find people that want to collaborate on this side. And I think, you know, kind of especially seeing how Silicon Valley now also invests in eyewear. And I think the next thing we will see is eye trackers in VR and so on and how close those systems are. I would really like to, you know, have this collaboration. So just try to catch me afterwards or just send me also a mail would love to keep in touch. Great. Thanks. And microphone number two, please. Well, thank you. Amazing talk. And how important do you think will be the combination of external signals like squinting or looking at something and internal signals or like directly measuring the brain waves or with an EKG? Do you think that is important to combine those or? It's important to combine those. I actually started looking into a lot of the EEG signals and so on. When I started this work, but it's very hard to get into your skull. So EEG, also brain computer interfaces, I think will take a while to work. And you also have this trouble of the signal processing is relatively hard. And because I'm lazy, I started with the eyes and the eyes give you actually already a lot of information you want to get from the brain. I like to think about combinations. So for some things we already working with also EEG. And I think it's crucial to find the right type of sensing so that it works throughout the everyday life. I think one example was also JINs and now with the JST project, what we really trying to do is find simple sensors to give you some information about your brain activities and so on. I think it's really a combination of multiple sensors. And I think, as I said, the eye is very, very interesting. Yeah, thanks. We have time for one more brief question. Number four, please. When I was wearing AR glasses, my brain messed the distance to objects so I couldn't grasp anything. It just wasn't there where I expected it. Is there any chance this problem will go away? This depends highly on the lens. So there's still this problem. It's also in VR. I mean, VR has this trouble with the perception. I think you will have always this issue, I found AR, so the HoloLens quite good for this. But here the problem is also you cannot grasp something because the distance to the object is too far. So I'm still wondering, I haven't seen something that's really kind of graspable in VR where you have the right distances or so on. I think either your brain has to get used to it, which I think it does after a while. It gets the distance measurements correct or we have to employ some other tricks to do that. What you can do for that, I'm not so sure. Thank you. Great. And with that, we're at the end of the talk. Thank you so much, Kai. Please help me thank Kai Konso.
 I'm inviting you to a little imagination activity.  Imagine that you can take all of your experiences that you have made at this Congress, take  them home with you and share them with your friends and your family.  Everything.  The first time that you tried mate.  The moment, endless moments of going through the labyrinth of this Congress hall and getting  lost for the hundredth time, trying to enter the club downstairs and feeling like you can  cut the smoke with a knife or actually being very surprised and seeing Snowden appear on  stage.  So all of these wonderful memories, imagine you can record them and you can take them  home with you.  Today, this kind of recording activity or possibility is actually possible to make.  This technology that could theoretically record everything 24 seven.  Kai Kunze will give us an overview of sensual and actual actuation technology and he will  tell us what the future holds for AR and VR.  He has brought with him some experience, some projects that are actually happening right  now and will give us an overview of how we are already pushing the boundaries today of  human sharing experience.  Kai is a researcher in wearable computing in AR and VR and really cool.  He's also a founding member of the Japanese superhuman sports society.  Thanks, is this on?  And with further ado, Kai Kunze, enjoy.  Is the mic on?  Yeah, okay.  Thanks a lot for the great introduction and thanks a lot also for waking up so early  in the morning to come and join the talk.  It's always a special occasion for me to talk at the Chaos Communication Congress and I  hope then for the next 30 minutes or so on, I won't be wasting your time.  So the talk will be about kind of from superhuman sports to amplifying human senses and as the  introduction said already, we are in the year zero of consumer available VR devices.  So Oculus, the wife, and also the Sony VR system and the cheaper cardboard variants  like Google cardboard and so on.  And we are in the brink of getting consumer level AR.  And you know, kind of already my grandparents have HoloLens.  So yeah, in this case, I always like to include pictures of my grandparents because they are  awesome.  I always try out any type of technology I give them and they provide valuable feedback.  However, I have to start with a big disclaimer and I will disappoint you, especially regarding  the great introduction I got for the talk.  Please don't trust me, especially about predictions.  I was working 15 years now in the variable computing field and around 2004, I thought,  you know, today we would run around like this.  I see a lot of you wearing your head mounted displays and having your one handed keyboard.  Nobody.  Dammit.  So I'm really, really bad with predictions.  That was me at the time.  One handed keyboard, Q-Big.  This is a belt integrated computer from ETH.  At the time, you have to remember we didn't have smartphones, so we had to carry our computing  somewhere else and then, you know, kind of seriously a couple of years back, I also thought  this would be a thing.  So Google Glass and this type of stuff.  So don't trust me about predictions and don't trust me on what's coming next.  However, for this talk, you might wonder why did you get up so early in the morning then.  What I will do today is I will talk a little bit about a hot topic in Japan right now and  that's super human sports.  The idea for super human sports is to use technology to enhance human abilities over,  you know, kind of what we usually can do.  And because, you know, we are in Japan, we of course founded a society.  So since one and a half years, we have the super human sports society disclaimer as said  in the introduction.  I'm one of the society members.  And also one of the, so originally the idea came from Inami sensei and Rekimoto sensei  and Nakamura.  I usually don't like name dropping, but especially if you're interested in human augmentation  and human computer interaction, check out all the works from Inami and Rekimoto and also  follow them closely because they are two of my personal heroes.  And then, you know, kind of we have the society now.  We're trying to expand it internationally, so since a couple of months we're working  with TU Delft, the Sports Engineering Institute.  And I don't want to bore you longer.  However, I just want to highlight one person in the advisory board and that's Robert Riener  because Robert kind of saved 2016 for me because, you know, it started all with Boy and so on  and then Brexit and Trump, but Robert organized the Cybertlon in Zurich.  I just wondered how many of you heard about the Cybertlon?  Oh, okay, a couple.  Whoa, nice.  Has anybody been there?  Oh, yeah, even great.  Cool.  My kind of audience.  Awesome.  So in this case, this was an international competition for disabled competitors using  bionic assistive technology, so exoskeletons and so on.  And you know, kind of that was just great and it's in the terms of what I want to talk  to you about today.  But what I want to talk to you about today is then super human sports and I kind of selected  these type of topics, so augmenting our body, augmenting the playing field, augmenting training  and cheering.  And I will show you a couple of examples from researchers in the society.  So first, going over to augmenting our body, there's a lot of work also mechanically that's  not so interesting for us computer scientists.  However, it looks cool, so I included it.  One is a company called Skeletonics.  They manufacture these exoskeletons.  They are completely moved by muscle movement.  So nothing IT here, but still cool.  And then another thing is the bubble jumper.  That's an idea that came out of a hackathon.  We organized at KO and we started, you know, kind of with first sketches and so on.  But the important part also for our hackathons is you have to demo.  You have to implement.  It should be playing.  Oh, yeah.  So you see the combination of a bubble ball and sky runner and to make a very safe and  nice sport.  Yeah, coexistence of safety and force.  I'm not so sure about the safety.  And now getting over this to something a little bit more serious.  Spider vision.  This is now work from Inami sensei.  In this case, this is really about extending the human field of view.  And it's very, very simple.  They use the DK one and they have two cameras, one in front, one in the back, and they overlay  the two video feeds on top of each other.  Like you will see right here.  And the interesting part is at the beginning it looks really terrible, but after two or  three minutes using this, you can really separate the two images and you can even read or do  other things.  So you know, kind of then you can be environmentally aware.  The only trouble is, and that's something you see here, because you're wearing a head  mounted display still VR set.  You know, there's an offset from the camera to your actual eyes.  So it's very hard to interact with your environment.  But that's kind of the initial work where you want to get at.  The next work I want to show you is the synesthesia suit.  It's a full body haptic feedback suit made by Misoguchi and Minami Sawa, two of my colleagues  at KOMedia Design.  And this was first done to promote rest.  That's a VR game for the PlayStation.  And what it gives you, you see it here on the right side, is it has a haptic feedback  vibrators 24 on the whole body.  So you can get a touch feeling on your whole body.  And the idea Misoguchi sensei wants to do with this is he wants to give you simultaneously  visual as well as auditory and haptic and touch feedback to put you in some kind of  trans state.  And if you try, you know, rest with this suit, it's quite an experience actually.  So you get, you know, visual auditory and touch feedback at the same time.  Then moving over to augmenting the sports field.  Here I want to highlight a work from Rekimoto sensei, Aqua Cave.  You see a water tank you can swim in, but the swimmer actually wears 3D shutter glasses  and you have a 3D display around the water tank.  So you can then really dive in the great barrier reef or you can do competitions with sharks  and similar things.  Then augmenting training.  Here I will show a little bit of old work.  Maybe some of you will know, but I just find it cool enough to still include it.  And that's galvanic vestibular stimulation by Maeda and also in Ami sensei.  And what they were doing in 2004 is they apply a tiny current behind your ear.  So you have anode and cut out behind your ear.  And what happens if you apply this tiny current is you're messing with your equilibrium, with  your sense of equilibrium.  So you will tend towards the anode if you do that.  And then now the reason why I show this video is because this would be impossible to do  for a researcher in Europe or US.  You see a student wearing this device.  You see the professor with a control unit.  And you can steer the person just left and right.  And I have to say I tried it, it's not safe to try out.  We don't know about long term effects.  But it's interesting and it's also user dependent.  So depending on the user and depending on your skin, you might need more or less of  this electricity.  And it's an interesting thing to think about, especially for sports that require equilibrium  or other things.  And then another work, this is work from one of my students, Takuro.  He was working on peripheral vision glasses.  And his first idea was to just give you notifications in a way that other people don't realize.  So I could look still into your eyes and you don't realize that I might get some notification  on my peripheral vision.  I can still keep eye contact.  So far if you look at your watch, the other person will realize that you're looking there.  So he started with this idea.  And the boring thing is, yes, it works for notifications.  So you can have up to eight different types of notifications.  I don't know why the video stopped, but that's actually not so important.  You can have eight different types of notifications.  The other person cannot see that you're getting them.  However, what he really wants to do, and here we get to the interesting part, is he wants  to try to influence movement over your peripheral vision.  And there's some other research, some interesting research from Furukawa et al. also, augmented  human conference.  They use some patterns on the floor to guide pedestrians, to influence the movements of  pedestrians.  And then Takuro wondered if it's possible using then also peripheral vision glasses.  And this is the first test.  This is partly unpublished, but I thought I want to share it with you.  So in this case, what he's using is he's using some lines that move along this way.  And he lets people walk.  And guess what happens if you let the lines move faster?  Yeah, I heard.  So people will move slower.  So walk slower.  So it's already possible to maybe use it to influence speed.  And the next thing he wants to try is also try to influence direction.  So the next thing then is augmenting watch and cheer.  So in this case, I want to show effective wear.  That's glasses that can detect your facial expressions in a very cheap or simple way.  That's Masai.  And you see kind of the faces he makes.  He wears his own self-made glasses.  And how this works is he uses just photo-reflective sensors in the glasses frame.  And he detects the distance to the skin.  And with this, you can get user-dependent very well what type of expressions a person  is doing.  So you see then the, I like the training phase.  So here you see the photo-reflective sensors and the distance measures between the sensors.  And then kind of for the start, he has to then calibrate the sensors once.  And here you see the principles.  So for example, for smiling, kind of your cheeks move up.  So it's easy to recognize that you're smiling and even laughter.  So eight facial expressions are recognizable.  And yeah, and I like just Masai making faces.  That's the normalization phase, then the training phase, and then the recognition phase.  That's user-dependent, but it's very cheap.  And if you want to build this, especially smiling, it's easy.  You just need one photo-reflector here above the cheekbone, and then one maybe for reference.  And then it's easy to implement.  So of course, kind of what we can do with this is we can put it into a VR headset, and  then your character in VR can have the same facial expressions.  You can think about remote participants in an event or so on, and then also showing the  mood or so on of the remote participants.  It's not playing.  Yeah, but I think you get the idea.  I don't know why the video stopped.  And also more interesting then is you can also maybe use this for design.  So if you want to design a very, very scary dungeon, you can kind of detect how people  are reacting to some elements in your VR environment and then do the correct adjustments.  And also you could wear it throughout the day and just get an overview over your daily  activities so you know kind of how much smiling versus frowning are you doing.  Also related to some of the activities you have.  That's interesting, and that's in line with some work also from Rekimoto Sensei who did  the happiness counter.  So here the idea is he uses smile as an interaction modality.  So in the morning when you wake up and your alarm clock rings, you don't have to push  a button but you have to smile to your alarm clock.  That's a little bit scary or creepy.  And because we are with this, this next one is a little bit of a joke.  You know kind of if you recognize that you don't smile enough, we can also make you smile.  So this is now, Tuniji Smile is now activated over the electrodes.  As I said, this is a little bit of a joke.  I don't recommend anybody or I hope nobody will make these type of technologies in the  future.  But here the interesting part is students were playing with EMS and we recognized one  of the main, most of the nerves to control our face actually come over from this side  of the face.  So what you can do is if you put electrodes on different parts here, you can actually  activate different muscles.  The students wanted to get smiling working but first when they tried it on me, my eye  was blinking and other things.  You can also do eye blink.  And then, I mean this is a little bit on the joke side as I said before.  However if you want to have references for more serious electric muscle stimulation work,  I refer you to Max Pfeiffer and also Pedro Lopez.  They also have a kit out, so two references to URLs if you want to build it your own or  if you want to build your own EMS and also their source code for all of their papers  is I think out there on Bitbucket and GitHub.  And this is also just cool, you know, cruise control for pedestrians and also muscle plotters.  So using your body as output for a computer.  However, now getting back to the super human sports.  So the idea also for the super human sports society is to work towards Olympic 2020 and  we want to create actually an augmented game culture in Japan.  So not only inventing new sports and using new sports to pursue amplifying human senses,  but also creating this culture to make these sports.  So we are organizing workshops.  This is an example from Wycombe beginning of this year.  So you know, kind of you have a lot of devices.  You get together researchers, artists, designers, but also local people that are just interested  in participating.  We do one day of ideation and trying out different types of technologies and research outputs.  So this is for example also check in one of the researchers.  You see all of the four people, you know, you have a view of all four people.  Anyways and then at the end trying to design these games together.  So these are some more pictures from some of the workshops.  And here the idea is first of all, as I said before, we want to create a proof of concept  by making and playing sports and augmented sports.  So not just ideation sketches or so on, but really demonstrations, demo or die.  And then also getting fit ourselves so far it didn't really work for me, but maybe in  future and then also creating this augmented game creators for the time being.  And now you might wonder after these slides, what's the scientific impact?  I mean it's a lot of fun to do this, but you know, why am I focusing on this?  And if you heard my previous talk two years ago, might know that I'm interested in recognizing  and improving cognitive activities, so concentration, attention, comprehension.  And I focus a lot on eyewear computing.  That's my old talk, a little bit of background about myself.  I was working with Jin's Japanese glasses company on sensing glasses that have accelerometer  and also some low level eye tracking in them.  And now since actually December, I have some good news.  I have support from the Japanese government, especially as JST, to sponsor Collective Open  Eyewear platform.  And I'm also looking for collaborators to work on this because I think this is crucial  to start now.  Because I don't want to end up in a situation that we have today with our mobile phones  that we have just more or less, or that most people just have two providers, two big companies  that tell you what you are able to run on your mobile phones.  And I think eyewear for me, believe it or not, because my track record is not so good  predicting the future.  However, I think eyewear is the next stepping stone, it's the next thing, and I really want  to have an open eyewear platform out there.  So that's the background for myself.  And I think for me, the super human sports is a great testing ground for these type of  technologies and then really not only doing sensing, but also really trying to amplify  human senses.  In this case, I have to hurry a bit because I'm behind time and I want to still show a  demo.  However, here the idea is that sensors, digital sensors are in some parts already better than  human sensors.  So you can have higher frame rates.  You can have broader spectrums, so you could see infrared and so on.  However, what is lacking from our side is the interface to it.  So I wonder really, can we create new and amplified sensors based on digital technologies?  And just going with the digital camera system analogy, I would just want to give you a small  example at the end of the talk now.  And that's squint to zoom.  So if I would be standing here and there's a sign in the back and I want to read the  sign and I cannot do it with my normal eyesight, I will squint.  And wouldn't it be nice if I could wear some kind of glasses that automatically if they  recognize that I tried to read this and squint would just zoom it in.  And I don't even realize that I'm using technology.  And unfortunately, I wanted to show you a demo on the HoloLens, I have one here.  However, I didn't manage to get this working on the HoloLens.  And I'm happy that I have a student, George, who could set up a demo on a laptop that gives  you a feeling how this works.  So can you switch?  Yeah.  So in this case, this is unfortunately just on the laptop, but you can get an idea.  In this case, I'm using a Tobi eye tracker to track my eyes.  I have a camera and just a simple open CV to track my face.  And then if I now go ahead and squint, it zooms in.  I can show you also this on a document.  So if I squint, it zooms in, if I don't, it stops.  And that's actually a very, very simple type of interaction you could do.  I hope that could show it to you on an AR system, but that's maybe something for next  year.  Here again, as I said, kind of the desktop eye tracker.  And I can show you a little bit how it works.  So this is just an open CV system.  I think George uses the eyebrows and the relation to the nose and the size of my eyes.  So if I squint down, this relation changes and it zooms in.  And if I stop, it zooms out.  So this should give you a small idea of what I'm trying to create or what I want to create.  Things, technology that's easy to use, that you don't even realize that you're using,  similar to the analog glasses and so on.  And also not only for sensing, but hearing and other senses as well.  So this brings me to the end of my talk.  I just have some obligatory thank you slides.  So these are all people I'm working with and that contributed to the research you've been  seeing.  And I have a special thank you slide for the students that actually did the work and especially  George who made the demo you just saw, but also Masai for the effective wear and the  rest of the crew.  And now this is the end of my talk.  So now I think there's a time for a couple of questions, remarks or violent descent.  So thanks a lot for your attention.  Wow.  Really cool stuff.  That was so cool.  You have a chance to talk to Kai right now to ask him questions.  There are four microphones in the room.  So find yourself behind one of those microphones if you have a question.  Also the internet is always able to ask questions if they are awake by now.  Does the internet have anything?  Yes, one question from the internet.  Yes.  Is that code for eye tracking available?  So the which one?  I can recommend you pupil labs.  So the eye tracking code right now I'm just using Tobi IX.  So that's kind of Tobi proprietary.  They have an API.  It's just a desktop eye tracker.  However, if you want to have a mobile eye tracker, I would recommend you the pupil labs.  That's an open source eye tracker.  They just keep the latest revision of their hardware close source.  And they have all of their code up on GitHub.  So it's very, very easy to adjust.  And they also have settings for VR headsets.  So you can easily use it.  You can just, you know, kind of use, build your two own cameras.  That's easy.  Use their code if you know kind of you have budget constraints.  The Tobi IX is okay, but I don't like Tobi or SMI so much because they use close source  APIs.  And it's hard to get to the actual images because, you know, you want to have, for example,  pupil dilation because that gives you information about stress and other things.  And that's not so easy.  Great.  Actually, let's go to microphone number six, please.  So I have just a quick question about how open you are to be able to collaborate with  different universities from around the world to use your technology and research.  I work in the effective sciences department in the University of Geneva.  And we actually use a lot of motion tracking.  This eye movement tracking would be super interesting for effective science research,  especially as well our facial expressions.  We use that a lot.  So effective eye movement would also be interesting.  So you've discussed how to use it for like opening up possibilities in the real world  and actually enhancing everyone as a general population, but how open are you to using  it for research as well?  Yeah, very open.  I mean, that's also the reason why I give the talk.  I mean, now finally I get also a little bit of money to fund this from the Japan side.  And I really want to find people that want to collaborate on this side.  And I think, you know, kind of especially seeing how Silicon Valley now also invests  in eyewear.  And I think the next thing we will see is eye trackers in VR and so on and how close  those systems are.  I would really like to, you know, have this collaboration.  So just try to catch me afterwards or just send me also a mail would love to keep in  touch.  Great.  Thanks.  And microphone number two, please.  Well, thank you.  Amazing talk.  And how important do you think will be the combination of external signals like squinting  or looking at something and internal signals or like directly measuring the brain waves  or with an EKG?  Do you think that is important to combine those or?  It's important to combine those.  I actually started looking into a lot of the EEG signals and so on.  When I started this work, but it's very hard to get into your skull.  So EEG, also brain computer interfaces, I think will take a while to work.  And you also have this trouble of the signal processing is relatively hard.  And because I'm lazy, I started with the eyes and the eyes give you actually already a lot  of information you want to get from the brain.  I like to think about combinations.  So for some things we already working with also EEG.  And I think it's crucial to find the right type of sensing so that it works throughout  the everyday life.  I think one example was also JINs and now with the JST project, what we really trying  to do is find simple sensors to give you some information about your brain activities and  so on.  I think it's really a combination of multiple sensors.  And I think, as I said, the eye is very, very interesting.  Yeah, thanks.  We have time for one more brief question.  Number four, please.  When I was wearing AR glasses, my brain messed the distance to objects so I couldn't grasp  anything.  It just wasn't there where I expected it.  Is there any chance this problem will go away?  This depends highly on the lens.  So there's still this problem.  It's also in VR.  I mean, VR has this trouble with the perception.  I think you will have always this issue, I found AR, so the HoloLens quite good for  this.  But here the problem is also you cannot grasp something because the distance to the object  is too far.  So I'm still wondering, I haven't seen something that's really kind of graspable in VR where  you have the right distances or so on.  I think either your brain has to get used to it, which I think it does after a while.  It gets the distance measurements correct or we have to employ some other tricks to do  that.  What you can do for that, I'm not so sure.  Thank you.  Great.  And with that, we're at the end of the talk.  Thank you so much, Kai.  Please help me thank Kai Konso.
33c3-8225-beyond_virtual_and_augmented_reality_0_4031.png 33c3-8225-beyond_virtual_and_augmented_reality_4032_4223.png 33c3-8225-beyond_virtual_and_augmented_reality_4224_4271.png 33c3-8225-beyond_virtual_and_augmented_reality_4272_12359.png 33c3-8225-beyond_virtual_and_augmented_reality_12360_21551.png 33c3-8225-beyond_virtual_and_augmented_reality_21552_21959.png 33c3-8225-beyond_virtual_and_augmented_reality_21960_26687.png 33c3-8225-beyond_virtual_and_augmented_reality_26688_30155.png 33c3-8225-beyond_virtual_and_augmented_reality_30156_30971.png 33c3-8225-beyond_virtual_and_augmented_reality_30972_32291.png 33c3-8225-beyond_virtual_and_augmented_reality_32292_32399.png 33c3-8225-beyond_virtual_and_augmented_reality_32400_32519.png 33c3-8225-beyond_virtual_and_augmented_reality_32520_33887.png 33c3-8225-beyond_virtual_and_augmented_reality_33888_33935.png 33c3-8225-beyond_virtual_and_augmented_reality_33936_34211.png 33c3-8225-beyond_virtual_and_augmented_reality_34212_34283.png 33c3-8225-beyond_virtual_and_augmented_reality_34284_37235.png 33c3-8225-beyond_virtual_and_augmented_reality_37236_39167.png 33c3-8225-beyond_virtual_and_augmented_reality_39168_40559.png 33c3-8225-beyond_virtual_and_augmented_reality_40560_42023.png 33c3-8225-beyond_virtual_and_augmented_reality_42024_42191.png 33c3-8225-beyond_virtual_and_augmented_reality_42192_43439.png 33c3-8225-beyond_virtual_and_augmented_reality_43440_44375.png 33c3-8225-beyond_virtual_and_augmented_reality_44376_44627.png 33c3-8225-beyond_virtual_and_augmented_reality_44628_45827.png 33c3-8225-beyond_virtual_and_augmented_reality_45828_48131.png 33c3-8225-beyond_virtual_and_augmented_reality_48132_48659.png 33c3-8225-beyond_virtual_and_augmented_reality_48660_49547.png 33c3-8225-beyond_virtual_and_augmented_reality_49548_50315.png 33c3-8225-beyond_virtual_and_augmented_reality_50316_50628.png