Weaviate OSS Smart Graph
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Luijt, Bob van
Weaviate is an open-source smart graph that aims to allow anyone, anywhere, any time to create their own semantic search engines, knowledge graphs or knowledge networks. Weaviate is RESTful and GraphQL API based and built on top of a semantic vector storage mechanism called the contextionary. Because all data is stored in the vector space, Weaviate is ideal for; - Semantically search through the knowledge graph. - Automatically classify entities in the graph. - Create easy to use knowledge mappings. Because the use of formal ontologies are optional, Weaviate can be used to create a P2P knowledge network which we want to present during this conference. This is a follow up after the initial design was shared during last year's FOSDEM. Problem Creating a knowledge graph can be a complex endeavor, let alone the integration of semantic search models. Bain & Company research under US enterprise CTO's shows that 59% of them believe they lack the capabilities to generate meaningful business insights from their data, and 85% said it would require substantial investments to improve their data platforms. Solution Weaviate aims anyone to create large, enterprise-scale knowledge graphs as straight forward as possible. Weaviate's feature set allows anyone to; - Semantically search through the knowledge graph. - Automatically classify entities in the knowledge graph. - Create easy to use knowledge mappings. Weaviate's Contextionary Weavite's Contextionary is the semantic vector storage mechanism that stores data -unlike traditional storage mechanisms- based on its semantic meaning. For example, if someone stores information about a company with the name Apple, this data object would be found closely related to concepts like the iPhone. Because of the algorithmic use (as opposed to retraining) of the pre-trained machine learning model, Weaviate is able to learn new concepts fast and near-realtime. This allows the user to update and manipulate the knowledge graph directly. Demo & Use cases During the session, we want to show a few -recent- use cases to demo how Weaviate can be used. The demo will include; querying; semantic querying; adding concepts; going from an ontology to a schema; and more. Knowledge network Because of Weaviate's contextionary, a formal ontology is optional (e.g., "a company with the name Netflix" is semantically similar to "a business with the identifier Netflix Inc.") this allows multiple Weaviate to connect and communicate over a peer to peer (P2P) network to exchange knowledge. Aka, the knowledge network. During the session, we want to demonstrate the first prototype of this network.
 Yes, thank you so much. Well, thank you for having me. So it's indeed is the last of the days. So it's like outside and I saw like a lot of people are drinking beers and stuff. So it's like I hope that there are still people in the room. So thank you for listening. So this is like last year my colleague, he was here and he he presented for the first time we've ate and what we're trying to do. And we've done a lot in the year. So today I want to I want to show you actually an update and way more things about we've ate and a lot of things have changed also the things that we've changed into. So quickly agenda what I want to talk about. So first I want to talk about what it is because they were you know, we're new, we're new on the scene. So what is it? Secondly, I want to talk about what has changed since last year or presented here and this year. I want to talk a little bit of the about the technology and last but not least, of course, I want to give a demo. I want to show it in action. So that's the that's what I'm going to do. So first about the about we've hit. So we've hit is an open source smart graph. That's what it is. And what we mean with that? Well, the fact that it's open source that is the usual suspects, right? So source lives on get up. But you can use that using Docker, Docker compose and Kubernetes. So we have that all out of the box for you. It is smart because of something which we call the Contectionary. And if you never heard about the Contectionary that's fine because we invented it and I'm about to explain to you what it is and what it does. And I want to use I'm going to use one buzzword. Sorry for that. But what I what is it has a serious it's it has serious notion. So sometimes people talk about like the AI first architectures. So like can we build systems that have like these machine models built into them that we can build new solutions and actually new ideas. And that's what we try to solve with semantic models. So what makes it in our case what we like to call smart is the building semantic model. And the graph is well, I don't have to in this room, I don't have to, you know, share what the graph is. But the we we've chosen to use graph QL. And the reason why I've done that is because we see, of course, there's like a lot of graph experts, you know, you sparkle or sorry for those kind of things. But sometimes when we have developers who find it a little bit more difficult, we see that they really like to work with graph QL. So we really embraced graph QL. And it's like completely 100% graph QL based. So that's how we define what our smart graph is. And with a smart you can do three things. First thing is semantic search. So what you can do and that's also what I will demo to you what we know from traditional search, if I may call it like that, we search for keywords. Right. So if we write about the company Apple, we actually need to search for Apple. But in we if you don't necessarily have to. So if you add company with the name Apple, but you search, for example, what's the company, the business related to the iPhone, it will still find your data object company with the name Apple. We also can do based on that is automatic classification. So we can automatically make edges in the graph based on the semantics of your data object. And last but not least, we can do knowledge representation. And this is often what's always also referred as like nowadays, it's like, it's the knowledge graph, but it's like, I wouldn't say we're necessarily a knowledge graph, but you can create those similar representations. So those are the three things that the three use cases that we can help you with. So something important to share based on what we did last year. So the best way to explain is a little bit on time. And what I mean with that is like that we saw, of course, a lot of databases in the past that were more relational based, right? So just, you know, a road column structure, row column structures and tables. And then of course, we got these graph databases. So for example, the people from New York and room, so they made a beautiful, beautiful database like that. And we chose a year back to store information with Janis. And what we did was that we had that semantic element that what I will explain the context narrative, I will explain what that is. We had that as a feature. But last year, we decided to just double down on that semantic element. So we got completely rid of the implementation in Janis. And we created basically everything ourselves. And that means that we now only have that context, you know, to store that information in. And we've actually, this is a crown. So we actually figured out like this is actually, we're really happy that we made that decision, because now we could really bring something new, you know, to the to the stage, something different and a different way to handle your data objects and work with them. So we store everything in a semantic space. That's all we do. And now if you go like, what semantic space, I'm going to explain what we mean with that. So but just keep thinking when I talk about semantic space, I'm talking about the context. So imagine it like this. If you go to a grocery store, and you have a shopping list, and on that shopping list, you might have four items, a banana, washing powder you're looking for an apple and a piece of bread. So if you go in the supermarket and you find a banana, you know that an apple is going to be closer by than the bread, or the washing powder. And if you move towards the bread, you know that you're actually getting closer to the washing powder, moving more away from the fruit. That's how we store data in the space. And so that's the metaphor to actually imprint the problem that we solve. And we do that to something that we that we call the Contectionary. And I want to give you a little bit of background where we're coming from, and what's different to other solutions that around here. So if you go all the way back again time to the, to the, to the, to the, to the, to the, to the, to the 1950s, it was like a famous quote, and I said like a word can be characterized by the company it keeps. And what that basically means that you would say that the word Paris would be more closely related to France than it would be to Holland, for example, or the US. And New York would be more closely related to the US than, for example, Spain. And that basically went for all words. And a lot of research was done there. And then we jumped forward. And then with the whole machine learning boom, we saw there was a lot of work being done with, with, based on machine learning and these word embeddings. And then what we also saw, there was, there was like, we got first word to vac, got glove. And nowadays have what they call in the academic realm, which they call a state of the arts called Bert. But if we now put on our engineering hat, we really felt that we were going to be in love with glove. And why did we fall in love with glove? Because Bert has multiple representations of a word in that data set. But glove doesn't. Glove has one vector representation for every word. And the critique that it often got was that, well, if you, for example, have to name Apple, Apple can mean, of course, fruit, but it can also mean the company. And that we wanted to solve the problem in a different way. But as engineers, we were very happy with this, because now we could index it. We could index those words in a storage mechanism. So if you start a WeVH, you can imagine it like this. It's an empty space that you start. You choose a language, not a programming language, but a spoken language. So let's say, for example, English, and the, the space is filled with all those words. So for example, if you find Apple, you find nearby fruit, you find nearby company, and you might also find iPhone. And these representations that we store, they have a 600 dimensional representation, which is very fancy, but it has to do with compression, that kind of stuff. And the thing that we did is this. If you store a data object like this, so for example, the class company with the name Apple found it in 1976, et cetera. And in, when I demo, I will show you how that actually works. But if you start information, what WeVH does, is that it creates a string of the words and the concepts that are in there. And it takes those concepts and it funnels them down. How does it do that? It says like, it takes the Euclidean distance. It combines it with the, it's, this sounds very fancy, but with the logarithmic function based among the occurrence of the word. So for example, company might occur less often than the word Apple. So then we say the word company is more important. And then we even work with word boosting, so that you can say certain words are important in my data object. But what we do with that is that we create our first object in our graph. And it gets its own factory presentation. So now if we have an empty refiate with those words in that, and we store our first data object, as you can see there, there it is. It lives, that's where it lives in the vector space. That's what we mean. So now if we query for, let's say, a company, an iPhone, it can look in the nearby space and find the data object. So now without even having iPhone in the data object, we can still find it. And that's the thing that we created. Then we thought, okay, this is, this is our thing, right? This is our, our golden goose egg, because now we have different ways of creating graphs and actually query through them. So this example, for example, might look something like this. So if I have a data set with companies, and this would be my GraphQL query where I say, get, give me things which are companies, and I want to have their names, but explore them by iPhone. I might get this result, Apple. And as you see, iPhone is nowhere in the data object. And if I have that same data set, then I would say, well, a little bit more abstract, organize these companies on the concept of Redmond, I might get back Microsoft. And that's how we structure our graph. So, and so that's basically what we've had. And as a developer, we've had comes with a few features. So the first thing is that's that's context. There comes all out of the box. You don't have to do your training. You don't have to set it up, whatever it comes all out of the box. Or basically, all out of the container, I should say. Adding data happens to the HTTP API, but calling data to the GraphQL API. It's completely containerized. So you can run it wherever you want. And because we use that vector space, it's very, very scalable. You can scale this that space very, very tremendously big. So I think the biggest one that we ever tried was a few was a few billions. It gets pretty big. And something that we have in the pipeline, and maybe I can show you that next year, but it's that we also can create peer to peer networks of VVS. So that we can point to semantic elements in different graphs so that we don't have to agree anymore on our ontologies or on our schemas. But that's like that's that's in the make. So a little bit about GraphQL. So I have so when I demo that. So how we structure our GraphQL, this is the UX, if you will, of our GraphQL. So we have a get function first. The other one is an aggregate function. I'm going to talk about the get function. We have a semantic kind, which we make a distinction between things and actions, nouns and verbs. We have the class, the classes, the property. A property might have a reference, and then the property itself. And what you can do, you can have these semantic filters on top of that. So there you see, for example, the explore filter, and they can search for concepts, but you can even move away from concepts of move towards concepts. And I will demo that to you in a bit. Well, demo. So now we get to the demo. So now you might want to say, how does that actually look? So what I did is that I spun up a docker container. If you want to do it yourself, if you go to our website, semi.technology, and then you simply click we've yet and then you find all the documentation. The installation gives you just a we've yet, but what I'm going to demo to you is this news publications that are set. And if you click that one, there's just one simple docker compose comment that you can run that you can play, you know, around with it yourself. So there's a meta endpoint, which I'm just running just to make sure. Yes, so that is running well. So let's first look at the schema. So this is an example of schema. So here you see I have the class publication, and I have I have the name name, so the name of the publication. But for example, I have so I set the headquarters geolocation, which is our geo coordinates has articles, etc. etc. This is how we structure the schema. That's important because we will see that back when we use GraphQL to query. And the things that we actually store look like this. So for example, here you see an article, the article has an ID, a beacon is a reference in our graph. But why do we call it beacon? Because we do it in the space. So it's a beacon in the space. And then here you see, for example, a summary of the article, the title of the article, and the URL, where the article actually comes from. So that's how it's structured. So we've created a simple GUI that you can use to actually, you know, look through and search through the to the graph so you can visualize stuff. But I want to I want to dive into the the GraphQL queries. So let me show you a simple query. So if I say get things, and I want to have, I want to have a publication, and I want to see the name, this would be a valid query that you see, rising folk financial times, why it New York, an economist, etc. Now what I now can do is that I can say, well, I want to I want to explore for the concept, let's say, business. And I'm going to limit it to three results just that it's easier readable. So I know I say, do the same query, but explore based on the concepts of business. And I want to say, I want to explore for the concept of business. And I want to explore based on the concepts of business. So if I now run this query, you see financial times, the New York Times, the international Times, etc. But you see the word business, the the meta tech, it's nowhere there. That's what comes from the context. Or if I would say fashion, and I would run it, then you see it starts, for example, with Vogue. So that's how we structured it and how it works. But that's not not only going for like small strings, but also for larger larger text objects. So for example, if I would say get things, article, and I would say show me the title of the article, and run the query. So now you see all these articles about a variety of topics. You see Brexit, so you can see when we when we actually pull to that end. But just a variety of topics in here. But if I now say, well, I want to have those articles, I'm going to use the explore function again. And I can I say, well, I want to explore for the concepts, let's say music. And I'm going to limit the results again, just for the sake of readability. So same query, but now based on the articles. So if I run this, you say like you say, fair enough, the first one has the word musical, but then it's about Gwen Stefani. And then it's about John Lennon. So you see the word music is not necessarily in there, but it organizes it like that automatically. So now even if you want to filter further in this in this graph, what you can do is that you can say, well, the question that actually how do we do pagination, because if you have a 600 dimensional space, what would be the next page? So what we've done is this. So we said, well, we can actually, for example, move towards a concept. So I can say, well, move towards the concept, for example, of the Beatles. So I guess you already know what will happen if I do that. So and I give it a certain force, the force is how strongly you want to push towards this concept inside the vector space. So let's say it's a little bit arbitrary, but let's say 85%. So if I do that now, you see that John Lennon, the article with John Lennon comes first. And if I say like, I'm more like a Stones person, I hate the Beatles, then of course you can also do move away from the concept of the Beatles. Same query. And we see John Lennon is gone. And now the question is like, okay, so what makes it so like now the traditional graph people are going to like, yeah, but I haven't seen really the graph in action yet. So that's just very simple because you could say, for example, has authors. And we can say on author, the author is a name. So this is how we structure the graphs. So now you see, so you see the graph object here. That's this first the title of the article. And it's a has authors and then actually the authors that are related to this to this article. I think if time allows it, there's one more thing. Can I still quickly show one more thing? Yes. So the I just want to see which I completely forgot because there's another problem that we also tried to solve with this. And I want to show you that going back to the publications. So quickly going back to the publications, things, publication, and then they say a name of the publication. When you glanced over this, you might have noticed that we have all these publications, but we have the International New York Times, the New York Times Company, and the New York Times in there three times, which is a problem because of course you want to represent concepts. But in a database, we have the same concept represented three times. So we have something for that, which we can do is that we can say, well, we want to group concepts together. So then we say, I'm going to do the type merge them together. I'm going to give a four. So how big do we need to look in the vector space before merging? And then I can say, well, do that with 5%. So if I now run this query, you'll see that it merged together the International New York Times, New York Times Company and the New York Times. But if I now do a graph query, so I say has articles on article, the title of the article, it even now merges together those different articles from those different publications into the same concept. So that's what we have. That's how it works. Oh, way more features, the automatic classification. I didn't even get to show you that, but you can play around with it because, well, we're at FalsNAM, so this software is open source. You can play around with it. You can set it up yourself. You can create your own graphs, or semantic graphs, I should say. That's my story in a nutshell. Thank you all for listening. And if you like it, and if you go to a website, then I just have one question. So this is a website. If you go here, then you can sign up to a newsletter if you want to learn, or you can click on the get off star button if you want to promote a little bit. So that's it. But this is a website. Thank you very much for listening. Thank you. That was a very great last talk for all that group. Thank you very much. Thank you, for everything. Thank you. Thank you. Thank you. Thank you. Thank you. Thanks. In that case, I suppose your model will have to cook up vector embedding for that. Is that actually sustainable if I put it on a server and then everybody tries to run your model to get their embedding? How did you solve the resource problem? So two questions. Sorry about that. No. Why are you sorry about it? That's a great question. So let me start with the first answer to the first question. So, and sorry if I went over that too quickly because it's, this whole everything what I told is also on the website in detail, but that's what's happening here. We always use the same algorithm. So if you would have a data object with a longer sentence like for example the summary or the title of the articles that you've seen, it applies the same algorithm. So it says like first I take all these individual words, then I find the center position between those words, then I weigh them based on the occurrence. So certain words are seen as more important than others. So it weighs it towards that. And then we have this optional word boosting that you can say, well in this specific case, this word is very important. So move more towards that. That's how we create those vector positions. So regardless if you're querying or if you're adding data, that's how we do it. So that's why we also became agnostic about the fact that gloves about single words. And because we learned that if we saw we the first prototype we did way back was very simple. We say, okay, I have the word Apple show me what's nearby. And then as you would expect as glove does, it says like, well, I found iPhone, but also found fruit. And then we did something very simple, which okay, now go and sit in the center between Apple and iPhone show me again what happens. And then you see actually that that's successful. And if I if I may, I can actually quickly show that I'm the last talk. So I should have waited a little bit said, go ahead. And then thank you. So there's a context, every endpoint, where you can say, well, concepts. So if I now literally do what I just said, so if I would say, Apple, then you see your Apple iTunes Google preview on this. And now, of course, in this example, we don't see, of course, the fruit. But let's say if I would do Apple and not based on the company, but on fruit. So I concatenate them, Apple and fruit. You see how it now starts to get better and better in those results. So and that's how it the algorithm works. So you can play around with this also yourself on this on this endpoint. And the other question. Yeah, sure. So we, we, of course, we're also like a we were also a business. So we have like a the core is like open source, but we built like a shell around that. So we currently have six companies using this on a large scale in a variety of industries, also retail, oil and gas, all those kind of things. And these graphs get pretty big. And especially if you scale the communities cluster, it's fast, which is something I'm now I could say that we fensely like architected that, but it was actually something we got for free because we just the data model is just vectors, only vectors. And that's of course very fast to scale and to search through. So the answer to that question is yes. Can you use this for translation? Like you said, your vector space is built in one language. But can you compare vector spaces by looking at one concept is associated with all these things. And in your second vector space, another concept is also associated with the same things. And you've already linked some some kind of translate use after translation. I don't know. And I love the idea. So we're definitely going to try that out. I don't know. I don't know. It's a great idea. We haven't we haven't tried that yet. So what we currently do is that we just have you have a we've written in a language. So, um, uh, dutch, French, English, of course. But we haven't tried that yet. So, um, if you don't mind, then it would be fantastic to or of course you can try that yourself or we can do it together. But that would be fantastic. It's a it's a great idea. And I don't know. Thank you. Thanks very much. Um, that we are officially over with the death room. Nice and I hope you all enjoyed it. Enjoying for them tomorrow. Enjoy the evening, of course. And maybe you next year. Thank you. Thank you.
 Yes, thank you so much. Well, thank you for having me. So it's indeed is the last of the  days. So it's like outside and I saw like a lot of people are drinking beers and stuff.  So it's like I hope that there are still people in the room. So thank you for listening.  So this is like last year my colleague, he was here and he he presented for the first  time we've ate and what we're trying to do. And we've done a lot in the year. So today  I want to I want to show you actually an update and way more things about we've ate and a lot  of things have changed also the things that we've changed into. So quickly agenda what I want to  talk about. So first I want to talk about what it is because they were you know, we're new,  we're new on the scene. So what is it? Secondly, I want to talk about what has changed since last  year or presented here and this year. I want to talk a little bit of the about the technology  and last but not least, of course, I want to give a demo. I want to show it in action. So that's  the that's what I'm going to do. So first about the about we've hit. So we've hit is an open  source smart graph. That's what it is. And what we mean with that? Well, the fact that it's open  source that is the usual suspects, right? So source lives on get up. But you can use that using  Docker, Docker compose and Kubernetes. So we have that all out of the box for you. It is smart  because of something which we call the Contectionary. And if you never heard about the Contectionary  that's fine because we invented it and I'm about to explain to you what it is and what it does. And  I want to use I'm going to use one buzzword. Sorry for that. But what I what is it has a serious  it's it has serious notion. So sometimes people talk about like the AI first architectures. So  like can we build systems that have like these machine models built into them that we can build  new solutions and actually new ideas. And that's what we try to solve with semantic models. So  what makes it in our case what we like to call smart is the building semantic model. And the  graph is well, I don't have to in this room, I don't have to, you know, share what the graph is.  But the we we've chosen to use graph QL. And the reason why I've done that is because we see,  of course, there's like a lot of graph experts, you know, you sparkle or sorry for those kind of  things. But sometimes when we have developers who find it a little bit more difficult, we see that  they really like to work with graph QL. So we really embraced graph QL. And it's like completely  100% graph QL based. So that's how we define what our smart graph is. And with a smart  you can do three things. First thing is semantic search. So what you can do and that's also what  I will demo to you what we know from traditional search, if I may call it like that, we search  for keywords. Right. So if we write about the company Apple, we actually need to search for  Apple. But in we if you don't necessarily have to. So if you add company with the name Apple,  but you search, for example, what's the company, the business related to the iPhone, it will still  find your data object company with the name Apple. We also can do based on that is automatic  classification. So we can automatically make edges in the graph based on the semantics of your  data object. And last but not least, we can do knowledge representation. And this is often  what's always also referred as like nowadays, it's like, it's the knowledge graph, but it's like,  I wouldn't say we're necessarily a knowledge graph, but you can create those similar representations.  So those are the three things that the three use cases that we can help you with. So something  important to share based on what we did last year. So the best way to explain is a little bit on  time. And what I mean with that is like that we saw, of course, a lot of databases in the past  that were more relational based, right? So just, you know, a road column structure, row column  structures and tables. And then of course, we got these graph databases. So for example, the people  from New York and room, so they made a beautiful, beautiful database like that. And we chose a  year back to store information with Janis. And what we did was that we had that semantic element  that what I will explain the context narrative, I will explain what that is. We had that as a  feature. But last year, we decided to just double down on that semantic element. So we got completely  rid of the implementation in Janis. And we created basically everything ourselves. And that means  that we now only have that context, you know, to store that information in. And we've actually,  this is a crown. So we actually figured out like this is actually, we're really happy that we made  that decision, because now we could really bring something new, you know, to the to the stage,  something different and a different way to handle your data objects and work with them.  So we store everything in a semantic space. That's all we do. And now if you go like,  what semantic space, I'm going to explain what we mean with that. So but just keep thinking when  I talk about semantic space, I'm talking about the context. So imagine it like this. If you go to a  grocery store, and you have a shopping list, and on that shopping list, you might have four items,  a banana, washing powder you're looking for an apple and a piece of bread. So if you go in the  supermarket and you find a banana, you know that an apple is going to be closer by than the bread,  or the washing powder. And if you move towards the bread, you know that you're actually getting  closer to the washing powder, moving more away from the fruit. That's how we store data in the  space. And so that's the metaphor to actually imprint the problem that we solve. And we do that  to something that we that we call the Contectionary. And I want to give you a little bit of background  where we're coming from, and what's different to other solutions that around here. So if you go  all the way back again time to the, to the, to the, to the, to the, to the, to the, to the  1950s, it was like a famous quote, and I said like a word can be characterized by the company it  keeps. And what that basically means that you would say that the word Paris would be more closely  related to France than it would be to Holland, for example, or the US. And New York would be more  closely related to the US than, for example, Spain. And that basically went for all words. And a lot  of research was done there. And then we jumped forward. And then with the whole machine learning  boom, we saw there was a lot of work being done with, with, based on machine learning and these  word embeddings. And then what we also saw, there was, there was like, we got first word to vac,  got glove. And nowadays have what they call in the academic realm, which they call a state of the  arts called Bert. But if we now put on our engineering hat, we really felt that we were  going to be in love with glove. And why did we fall in love with glove? Because Bert has multiple  representations of a word in that data set. But glove doesn't. Glove has one vector representation  for every word. And the critique that it often got was that, well, if you, for example, have to  name Apple, Apple can mean, of course, fruit, but it can also mean the company. And that we wanted  to solve the problem in a different way. But as engineers, we were very happy with this, because  now we could index it. We could index those words in a storage mechanism. So if you start a  WeVH, you can imagine it like this. It's an empty space that you start. You choose a language,  not a programming language, but a spoken language. So let's say, for example, English, and the,  the space is filled with all those words. So for example, if you find Apple,  you find nearby fruit, you find nearby company, and you might also find iPhone. And these  representations that we store, they have a 600 dimensional representation, which is very fancy,  but it has to do with compression, that kind of stuff. And the thing that we did is this.  If you store a data object like this, so for example, the class company with the name Apple  found it in 1976, et cetera. And in, when I demo, I will show you how that actually works. But  if you start information, what WeVH does, is that it creates a string of the words and the  concepts that are in there. And it takes those concepts and it funnels them down. How does it  do that? It says like, it takes the Euclidean distance. It combines it with the, it's, this  sounds very fancy, but with the logarithmic function based among the occurrence of the word. So  for example, company might occur less often than the word Apple. So then we say the word company  is more important. And then we even work with word boosting, so that you can say certain words  are important in my data object. But what we do with that is that we create our first object in  our graph. And it gets its own factory presentation. So now if we have an empty refiate with those  words in that, and we store our first data object, as you can see there, there it is.  It lives, that's where it lives in the vector space. That's what we mean. So now if we query for,  let's say, a company, an iPhone, it can look in the nearby space and find the data object.  So now without even having iPhone in the data object, we can still find it. And that's the  thing that we created. Then we thought, okay, this is, this is our thing, right? This is our,  our golden goose egg, because now we have different ways of creating graphs and actually query  through them. So this example, for example, might look something like this. So if I have a data set  with companies, and this would be my GraphQL query where I say, get, give me things which are  companies, and I want to have their names, but explore them by iPhone. I might get this result,  Apple. And as you see, iPhone is nowhere in the data object. And if I have that same data set,  then I would say, well, a little bit more abstract, organize these companies on the concept of  Redmond, I might get back Microsoft. And that's how we structure our graph. So, and so that's  basically what we've had. And as a developer, we've had comes with a few features. So the first  thing is that's that's context. There comes all out of the box. You don't have to do your training.  You don't have to set it up, whatever it comes all out of the box. Or basically, all out of the  container, I should say. Adding data happens to the HTTP API, but calling data to the GraphQL  API. It's completely containerized. So you can run it wherever you want. And because we use that  vector space, it's very, very scalable. You can scale this that space very, very tremendously  big. So I think the biggest one that we ever tried was a few was a few billions. It gets pretty  big. And something that we have in the pipeline, and maybe I can show you that next year, but it's  that we also can create peer to peer networks of VVS. So that we can point to semantic elements in  different graphs so that we don't have to agree anymore on our ontologies or on our schemas. But  that's like that's that's in the make. So a little bit about GraphQL. So I have so when I demo that.  So how we structure our GraphQL, this is the UX, if you will, of our GraphQL. So we have a  get function first. The other one is an aggregate function. I'm going to talk about the get function.  We have a semantic kind, which we make a distinction between things and actions, nouns and verbs.  We have the class, the classes, the property. A property might have a reference, and then the property  itself. And what you can do, you can have these semantic filters on top of that. So there you  see, for example, the explore filter, and they can search for concepts, but you can even move away  from concepts of move towards concepts. And I will demo that to you in a bit. Well, demo. So now we  get to the demo. So now you might want to say, how does that actually look? So what I did is that I  spun up a docker container. If you want to do it yourself, if you go to our website,  semi.technology, and then you simply click we've yet and then you find all the documentation.  The installation gives you just a we've yet, but what I'm going to demo to you is this news  publications that are set. And if you click that one, there's just one simple docker compose  comment that you can run that you can play, you know, around with it yourself.  So there's a meta endpoint, which I'm just running just to make sure. Yes, so that is running well.  So let's first look at the schema. So this is an example of schema. So here you see I have the  class publication, and I have I have the name name, so the name of the publication. But for example,  I have so I set the headquarters geolocation, which is our geo coordinates has articles, etc.  etc. This is how we structure the schema. That's important because we will see that back when we  use GraphQL to query. And the things that we actually store look like this. So for example,  here you see an article, the article has an ID, a beacon is a reference in our graph. But why do we  call it beacon? Because we do it in the space. So it's a beacon in the space. And then here you see,  for example, a summary of the article, the title of the article, and the URL, where the article  actually comes from. So that's how it's structured. So we've created a simple  GUI that you can use to actually, you know, look through and search through the to the graph so  you can visualize stuff. But I want to I want to dive into the the GraphQL queries. So let me  show you a simple query. So if I say get things, and I want to have, I want to have a publication,  and I want to see the name, this would be a valid query that you see, rising folk financial  times, why it New York, an economist, etc. Now what I now can do is that I can say, well,  I want to I want to explore for the concept, let's say, business. And I'm going to limit it to three  results just that it's easier readable. So I know I say, do the same query, but explore based on the  concepts of business. And I want to say, I want to explore for the concept of business. And I want to  explore based on the concepts of business. So if I now run this query, you see financial times,  the New York Times, the international Times, etc. But you see the word business, the the  meta tech, it's nowhere there. That's what comes from the context. Or if I would say fashion,  and I would run it, then you see it starts, for example, with Vogue. So that's how we structured  it and how it works. But that's not not only going for like small strings, but also for larger  larger text objects. So for example, if I would say get things, article, and I would say show me the  title of the article, and run the query. So now you see all these articles about a variety of topics.  You see Brexit, so you can see when we when we actually pull to that end. But just a variety  of topics in here. But if I now say, well, I want to have those articles, I'm going to use the explore  function again. And I can I say, well, I want to explore for the concepts, let's say music.  And I'm going to limit the results again, just for the sake of readability. So same query, but  now based on the articles. So if I run this, you say like you say, fair enough, the first one has  the word musical, but then it's about Gwen Stefani. And then it's about John Lennon. So you see the  word music is not necessarily in there, but it organizes it like that automatically. So now  even if you want to filter further in this in this graph, what you can do is that you can say, well,  the question that actually how do we do pagination, because if you have a 600 dimensional space,  what would be the next page? So what we've done is this. So we said, well, we can actually,  for example, move towards a concept. So I can say, well, move towards the concept, for example,  of the Beatles. So I guess you already know what will happen if I do that. So and I give it a  certain force, the force is how strongly you want to push towards this concept inside the vector  space. So let's say it's a little bit arbitrary, but let's say 85%. So if I do that now, you see  that John Lennon, the article with John Lennon comes first. And if I say like, I'm more like a  Stones person, I hate the Beatles, then of course you can also do move away from the concept of the  Beatles. Same query. And we see John Lennon is gone. And now the question is like, okay, so what  makes it so like now the traditional graph people are going to like, yeah, but I haven't seen really  the graph in action yet. So that's just very simple because you could say, for example, has authors.  And we can say on author, the author is a name. So this is how we structure the graphs. So now you  see, so you see the graph object here. That's this first the title of the article. And it's a has  authors and then actually the authors that are related to this to this article. I think if time  allows it, there's one more thing. Can I still quickly show one more thing? Yes. So the I just  want to see which I completely forgot because there's another problem that we also tried to solve  with this. And I want to show you that going back to the publications. So quickly going back to the  publications, things, publication, and then they say a name of the publication. When you glanced  over this, you might have noticed that we have all these publications, but we have the International  New York Times, the New York Times Company, and the New York Times in there three times, which is a  problem because of course you want to represent concepts. But in a database, we have the same  concept represented three times. So we have something for that, which we can do is that we  can say, well, we want to group concepts together. So then we say, I'm going to do the type merge  them together. I'm going to give a four. So how big do we need to look in the vector space before  merging? And then I can say, well, do that with 5%. So if I now run this query, you'll see that it  merged together the International New York Times, New York Times Company and the New York Times.  But if I now do a graph query, so I say has articles on article,  the title of the article, it even now merges together those different articles from those  different publications into the same concept. So that's what we have. That's how it works.  Oh, way more features, the automatic classification. I didn't even get to show you that, but you can  play around with it because, well, we're at FalsNAM, so this software is open source. You can play  around with it. You can set it up yourself. You can create your own graphs, or semantic graphs,  I should say. That's my story in a nutshell. Thank you all for listening. And if you like it, and if  you go to a website, then I just have one question. So this is a website. If you go here, then you  can sign up to a newsletter if you want to learn, or you can click on the get off star button if  you want to promote a little bit. So that's it. But this is a website. Thank you very much for listening.  Thank you.  That was a very great last talk for all that group. Thank you very much.  Thank you, for everything. Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thanks. In that case, I suppose your model will have to cook up vector embedding for that.  Is that actually sustainable if I put it on a server and then everybody tries to run your model  to get their embedding? How did you solve the resource problem? So two questions. Sorry about that.  No. Why are you sorry about it? That's a great question. So let me start with the first answer  to the first question. So, and sorry if I went over that too quickly because it's, this  whole everything what I told is also on the website in detail, but that's what's happening  here. We always use the same algorithm. So if you would have a data object with a longer  sentence like for example the summary or the title of the articles that you've seen, it  applies the same algorithm. So it says like first I take all these individual words, then  I find the center position between those words, then I weigh them based on the occurrence.  So certain words are seen as more important than others. So it weighs it towards that.  And then we have this optional word boosting that you can say, well in this specific case,  this word is very important. So move more towards that. That's how we create those vector positions.  So regardless if you're querying or if you're adding data, that's how we do it. So that's  why we also became agnostic about the fact that gloves about single words. And because  we learned that if we saw we the first prototype we did way back was very simple. We say, okay,  I have the word Apple show me what's nearby. And then as you would expect as glove does,  it says like, well, I found iPhone, but also found fruit. And then we did something very  simple, which okay, now go and sit in the center between Apple and iPhone show me again  what happens. And then you see actually that that's successful. And if I if I may, I can  actually quickly show that I'm the last talk. So I should have waited a little bit said,  go ahead. And then thank you. So there's a context, every endpoint, where you can say,  well, concepts. So if I now literally do what I just said, so if I would say, Apple, then  you see your Apple iTunes Google preview on this. And now, of course, in this example,  we don't see, of course, the fruit. But let's say if I would do Apple and not based on the  company, but on fruit. So I concatenate them, Apple and fruit. You see how it now starts  to get better and better in those results. So and that's how it the algorithm works.  So you can play around with this also yourself on this on this endpoint. And the other question.  Yeah, sure. So we, we, of course, we're also like a we were also a business. So we have  like a the core is like open source, but we built like a shell around that. So we currently  have six companies using this on a large scale in a variety of industries, also retail, oil  and gas, all those kind of things. And these graphs get pretty big. And especially if you  scale the communities cluster, it's fast, which is something I'm now I could say that  we fensely like architected that, but it was actually something we got for free because  we just the data model is just vectors, only vectors. And that's of course very fast to  scale and to search through. So the answer to that question is yes.  Can you use this for translation? Like you said, your vector space is built in one language.  But can you compare vector spaces by looking at one concept is associated with all these  things. And in your second vector space, another concept is also associated with the same things.  And you've already linked some some kind of translate use after translation.  I don't know. And I love the idea. So we're definitely going to try that out. I don't know.  I don't know. It's a great idea. We haven't we haven't tried that yet. So what we currently  do is that we just have you have a we've written in a language. So, um, uh, dutch, French,  English, of course. But we haven't tried that yet. So, um, if you don't mind, then it would  be fantastic to or of course you can try that yourself or we can do it together. But that  would be fantastic. It's a it's a great idea. And I don't know.  Thank you. Thanks very much. Um, that we are officially over with the death room. Nice  and I hope you all enjoyed it. Enjoying for them tomorrow. Enjoy the evening, of course.  And maybe you next year. Thank you. Thank you.
graph_weaviate_0_179.png graph_weaviate_180_779.png graph_weaviate_780_1091.png graph_weaviate_1092_1463.png graph_weaviate_1464_1919.png graph_weaviate_1920_3587.png graph_weaviate_3588_4355.png graph_weaviate_4356_4595.png graph_weaviate_4596_5039.png graph_weaviate_5040_7307.png graph_weaviate_7308_8627.png graph_weaviate_8628_8855.png graph_weaviate_8856_9479.png graph_weaviate_9480_12407.png graph_weaviate_12408_14327.png graph_weaviate_14328_14651.png graph_weaviate_14652_14843.png graph_weaviate_14844_15023.png graph_weaviate_15024_15167.png graph_weaviate_15168_15707.png graph_weaviate_15708_16727.png graph_weaviate_16728_17483.png graph_weaviate_17484_17939.png graph_weaviate_17940_18059.png graph_weaviate_18060_18119.png graph_weaviate_18120_18431.png graph_weaviate_18432_20003.png graph_weaviate_20004_20279.png graph_weaviate_20280_28367.png graph_weaviate_28368_28451.png graph_weaviate_28452_30503.png graph_weaviate_30504_32747.png graph_weaviate_32748_37475.png graph_weaviate_37476_37600.png