Googlebot needs to have a reference to your website.


Google Search: State of the Union (Google I/O'19)

Typically, that's a link that someone else, you know,


recommends. Or like someone goes like, this is great content.


You should check it out. But you also want to make sure that


you link us to all the other pages. You should have some


sort of internal linking structure and some sort of page


structure that tells us what the rest of the website looks


like. And last but not least, if you link, use a link. This


shouldn't be rocket science, but I think it's worthwhile


calling this out. This is a link. It has an href, and that


is a URL in that href. Good link. A plus, right? So, please


give us proper unique URLs for your site so that we can


actually discover and crawl them properly as well. Also, if


you don't want us to do that, if you would like some things


to not end up in search, please use the robots.txt to tell us


so. We will respect that and help you not have something in


search that you don't want in search, right? Also, if you


want us to discover, links are not the only way to make sure


that we see content on your website. Another possibility is


to upload an XML file, for instance, through Search


Console or tell us there's a site map with all the URLs that


you consider worthwhile discovering for Googlebot. Also,


if you have RSS feeds, you can give them to us using a thing


called web sub, and then we basically will start crawling


your RSS feed entries as well. If you have structured data,


and we're going to talk about structured data later, right?


So, we have structured data for jobs and livestreams. If you


are doing livestreams or have job offers on your website, you


can use an indexing API to tell us about these as well. Last


but not least, if you're really excited about something new


that you put in and don't want to use the other ways, you can


also literally go into Search Console. We're going to look


into that tool a little later and add it manually in Search


Console. All right. Now, discovery is just the very first


step. As I said, it's the beginning of this beautiful


relationship, right? But it might take a while for us to


actually get around crawling. 130 trillion pages plus, and I


mean the web keeps growing. That's a lot of stuff for


Googlebot to look at. So, we can't just immediately do that.


So, it will take some time for us to come to your sites and


web pages to basically crawl them in. But don't worry about


that. If you're worried about if we have discovered it


already, use Search Console to see that we have discovered it,


when we have crawled it, and when we will, like, crawl it


last time. So, you can use Search Console to learn more


about that. But we'll talk about that in a little more


detail later. Okay. Now, when I say crawling, I wonder if


there's, like, web developers in here that are like, okay, but


what is crawling exactly and does that work with my web


application or PWA, right? Well, crawling very fundamentally


is an HTTP request, but your browser doesn't. It's a GET


request to the URL that you provided us or that we


discovered. And that is great for websites that are just


HTML. But modern websites are actually not just plain HTML.


They look more like this, where you basically have a very


barebone HTML structure that loads JavaScript and then


JavaScript generates the content on the fly. So, how do we


actually get that into our index? How do we understand what


the content of this website actually is? Well, for that, we


have to render. Now, rendering is a whole different can of


worms and it's a beautiful topic with lots of interesting


details. We have a session on that at 1.30 together with


someone from the rendering team, Zoe, my lovely coworker


from the rendering team. Stage 3, if you want to join us for


that or check out the livestream later or the recording if


you're watching this on YouTube. Definitely worth learning


more about rendering. And there's one thing that is quite


cool that I would like to take away already, but we basically


go in much more detail later. And that is that we are now


actually using a modern Chrome in Googlebot. So, Googlebot.


It has been the number one thing that you guys are. You all


asked us, right? You asked us, hey, when do we get the latest


Chrome? What Chrome version is this? Why can we not have the


latest Chrome? When is it coming? Will it be updated? All


these questions, we listened. We took them very seriously, but


we also make sure that we have a good strategy in place and


that we have a sustainable strategy in place as well. And


you don't have to worry about things anymore. So that's why


we decided to not only update to the latest Chrome, but keep it


evergreen. So that's the big news here, really. This is not


just a one-time update and then you have to wait again. We'll


keep continuously updating Googlebot with this new Chrome


release coming out. Also, there's a blog post if you want


to learn more about what that means to you as a web developer


or publisher. Highly recommend checking that one out, too.


Right. I heard a lot of people. We announced this in the dev


keynote really briefly and a few people asked us questions


already. Not a few. A bunch of people asked us really good


questions. I love these questions. Keep them coming.


But the number one concern that is underlying all these


questions is do I have to worry about this? What does that mean


for me and my site? Is this a problem? And I can tell you, no,


don't worry about it. It's cool. We will not just start using


the modern Chrome. We have been doing that beforehand. We have


very carefully tested on an increasing number of sites and


today I'm happy to announce that if you have a URL on the web


that is being crawled and rendered, it is rendered with


the modern Google Chrome. So no worries about that. We tested


this out very carefully. Also, we keep monitoring rendering


carefully. If something goes wrong, we'll probably be very


quickly to notice that and fix that as well. So don't worry.


We have you covered. And also, if you want to learn more about


the user agent, a few people have seen, you're still seeing


the old user agent, that's because we don't want to break


sites that don't know about this yet. So we'll give you a heads


up in Google Webmaster Blog. So stay tuned on our blog whenever


we are about to change the user agent. So please prepare your


sites. Do not hard code the user agent exactly. Use something


like pattern matching to be a little more flexible. But


basically we will update the user agent, but we haven't done


that now because we want to make sure that we're not breaking


your content. All right? Cool. If you want to learn more about


that, as I said, please join us 1.30 p.m. tonight. It's going


to be fun. All right. So that was very brief. And the


session is going to cover more details. If you are using


JavaScript frameworks, not everything will just magically


work now. A lot of things have been fixed. We have over a


thousand new features. But if you want to learn more about how


to work with your specific framework and improve SEO on


your specific sites, definitely check out our YouTube video


series. It's called JavaScript SEO. You can find it on


YouTube.com slash Google web master. Sorry. Nearly had a slip


there. Right. But that's not the only thing that we are


doing. We also thought, hmm, the web usage patterns have


changed. Who here has a smartphone? Who here does not like


to raise their hands when questions are asked? Caught you.


So like pretty much many people have smartphones and use


smartphones to browse the web. We thought it's time for Google


bot to step up its game. Right? So we want to represent the way


that users actually browse the web. So to do that, we started


a multiyear effort called mobile first indexing. We announced


last year at I.O. and we are happy and proud to update you


that we are now over 50% of the sites that we are indexing are


indexed mobile first. So Google bot has stepped up its game and


it's using a smartphone user agent to crawl your websites.


So as not only we are parsing, sorry, indexing and rendering


most sites with a smartphone user agent, we are actually


also using, sorry, seeing more than 50% search results now


being actual mobile sites. And I think that represents the


users on the web much, much better than it had been before.


But none of this would have been possible without your support.


Every single one of you who is making websites that work great


on mobile has helped us so much and we are very grateful for


that. Thank you so much for your support and I hope that you


get your continued support on this. So this is for you all.


Thank you very much for helping us with this one.


So there's one thing that I would like to note and that's


that mobile first indexing does not necessarily mean that we are


only indexing mobile sites. We are trying that with pretty


much every page, even old desktop pages. So is there something


that you should be doing to make sure that you are doing fine


in mobile first indexing? Well, I mean, the thing is, if you


are making a website these days, you want to make sure that


the websites that you are serving to mobile and desktop are


more or less on par, right? The content should be more or less


the same, the text, the images, the videos, but that's not all.


It also matters to include things like alternative site links,


canonicals, meta tags for search engines should be also there.


Structured data, very important, should also be in your mobile sites.


John is going to talk about cool stuff happening in structured data as well.


So make sure that this works. And what we noticed is that


basically the thing is, if you use multiple different URLs,


that's a little tricky sometimes to do. So we highly recommend


that you actually try to make a responsive website that responds


to the different sizes of screens and not use like a www URL


and an m.url because then the question is what should go into the index,


what should we serve? So you can make your life easier by having just one version.


Also, we are probably going to make sure that we alert you


if there are changes to mobile first indexing and we have a lot of stuff


that is still to be done on our site, but we'll keep you updated


as the updates come. So definitely keep an eye on our blog,


our Twitter feeds, and yeah, I mean the question is,


can we maybe eventually switch to mobile first indexing for everyone?


The 100% is what we are striving for, so definitely keep an eye


on what we are saying on Twitter and our blog.


Right. With that said, once your page is actually in the index,


it might show up in search results. So is there something that you can do


to make it stand out a little more? And I think the most straightforward


thing for the simple search results that we are having here in this example


is you want to make sure that the title is actually very useful for the user


and makes clear what the page is about. But that's not the only thing.


Titles are something that you give us, but we might not exactly show them


the same way or we might change them slightly for the different queries


that people are asking. So don't worry if your title doesn't show up exactly


like you specified it in the search results. That's a good thing.


We are trying to help the user understand better what your pages are about,


so do not worry. Just give us a good title to start from and you'll be fine.


The same goes for the meta description. If you give us a meta tag


for the description, we'll show the little snippet and we'll try to match


what is the best snippet from what you've given us and what is on the page,


what's the best thing that we can show for this query for this user


in the search results. Right. We also try to understand the website


when we are indexing not just from these two things.


We are looking at every text that is on the website, headlines, image descriptions,


alt text, that kind of stuff is also very important for us in indexing


and helps us better understand the page and better show it.


And then basically we want to show your pages in the best possible light.


Speaking of which, best possible light, I think images are a great way


to even boost that further. This text stuff is nice, but images are something


that helps people understand the information you're presenting better


and more visual and I think they are a great opportunity for your site.


So John, would you like to talk a little more about images?


Okay. Fine. All right. So thanks for the intro on how Google search works, Martin.


I think that will be really useful. But let's talk about images a little bit more.


So for example, what if I'm looking for a Halloween costume?


Or what if my website is selling Halloween costumes?


How do I get those into Google and make them findable?


So we believe that traffic from Google Images is very valuable


and applicable to all kinds of different sites because of how we've seen users adapt to images.


People come to Google Images for all kinds of visual intent.


You might be coming for an outfit for a birthday party, designing your living room,


planning a wedding, finding a Halloween costume,


or maybe you're looking for a vegetarian recipe for friends coming over.


Or maybe you're just using GIFs to respond to a text that you received.


So we realize that with all of these things that users come to Google Images for,


from planning an event to learning about fashion,


they're not just looking for the image.


They're looking at the image as a gateway to the content underlying it.


The image is sometimes just an ideal snippet to show in search.


So we've done a number of things to improve Google Images for users


and for developers like you all.


And as a result, publishers are benefiting more and more from engaged users.


So let's take a quick look at what Google Images has been working on.


So here's an overview very briefly of how we've evolved Google Images.


Consider I want to build a tire swing with my niece.


Previously, when I searched for kids' tire swing,


I would just see all kinds of images of, well, tire swings, I guess.


But it was hard to tell what I would get from there.


So now, when I look at the results in Google Images, I get a lot more context.


So it really helps me understand a lot more what the pages are going to be about,


whether that's a guide about how to build a tire swing,


or if it's a tire swing I can buy online.


And this additional context means that I can intentionally navigate to the content


and to the site that's most relevant to my interests.


And similarly, there's a lot of metadata that's shown here in the search results


that really makes it easier for me to understand what I'm going to get.


So now I have a bunch of tire swing pages bookmarked to check out with my niece,


so I'll get started on that.


So tire swings look really cool, and I'm going to take a ton of photos


as I move along and put these photos on my blog.


But how do I get these into Google Images then?


What recommendations do I have to follow?


Well, for Google, it's pretty straightforward.


I mean, it doesn't get simpler than this.


First, you really need to make great websites and have fantastic images.


These are things that we really watch out for.


It's kind of obvious, I guess.


And then on those pages, we have to see those images prominently


for visitors and for search engines.


It really has to be clear to us what this page is about


and that the image plays an important role here.


So you should give us more context about those images


by placing text around the images and by using appropriate alt text


where it makes sense to understand more about the image


and how it might be interesting in search.


And similarly, if something is important,


don't embed it in the image, but rather keep it in text form


so that we can really find the text and understand what this image is about


rather than trying to decode the image.


And structured data.


We've heard about structured data quite a bit so far.


Structured data gives a lot more context about the images.


So that's really something that I'd recommend using whenever you can.


With structured data types like product and recipe in particular for images,


you can give us information about pricing, availability, ratings,


cooking time, ingredients, and a bunch more.


We'll have additional details on that later on in this session.


And finally, this is kind of easy as well,


use descriptive file names and a clean URL structure whenever you can.


So it's not that we won't be able to recognize an image if it just has a number as a file name,


but it does help us a little bit.


So similar to how Martin showed with links where we really need to get HTML,


with images, HTML is also important,


and it really makes sense to do something that's semantically correct


so that it's clear to everyone who's looking at the HTML page what this is.


So in particular, we don't pick up images that are only embedded using CSS styles.


That's really hard for us to tell that it's supposed to be an image


because it's completely different locations,


whereas if it's an image tag, we can pick that up,


we see an alt text for that, and we process that right away.


So the basics will help you get started


and enable your site to be shown in Google Images,


but you can go further. I'm sure you want to do that.


So responsive images, for example, are a great way to provide different resolution images by device type.


These can be implemented in a search-friendly way


using the picture element and the source set attribute.


And another thing a lot of modern sites do is lazy loading.


So with that, you can speed up the loading of your pages.


We recommend using JavaScript that leverages IntersectionObserver


and using a polyfill where you need to do that.


And that new lazy loading that you might have heard of in the developer keynote


that's coming to Chrome, that just works by default.


So if you're going to go that direction, that will work perfectly.


And finally, we've talked about this a little bit.


Martin mentioned this, I think, twice.


Robots text. So Googlebot is a polite crawler.


If your images are blocked by robots text,


then we can't crawl and index those images.


So if you want them indexed, then make sure they're accessible.


So that sounds pretty good, but we also have one new thing when it comes to Google Images.


So let's take a look.


Woo-hoo. So we're really excited to announce a new program that will be available soon,


specifically for websites with fantastic large and high-resolution images.


Users really love seeing larger images since they can provide a lot more context about a page.


And accordingly, showing bigger images in search can give your pages a little bit more attention.


So if you have large images on your website


and you want to make those accessible to users directly through Google products,


you'll be able to opt in to displaying these soon.


So we'll have details about that.


And the current plan looks something a little bit like this.


So we'll have a combination of a setting in Search Console


where you can tell us this site is interested in taking part in this process.


And additionally, per image, using structured data markup like this,


you can tell us which images you'd like to have indexed.


So here's a rough idea of what that will look like.


We'll have full details on this later on, including a license URL that you can use,


some sample code, more information on the Search Console functionality.


So I'd keep an eye on the Webmaster Central blog to find out more.


Wow, that's a lot.


So another cool source of traffic is Google Discover.


So the Google Discover feed is a feature in Google Search


that helps users stay up to date on all of their favorite topics


without actually needing any queries.


So it's kind of like Search without having to search.


Users get to their Discover experience in the Google app


on the Google.com mobile home page


or by swiping right on the home screen on a Pixel phone.


It's grown significantly since launching in 2017


and now helps over 800 million monthly active users get inspired


and explore new information, perhaps from your website as well.


So Discover surfaces the best of the web regardless of publication date,


from recipes all the way over to human interest stories


to fashion videos and more.


We have a guide for optimizing for Discover that's linked here.


And of course, images play a big role there too.


So as you can imagine, having large images in Google Discover


can make your pages particularly compelling to users.


Should you opt in to displaying large images in Google product,


we'll also be able to show these directly in the Discover feed like that too.


So if you're curious, sign up or watch out for the Webmaster Central blog


and learn more about the information there.


So another thing that's coming to Google Images fairly soon


is the swipe up to navigate interaction.


So when your website uses AMP mobile pages


and a user clicks on an image preview,


then Google Images will automatically prefetch and render a preview on the bottom.


That lets the user swipe up to instantly interact with your content.


No more page loading delays, instant gratification


thanks to the prerendering support in AMP.


So NetMet, your existing AMP pages give you better visibility,


they work in a new place, and they get you engaged audiences interested in your content.


So we looked at flat images, but how could it be taken to the next dimension?


Imagine you're studying human anatomy and wanting to learn about muscle flexion.


What if you could view a 3D model, in this case from a partner visible body,


directly in the search results?


Or how about a model that you can pull out and see directly on your desk?


So it's one thing to learn about flexion or extension,


but it's another thing to see it right in action in front of you.


So, for example, if you work with a furniture store website,


you could enable users to see your products in 3D directly in the search results


and to pull them out and view them in augmented reality directly from search.


How cool is that?


So today we're working with a small set of partners


to start bringing this content from websites into search.


We hope to be able to index more models from other sites in the future.


There are various tools that you can already use to bring your 3D content to the web.


For example, the open source model viewer component


makes it really easy to add 3D content


without needing to have any specialized knowledge.


So this is the same viewer that powers essentially the features in search as well.


So, shifting gears a little bit.


What other elements are critical for search on a website?


Well, structured data is one we've talked about a few times,


so let's take a quick look at some of that.


So once upon a time, search was really bland and blue.


Links to great websites, but just a small text snippet


with information about why they're good results for the user.


This is what search looked like in 1998.


It's been a while, so how can you make your search results stand out


a little bit more than just this?


So in comes structured data.


In the old days on websites, it's mostly about text on a page.


So here, basically HTML.


There's some list items that are marked up,


but we don't really understand what this page is about.


With structured data, you can provide the same information


in a machine-readable way directly on your pages.


There are multiple supported formats that we provide.


We recommend using schema.org vocabulary


with JSON-LD structured data markup.


So to get started, I recommend going to the search developers documentation


and review the search types that you find relevant there


and implement them maybe on a test page.


I'd recommend starting with just one type


so that you can get a feeling for the steps that are involved.


It's easiest to take just a sample page from your website,


copy it into a temporary location, and work on that.


Use the appropriate testing tool to make sure that it's working.


There are different tools for different types, so watch out for that.


And when your sample page works,


then you can add structured data to your real site.


So when will it be shown?


Remember how Martin mentioned that crawling and indexing takes time?


Well, it also takes time for structured data to be visible in the rich results.


So keep in mind, structured data can make your pages eligible


to be shown in the rich results, but it doesn't guarantee it.


So if over time you see it doesn't work, we recommend checking three things.


First, the markup needs to be technically correct.


Second, it needs to be implemented in a way that's compliant with our guidelines.


And finally, the overall website quality is also critical


for developers as well, because if Google systems aren't sure


about the quality of the website, then your work, the structured data,


might not result in rich results being shown.


So how do you work on quality?


I recommend trying to get really honest and hard feedback about the website.


This is a screenshot from an older blog post with ideas to look at.


A good source of feedback is from your users,


maybe doing a user study, getting feedback from other users.


And when in doubt, I'd also recommend checking in with other people


who've had similar problems.


In other words, check in with your peers.


So I think he's sitting back there.


Hi, peers.


And actually, what I meant is checking in with your colleagues and Webmaster Peers.


So for example, there are lots of awesome folks in the Webmaster Help forums


who've probably seen a lot of similar cases


and can help point out issues to work on.


So what are some newer types that you might not have heard about?


We'll run through this fairly quickly.


One of the recent types is the how-to markup.


This is for step-by-step tutorials on real-world tasks.


So this is a great way to help users find out what they should be doing


to kind of do the steps that you're looking for.


So you can specify the steps with videos, images, and with text.


How-to markup is pretty easy to do.


Structured data is specified in our developer documentation.


So it's mostly just a matter of filling that out.


Another cool form is for question and answer pages.


So QA pages are Web pages that contain one question


which users are commonly seeking an answer for,


often about a product or service that you have.


For example, you might use this for individual custom support requests


that your business gets.


And similar to QA pages, there are frequently asked questions pages


that have multiple questions on them.


So that's a great way to help answer questions about your products and services.


And there's structured data for more than just really fancy features.


Dates matter as well.


So we have a bunch of guidance on dates


and how you implement them in search in the link here.


And the short version is kind of that it's important that you be consistent with dates


so that we really can confirm that the date and the time and the time zone


are actually matching across your pages.


So you can do that with structured data, of course.


Specify the date in a machine-readable way.


Include the correct time zone


and make sure that it really matches what is shown on the page.


And there are lots more types of structured data


and rich results that you might want to implement.


These are just some of the newer ones.


As site owners often ask us, like, how do you manage this?


How do you keep track?


Like, Martin, do you have any ideas on what people should be doing?


I do. Thank you so much for John's fantastic introduction


to all the new structured data stuff.


And if you want to basically manage that and maintain that,


I highly recommend checking out Search Console.


Search Console is pretty much your window into the world of search


when it comes to your sites.


And how that works is you have to tell us or prove to us


that you actually are the legitimate owner of the domain.


You can do that with various ways.


You can use a token in your HTML.


You can use something else.


They have a few methods available.


You can also use a domain name, which is kind of nice.


And that's the very first step.


The next step is that you basically do what the setup instructions tell you.


And if you're building websites for other people,


so if you're like an agency, for instance,


you can use an API to do that.


So you don't have to basically do that manually


for every site that you're managing.


It takes a little while to verify your site and aggregate all the data.


But after a couple of days, once we have the data available,


we basically show you all the details.


By the way, if you're using the domain name, that's quite cool


because it includes all the subdomains


and all the different sites under this domain name as well.


So that's something you want to check out.


That's a relatively new domain properties.


So if it's been a while since you used Search Console


or you've never used it but you used Webmaster Tools,


you might want to have a look again because the UI has changed a lot.


And we are still expanding it to match the functionality


that the old tools had.


So we're not done there yet,


but pretty much everything vital has been moved,


so you can definitely use the new reports that we have in the new tool.


But feel free to switch to the old tool whenever needed.


That's absolutely fine.


But we don't just want to make a new, shiny interface.


We actually thought about it from the ground up.


How can we help you achieve the tasks that you have to do


quicker and easier, right?


So what can we do to help you be more successful in search as a site owner?


And we think that one of the great new reports that we have here


is the Index Coverage Report that helps you identify which pages are in the index,


which are not, and why they are not,


and if there's problems that you should definitely look into.


All this is visibly straight in this report, right?


And we can also help you identify and recognize issues


that we just detected as they come up.


So for instance, what if you deploy something


and a set of pages just suddenly return server errors or 404s?


We can tell you about that. That's really, really useful.


Also, once we have notified you of the issue,


you can basically drill down into samples of this issue occurring on your site,


double check that the issue actually still exists,


and then work to resolve it.


It's pretty easy to check that you fixed the problem.


There's a button that you can say Verify Fix.


And once you know that you have fixed the problem,


you can ask us to reprocess that URL


and that URLs that are affected by this issue,


and we bring you back into the search results really quickly.


So this is pretty unique. It's a unique feature.


And it has helped users to significantly be faster


at improving your websites.


Also, we have this lovely performance report


that allows you to understand how you're performing in search results,


and it gives you 16 months of data, so that's pretty cool.


You can see that evolve over time.


We also show you Discover Feed performance.


It's a separate feature and really useful as well.


And on top of that, we also bring this information into search results.


If you search for your own content and you're logged in,


you'll see it right in the search results.


You don't have to actively go to Search Console anymore.


So that's pretty cool.


AMP pages, structured data, all that has aggregated reports as well.


So if you want to dig into these details, definitely do that.


Also, we have a new speed report.


Sign up for the test.


We tweeted it from the Google Webmasters account.


Definitely sign up. Give us feedback before we roll it out.


And if you want to participate, that is fantastic.


And I think, to sum it up,


Search Console wants to be the dashboard for the features of the future.


We want to give you control over your sites in search,


and that's what we want to use Search Console for.


For an example, Duplex on the web.


You heard about it.


You can use robots.txt to let us not do that.


But I think it's pretty cool to be able to perform transactions on the web automatically.


You don't have to do anything like that.


We do the machine learning for you that basically enables all these features.


And if you want to prepare, verify your site in Search Console.


New features are coming to Search Console to help you.


We'll give you more updates on the Google Webmasters blog,


so definitely subscribe and check that one out.


And I think, to just quickly sum it up,


I believe Googlebot is now mostly smartphone,


and we are continuing that journey, so that's pretty cool.


JavaScript is welcome as well.


Users love really high quality, large images,


and showcasing your site in Google products like Search and Discover.


So we'll have more on that soon.


And there are many ways that your content can be highlighted in search,


so it's worth checking out the search gallery in our documentation


to find out what works for you.


And last but not least, use the Search Console to get the most out of search.


And I think with that, we just want to let you know the web keeps evolving.


It's a powerful platform, and Google Search is here to help your users and others


unlock this potential via Google Search.


Thank you very much for coming. Have a fantastic day. Enjoy.


Does Google have an index limit? #AskGooglebot

Today's question is from Digital Mosaics, who is asking,


is there a maximum limit to how many web pages with HTML


extension, so static web pages, Google


can index for one website?


Well, in short, no.


There is no limit.


But let's take a quick moment to cover two related topics.


First, static pages versus dynamic pages.


When content is made available to search engines,


they don't know if it's from a static page, so a file that


was added and has never changed, or if it's


created by something on the server, so a dynamic page.


We can't tell.


And it doesn't matter to search engines either way.


Sometimes, sites try to make dynamic pages appear


to be static pages by adding an HTML extension.


But that doesn't change anything.


There is no need to do that.


The other aspect is a limit for indexing.


We don't have any limits for indexing.


However, we do try to focus our resources in places


where we think they make sense.


With that in mind, our algorithms


will generally try to focus indexing on pages


that make sense to be indexed.


So if it's a completely new website,


we probably won't go off and index millions of pages.


However, if we've seen that it's worthwhile,


we'll be happy to do just that.


This process is fully automated.


So if you're building out your website,


our advice would be to make sure it's awesome so


that our algorithms easily understand


that it's worth spending a bit more time to get it all indexed


and shown in search.


Thank you for the question, and till next episode.


JavaScript SEO office hours, April 8th 2020

Hi there, and welcome to another JavaScript SEO Office Hours


recording or public Office Hours Hangout.


With me are 10 people in this Hangout tonight,


and you have submitted various questions on YouTube.


So let's see if we find the time to go


through all of these questions.


Before I jump into the YouTube questions,


anyone here in the Hangout having any questions so far?


Three.


Yes, there you go.


WebSockets.


Obviously, Googlebot currently doesn't support WebSockets


and kind of things.


Do you think that will ever change?


Because sometimes you see things like, I mean,


I can understand why normally.


There's things like Firebase that has its live connection,


and there's some other things that sometimes use WebSocket.


And Blazor as well, which is a new thing, which is kind


of compiles in soon.


So do you think that will ever change,


or you think it's always going to stay as it is?


So while I cannot make any comments on the future,


really, because I don't know what's going to happen,


the fundamental goal for Googlebot and Google


and Google Search is to make the world's information generally


accessible, including the web, obviously.


And so if we see a major trend going


to WebSockets for essential communications,


we will eventually probably have it as well.


At this point, it is a very niche technology


in the sense of that most of the crucial content that people


consume does not come over WebSockets,


or it has a fallback mechanism.


And as you said, we are not supporting it


at this point in time.


But I can't make predictions for the future.


There's nothing.


It's not that I am like, oh, yeah, we


are about to release this.


No.


There's no plans to be communicated


at this point for WebSockets.


But very good question.


Cool.


I will be typing these questions down as well so


that I have a running log of questions


so that I can identify more frequent questions, which


usually is a hint for us that we need


to be addressing these additionally in the documentation.


So if you see me typing, that's not because I'm like,


I don't know, chatting with my coworkers


with John or something.


But it's that I am trying to keep track of what gets asked.


Thank you very much.


Any other questions from the audience?


If not, then I'll start with some questions from YouTube.


I've seen that multiple people on YouTube


submitted questions around lazy loading, which


is really interesting for me because it tells me


that I might want to look into our lazy loading


guidelines again, as it's apparently not clear yet.


So to start with one, Lighthouse recommends


lazy loading off-screen images.


I usually do this with JavaScript,


and I use either a placeholder or a very low quality version


of the image as the initial source.


Is this low quality version or placeholder


likely to be indexed over the lazy loaded images


with JavaScript?


That depends on your implementation.


If you use the testing tools, you see what we are seeing.


If we are seeing the high quality versions


or the non-placeholder images that you load using


JavaScript, that will be fine.


If in the testing tools, you see that the rendered HTML contains


the low quality version or the placeholder image, that


means that there's something in your lazy load implementation


that isn't quite right, and we would not


see the higher quality version or the actual image lazy loaded


by JavaScript in that case.


So use the testing tools to determine


if your setup works for us.


The other thing is you can also use the native lazy loading


attributes for images so that you


have something that degrades nicely or actually


progressively enhances really.


So a browser would only see the regular image,


and you would specify just the image source that you want,


the high resolution image source that you want.


And then Googlebot would also see that,


and you would lazy load on top of it


when the browser supports it.


So that's not an issue there, and it's


built into the browser, so you don't


have to worry about implementation problems


with your JavaScript implementation.


Question.


How can I ensure that a one-time notification on each page


is not indexed in Google?


Is an onload event a solution because


of this one-time notification we have now found on searches,


including the term corona?


Well, the onload, there's like two different questions, really,


in this one.


Do we not see things that run on or in the onload event handler?


No, we're going to see those.


So that is not a very solid way of making sure that we are not


seeing your notification.


We might sometimes not see things


when they only trigger on the onload event,


but that's very unlikely, and that is probably


because weird or shoddy JavaScript.


I would very simply check if the navigator user


agent is Googlebot, and then if it is,


don't show the notification.


That's one way of doing it.


An alternative way is to hide it behind a user interaction,


so only users that scroll or click or do something


on the page somehow see the notification.


I think for corona warning notifications,


you just want to have whatever.


It's basically the same situation


as with the cookie banner.


So you could definitely use something like a button


to actually only show the notification when


it has to, when the user has interacted with the page


or when the user clicked on the page or something like that.


If you don't want to wait for an interaction,


I would go for the loading it only when Googlebot is not


there, or basically sniffing out if it's Googlebot or not.


It's not a great practice, but it is not cloaking either,


because what you're doing is not swapping out


content to mislead the user.


You're just loading additional content.


as if you would do server-side or dynamic rendering, where


the content might be slightly different,


but it is still the same content that the user would expect


coming to your page.


Unless the notice for corona would completely


remove the entire content and only load,


that would not be a great idea.


But use that wisely and carefully,


and you won't be within cloaking limits, so you should be fine.


Questions from you all, or should I continue with YouTube


questions?


AUDIENCE 1


On the topic of cloaking, can you go a bit more in detail?


What is considered cloaking, and where is the gray line?


I don't know.


MARTIN SPLITTENBERG So that is a really interesting question,


where is the gray line?


So where is the gray line?


I can't do much in terms of giving you


too much detail for that.


But fundamentally, cloaking means misleading the user.


That means if I see that Googlebot is requesting


my site, and I say, this website is about kittens


and butterflies, and then when it's


a user going to that website instead of Googlebot having,


I don't know, like an online drugstore


or trying to sell knockoff products or something


like that, that would be very much


against the intention of the user,


and that would not match what we would show in search results


if the user searches for cute kitten or something like that.


So that's very, very clearly cloaking.


What isn't cloaking is if my website content is slightly


different, because we all know that with responsive web


design, we might have slightly different content


to begin with.


On a mobile phone, I might only load one product instead


of 10 products or something like that,


and then have the user click through multiple pages


or something like that.


That's not cloaking.


That's just slightly different content,


depending on what the browser can do


or what the device capabilities are.


That is fine.


If you show slightly different content for Googlebot


than for real users, like a notification or a pop-up


that doesn't show when Googlebot comes in, that is mostly fine,


unless it is a pop-up that has 90% of the content in it,


and on the actual page is only an image,


then we are again, we're like, does that still


fall within the grounds of the user sees what the user expects


or what we saw as Googlebot?


But generally speaking, as long as you're not


misleading the user, you're definitely on the safe side.


What you shouldn't be doing is you shouldn't


be misleading your user.


Anything that is within reasonable bounds of that


is not a problem.


AUDIENCE MEMBER 2 OK, because I can think of two examples.


Like, for example, loading this dynamic approach,


like server-side rendering some part of the page,


but then actually enhance the behavior with JavaScript.


But the content actually is the same, OK?


Or a more extreme case, the second case,


is actually loading the translated version


via JavaScript.


MARTIN SPLITTENBERG That is a little tricky one.


So the first case is definitely not cloaking.


The first one is just slightly different content,


but it has the same topic.


It has the same, I have the same intention when I go there.


When you load completely different content


in the sense of a completely different language version,


that is trickier.


We do have mechanisms for that kind of stuff.


So for instance, you can use an alternative URL to something


that is linked via hreflang.


That would be safer, whereas when we're like, OK,


so this website is about, I don't know, cats.


And it very clearly is about cats.


But then when I go there from my browser and it says,


katzn, that might not always be easy for us


to distinguish that this is just the same content just


translated.


So I would be very careful doing that kind of stuff.


But you see there are certain gray areas.


And it might be that this works, actually.


I don't know.


But that sounds to me like, there is potential


for this going wrong.


So I would probably tread carefully.


But just loading slightly different versions


of the content because of server-side rendering


versus client-side rendering, that's not something


that you would need to worry about, definitely.


AUDIENCE MEMBER 1 OK, thank you very much.


MARTIN SPLITT


Title canonical and the site content


itself is rendered on the client side.


Will this affect our ranking?


No, unless it's broken and we're not seeing your content,


it will not affect your ranking.


If your website is very, very slow,


that might affect your ranking because speed


is a ranking factor.


But it's only one out of hundreds.


So you shouldn't worry too much about this.


But definitely test, test, test, test if you do that.


I also noticed that GSC, Google Search Console,


is warning us on mobile issues for tag archive pages


due to its rendering on first load


as the CSS gets injected in later.


Any recommendations?


Try to load the critical CSS as quickly as possible.


That's also a recommendation in Lighthouse, by the way.


And then only load CSS in later.


That is not essential for the initial render.


That should make these warnings go away


because that might actually cause trouble


when we try to figure out if a page is mobile-friendly or not.


So I would recommend to get as much critical CSS inlined


into the page as possible, but not too much


for the obvious reasons that then your HTML is too large


and so on and so forth.


Lighthouse has pretty good guidance on that.


So check out Lighthouse for your CSS troubles.


Questions from you all?


Because maybe these inspired you.


OK, I take one more from YouTube,


and then we'll see if questions come up here.


Hi, Martin.


What is your recommendation for the most optimal way


of troubleshooting if a, sorry, there's


someone trying to get into the chat.


Sorry, there's someone trying to get into the,


oh, the pop-up has disappeared.


OK.


of troubleshooting if a JavaScript-based website is


having issues with slow indexing, where content is not


indexed immediately due to longer processing times?


Not 100% sure what this means.


If this is aiming at the two ways of indexing,


don't worry about them.


We have discussed that last time.


If you do see that users are taking a very long time until,


or for users, your website is very slow,


that's something that you want to improve on.


I highly recommend using the web page test or Lighthouse


to get a feeling for how fast your website is


on people's devices.


But generally speaking, slow indexing


because of long processing times is not really


that much of a concern these days.


Anyone having a question to jump in with?


Yeah.


Can you hear me?


So.


Yes.


No.


No, it's work.


I have a little bit trouble with my mic.


So I have a question about structured data.


We have a dynamic render page to this structured data.


And we don't know if Google can get this normally


with the render engine from it.


Or it's a little bit difficult for Google


to see the structured data then.


We didn't know has the structured data or not.


We don't see anything in the Google search results.


Right.


That's a very good question.


When you say dynamic rendering, do you mean as in,


like, you are using something like rendertron


or pre-render.io?


Or do you mean server-side rendering?


So we don't do pre-renderings.


We only deliver at the moment.


Client-side.


Yeah, client-side.


So the Googlebot, I think the Googlebot


do pre-rendering with Puppeteer.


Also, you talk about in your stream.


Yes.


That's what I mean.


Yeah.


Yes, yes.


We are not actually using rendertron,


but we do render pages.


So that we have to do.


I can show you something.


So to give you an example for, no, no, no, no, no,


none of these.


To give you an example of what you can do is,


I have created a few test pages to show this.


Here, we have a website that, wait, hold on, does not


actually, I will try to make this larger so that it's


easier to read, actually.


Give me a second.


I think, yeah, this should be better.


We don't need that at this point.


So this website does not have any structured data in it,


as you can see.


There's no structured data here.


But it does use Google Tech Manager.


And if you check out this one, then you'd


see that it does have some structured data using Google


Tech Manager.


It's injected dynamically.


And when I want to know if Google actually sees this,


I can just go to goo.gle rich results.


So the rich results test, I don't need that anymore.


And then if I run this, I will see, I know.


Thank you very much.


I will see if it gets picked up or not.


And it gets picked up.


In this case, it has successfully


picked up that there's some organization markup


and that there's a logo in there.


And you can use the rendered HTML


to see if your structured data shows up.


If it is in the rendered HTML, that


means that we are also seeing it.


So generally speaking, it should not be a problem.


If you need some support for that statement,


then you can check out our developer documentation.


Because under Guides, Enable Rich Results,


there is Generate Structured Data


with JavaScript that has a lot more details than what I just


described.


But basically, we have documentation for this.


Now, the question is, why am I not seeing this


in Google search results?


That's because structured data is a necessity or a requirement


to actually be eligible for rich results,


but does not mean that we will always display rich results.


So even if your structured data implementation is correct,


we might choose not to show rich results because for, I don't


know, various reasons.


There might be a faster website.


There might be a website that has higher ranking information.


There's lots of factors to consider.


It's basically a ranking question,


and I can't really comment on ranking questions.


But you make yourself eligible by having validating structured


data, and you can use the rich results test


to see if we are picking it up.


So to understand this, if I add this literature rich result,


it's not guaranteed that Google showed this.


Correct.


But if Google showed nothing structured data from anything


else with the search result, this is OK.


This can be the problem from size ranking or something else.


It can be all sorts of things that make it not show up,


mostly ranking-related things, as long as we are picking it


up.


And also, Search Console should show you


if we have picked up the structured data.


If it is in Search Console and or you


can see it in rich results test, then you


have successfully implemented it on the technology


side of things.


Doesn't guarantee that it shows up in search results.


Perfect.


Thank you.


You're welcome.


I have a question in the chat.


I have a question from my developer partner,


who doesn't want to ask himself directly.


That's fair.


We sometimes see websites which check the user agent in order


to set specific behaviors for the Googlebot.


Can it trigger something?


Can it be seen as potential cloaking?


And if so, how does Google work?


It's very common here in our bilingual nation.


Interesting question.


We kind of scratched on that earlier on.


If you are changing content so that we


are under the impression that you are misleading the user,


that would be considered cloaking.


Just user agent sniffing and then loading


slightly different content.


Or for instance, the example I gave is, what is fine to do


would be to say, oh, this is Googlebot,


so I'm not showing a corona information banner.


That's fine.


Whatever.


This is not primary content to your website anyways,


so you can just hide it out.


If you're using it to swap out or redirect


from a page that is about kittens


to a page that is about cheap drugs online, that is cloaking.


So generally, when we have the feeling


or when we get content that is wildly different,


that is considered cloaking.


Where is the border between legitimate content changes


and illegitimate content changes, aka cloaking?


That is a tricky question, and you


will want to stay away from the murky waters


in the middle of that spectrum.


So as I said, like, server-side rendering something


or hiding a notification from Googlebot,


that's definitely not on the dangerous side of the spectrum.


Completely going somewhere else and actually


showing completely different content, that's a risky one.


We had the example of translations.


What if I auto-translate the content under the same URL?


That is in the murky waters where I say it might work.


It might not work.


I would probably steer clear of that and not risk it.


It's a tricky one.


But generally speaking, small content adjustments


do not run under cloaking.


That's where the bilingual comes in.


We have some really funky decisions


made by companies on which language they want to show you.


If a contest is not available in some provinces or some regions


in one language version and but it's


so let's say you have a website in, I don't know, Canada,


where you have French or whatever they consider


French and English.


And you have some content that is only available in English.


What I would do if it's only available in English


and I'm on the French side is I would either


have a page that says, oh, this content


isn't available in this language or in this province


or whatever.


How about you go there and have a link to the other thing


or just do a 404.


But don't try to be clever and don't


try to force different content on users


where they don't expect it.


It's not a great user experience.


And I'm not sure if Googlebot will handle this gracefully.


So you definitely want to be very carefully testing this


or avoid this to begin with.


I think that's my guidance on that.


Do we have questions in the audience


or should I go back to the submitted question?


Extended question to the question before.


So I've tried this with my page.


And we add the 3D structured data.


And I see I get no test result with this 3D structured data


but with another structured data.


Is the testing tool cannot actually at this moment


testing the 3D structured data like the one from Wikipedia


with Tiger?


I think that is a definite possibility because I think


the test currently is limited.


I'm not sure where the we have something


saying that there is a limitation to this.


But I'm not sure.


Here, supported types.


So currently, we support a bunch of types.


But we don't.


Actually, I can put it in the chat


if you want to check the chat.


That's the piece of documentation


that explains which types are supported.


What you can do, though, is in the structured data testing


tool, and actually, I'm going to run this real quick


through screen sharing one more time.


I will be sharing this entire screen.


If you are running this and you don't see your structured data,


what you can try is you can, oh, come on.


Here we go.


Use the rendered HTML.


And you can A, look for the, wow, thanks.


You can A, look for the structured data


to be present here.


But you can also use it to actually take it from here


and go into the structured data testing tool.


Structured data testing tool.


And that one should hypothetically


support this better because it has a few more


things in support.


It should show up here, but even that is not a guarantee


that it's correct.


You would have to manually make sure


that everything that we say is required


is actually present there.


3D data is a tricky one because it's relatively new.


And I'm not even sure if it is public-public yet


or if you are in a early access program.


Let's see.


But that's relatively easy to find out.


We can go in.


Wait, where's the structured data gallery?


It doesn't look like it is generally available


at this point, the 3D data.


At least it's not showing up in here.


So you might be in an early access or early adopters


program.


Yes, 3D and AR results are currently


limited to people in the early access program.


You can use a form to express interest if you're not


having done that already.


Thank you.


OK, good.


I need to call to my boss, and we


need to add to the early access program.


Then the most of the page is 3D data.


In that case, yes.


Thank you very much.


You're welcome.


OK, other questions?


Oh, a question on YouTube has been answered.


Oh, one of the lazy loading questions.


That's cool.


In that case, I may have a look at YouTube one more time.


Hi.


Does Googlebot have some issues crawling isomorphic pages,


and does it understand client-side routing,


or is it still safer to do the routing on the server side


and avoid page rehydrations on the same URL?


No, Googlebot does not have fundamental issues


with isomorphic pages.


Isomorphic pages, for those of you who are wondering,


are basically just like server-side rendering


plus hydration, where you run more or less


the same JavaScript on the server side


as you do on the client side.


We support that.


Client-side routing is fine as well.


And rehydrations are OK too.


Just make sure that it's implemented properly


and that you test it with our testing tools


to see if the content that you expect


is actually visible to us in the rendered HTML.


You can use the Google Search Console URL inspection tool.


You can use the rich results test.


You can use the mobile-friendly test, any of these work.


And with the rich results test and the mobile-friendly test,


you can do it on local development and URLs as well.


If you use a tunneling tool like local tunnel or ngrok


or something like that, you can basically


plug any URL in and see the rendered HTML that we see.


So that's quite nice.


This article compiled the quotes from John Miller, Gary,


and Martin over time.


Some of those quotes seem to contradict each other.


There's a guide from an article from oncrawl on lazy loading.


And it seems to contradict each other.


The guide is from 2019, from April 2019.


It's like a year old now.


And the contradiction comes from the fact


that they have actually read the article earlier because I


saw this question earlier.


If you look at Google Search over time,


you will see that it keeps changing and keeps improving.


Our mission is to basically make sure


that we understand the web that you create,


so we keep improving things.


That also means that quotes from a month ago, from a year ago,


from five years ago, might no longer be true.


Things evolve.


In this article, there are a few things


that look like they contradict each other


and not necessarily contradict each other.


And actually, this is elaborated in the question as well.


I know lazy loaded images are, in principle, indexable


if done properly.


But is it worth the risk to lazy load important images,


like product images, even if they are below default?


Yes, it is worth it.


Because especially if you use the native lazy loading,


there is no risk whatsoever.


Because it is an image element that has the actual product


image as the source.


It's just loading lazily when the browser supports it.


So even crawlers that don't run JavaScript


will get the high quality image.


But users on browsers that are more modern


will have that support and actually get a slightly better


experience, or actually much better experience,


depending on your network speed and the price of your network,


especially on mobile.


Lazy loading is quite an improvement.


And I would definitely recommend it.


I would be careful with custom implementations these days,


because why?


If some browsers have it and other browsers fall back


to less great but still solid behavior,


I think that's worthwhile taking or using this opportunity.


Also, is NoScript actually helpful for image indexing


for lazy loaded images?


Or is it like John Miller said, Google ignores it?


The quote there is slightly out of context, I think.


I actually haven't, I don't remember.


I think the quote is slightly out of context there.


Because we do ignore content in NoScript, except for images,


interestingly enough.


For images specifically, we have a workaround


that allows you to use images in NoScript,


and we will index it.


But we're not sure how long that's going to stay that way.


And if we really need this, if engineering finds out


they don't need to do this, they don't need to support this,


they might remove it.


We have seen with the pagination situation last year


that that can lead to confusion.


It does happen every now and then.


There's many, many engineers working on this.


And sometimes they decide that, well, we


get the signals elsewhere, so it's no big deal.


Then they run an experiment, find out,


yeah, it really is no big deal.


But then we have to communicate the changes.


So I would shy away from NoScript


because there's better alternatives.


The alternatives are using native lazy loading,


which is a fantastic progressive enhancing way,


or using a as robust as possible JavaScript implementation


to do lazy loading.


So I would not use the NoScript fallback.


At this time of recording, today, April 8, 2020,


as far as I'm aware, we are still supporting the NoScript


workaround for images.


But that's specifically for images.


It does not work for the other things.


Cool.


Questions in the meantime?


SPEAKER 1 I've got another quick one.


That's OK.


When you use the testing tools or anything


and render a page and view the rendered HTML,


it flattens iframes out in there.


Are they considered just part of the content


for ranking purposes?


Or do we not know?


Is that secret sauce?


MARTIN SPLITTENBERG It's not secret sauce.


And it is observable in the testing tools, too.


In certain cases, we will flatten iframe content


into the document.


That is when the iframe is large enough


and when the content is.


I'm not sure what the other signal


is that we are using to consider it as inlinable.


But if it's inlined into HTML, it


is at least sent to indexing.


What I don't know at this point, because that is also


a question about indexing and somewhat related to ranking,


is how ranking slash indexing actually


considers this content.


It is marked as a flattened or injected or inlined.


I think inlined is the term we're using.


It is marked as inlined content.


I'm not sure how it's used in ranking.


I'm not sure how exactly it is used in indexing.


So I would assume it gets inlined


in certain conditions.


And if you see it inlined in the testing tools,


that means that we are at least seeing the content


as part of this document.


Yeah.


Cool.


So very good question.


Other questions?


No?


All right, then I'll take another one from YouTube.


There is a website that is dynamically


excluding part of the content on its mobile version


using JavaScript instead of CSS as a responsive solution,


a read more button.


The content is being sent by the server,


but excluded in the client side.


Mobile-friendly tests and GSC seem to not recognize


that piece of content.


Is that correct to affirm that the hidden content may not


be read by Google?


In your case, I'm not 100% sure what the details


of this implementation are.


But if you say read more button, then I'm


assuming that you mean an actual button


that we would have to click on to load


the additional content.


If that is the case and the content is not


there in the rendered HTML, no.


We're not going to see it, and we're not


going to see it because there is a user interaction required.


We are not interacting with your page.


We're not clicking on anything.


But generally speaking, if you don't


see the content in the rendered HTML in the testing tools,


it means that we are not seeing it


when sending it to indexing.


So yes, you want to be very careful


with that implementation unless you


want to exclude that content, then mission accomplished.


Do we have a follow-up?


Yes.


AUDIENCE 1 Actually, this implementation, the content


is being passed by the server.


And it's being suppressed in the client side.


MARTIN SPLITTENBERG What does that mean, it's being suppressed?


AUDIENCE 1 Like dot remove with JavaScript.


MARTIN SPLITTENBERG Aha.


OK, then we're not seeing it.


If you remove it from the DOM, we're not seeing the content.


AUDIENCE 1 OK, awesome.


That's my main question because we


are discussing if we just pass it by the server,


was enough to render and to interpret it.


So it's not, right?


MARTIN SPLITTENBERG No.


If you're removing it and we're rendering it,


we're not seeing the removed content because we are sending


what is rendered to indexing.


So the tools show you we have successfully


removed the content you wanted to remove,


and we are not seeing it in indexing.


AUDIENCE 1 Awesome, thank you.


MARTIN SPLITTENBERG You're welcome.


OK, other questions?


OK, building up on this topic, what about the content that's


on mobile, for example, that is hidden or invisible,


that, for example, a sidebar that would appear on desktop,


but on mobile, it requires a user interaction,


like a tap on show sidebar to see the sidebar.


But the sidebar content is actually in the HTML.


So would Google even consider that content?


Because it's not actually visible


until a user clicks on show sidebar.


But it's there.


I mean, on desktop, it's there as well.


It's in the HTML.


But on mobile, it's not visible by default.


So let me be very careful here.


Anything that is in the DOM is considered for indexing.


So whatever we render, that goes into indexing.


That means something that is invisible,


but present in the DOM, in the HTML, we will see that.


Whatever is in the rendered HTML that the testing tools give


you, we will use that for indexing.


Now, if it is invisible, we might consider certain cases.


For instance, if it is invisible and matches topically,


then we would consider it maybe not


as important as the visible content,


but we would still consider it.


So we might not show it in bold in the snippet or something


like that, but we will still consider it.


If it is unrelated or if it feels


it is a spamming technique to basically just add


additional content that the user doesn't really see


and doesn't really need or isn't really


beneficial to the user at this point,


then we might exclude it from the actual weighting.


But fundamentally, if it is related, useful content,


even if it is not visibly visible,


as long as it is in the DOM, we will see that.


So the rendered HTML is what goes into indexing.


And then indexing makes decisions based on things like,


is it visible or not?


But that doesn't mean that we are not seeing it or not


using this for indexing.


AUDIENCE MEMBER 1 OK, thank you.


I've had this question for so long.


MARTIN SPLITTENBERG No worries.


And it is a tricky question because a lot of people


understand different things when they say not visible.


It's like, it's not visible, and then it turns out


they mean not in the rendered HTML.


And it's like, well, if it's not in the rendered HTML,


no, we're not seeing the content.


Whereas, like, yeah, but it's, and then you


can do like stupid things when you try to hide stuff.


It's pointless.


Or even if it's, that's a tricky one,


if you require a user interaction


to add it to the rendered HTML.


So like, if you have to click an actual button to load the thing,


we would also not see it.


And it's not really present then.


So there's like gray zones of that.


But then very fundamentally, we are


seeing invisible content if it's part of the rendered HTML.


MARTIN SPLITTENBERG Thanks, now it's clear.


It's finally clear.


MARTIN SPLITTENBERG Awesome.


To answer Milena's question, what's the one question


you are itching to answer today?


I already answered that one.


That was the lazy loading guide question.


Was like, oh, there are so many comments and quotes from you


all, and they're contradicting each other.


And I'm like, well, the article is a year old,


and also the comments were collected over a long period


of time.


That's something fundamental with SEO.


The web is moving forward really quickly,


which is amazing because that means


that we get to improve and build new cool stuff on the web.


And Google Search is trying to keep up with the web


as it evolves, so things change.


Things change constantly, and the only constant thing


is change.


But that also means you want to be very careful when you read


outdated articles or when someone says like,


when I tested this five years ago,


usually whenever I hear someone talking about something,


I'm like, that's outrageously wrong.


And I ask, so where does this come from?


Then usually they say, oh, like three years ago.


I'm like, yeah, three years ago we didn't have headless Chromium


in Googlebot.


We didn't have evergreen Googlebot.


So since 2018, this has dramatically changed.


So the best recommendation I can give you


is whenever you hear a quote and you're not 100% sure what's


true or if it's true or not, test it.


Test it.


A good article should tell you, this is what I did.


This is where I put it in.


So for instance, there's an article


that explains that Google Tag Manager and structured data


does not work with Googlebot.


And that article is like two years old,


and they have a piece of code that you can copy onto your page


and try it out.


And I took that piece of code and I put it


into the structured data testing tool.


And yes, structured data testing tool does not show it.


Structured data testing tool is not the latest and greatest


in terms of using the right infrastructure.


The rich results test, however, does


use the actual indexing infrastructure


and shows you what we are actually


seeing when we index your page.


And that one showed the structured data.


I'm like, aha.


So the source of these conclusions


is outdated because structured data testing tool is outdated.


But because this article was very transparent and self-contained


when it comes to how to test the hypothesis that they started


with before they came to the conclusion,


I could go along and say, yes, the article is right.


That was the case back in the time.


It is no longer the case as seen here


by taking that exact example, putting it into the actual tools


and seeing that it works, where it didn't work when


the article was written.


So always take everything with a grain of salt,


including the things that Google has said.


If I say something today, it might


be valid today and tomorrow, but not in two years,


not in one year maybe.


Make sure that you have the latest sources.


Our documentation, we are not always 100% up to date,


but we are trying our best to stay as up to date as possible.


And usually, we draft the documentation


before the thing is launched.


And when it launches, we update the documentation in one go.


If not, then we usually communicate that very clearly.


So make sure that you are not jumping to conclusions.


Test for yourself and try to find the most up to date


source of information.


Now, another question from the chat.


Giacomo is asking, for websites like e-commerce that


are using structured data for products, when we need


a very quick update on availability


for a specific product out of stock or something similar,


would you suggest implementing structured data


in the HTML without JavaScript?


Will Googlebot parse structured data


after getting the non-rendered version of the website


without waiting for the internal Chrome service


to render page queue in the render list,


or structured data will pass anyway


after the rendering phase?


The queue is at median five seconds.


So assume every page gets rendered these days,


and that does not make that much of a difference.


However, that being said, in the guidelines for structured data


and JavaScript, I noticed that we do cache aggressively.


That means if your JavaScript needs


to update to actually reflect the new change in structured


data, which might not be the case.


If you use Google Tech Manager, then the JavaScript


does not need to update because it's just a data change.


If your JavaScript needs to update to actually reflect


the new structured data, then you definitely


want to make sure that you use proper caching, in this case,


like something like long-lift caching with hashes or version


numbers or something like that.


Because if we use an outdated cached version


of your JavaScript that contains the structured data as part


of the asset, we might not see that updating very quickly.


So I highly recommend making sure


that your JavaScript is updating and caching properly


using Google's tools.


Basically, whenever you see an update,


you want to make sure that what we are crawling


is actually the latest version of it


rather than an outdated version.


If you don't version your assets,


that is very, very hard to debug,


just to give you an example.


Generally speaking, putting the structured data in HTML


is probably always going to be more robust,


but there's nothing inherently wrong or faster or slower


regarding JavaScript and structured data.


What's the average expiration date for technical statements


from you?


Oh, my goodness.


I don't know.


Sometimes it is years.


Sometimes it is months.


Sometimes it is weeks.


If you asked me about the Google Chrome version that


is being used to render, if you would have asked me that two


weeks before IO last year, so let's say


you would have asked me that end of April 2018,


I would have said it's Chrome 41.


That would have immediately changed in May.


I would have hinted at that to potentially change soon.


But some people then ignore the fact


that I say, but this is about to change,


and then just quote me on, like, Martin says,


Chrome 41, which is true.


I said that.


They're just ignoring the sentence I said afterwards.


So be very, very careful when people are quoting me.


Generally speaking, just point to our guidance.


Our guidance is 99% likely to be the actual source of truth.


Average expiration, I don't know.


A month, maybe.


Don't know.


Cloking question.


If we use dynamic rendering on a page with infinite scrolling,


would it be OK if the static HTML version that crawls


see have href links to the paginated?


No, that's fine.


That's absolutely fine.


If you have pages as in previous and next or pagination


links in the version that only Googlebot sees,


that's not a problem.


That is OK.


That's perfectly fine.


That's not an issue.


All right, one more question from YouTube, maybe.


Would you generally recommend implementing dynamic rendering


on e-commerce sites that use JavaScript to display products


on category pages?


My feeling is that this should make indexing


more safe and accurate, especially


for crawlers other than Google.


We've said it multiple times.


As far as we're concerned, dynamic rendering


is a workaround.


I would not recommend dynamic rendering to anyone


who can reasonably switch to server-side rendering


and hydration because server-side rendering


and hydration gives a more robust rendering to crawlers


that don't understand JavaScript,


as well as a better and usually faster way of rendering things


for users as well, whereas dynamic rendering is only


useful for bots, especially those


that don't run JavaScript.


So dynamic rendering is considered a workaround.


I would not necessarily encourage people


to do that unless it is the most viable option for them


until they can consider server-side rendering,


or if server-side rendering is an investment they are not


willing to make or they can't make


for whatever technical reasons.


But generally speaking, if you come


to server-side rendering and hydration,


sure, go for dynamic rendering.


That will definitely help you with crawlers and bots


that don't run JavaScript or don't


run JavaScript as reliable.


There's nothing wrong with it, let's say.


But understand that dynamic rendering,


even though it's a workaround, does incur infrastructure cost


and does mean maintenance on your end


and can hurt you if it's implemented incorrectly.


Everything that is implemented incorrectly


can hurt you, including HTML, static HTML.


So it's just more complexity, and you


want to be careful if you really need that additional complexity


to reach whatever goal it is.


If you don't really care about search engines that


don't support JavaScript, as far as I'm aware,


Bing supports JavaScript.


Google supports JavaScript.


If your rendered HTML in the Google tools looks fine,


at least we are not concerned.


I'm not sure how you test it in other search engines.


But unless you have a good reason


to implement dynamic rendering, I wouldn't.


OK, do we have other questions?


No.


Do you guys have questions?


Do you, ladies and gentlemen and others, have questions for me?


So if nobody has a question, I have a question.


Go ahead.


Yeah.


So it's for the type of cloaking question.


So we have 3D data and a table with information


of this 3D data on our page.


So we have a table with information


And we have a table with information


The Google bot is coming to us with the mobile version.


I can see in the GSC.


So and the point is, in the mobile version,


we show the user first the 3D model.


And secondary, he can click on a button.


And the table is showing.


And 3D model is displaying it.


It's displaying now.


So it's cloaking if we say, OK, both content


is at the same time in the content.


But if the Google bot comes, you see first the table.


No, that's not cloaking.


OK, perfect.


All right.


Awesome.


I think it's time for the last couple of questions.


Now is your chance.


I ran out of YouTube questions, as far as I can tell.


I can refresh the page.


Maybe there's an additional one.


I don't know.


No, no, no, no.


Doesn't look like it.


No, no one asked.


Oh, actually, no, that's not true.


There's one more question.


Can FAQ, schema, and special announcement


be read and indexed by Google delivered


through JavaScript that is powered through a third party


tool or platform?


Possibly.


It depends on the implementation.


It depends on if they let us crawl, or actually,


if they let us request their resources.


So let's say you are using urexample.com,


and you are using a service from structureddata.org.


And structureddata.org says, oh, our API.js shall not


be crawled by robots.


So they say, like, they block it in robots.txt.


And what happens then is if we come to your website,


and your website loads that JavaScript file,


we can't load it.


Because they are saying, we can't load


that page using our robot.


So Googlebot can't fetch the JavaScript,


so the JavaScript doesn't run.


And then the content doesn't display it.


That is a scenario that happens more often than you might think.


And there's other ways as well.


There can be issues on their side.


There can be problems with your JavaScript.


So you have to test your implementation.


Fundamentally, technology-wise, it


is absolutely fine to have FAQ schema


and special announcements generated by JavaScript


following our JavaScript and structured data guidelines.


But you want to test that very carefully.


If the test shows you, yes, all good, then you are safe.


If not, then, well, then you want


to either talk to your third-party provider,


or you might want to migrate away from that third party


if that's something that you care for.


Test, definitely test.


OK, time for our last question from the audience,


if anyone has one.


You can also write it in the chat.


If you don't want to speak up, that's also fine.


I'm flexible either way.


Cool.


Wait, chat has a question.


Is there a rendering budget for websites?


No.


Asking this question related to my previous one,


because I can see that using Google Tag Manager


for structured data, the big e-commerce site


after a Google visit, we have not the structured data


updated.


There's also for more than a week using HTML version,


we get the data refreshed way more faster.


That is possible, but there is no such thing


as render budget.


This can be a caching issue, possibly.


Without looking at the specific website,


I can't really make a judgment on that.


But if you are seeing issues, you're


very likely running into caching.


And then if you really care very much for the data


to update very quickly, then either host


and version the JavaScript yourself,


or consider having it in the HTML.


But again, caching is king here.


And also, if we are not crawling it very often,


then I don't think that's going to be even faster.


But I think from what you say is that we


are crawling quite frequently, as the HTML does


have the update, whereas we don't when


it's using Google Tag Manager.


So I guess Google Tag Manager might be also


affected by caching, which would surprise me.


That's not impossible.


OK, excellent.


I would like to thank you all very much


for these fantastic questions and joining me here live


as well as on YouTube to give us all your questions


and discuss them here in this forum.


The next JavaScript SEO office hours


will be in approximately two weeks' time.


I'll update the YouTube community


feed with the actual dates, and I will also


post the link there again.


Thank you very, very much for being fantastic.


Stay safe, stay healthy, and all the best for you.


Hope to see you soon again.


Bye-bye.


Stay healthy.


Trying my best.


Bye, everyone.


English Google Webmaster Central office-hours from September 4, 2020

All right, welcome, everyone, to today's Webmaster Central


Office Hours Hangout.


My name is John Mueller.


I'm a webmaster trends analyst at Google in Switzerland.


And part of what we do are these Office Hour Hangouts, where


people can join in and ask their questions


around their websites, SEO, and search,


and those kind of things.


Bunch of stuff was submitted already


on YouTube, which is great.


But if any of you want to get started with the first question,


you're welcome to jump on in.


Hi, John.


Hi.


I have just got a question from one of our clients.


So the question is about language tag.


Now, we know that we can add language tag


at the header section of the website,


or we can create XML sitemap to show other version.


So now, which one is better, actually?


Or both are the same?


So I think you mean the hreflang attribute?


Yes.


Hreflang?


Yeah.


Both of those are equivalent.


We treat them exactly the same.


OK.


The reason he's asking this question,


because the site is built on WordPress.


And if we want to add it to the header section,


we need to use some plugin and client-based effort that


may reduce the speed, something like this.


That is why he is insisting to use XML sitemap.


So do you think this is really a factor?


I don't know if it would really affect speed,


but it's certainly an option just to use a sitemap file.


And you don't have to use the same sitemap file that you


use for the rest of the site.


You can have your normal crawling sitemap file,


and you can set up a separate hreflang sitemap file just


for those language connections.


So that's something that sometimes it makes it easy


to put it in a sitemap file.


Sometimes it's easier on the page for debugging.


It's really up to you.


Thank you.


I guess there's also the option of HTTP headers.


So that wouldn't require changing the HTML code,


but it's still a bit more complicated to implement.


Yeah, I guess especially on WordPress,


you'd have to do even fancier things.


Yeah.


Cool.


Let me jump through some of the questions that were submitted.


And hopefully we'll have some time towards the end


as well for more questions from you all.


Or if anything comes up in between that fits,


we can take a look at that.


So the first one that I have here,


I don't really know how much I can say here.


So it's about a domain migration.


We migrated from one domain to another on August 6.


And everything went bad, essentially.


And there's a forum thread with a lot more information there.


And from a first glance, looking at the sites there,


it seems that it's kind of OK.


So I suspect I'd need to look into that a little bit more


in depth.


So I'll take a look there and also


see if there is any escalation from the forum thread


that we can help with.


Why exactly websites that have been hit by the core update


can't recover before the next core update,


even if they make good improvements?


Some algorithms are launched one time for a couple of months.


Or how does that work?


So with core updates, we're essentially


trying to re-understand how the relevance of the search


results are.


And it's not something that requires a site


to kind of wait for the next update


to have a chance to be seen differently.


They can continue working on things.


And things can improve over time.


It's possible that our next core update


will make a bigger change in the same direction


that you've been working.


And you'll see a bigger change in your site's performance


as well.


But in general, sites don't have to wait


for the next bigger update in order to start seeing changes.


So from that point of view, I wouldn't just


stop working on things once you think


you've done the right thing.


But I'd continue working in that direction.


And you should see at least some incremental improvements


over time.


Are there any tricks to check a website or page


if it has quality issues?


Like this, for example, the page doesn't show up


on the first position for its own unique content.


So something can be bad there.


I remember in the past, Google hides rich snippets


for low-quality pages.


I don't know if there's any trick to see the way


that our algorithms assess the quality of a website overall


or the quality of individual pages,


because we do look at different things.


With rich snippets, I think it's slightly different just


because of the way that these are implemented there.


So with rich snippets, if you've implemented them


in a way that is valid with the validators,


so valid from a technical point of view,


if it's valid from a policy point of view,


and if your website just isn't of the quality


that we would like to show rich snippets for,


you can sometimes do a site query


and check to see if the rich snippets would show up there.


It's not something from our side that is done by design.


So it's not like a feature that was added from the rich snippet


seeing that you could debug things this way,


but more of a side effect.


And it's also something where it's


possible that at some point this kind of side effect


will get cleaned up to make it more consistent.


I don't know if that's really important,


because not a lot of people just do site queries on their own.


But it is one of those things that's


kind of more of a side effect rather than


a purposeful decision.


And for other kind of quality issues,


I don't really think there is a clear way


to kind of understand how Google would see that.


In general, when you run across quality issues on your website


or when you suspect you have quality issues on your website,


then that's something where I'd recommend getting input


from external people as much as possible


rather than just purely looking at the search results.


So that's something where my general recommendation is


to take the questions that we have in the blog post


that we did for core algorithm updates sometime last year


and look at them with people who are not directly associated


with your website to see what do they think about your website


when they're kind of experiencing it


for the first time, when they have


to complete a task on your website


compared to other websites maybe,


and to really try to get that objective input from someone


who's kind of fresh, who's not skewed


from a technical point of view, who's


not skewed because they know you and they think you always


do fantastic work.


So that's usually my recommendation there.


We're considering changing our navigation on our e-commerce


site, and we have a couple of queries.


Can you tell relatively quickly whether a new navigation will


have a positive or negative effect on a site's ranking


once deployed, or does it take a couple of months for things


to bed in and see?


Also, can it affect things from an SEO point of view


depending on where the links are displayed in the navigation


or not?


Do you have any other tips or advice on this


and whether one can revert back to the old style of navigation


if it has a negative effect?


So the last part is definitely the case.


If you make a bigger site-wide change on your website


and you revert that, you basically


have the previous situation.


It's not the case that these kind of things


will just flip back and you will have the previous state


for your whole website.


But rather, we kind of have to reprocess that and understand


your site again with the new state.


In general, when you're making bigger site-wide changes


on a website with regards to the internal navigation,


that can definitely have an effect on SEO.


Usually it does.


I mean, it's one of those things that SEOs sometimes


look at when they look at a bigger website.


My main recommendation there would


be not to change the URL structure unless you absolutely


need to.


So that's kind of on the side, because changing the URL


structure on a website site-wide is something


that does take quite a bit of time to be reprocessed.


So if you can exclude that and purely focus on the navigation


part, that makes it a little bit easier.


And with regards to the navigation,


the navigation for us as a search engine,


I guess, serves two purposes.


On the one hand, we need to be able to discover


all of the pages on your website,


so to be able to crawl through everything


and find all of the content.


And on the other hand, we need to be


able to understand which pages are relevant or important


in which context.


So which of these pages belong together?


Which of these pages is the most important one?


That kind of thing.


And that's something that you can provide fairly well


with a navigation on a website.


So one extreme might be you link all pages to each other.


And that makes it easy for us to crawl,


because we have to look at one page,


and we find links to all of the other pages.


But it makes it hard for us to understand


which of these pages belong together


and which of these pages are important.


So finding a balance between everything really flat


and everything really deep where you


have to follow one link after another to find everything,


that balance is sometimes a bit tricky.


But it is something where you need


to look for that balance instead of just purely taking


an SEO crawling tool and saying, oh, it says my crawl depth.


It's five.


I need to change it to four.


That's probably not the right approach.


But sometimes these tools can give you some input on things


that you might have missed.


AUDIENCE 1 The URL parameter tool,


since e-commerce sites tend to use it, especially given


the parameters and everything, will that ever


be moved to the new search console?


Is it planning to be deprecated?


Are there any plans with the tool?


The data there has been missing for a really long time.


But that's not because we want to deprecate it.


It's just because things are weirdly stuck on our side


with the various teams that are involved


with creating that data.


Internally, we use similar data already.


It's not that we don't follow that input at all.


It's just the data that we display in Search Console


is kind of stuck just before it reaches Search Console.


And my understanding is that that should


get resolved fairly soon.


But I've been hoping that that's fairly soon for a while now.


So I've been nudging a little bit more.


Hopefully that'll get better.


With regards to moving to the new Search Console,


my understanding is that they do want


to kind of keep that functionality


and move that to the new tool as well.


I expect, especially for larger websites,


we'll have some really cool stuff coming out as well


over time that, yeah, it'll be pretty cool.


I don't want to pre-announce anything,


so I'm not going to go into more any details there.


But it definitely is something that makes sense to focus on.


Like you mentioned, for e-commerce sites


with a lot of parameters, you can clean up some things there


and make it a little bit easier to crawl and index your site.


But just out of curiosity, if you're already doing that using


from what I've seen, there are two situations where


the URL parameter tool makes sense.


Both are for bigger websites with a lot of parameters.


But one is if you really can't set those parameters


with canonicals and things on your site and internal linking


and all of that, then it's kind of a way to kind of bridge


that.


And the other is if you have an extremely large site


and you have individual parameters that just blow up


the whole crawl space, then that's


something where it can make a big difference.


So those are kind of the two situations.


And it's really in both of those,


it's for sites that are a little bit bigger.


It's not the average kind of, I don't know,


100,000 page e-commerce site.


It feels kind of small in regards to what


those tools are kind of meant for.


AUDIENCE MEMBER 1 Hey, John.


Since we are, just as current here,


since we are on Search Console as well,


I've been facing some issue with the Search Analytics API.


So I'm just wondering how we could actually look into that


or raise it in a sense that if I look at my search query


data or my search analytics data,


I see impressions and clicks.


But when I'm trying to pull the data via the API,


it doesn't return any entries at all.


And to add some context, half my, like on one of my domains,


it's working perfectly fine.


On one of my subdomains, it's not working.


JOHN MUELLER OK.


My guess is that has something to do with the way


the site is verified.


Maybe you have something like HTTP verified in the API


and HTTPS verified in the UI.


Because the back end for both the API and the UI


are exactly the same.


So what commonly happens is that the absolute numbers are not


exactly the same because the queries are processed slightly


differently.


But it shouldn't be the case that you have no results in one


and lots of results in the other.


That definitely shouldn't be the case.


So basically what I did was that I used the exact same domain.


So imagine if I have www.x.com.


And that's what I verified.


And I took that exact same domain user.


And I plugged it into the Search Analytics API test


query on the browser page.


It returns a 200 but without any data.


So that's the issue I'm facing.


So it's valid.


But there's no data that's coming back.


JOHN MUELLER OK.


That seems weird.


So I mean, you're welcome to drop your domain here


in the chat.


And I can take a look at that afterwards.


But one thing you might also want to try in the meantime


is to check out some of the other tools that


use the Search Console API just to see if maybe there is


something unique in the way that you had that query compiled


versus how other tools would pull that.


Like, Mihai has a nice plugin for Google Sheets


that lets you also use the API to pull in data.


And that way you can kind of double check,


is it that Search Console is not giving me any data


because my domain name is slightly different than what


it expects?


I don't know.


Or is it because I'm doing something slightly unexpected


where maybe I'm copy pasting something slightly different?


MIHAI APERGHISLOOO-MUNDEZHUHAS.


Cool.


Thanks, Mihai.


I saw your message.


Hi, John.


Thank you for taking your time.


So I have a question regarding featured snippets.


So we have encountered some occasions


in which there has been a featured snippet in place


in search results.


But afterwards, after some qualitative analysis


of search results, we came across the same term


that we looked up on Google.


And we found out that there is no featured snippet anymore.


So there are two questions for us now.


Is there any way we can actively approach a featured snippet


by structuring our data in a certain way on our site?


And if there happens to be a case


that a featured snippet goes missing,


what does that indicate for the term and for the search result


as such?


So is the featured snippet not attractive anymore?


Should we not try to get the featured snippet


if Google removes the featured snippet from search results?