which people install in order to enhance


Build a successful web presence with Google Search (Google I/O '18)

the functionality of their site.


And one of those things are plugins.


Now, plugin here is defined as any kind of add-on


that people would add to their site.


So for example, things like a shopping cart or maybe a way


to add reviews or a comment plugin, things like that.


So while it can enhance the functionality of the site,


it can also significantly alter the functionality


of the site in terms of performance and other factors.


So I wanted to give you a few tips on what to do


if you're building plugins.


First of all, make sure that it doesn't slow down


the performance of the site.


So in order to do this, have a test site, install the plugin,


and then use our speed tools to make sure


that the site with the plugin is doing just as well as the site


without the plugin.


This is webpagetest.org, which is one of the performance tools


that we have.


And the neat part about it is that it will give you


a super detailed breakdown of what loaded and when.


So you can see how your plugin is affecting


So test that out.


Then, if you're building a comment plugin,


and if you've been on the internet in general,


you will know that there are a lot of comments


out there which are maybe a little bit less


valuable than other comments.


And especially in some cases, they're altogether spammy,


or they're bots which are going around and posting


auto-generated stuff in order to create links


that they're hoping search engines will follow


to some spammy websites.


So this is not pleasant for any user.


And you can actually help them out a little bit


if you're building a plugin like this by adding


a specific type of annotation to those links by default


so that search engines know not to trust them.


This is a link attribute that we call nofollow.


And basically what it does is it will just


tell the search engines, don't follow this link.


Don't trust it.


So if you're building a comment plugin,


definitely consider adding this to the links in the comments.


Finally, and kind of most unfortunately,


we've noticed that one of the main vectors for attack


on websites and attacks on websites are increasing


is hacks through plugins.


So a lot of hackers and other malicious people


will get access to a site through an old plugin.


And if you're building plugins, there's


a few things you can do to make sure that your users are not


affected by this.


First of all, make sure that every time you add an update,


everybody who has this plugin automatically receives it.


Then make sure to follow coding best practices


so that there's no back doors that the hackers can exploit.


And finally, if you get tired of this plugin


and decide not to support it anymore,


make sure it's clear to people that this is not supported


so they don't go ahead and install


something that is actually making


their site more vulnerable.


Themes are another thing that is very closely related to CMSs.


And a lot of people install them in order


to improve the appearance of their site


and make it look cooler or give it a specific feel.


So they can change how the site looks


and how users perceive it, but they can also really affect


performance, and they can also affect mobile friendliness.


So again here, test your theme.


Make sure that it's responsive.


You can do this with the mobile friendly test


that John was showing earlier.


And for performance specifically, we


recommend, again, having a test site, and then with the theme


and with another theme, having a look at how it performs.


Lighthouse is one of our speed tools,


which is really useful in this case


because it's in the browser.


You don't need to have had the site processed by search


in order to test it.


So here is a blogger site that we used


for purposes of this example.


We install the theme, and then we


use Lighthouse to do the performance testing.


So you see how long it takes actually


until what they call the first meaningful paint, which


is when all the elements appear to the user.


So the overall score for this team was not super great.


And also the specific user metrics were not great either.


But then we went ahead and we switched to another theme.


So you can see here it's much, much faster to load,


and the user can interact with it much faster as well.


And consequently, the score is also great.


So if you're building themes, definitely


make sure that it's performant and also that it's


responsive by using the free tools that we provide.


So there's a lot of stuff that we covered today.


And hopefully for any of these things


that we've told you that you might be building,


you now have enough tools and equipment


to go ahead and make improvements


and make your users happy and more successful in search.


We know that there is many, many different details and links


that we provided.


So if you have to remember just four things,


pay attention to what John is going to tell you right now.


All right.


Thanks.


Wow, that was a lot.


OK.


Looking back at these things, there


are a few common elements that we covered


that came up again and again.


So first is remember the basics.


URLs, titles, and descriptions, they do matter.


They do play a big role when it comes to search.


They play a big role in how people come to your site


through search.


Secondly, remember to take advantage of structured data,


like the Rakuten example in the beginning.


Obviously, they saw a big change in the traffic from search,


even without ranking changes, just


by making their search results look a lot more visually


appealing.


And then take advantage of all of the tools and APIs


that we have available.


So use the Search Console.


Understand how Search Console works.


Use the APIs from Search Console to make it better


for your users, people who are using your products,


to really create fantastic web presences.


And then finally, especially if you're


making something for other people


to create web presences in, make it as easy as possible for them


to do the right thing as well.


So make it easy for them to put the right fields in


to add data about titles and descriptions on pages.


Make it easy for them to really create high performance web


pages.


So these are only some general tips,


I think, to get started with.


There are obviously a lot of different aspects that come


into play with search.


But we think these are aspects that are really critical


to start with.


We have a lot more information in our developer center,


developer guides.


We have a Search Console help center


with a lot of more information about search in general,


and specifically about Search Console.


If you have any more questions today,


we'll be in the web and payment sandbox area later today.


So feel free to come by there as well.


And finally, there are, of course,


other ways to reach out to us online as well.


So you can find us on Twitter.


We do live office hours hangouts where you can join us


with a YouTube live hangout.


We're available in the Webmaster Help Forum


if you have any questions.


So don't let questions kind of stick around.


Make sure you get answers to them from us if you need them.


We hope you found this introduction


into search interesting.


Thank you all for coming.


We wish you and your projects more success


online through Google Search.


Thank you.


Does a .jobs domain help rank for "jobs"? #AskGoogleWebmasters

Today's question is from Ivan Hydra.


Ivan is asking, does a.jobs domain


improve the ranking in Google, specifically


for jobs websites?


This is a really common question that comes up


for the new top-level domains.


In short, no.


You don't get a special bonus from having a keyword like that


in your top-level domain.


Anecdotally, you can see that by searching naturally


for anything that interests you.


I venture a guess that the top results


don't have those keywords as a domain ending.


Often, it's not even in the URL at all.


That's by design.


Just because a website has a keyword in its domain name


doesn't mean that it's more relevant than others


for that keyword.


In short, you don't need to put keywords in the domain name.


From a practical point of view, it's


worth also keeping in mind that businesses evolve over time,


and moving domains is hard.


So it often makes sense to pick a domain name


that you can use for the long run,


and not necessarily one that just matches


what you're offering today.


For example, if you focus on making awesome blue widgets


and build your website on bestbluewidgets.com,


what do you do when you start offering red widgets?


So instead of spending too much time on the domain name


or top-level domain, focus instead


on building up a site that you can continue


to use for the long run.


I hope you found this video useful and insightful,


and I'll see you next time.


The past, present, and future of robots.txt

Welcome to a new episode of Search Off The Record where we have fun, sometimes talk about


search and get Gary in trouble.


I'm the aforementioned Gary.


I'm joined today by Lizzie, tech writer on the search team.


Hey, happy to be here.


And Martin, who is a search advocate on the search team, just like me.


Hi, everybody.


Today, we also have a special guest because John doesn't love us anymore.


So he decided to skip this episode.


But today we have David Price.


And I'm super excited about this.


Hello, Dave, why are you here?


And what are you doing at Google?


I am a product counsel, which is a lawyer who looks after legal compliance and legal


issues for one or a few of Google's products.


So we are legal generalists.


We look at any and all legal issues that may affect our products.


And we are product specialists.


So my team and I are sort of the legal experts for the search team.


And then kind of on the other side, we are the search experts for the rest of Google's


legal and other similar teams.


That is fascinating.


Side note, Dave is one of the more exciting people to work with at Google because his


advices are always interesting and also very challenging sometimes.


We work together for quite a few years.


And it's very, very fascinating to work with him.


On the robots.txt topic, I want to touch upon a few things.


Perhaps we can dive into its history.


How is it relating to Google?


But I'm also very, very, very interested in how or when did you first encounter robots.txt


and how did you get to know about it?


That's a really good question.


I have to admit, I don't know when I first heard about robots.txt because it's such an


old and well-established thing that if you spent time on the web in the 90s even, which


I did, which probably showing something about me, you heard about it.


It was sort of the thing that was that sort of hidden Easter egg in a bunch of websites


that they had this file that you could go to.


And sometimes it would have funny things that it would say to engineers.


But for the most part, it was there to sort of manage the flow of automated accesses to


the site.


And I remember being a sort of young student at the time looking at it and thinking, well,


this is really interesting.


Look at all these sites.


Look at the flow of information that's happening.


You have these cool things called search engines.


They seem interesting.


I wonder if you can make a career in those.


And so, yeah, it was one piece of a really interesting puzzle that I'm very privileged


all these years later to still get to work on.


That is fascinating.


Does that also mean that you were a webmaster by any chance before joining Google or?


Never in any kind of really serious way.


I think like everybody who kind of grew up on the web during a certain period of time,


I had a few web pages that I ran.


I kind of knew how to run a web server, but I never really operated a website as anything


more than sort of a very minor hobby.


But if you were operating a website, then basically you also dealt with robots and automated


crawlers.


Yeah.


I mean, I think it was interesting because I think at the time, even when I was getting


started, there was a lot of focus on third party hosting.


I remember, I think in high school when I was a teenager, my school newspaper put up


a website, which was sort of a brand new thing for school newspapers to be doing at the time.


And I remember setting it up.


I remember working with a friend of mine to get the hosting arranged.


I assume we had a robots.txt file, but I have no memory of this at all.


Interesting.


You haven't checked your robots.txt.


I haven't checked the robots.txt file of my school paper, no.


That's shameful.


I'm sorry.


I know.


I'm really falling down on the job.


Another interesting bit from what you said was you specifically said something like automated


clients.


In our robots.txt documentation, we usually use the word crawler, but that also leads


to the question, like what does constitute a crawler?


It's a really good question.


And it's one that I will get from time to time from folks who are working on different


efforts at Google.


And I think it's sort of something that people tend to look past because there's a lot of


things that are just so obviously crawlers.


And so obviously things that are meant to be captured by robots.txt that there is a


sort of whole line of interesting questions that you kind of back off from the notion


of having a fully automated system that runs a giant queue of URLs to crawl and runs 24


seven all the time.


Kind of at the other extreme of things that happen on the web.


When you open a web page in a browser, your browser is going to deploy some programmatic


automation to access a bunch of different resources that are linked from the page that


you loaded.


So you might load dozens of URLs as a result of one click on your web browser and nobody


thinks that's a robot.


And it's quite obvious that that's sort of the opposite of a robot.


It is a thing that is done at a user's request, sort of an obvious and logical outflow from


what the user has decided manually to do.


And so often when I get this question, I kind of try and think about that manual versus


automated question.


And is this something that is sort of aligned with something that a user is doing where


you have a person in the loop somewhere saying, yes, I want to initiate this thing.


And then you want to make sure that you're not saying it's a person who's clicking start


to the crawling program, but rather something where you expect to have kind of people in


the loop constantly using something and constantly saying, well, now I'm interested in this page


or that page.


Very much akin to a browser.


And something that's not that, something that sort of runs with full automation, that is


going to be a robot.


And if it operates at Google, then it definitely needs to obey robots.txt.


Yeah.


I'm trying to think of or crawler setups.


And for example, Googlebot, that would be the crawler.


That's a completely automated system that just goes out and we give it a list of URLs


and it will bring it back whatever it finds.


And then we have in Search Console tester tools, for example, where the webmaster can


log in and then they press a button and then still Googlebot or something that behaves


as Googlebot would go out and fetch the page that the webmaster requested and bring it


back from the webmaster site.


And that would probably be the kind of crawler that's more close to a browser.


I think that's absolutely right.


I think you can end up in a situation where, I mean, the thing you just described is something


where you probably want to obey robots.txt because you're helping a webmaster test,


you know, can Google crawl this page?


And if the answer is no, because it's roboted out, that's certainly something that the webmaster


will want to know.


But in principle, that's something where there's a human in the loop.


And so you can imagine other situations where you have some kind of testing tool where Google


would accept some kind of click from a webmaster or somebody and go out and fetch that content


that was requested and maybe not necessarily be limited by robots.txt in that situation


because there's a person at the start of that process pushing a button to say, yeah, I want


this thing.


So you can see two things that are kind of blurry on the line where I actually don't


know the answer to what if we should consider it a robot or not.


So for instance, there's like the social media bots.


So when you post, when the user posts a link to a page on social media, certain social


media platforms are then making an automated request to that URL, which is not really something


that I proactively as a user have requested.


But the way that things show up, use these, I think it's called Open Graph tags that are


fetched from these websites.


And I'm not sure if these, let's call them robots, are actually following robots.txt


directives or not, but I know that they call themselves bots.


And it's not really something that I as a user posting a link to a social network necessarily


intended to happen.


It just happens because that's how the specific social network works.


And then there's another thing that's actually a Google thing.


What if I, as someone who has an e-commerce shop, am opting in to say Google Shopping


so that like Google Merchant Center can make requests, but then these requests happen somewhat


outside of my control because I say, yeah, sure you can do that.


But I don't really request Google to do specific requests to my website.


And then the robots, so to speak, so Google's product side has to figure that one out.


And I think they've, do they follow robots.txt?


I guess they do.


I don't know.


Maybe not because I, as a website owner opted in already.


Is that a robot?


It's a good question.


I mean, this is all starting to sound like what we used to do in law school, where you


throw out the obvious example and then you say, aha now, but what if we change this?


And now I'm having flashbacks to my professors in school.


But yeah, to your first point, the sort of social media fetcher thing where you have


a user who is posting a link to a web resource and maybe that user isn't necessarily saying,


yes, I definitely instruct you to go get that thing for whatever use I'm putting it to.


But nevertheless, there's some really good reason to go get it right then and there.


Like maybe you need to produce a good pithy summary of that to put on your social media


post, or maybe you need to scan it for say malware or other kinds of abuse.


I think the way that would shake out is that that's something that's done once when a user


sort of hits a button.


The thing that I would probably say about that is that is not necessarily automated


enough to be sort of hard and fast.


You must check robots.txt and if it says no, then you can't do that fetch because what


you're doing is you're doing something at the result of an immediate user action.


And that's kind of the way I tend to think about these things is a user said, hey, I


want to sort of in some way engage with this piece of content on the web and it's kind


of natural that you would want to look at that content on the web and understand what


it is.


And so the social media case, I think is a good example where you might say that that


is not a robot in the classical sense.


Now maybe that then means that there should be some other level of control or some other


set of questions if you're a webmaster and you say, hey, I don't want this fetch to happen.


But the question of is it a robot?


I think the answer is probably not.


And then the other, sorry, Martin, you had another question and I've totally forgotten


it.


That was the case where for a product, I as the website owner opt in to some program to


consume content from my website, but I don't necessarily know the bits and pieces of like,


I don't have control over the requests made specifically, but in generally I agreed to


have a program make requests to my website.


Yeah, it's a good question and it's something where we see different things kind of land


in different ways.


But I think the kind of high order bit here is robots.txt is a sort of scaled way to communicate


permission to the web ecosystem at large to do crawling.


I don't think anything would stop a webmaster from going out and saying, hey, for this particular


specific integration that you and I are doing together, I am cool with you going and doing


an automated access to my site regardless of whatever my robots.txt says.


It might be something where best practice is to adapt your robots file in that situation


to account for whatever you've agreed to do so that you have less sort of out of band


permissions and denials that are not captured in the file.


But that is maybe more of an engineering detail than anything else.


But I guess it still kind of consists of a, it fits our description of a robot because


it still is an automated system and it is kind of outside of the control of a user or


the webmaster because it does its thing as it needs to.


So it's, I guess it's still a robot, the second case that I described, but it's not really


something that is falling on the robots.txt territory.


So is a robot and a crawler the same thing?


Because we've been talking about like client, crawler, and robot.


That's a really good question.


My assumption is yes, that robots and crawlers are basically two words for the same thing,


which is a program that acts automatically to access a bunch of web pages without human


intervention.


But I'm curious what y'all think about that.


I mean, we are using them interchangeably in public talks, which might be confusing


and we should settle with one.


And in the RFC draft, the internet draft that we are going to talk about later, we are just


using one word, I think, and that's crawler because that's more specific because with


a robot, you can also have, for example, an assistant robot that you specifically written


for assistant, be that Siri or Alexa or our assistant.


I forgot its name.


Google assistant.


A Google assistant.


Okay.


That's why I forgot its name.


And robot is just a more loaded term while crawler is very specific in my opinion at


least.


And that's why.


Is robot like an umbrella term that could mean more things and crawler is like a subset


of that?


I mean, if you think about robots, then probably Googlebot is not the first thing that comes


to your mind.


It's probably something that like, I don't know, like from the Hitchhiker guides to Galaxy


movie, the robot, the depressed robot walking around, that's more likely to come to your


mind than Googlebot.


Context.


On this podcast, maybe Googlebot is the one.


I'm not sure.


But like I work a lot on robots.txt and on Googlebot and yet when I hear robot, the first


thing that comes to my mind is an actual robot versus Googlebot.


And if I work with these things and it, I wouldn't say it confuses me, but crawlers


are not the first thing that comes to my mind, then probably it's bad term and we should


try not to use it.


It seems like also it could be just a phrasing thing because if we want to say a crawler


crawls, it feels like awkward and repetitive.


Whereas saying like a robot crawls is more natural.


So I guess if you wanted to say a crawler does something, would you use a different


verb like fetch or goes and finds or something else to just not sound repetitive when speaking


or writing?


So on the dev side specifically, and now we are in the writing topic already, I don't


like certain things that we use in the context of crawlers.


One is the crawling because it's a very descriptive term, which the crawler does not do.


The crawler is downloading things from the internet or fetching.


And I would like to see more specific words on the dev side because there are things that


Googlebot is not doing.


Also not following links, for example, because Googlebot cannot follow links.


It's just a fetch system.


So yeah.


Right.


It seems like we give Googlebot maybe too much agency that Googlebot is going off and


doing things when really it's automated systems or other things that are instructing Googlebot


to do something.


Yeah.


I mean, maybe the thing to keep in mind is it depends on what you mean by Googlebot.


Googlebot, I guess, is the thing that webmasters see downloading stuff from their sites.


And then inside Google, there's that system, the set of fetches that Gary mentioned.


But then also there's sort of the stuff that sits on top of that, that's orchestrating


the crawl, that's prioritizing what do we fetch today versus what do we fetch an hour


from now.


And you could think of all of that together as the crawler if you wanted to.


But I think the key point from kind of the question of what does robots.txt have to say


is like, why is this happening?


And is it happening because of automation or because of something else?


And that also touches upon the question of how robots.txt became important, because there


were robots, especially in the beginning of the internet, which didn't necessarily behave


well on the internet and they were going rampant and ravaging sites.


Perhaps that's also why robots.txt was born, basically giving people some sort of control


over these rampant crawlers.


Why was that a problem that the crawlers were doing whatever they wanted?


I think if you kind of think back to the mid 90s, when all of this was getting off the


ground, there was a lot more or just the state of the technology then was reasonably limited


bandwidth, reasonably limited server capacity.


And so you could imagine that somebody with a computer in their dorm room might be able


to impose a pretty significant load on a fairly well known and prominent set of web services


at the time.


You didn't have the kind of galactic scale web service systems that you have today.


And so I think the concern that one sort of ill behaved script running on somebody's box


under their desk could cause some serious damage to your ability to serve was, I think,


a real concern at the time.


And if that happened multiple times and it was scaled across multiple different robots


running all over the web, then I think you have, especially in that technological environment,


a pretty significant problem.


Is that still a problem today?


Yeah, I think it could well be a problem.


Just everybody's got higher scale now.


The serving systems have higher scale, but then again, so do the hobbyists, many of whom


are running their hobbyist systems on high end cloud services, maybe even the same ones


that the sites they're hitting are on.


Another aspect to the webmaster side is that in the beginning, in the 90s, bandwidth was


extremely, extremely expensive.


I remember the first site that I had that was probably in 96, 97, it was just like an


HTML document with one image on it.


And it was serving for a few days, probably not even a week.


And I managed to incur like $50 worth of bandwidth traffic, and that was my allowance probably


for like a year.


So I was not very happy with it.


And then when I was looking at the server logs, then it was back then a very popular


search engine that was scrolling that page repeatedly many times a minute for reasons


that I could just not comprehend because like it's a simple HTML page with no real information.


It was just an image, yet the crawler was hitting it.


I didn't know about robots.txt back then, but I would have loved to have something like


that before I actually got that bill from my hosting company.


I think it's a really good point for a couple of reasons.


One is you're kind of getting at this broader question of what is a well-behaved crawler.


And sort of a necessary but not sufficient condition for that is that it obeys robots.txt.


But you could imagine that if you had allowed that search engine's crawler to access you,


then it would do that.


Then it would do it again and again and again, and now you're paying $50.


It's something that I know the search team thinks a lot about is how do we ensure that


we are giving good value for the bandwidth that we are pulling.


We want to do a good job of going and getting data when it's likely to be useful, balancing


between wanting to keep an up-to-date vision of what the web looks like right now versus


not hammering everybody's website with $50 worth of traffic every day.


And so that is a question of good behavior that kind of goes beyond the question of are


you obeying robots.txt and goes into just more general standards of polite internet


usage.


But the nice thing about the robots.txt standard is that that gives you the ability to say,


I don't think this crawler is well-behaved anymore.


At least it's not well-behaved to me.


And so I'm going to say in a very lightweight way, this particular crawler should not access


me anymore.


So it provides a fair amount of control and autonomy for webmasters in that regard.


If you like internet archeology, then there are some very, very old mailing lists where


people are complaining about situations that we just described.


And the original author or brainfutter of robots.txt actually chimed in with a proposal


which was the robots.txt.


And this was way before Google, like years before Google, which is kind of fantastic


somehow because it just shows that it was just something that Google decided to adopt


to help webmasters versus us pushing something because we think this is going to be good.


Yeah, I think this is something that is important is sometimes I'll give talks inside Google


on sort of the legal foundations of a search engine.


And one of the things I tend to point out is that a lot of these concepts really do


have their foundation in the web of the mid to late 1990s, before Google was even a thing.


Things like what is a search engine?


What should a website expect with respect to the population of search engines that are


on the web?


There's a notion that dates back to the mid to late 90s that if you put a website


on the web, you should expect that search engines will go and access you if you have


not used robots.txt to tell them not to do that.


And that that feature of the open web is something that you should expect.


And that expectation in various ways has been looked at legally and seemed to have a legal


effect in various ways as well.


In what way or can you say?


It's a concept.


So the lawyer's name for the concept is implied consent.


So the idea here is you have a website, you have some stuff on your website that you own,


maybe you have a copyright in it.


And there are various things that other participants on the web are going to do with that content


that in one way or another triggers one of your exclusive rights under copyright.


So the most basic one is users are going to go download your webpage and associated resources,


and they're going to make copies of that on their computer.


It's going to sit in their browser cache and things like that.


And the idea is that you never sort of wrote down a formal legal contract to say, yes,


internet users so and so, I hereby license you to make the necessary copies of my web


pages to render it in your browser.


That's just inherent to the way the web works.


And it's so sort of obviously inherent to the way the web works that you haven't seen


a lot of litigation over it.


You haven't seen people say, well, wait a minute, all these web users are actually copyright


infringers because of course they're not.


They're operating in a way that the webmaster expected and should have expected and is sort


of going to be deemed to have given a level of consent to.


And I think the same is true with respect to the sort of basic aspects of search engines


doing search engine-y things with content that they crawl.


Okay.


This is fascinating.


I love it.


It might be a weird question, but how come robots.txt hasn't evolved in all these years?


Because if you think back, and I've seen those mailing lists or posts on those old mailing


lists and the basic structure of robots.txt was still, or it was the same thing that we


have today.


Yeah, but why would you change a winning team?


Like it works.


Don't fix what's not broken.


It works, doesn't it?


HTTP 0.9 also worked.


It was not broken, but it became HTTP 1.


Yeah, and now it's HTTP 2 and we're on the brink of HTTP 3.


And that's because there are changes to the environment that, or like the situation or


the systems, the networks, everything that the way that we use it has changed so much


that while it worked, it wasn't optimal.


What would you say is not optimal in robots.txt right now?


What is missing?


What is not great?


So you're saying that it operates at such a basic level that it doesn't need evolution?


No, I'm not saying it doesn't need evolution.


I'm just saying that the pressure of pain of using it is not great enough for something


else to replace it or improve upon it.


Let me put it that way.


The things it does, it seems to be doing well enough for people not to worry about it too


much and kind of forget about it.


So I don't know.


Is there something that you think is missing or that needs to be changed based on how the


internet has moved forward or evolved?


I honestly don't know.


I just find it weird.


Okay, tracking back, I would be the first person at Google who would push back on changes


to robots.txt.


And I imagine that Paul Harr, one of our quality leads, and Dave would be the second and third.


But I find it utterly weird that the protocol has been around for like 20 years or 20 something


years and it essentially hasn't changed.


Like on the web, on the scale of the web, that doesn't happen often.


So it's either like utterly good or I don't know.


I think one thing that kind of accounts for the success and longevity of the protocol


is how sort of high level it is.


It is very simple in what it says to do.


It says don't access these resources using a crawler.


It doesn't try to do much more than that.


And then one of the things that you can kind of get into if you start applying semantics


to guidance like that, that maybe fits with a particular product offering that you may


have.


So let's say you say, well, I'm launching this new feature and I want to offer a bit


maybe within robots.txt that says yes or no to some aspect of that feature.


One of the problems is that that might make sense with respect to version one of the feature,


but features tend to evolve and they tend to evolve in ways that you couldn't have predicted


at the time you laid down that initial set of controls.


And so one thing that we often sort of view with some caution is the question of, well,


if we offer this control today, when we come back six or seven years from now and we're


on version four or five of the same feature, does that control still make sense?


And part of the challenge there is that if you have declared a control years and years


ago, then you are stuck with the interpretational problem of what does this opt-out mean in


a world in which the technology has moved along so much.


My legal colleagues who think about the US constitution, it's kind of a question of,


are you an originalist?


What was the original meaning of this opt-out?


And you have the same struggle to figure out the right answer to that.


Has the original meaning of the robots.txt protocol changed?


If you ask the legal originalist, they would say that's the point is it can't change.


There was some notion of what robots.txt meant back in the day, which is, again, fairly simple.


It hasn't changed that much because to Martin's point, it hasn't needed to change very much.


Here's what counts as a robot, which I think has kind of been worked out by consensus.


Here's what you can tell a robot to do or not to do.


And what it does or doesn't do is download content at a particular URL.


And all of those things are so inherent to the way the web works.


The notion of there's a particular URL and there's stuff at that URL that you might download


that the same concepts work equally well today.


I think one area where it could have evolved but didn't was indexing directive in robots.txt.


And that was fairly interesting to me because even before joining Google, I knew that Google


supports very specific indexing directive and serving directives in its robots.txt parser.


But then when we started working on the parser, open sourcing the parser, then after consulting


with many people at Google, we decided that we don't want to publicize that support.


And in fact, we want to remove that support.


And that would have been an interesting evolution.


I don't think I could have...


I didn't want to support it myself because I guess I would be an originalist.


It would be like if the original author said that it's for controlling crawling, then it


should control crawling.


But then on the flip side, we index a very small number of URLs that are actually disallowed


for crawling.


It is like on the grand scheme, it's an insignificant number of URLs, but we still index the URLs,


not the content.


And some people want to have control over that as well.


Yeah, I think it's a really good point about one of the reasons why adding a no index directive


to robots.txt is exactly what you said.


It adds a new concept that you have to think through and interpret.


What's an index?


What does it mean to index?


What does it mean to not index?


What counts as an index?


And then as you said, one of the things that the setup that we currently have facilitates


is a situation where search engines can come to know about particular pages on the web


that are important, that the web says are important because there's a bunch of people


linking to them, say, but the content of those pages is roboted out.


And sometimes for good reason.


There's a site that I use a fair bit, which is the repository of all of the big court


judgments in the United Kingdom.


It's called, I think, bailey.org, B-A-I-L-I-I.


And they have chosen to robot out the text of the court judgments for reasons that make


sense and are reasonable to them.


They want to, as I understand it, they want to make sure that if say one of these judgments


has content in it that maybe shouldn't have been made public, that they can at least stop


well-behaved search engines from going and getting that content.


But sometimes these are really important documents.


They may be sort of really socially meaningful and significant court decisions and a bunch


of people are linking to them.


And so, how do you square that circle that there's this really important and widely linked


to content that a bunch of people are going to be looking for, but there's a directive


to us that says, don't download this from the webmaster.


Probably the least worst outcome is the one that we've arrived at, which is we're not


going to get the content from that page, but we're going to take our best guess at what


is there and if it's the right answer to a search query, show that to the user.


To me, this is also interesting because it's very hard to explain to people the distinction


between crawling and indexing when it comes to robots.txt.


And that always made me wonder whether we are doing a bad job explaining this or there's


just like this very basic level perception that crawling is indexing.


And I don't know, but in our docs, I think we got better at this.


We've been working with Lizzie on the robots docs for months, which is a topic on its own.


How do you push changes to our robots documentation?


Because I don't know if our listeners got the sense, but we are passionate about this


topic.


Plus, it's a relatively sensitive topic.


So pushing changes is not necessarily all that easy sometimes.


Yeah, and I think there's a lot of nuance too, even with particular word choice like


do not versus should not, like something shouldn't do something, but does that imply that it


could still or that you can do something versus flat out don't.


I think it's also something that maybe it's one of those things that looks different on


the inside.


So if you're working on the search team at Google, you kind of get a sense of how the,


obviously I wouldn't claim to be an expert on any of this, but my job does give me kind


of an overview about how all this works and how there is plumbing to go from, okay, we


got this from the web.


Okay, we feed it into our system that builds the index and then it gets sort of from there


pushed out to data centers and ready to be served to users.


And each of those things is sort of a really significant step, often one that has a pretty


significant team behind it, working on it.


And then if you're outside Google, what do you see?


You see Googlebot goes and gets some stuff from your site and then a little while later


you see that stuff showing up in search results and everything that happened in the middle


is a black box.


Yeah.


And that's what we are trying to make it more transparent, the way we can iterate on the


robots.txt documentation.


Like I remember a few months ago, we revamped our robots.txt documentation quite considerably,


but it was structural change.


Basically we just rearranged the existing paragraphs in a more logical way.


And for that, I think we haven't even checked in with you, Dave, but then if we wanted to


change some of those normative keywords, like should, must, whatever, then that would be


probably some of the things where we would run to you and ask for advice, whether we


can do this or not, because those can be quite important.


I think that was a big part of what my team and I do for the search team, kind of at large,


like almost everything that the search team does that is public facing and significant


and touches on some issue that might have a legal valence to it.


That's kind of what my team and I are there for is Gary or someone like him will come


to me and my team and say, Hey, I propose to do this.


And a lot of the time the job of product counsel is to say, you could probably meet your aims


by doing it in a slightly different way, or have you considered the risk of doing it in


the way you propose might be as follows.


That's sort of my day to day.


That's what's in my inbox all day long.


But I would definitely agree that sort of the things that we say to the web about what


we are doing with robots.txt is definitely one of the more sensitive and important pieces


of wording that we have out there that's really important to get as exactly right as we can.


We did change one of those.


That was part of the structural change was to try to make it more clear what that doc


was about.


And it's how we're interpreting robots.txt, that it's not just like a copy of the standard.


It's how Google is interpreting it and what that means.


But it was just a title change.


The doc, that's what it was before.


But maybe it wasn't obvious enough without the rephrasing at the beginning.


Yeah, the RFC that was the, or the RFC draft that was the other thing that took a long


time to get out.


And I remember that we worked with someone who was back then working on the Google bot


team plus Dave plus Lizzie.


Oh, and the original author as well.


And we were sending a copy in circles to see if we can get something like this out because


to, and the reason we wanted to do this was really just that it was so weird that after


like 20, 25 years, the robots.txt protocol was still just a de facto standard.


Like it was not standardized.


It was just like an agreed upon thing that some search engines followed in certain ways


and others did not.


And then webmasters had to make sense of all that mess.


Yeah, I totally agree with you that it's a little late.


It does feel like an RFC or RFC style standard for robots.txt is maybe a quarter century


later than you would have expected based on the way the web has tended to work.


It's not like people went around using HTTP for 20 years and said, hey, maybe we should


write down how this works.


But then we also went even further and then we decided that we also want to open source


our own parser so people see how we actually parse and match the rules in robots.txt files.


That was also very exciting.


And I don't know if we worked with you, but definitely Paul Haar was involved and he was


the one who approved an open source release.


Something that the search team doesn't do too often because, well, reasons.


But that was also picked up really nicely and people appreciated it.


And some people also contributed in several languages like Java and I think Ruby.


They just made new versions available in new languages.


And I think Martin is also working on something exciting on that front.


Yeah.


We're bringing it to Node.js as well.


Or actually to TypeScript more specifically.


Wait, Node.js.


That means that Node.js, Node.js.


Node.js.


Don't make fun of my accent.


How dare you?


Actually I'm wondering about something.


When I started implementing the robots.txt parser in Node.js, I noticed that the Java


version I think doesn't use the exact same test suite as the C++ version.


How do we make sure that all the different versions are behaving the same?


I think I basically take the codified C++ tests as the truth.


That's a fantastic question.


And it has a very interesting answer.


Last year we hosted two interns, the search open sourcing team, who were working explicitly


on that.


Basically figuring out how we can test parsers against each other.


Basically the C++ one would be the source of truth, because that's what we use in production


as well.


And then we match the output of the C++ parser against the new language version.


And then compare.


And that meant that we had to...


I mean, the whole thing has to be cross language interpretation or execution, which is a very


interesting thing to solve, to say the least.


But our interns delivered and they shipped it right before they left, which was great.


And then that's what we do nowadays.


For example, recently we had to fix a bug in our parser.


It affected a very small number of sites, but we still had to fix it.


And the way I tested it was actually matching the old C++ parser against the fixed version


and see what it caused.


Basically we have certain test cases hard coded in the tester, in the comparison tool,


and it just does that.


So for example, if you are going to launch a Node.js one, then you can probably make


use of that tool.


I can ping it to you once.


That is awesome.


Yes, because we're getting there.


Slowly but surely we're getting there.


We also had a discussion recently, a few weeks ago, where we were sitting with Martin in


front of a whiteboard and trying to reverse engineer the RobustDXD parser because it was


so low level optimization that we could just not wrap our heads around it for a couple


hours.


But that was really, really awesome.


And also exhausting, surprisingly.


It was exhausting, but it was also awesome.


And I just love that you can see the code hasn't just been written by a team of people


who really made sure that everything is readable because they had to communicate to each other.


It's probably mostly written by a very small number of people, and then other people have


built upon it incrementally or something, right?


Yeah.


I mean, so the whole RobustDXD parser that started with Matt Cutts, who left Google by


now, but he was one of the quality leads and web spam lead for the longest time.


Or some people know him as the Google guy because he started communicating more on behalf


of Google to webmasters or with webmasters.


So he came up with the idea of supporting RobustDXD and then Jeff Dean, for example,


if you don't know him, Google his name.


Like he was one of the first people who actually wrote the parser in no time at all.


And back then it was very basic.


I think this was in 98 or 99.


It was very basic, just a few lines, but then our index was also not that complicated.


And then we could tell people that, well, you can just support this RobustDXD protocol,


and then you can restrict our crawlers access to your site however you wish.


And since then, we've been building upon that because some time after that, we also had


the special operators coming in, like for example, the star and dollar sign and comments,


for example, which are a nightmare to parse, but we had to have that sitemap key that's


also relatively new, as in like 15 years old versus 25, I guess.


There were many people, you can still count it on your two hands how many.


There were not that many people at Google who touched the parser because it's such a


basic building block of Google's infrastructure and we don't make modifications to it.


So all you had to do was optimize the living hell out of it so it doesn't consume too much


resources and then just launch it and use it.


I'm very happy that we could open source it and I'm sure that at least some people appreciate


it too.


Yeah, I was peripherally involved in the launch of the open source parser.


I think I spoke to the interns who did a bunch of the implementation work and I thought they


did a really nice job.


And I was really enthusiastic about it because one of the things that I'm always focused


on in my role is where can we be transparent to the webmaster community in ways that make


sense and that are sort of in everybody's interests.


And this, I think, is a great example of being super clear to the point of shipping some


code and saying, hey, here's the code that parses robots.txt is a really useful thing


because you can kind of see this is what we're doing on the inside.


And yes, ultimately it's about the standard rather than a particular implementation of


the standard.


And Gary, as you mentioned, we found a bug and the result of that is a change to the


interpretation of a few sort of strangely configured robots.txt files.


But at a high level, you can kind of see for yourself if I apply this to my own robots.txt,


what does it do?


What is Google's view of my robots.txt?


And you can see that for yourself using the code, I assume that is the same or very similar


to what's actually live in production.


It is actually the same thing.


In production, we use the same thing.


It's ported inside and we are just using whatever we published.


It was also interesting because since I published or pushed that bug fix, I was very curious


how soon after Googlebot will pick it up, the change.


And it was actually just like a day or two and it was already in production, which was


very surprising to me because sometimes these things take a while.


If you had a chance to change something about robots.txt, then would there be anything that


you wanted to change?


Or to rephrase, what do you think the future of robots.txt is?


Would it stay the same or?


Well, that's a good question because I don't know that those are the same question.


What will the future be is maybe different from what would we have done or what should


we have done a quarter century ago.


I think the reality is that the standard is so well established that it's going to be


hard to make kind of root and branch changes to the way it's implemented.


I have to imagine that from an engineering standpoint, you might wish that it was deployed


using maybe a more modern format or something like that.


But the one that we have is pretty nice because it's fairly compact and human readable.


It doesn't have any bloat to it.


It's just like rules and values and that's it.


I really like it for that.


It's easy to learn and relatively easy to master.


But I mean, obviously there's edge cases where you can construct really complicated rule


conceptions and stuff.


But I think, as I said, it has worked so well and none of us at least is like, oh, we need


to change this and we need to do this differently.


But I don't think, I'm sure there's people out there who are like, this should be JSON


or YAML or whatever.


But as you say.


Oh, there are.


Of course there are.


The more recent thing that I've seen that people were trying to push for is to move


robots.txt in the.wellknown directory instead of leaving it in the root.


And we were like, well, no, because no, this has been there for like 25 years.


Why would we move it to well-known?


And the second one was exactly that, transforming it into a JSON object.


And again, why?


Did they have answers to the why or have a compelling reason?


I mean, why would you bring it up?


I guess there must be something.


So they are trying to standardize how websites communicate with robots, as in crawlers.


And the idea would be that everyone is supplying robots with in JSON objects, which is still


human readable as well.


But it just adds that one layer of complexity over the very simple rules that you can write


in a robots.txt that doesn't make sense.


We see people screw up robots.txt files every day.


Now spice that up with a little bit of JSON and you have even more people that screw up


their robots.txt.


So from our perspective, it just doesn't make sense to add more complexity when there's


no benefit to it.


Like everyone can just read, just find the text files.


So why make it a JSON object?


And that's for today.


Thank you everyone for listening.


Thank you everyone for coming to this podcast and sharing their opinion and knowledge about


robots.txt.


This was the last episode of this year and we will hopefully see you next year as well.


Well, or hear you next.


No, you are going to hear us next year.


Bye.


See you next year.


Happy holidays.


Uffiye de luege.


Essential JavaScript SEO tips - JavaScript SEO

If you are building a website or web app using JavaScript,


you should take a few basic steps


to make sure your content is discoverable via search.


Let's look at a few SEO techniques


to help users find your content.


All of your pages should have a descriptive and helpful title


that describes what the page is about in very short terms.


For example, on recipe pages, avoid


the text that you want to read, and avoid the text that you want


to read.


You also should provide a description


of what the page will contain specifically.


For example, what makes this recipe special,


and what are its main characteristics?


So people have something helping them identify the best page


to fulfill their intended goal.


Both of this can be done by adding a title and meta


tags in your markup.


You can then inspect and then search for double slash title


and double slash meta to find them.


If you do not see all of your content in the markup,


you are probably using JavaScript to render


your page in the browser.


This is called client-side rendering,


and is not a problem per se.


Rendering is the process of populating templates with data


from APIs or databases.


Either on the server side or on the client side.


When it does happen on the server,


crawlers, as well as your users, get all the content


as HTML markup immediately.


In single page apps, the server often


sends the templates and JavaScripts to the client.


And the JavaScript then fetches the data


from the back end, populating the templates


as the data arrives.


As explained in the first episode,


the indexing for JavaScript sites happens in two waves.


Content that requires JavaScript to be fetched


will only be indexed in the second wave,


which might take some time.


In later episodes, we will cover how to overcome this,


and often also improve the user experience and loading


performance by using techniques such as dynamic rendering,


hybrid rendering, or server-side rendering for single page apps.


Another important detail is to allow Googlebot


to crawl pages from your website by linking between your pages


properly.


Make sure to include useful link anchor text,


and use the HTML anchor tag with the destination URL of the link


in the href attribute.


Do not rely on other HTML elements, such as div or span,


or use JavaScript event handlers for this.


Not only crawlers will have trouble finding and following


these pseudo links, they also cause issues


with assistive technology.


Links are an essential feature of the web,


and help search engines and users find and understand


the relationship between pages.


If you are using JavaScript to enhance


the transition between individual pages,


use the history API with normal URLs


instead of the hash-based routing technique.


Using hashes, also called fragment identifiers,


to distinguish between different pages


is a hack that crawlers do ignore.


Using the JavaScript history API,


on the other hand, with normal URLs


provides a clean solution for the same purpose.


Remember to test your pages and server configuration


when using JavaScript to do the routing on the client side.


Googlebot will be visiting your pages individually,


so neither a service worker nor the JavaScript using history


API can be used to navigate between pages.


Test what a user would see by opening URLs


in a new incognito window.


The page should load with an HTTP 200 status code,


and all the expected content should be visible.


Using semantic HTML markup properly


helps users better understand your content,


as well as navigate it quicker.


Assistive technologies like screen readers and crawlers


also rely on the semantics of your content.


Use headings, sections, and paragraphs


to outline the structure of your content.


Using HTML image and video tags with captions and alt tags


to add visuals, you help crawlers


and assistive technology to find this content


and surface it to your users.


In contrast, if you use JavaScript to generate


your markup dynamically, make sure you aren't accidentally


blocking Googlebot in your initial markup.


As explained in the previous episode,


the first round of indexing does not execute JavaScript.


Having markup such as a noindex meta


tag in the initial payload can prevent


Googlebot from running the second stage with JavaScript.


Following these steps will help Googlebot understand


your content better and make your content more discoverable


in Google Search.


Hi, Googlebot.


Did you see the new webmasters video series?


No, I did not.


Oh, what?


You missed out on so much stuff.


Really?


Yes.


Oh, no.


How could we have prevented that?


Tips for hiring an SEO specialist | Search for Beginners Ep 9

SEO is an acronym for Search Engine Optimization,


or Search Engine Optimizer.


Hiring an SEO specialist has the potential


to improve your site and save time.


But you also risk damage to your site and reputation


if you hire someone who gives you bad advice


or recommends bad practices and bad shortcuts.


Here's a general set of recommendations to have in mind


as you go about hiring an SEO specialist.


First, conduct an interview


with your potential SEO consultant.


A good SEO specialist doesn't focus


only on search engine ranking, where you appear,


but rather on how they can help your business


through improving how you appear.


Good SEO consultants tend to ask questions like,


what makes your business, content, or service unique


and therefore valuable to customers?


What does your common customer look like?


How do they currently find your website?


How does your business make money?


And how can search help with that?


What other channels are you using?


Do you use offline advertising?


How about social networks?


Who are your competitors?


What do they do well online and potentially offline?


If the expert doesn't seem interested


in learning about your business from a holistic standpoint,


consider finding another expert.


It's difficult to optimize properly


without knowing about a business's goals,


their customers, and other existing marketing efforts.


Be wary of an expert who gives unrealistic guarantees


of ranking first on Google.


No one can guarantee that.


And often those who try to guarantee that


do so by engaging in practices


against Google's webmaster guidelines,


like creating unnatural linking and doorway sites.


This could result in your pages or website


being permanently removed from Google's results


if there are particularly strong violations.


The second step in hiring an SEO is to check references.


You want to check with past clients


and confirm the SEO expert was able to provide


useful guidance and worked effectively with their teams.


You want to ensure their results were sustained


rather than temporary.


A good expert will help SEO become a part of your


general business operations.


They should be focused on long-term plans


and helping your brand.


Step three is to request a technical and search audit.


If you trust your SEO candidate,


give them restricted view,


not full or right access to your Google search console data


or analytics data.


Before they actually modify anything on your website,


have them conduct a technical and search audit


to give you a prioritized list of what they think


should be improved for SEO.


In the audit, the SEO expert should prioritize issues


and suggested improvements.


These suggestions should be based on data about your site,


should apply well to your online presence,


and should avoid unnatural practices


that may go against Google's webmaster guidelines.


The suggestions you should look for


will ideally focus on techniques and strategies


that target a human audience


as opposed to a search engine.


The audit also should estimate the overall investment


and the positive business impacts.


After interviews, reference checks, and technical audits,


you should be ready to evaluate your potential SEO experts.


A good SEO expert will prioritize ideas


that will improve your business


using the least amount of resources.


They will also suggest improvements


that may take more time in the beginning


but will promote growth in the long term.


They should feel like someone you can work


and experiment with, learn from,


and help forge a path ahead for you and your business.


Once hired, work with them to agree upon goals, metrics,


and how to track results beforehand


so you know what you are getting out of this.


Thank you for watching,


and check out our resources at g.co. webmasters.


Don't forget to subscribe to the Google Webmasters channel


and tap the little bell to receive notifications


so you will not miss any future videos.


JavaScript SEO office hours April 29th, 2020

Hello, and welcome to another JavaScript SEO Office Hours


Hangout.


I'm pretty happy to see that there's a few people here.


I'm trying to do these Hangouts now weekly


with an alternating time zone, basically


like an APAC-friendly time zone and a more North America-friendly


time zone.


North and South America, actually.


Doesn't matter.


Point being, I'm really happy to see a few people in the Hangout


today as well.


And we have a few questions.


We have very few questions.


So I think I'll start with the submitted questions,


and then we'll take questions from those who


are with me in the Hangout.


So the first question that we got


is the question that I have is JavaScript single-page apps.


I think that's what they mean.


Do they require hardcoded links?


What does that mean, hardcoded links?


I mean, OK.


So we built a 17,000-page e-commerce website,


and Google cannot crawl the pages.


They will not index them.


The thing here is, as far as I can tell,


I'm not sure what you mean by web developers need to know.


Web developers do know because it's in our documentation.


And as long as you generate links that are proper links,


and I mean a tags with an href that has a URL that we can


crawl, your discovery will not be impacted.


Now, again, links are important for discovery of content


and allowing us to understand the structure of your site.


With 17,000 pages, as far as we're concerned,


that's not a huge site.


So that should not be a problem.


You can also use the site map to submit the URLs to us.


But that's an optional additional measure


of making sure that we can discover pages.


Site maps like the site structure, so having links


is very, very important to allow us to understand your site


structure.


But I would reiterate, if you use JavaScript


to generate the links, if you use client-side rendering


and that generates all your HTML, that's not a problem.


As long as it is proper links.


And there's actually, on this channel,


later today, which is April 29th, I believe,


and 28th, 28th, 28th, 29th, 29th, April 29th,


we have this additional video, the Webmaster Conference


Lightning Talks on JavaScript and links.


Very, very basically said, as long


as you generate proper HTML links,


we'll be able to crawl them.


The HTML links have to have crawlable URLs,


and there should not be any other technical issues that


prevent us from indexing or rendering the page.


You can test that with the URL inspection tool.


You can test that with the mobile-friendly test.


You can test that with the rich results test.


Wherever you see the rendered HTML,


if your links are in the rendered HTML,


then you'll see the links.


And then we can crawl and potentially index


websites that are linked.


And yeah, I mean, you have to make sure


that there's no other technical problems,


like accidental no index or robots.txt


blocking off certain URLs.


And if that's not the case, and if you're not preventing us


from crawling, rendering, and indexing,


then everything will be fine.


So that's that.


If you want to elaborate on the question,


that's fantastic as well.


You can also send that in with more details in the next.


There's a post on youtube.com slash googlewebmasters


slash community for the next JavaScript SEO Office Hours,


if you want to elaborate on that question.


Second question we got is, how to do a client-side 301


redirect?


I have a static website, and I want to do a client-side 301


redirect.


How to do this in a way that it does not affect SEO?


A redirect always affects SEO.


Because what it does is it tells us, especially


if it's a 301 or a 302, this website no longer,


this page no longer has content.


Instead, you should go there, and then it just


takes us somewhere else.


So there's always an effect in terms of SEO,


because that means that the page that we originally indexed


is no longer having content.


It is now just taking us somewhere else.


A 301 redirect means an HTTP status 301 permanent redirect.


You can't do that client-side, because an HTTP status


is sent from the server.


What you can do client-side, though,


is, especially if you have a static website


and you don't have access or control over your web server,


optionally, if you have access to your web server,


configure your web server for a specific URL in HT access files


or in whatever you're using to configure your web server.


You select, I want a 301 redirect to this other URL


when this URL is being requested.


That's how 301 redirects work.


If you can't do that and you do only


have access to the client-side of things, meaning JavaScript,


then you can use window.location.href


and then give it the new URL.


And then that's also a redirect.


That's not a 301 redirect.


It is a JavaScript redirect.


That also works.


It has no obvious downsides.


It doesn't really matter to us.


But it requires that the crawlers that you care for


understand JavaScript.


If you have a crawler that does not understand JavaScript,


then that won't work.


You can still use a meta refresh tag in your HTML.


That would also probably work for search engines or crawlers


that don't understand JavaScript.


But for Google Search, it doesn't really matter.


You can use a JavaScript redirect.


You can also use a server-side redirect


if you want to have a 301 redirect.


But a 301 client-side redirect is, per definition, impossible


because the 301 is the HTTP status


code that comes from the server.


So if you're not changing your server,


you can't really have a 301 redirect on the client-side.


That is that.


And that were the two questions that


were submitted on YouTube.


As far as I can tell, we don't have more questions


submitted via YouTube.


So any questions from the audience?


Now is your time.


Hey, Martin.


Can you hear me?


So I'm using a framework that does bundling.


So you've got one big JavaScript file.


And the core framework part of it is 1 meg.


The bundle is 1.7 meg.


And what I'm seeing is it seems to be


hit and miss if Google can actually render that page.


It seems to go, that's just too big.


Don't want to bother with that anymore.


Sometimes it works.


Sometimes it doesn't.


And this company, who do the framework,


are working on a way to shrink it down.


That is amazing.


So are there any other ways to somehow try and get around it?


Maybe not bundling.


So it's lots of smaller files.


I wouldn't do that.


So the thing is, I'm guessing when you say,


sometimes it works, sometimes it doesn't,


you're seeing that in the testing tools, right?


Yeah, so I'm seeing where you get the error thing going,


fail to load file, fail to load file.


Let me guess.


It says other error.


Is that the reason for it, other error?


Yeah, I think so.


So here's the good news and the bad news.


The good news is you don't have to worry too much about that,


because that comes from the fact that the testing tools are


a lot less patient than the actual indexing and rendering


infrastructure, because we can't really use a cache.


We're usually avoiding caching in the testing tools,


because we want to actually give you


the result for the latest version


that you actually have on your server.


That comes with the downside of, as the testing tools use


the actual infrastructure that Googlebot would also use,


that we might get scheduled a little later.


And that is actually relatively independent of the file size


as well.


We might just be scheduled for fetching later,


and then the tools say, we can't wait an hour or half of an hour


or 20 minutes or 10 minutes.


We need the result now.


So then they basically just time out


and give you this unfortunate other error.


That's not great.


We are aware of that.


We are working on solutions, or we


are trying to figure out what we can do to alleviate


the issue a little bit.


But generally speaking, the indexing infrastructure


is a lot more patient.


That being said, so while you shouldn't exactly


get worried about it, it is annoying at testing,


but it wouldn't affect the indexing and rendering


performance or anything of that.


That being said, 1.0 megabyte and 1.7 megabyte,


that's 2.7 megabyte.


That doesn't fit on a floppy disk.


That is obviously, that's always how I think about these things,


is like, look, if your thing requires


me to have 14 megabytes transferred over the wire,


that means that you are literally shipping 10 floppy


disks to me.


I played entire and fantastic games on 10 floppy disks,


so why is your website that large?


But besides that, the bundle size


is definitely not fantastic for user experience,


because you are downloading a lot of stuff


that you probably might not need.


And that's something that, as you said,


your framework provider can help you with.


There are ways of doing things, depending


a little bit on what the framework is


and what the tooling looks like.


You might have options to do what's called tree shaking.


So if you are in charge of the application code,


you can usually use tools like Webpack or Rollup


to actually analyze what of the dependencies from the framework


is actually being used and removing that from the bundle,


so that everything that you don't use in the framework,


on the application code, you would not have to ship,


which is super helpful.


But that really depends on the framework and the tooling setup


that you can get to and how much control


you have over the application code that you are using.


So tree shaking is a great way of reducing bundle size.


I would not unbundle for the simple reason


that then you have a lot of network round trips,


and the browser has a maximum number


of simultaneous network connections


that it opens to a single host.


So let's say you have it all on a CDN, maybe,


cdn.example.com slash main.js, app.js, bundle.js, framework


or framework part one, framework part two, framework part three,


then they would have to be staggered, because that's too


many.


At some point, it's too many requests


to do at the same time, so that would also slow things down.


So actually, bundling is a good idea.


But that being said, there is also,


most frameworks have an option to do


what's called code splitting.


I know that sounds paradox now, because on one hand,


you want to have a bundle so that you don't have


too many requests separately.


But then again, you don't want to have one gigantic bundle,


because then you are basically downloading


the entire application code for every single page


that you're getting.


Even though you can cache that, it's still a lot of code


to download for the initial visit, for the first visit.


So what people tend to do is they


tend to split the bundle a little bit,


like, oh, we have, I don't know, some route.


Our home page needs, or our landing page


needs this much code, and then there's all the other code.


So we're like, we're splitting this out


and downloading just this bundle when you have your first visit.


We download the framework bundle,


and then the framework bundle gets cached.


We need to have that on any page, every page anyway.


So we are getting this one cached,


and then we have the bundle only for the home page.


And in the background, we are loading the other bits


and pieces.


Or if you quit in between and jump to the next thing,


then we download the next bundle.


And then basically, you are using the cache the most


optimal way, and you're breaking it down


into reasonable bundles.


But completely unbundling is not the greatest thing


to do because of the amount of HTTP requests that you have.


HTTP2, or H2, makes that a little better


because it can actually multiplex


over the same connection.


But A, Googlebot does not support HTTP2 yet.


And B, not everyone has a HTTP2 compatible browser version,


probably.


Or there might be other issues, especially


on high latency connections, which are basically


unpacked loss-affected connections,


which are mobile connections.


HTTP2 can also be a little bit of a performance bottleneck.


So try to figure out if your application code


and or framework provider support tree shaking.


That's the biggest factor that will reduce the size.


And maybe also code splitting along the routes.


If you search for tree shaking and code splitting,


you'll find a bunch of articles on how other frameworks do


that.


And I'm pretty sure if you have a commercial provider


behind the framework that you are using,


then they should be interested in actually getting


that shipped as well.


Yeah, they are aware of it.


I think they pretty much said tree shaking's out


the question with their design.


Oh, OK.


So yeah, they're looking at the idea of code splitting.


It's a very GUI type.


It's like grids and charts.


So there's a lot of stuff in there.


So I think they're going to give you checkboxes.


Do you want charts?


Do you want this?


Do you want that?


Yeah, OK.


So you're going to think of it that way.


Yeah.


That's the approach that jQuery took, I think.


Yeah, it's actually got jQuery under the hood as well.


Just connected to it.


In fact, it uses, you've got a choice of something


like five different frameworks that it sits on top of.


Hence, this massive thing.


Another symptom I saw was, so I'm actually


creating structured data using the package.


What I'm seeing in the search console


is the structured data appears and then disappears


and comes back and goes in the enhancement section,


implying that this is the intermittent thing that I'm


thinking of.


I'm just getting around to it.


Sometimes it's not.


Debug, because it could be.


That is hard to debug.


That's an interesting point.


I'm not exactly sure where and when


we are extracting the information


for the enhancements report.


This could be potentially that we are putting it or pulling it


from the wrong spot in the pipeline.


A short-term work around while your framework provider figures


out how to get that solved is potentially


considering dynamic rendering.


That's an option.


Because then you would work around that JavaScript bundle


issue.


But I'm not sure if that's a good idea,


because dynamic rendering, A, it's a work around.


B, it's a non-significant investment, I would say.


And C, it is additional complexity.


So you have to think very carefully if that's a problem.


I would say, as long as you're not seeing general issues


with impressions or performance in search,


then I would probably not change much,


because then additional complexity is just risky.


You're importing risk for certainty that


does not have a visible effect.


If you are seeing issues with the performance in search,


then maybe it's worthwhile thinking about it


while your framework provider figures out


how to reduce the bundle size.


But generally speaking, the indexing pipeline


is quite patient with situations like this.


And 1.0, sorry, 2.7 megabyte is not a size


where I would start to cry and scream.


If you would have said 10 megabytes, I'd be like, well!


Now we're definitely in hot water.


But 2.7 megabytes shouldn't be a problem on our side,


except for the testing tools.


Yeah, and I'm trying to do as much content, so


on stuff server side, so that it's not


reliant on the JavaScript that builds all the fancy tools.


The content comes straight out.


Hopefully, there's richer data at some point.


That's also always an option to say, OK, we


are trying to get everything that is content and landing


pages.


We get that as static as possible.


And we only load the framework for the ones,


for the pages that are really dynamic.


That's a solution that a bunch of people


have successfully implemented as well.


Awesome.


You're welcome.


Next question.


SPEAKER 2 Hi, I had a question.


I just checked last week.


And even though I think you said that the newer Evergreen


Googlebot is working like 100% since the beginnings


of this year, I checked my access logs.


And 80% of the Googlebot requests I get still


show the user agent with Chrome 41.


Is that something to be expected?


MARTIN SPLITTENBERG No, that's not something to be expected.


I would check if the IPs come from Google.


Because as far as I'm aware, we are not using the 41 string.


We do have different user agent strings, that's for sure.


But the Chrome 41 string is unusual.


I would not expect any service from our side


to still use that one since April 2019.


So that's like a year ago.


If you're seeing that, check if the IP range resolves.


If you do a reverse DNS lookup, it


should resolve into a Google namespace.


If it doesn't, then it's a fake Googlebot that happens.


Some people are like, or some services


are pretending to be Googlebot, but they're not Googlebot.


Then they haven't updated their user agent yet.


If you are seeing a Chrome 41 Googlebot coming from Google,


then it would be some service.


But I'm not aware of any service that would


be using that string anymore.


I know that we are still having some services,


like the read aloud, for instance,


that uses a completely different user agent.


But basically, I know that we can still crawl without saying


the Chrome version.


There's a few Googlebot strings that we have on our website.


And one does not have the version information.


And that still happens.


But the 41, that would surprise me


if we were still using that one.


Yeah, that surprised me, too, because we have a million-sized


website, but we have around a million requests,


or even more, up to 10 million from Googlebot every day.


So I checked, and it was like at least 3 quarters


were using this old version.


I don't remember if it was 41 or 42,


but it was not 70 or 80-something


that's the current version.


I will check again and let you know.


Yeah, I mean, fundamentally, just


make sure that it is an actual Google IP address.


If it does come from a Google address,


it wouldn't be the regular search bot.


Then it would be something strange.


And I would have to figure out what exactly that would be.


But I don't think that we are using this still


for anything in search, not that I'm aware of.


I think there's a few services that


used to use that string and weren't actually rendering.


And then it doesn't really matter if it's not rendering.


It doesn't truly matter.


But I'm not aware of anyone still using this.


I would have to research this a little bit.


But generally, you can rely on Googlebot to be able to agree.


OK.


OK, thank you.


For search, for search at least.


I don't want to comment on other Google products.


Yeah, I checked because we were adding our first client-side


render page.


And I wanted to make sure that Google could actually render


everything.


So yeah.


MARTIN SPLITT


They might still use the old string.


That's possible.


Thank you, not Google search.


You're welcome.


Yeah.


All right.


Oh, that comes through the chat.


Hi, Maxine.


Infinite scroll in Google-friendly ways.


Yes.


Oh, you have a link for me to check.


That's nice.


Ta-da.


So let's have a look.


I'm not looking at the screenshot.


I'm basically trying to figure out


how the page is supposed to look.


OK, let me actually open the website first and then


see if I can find something that is content that would be loaded


by infinity scroll or infinite scroll.


Actually, you know what?


I can probably share my screen real quick


to show you what I'm doing.


Because it's probably a little boring to just hear me go,


like, blah, blah, blah.


I want a specific Chrome tab.


And I want, actually, no.


If I do a specific Chrome tab, then you're


not seeing when I switch tabs.


That's also not great.


OK, in that case, I will share with you.


I'll share just Chrome.


A window.


It says a window.


That's good.


I think I want this window to be shared.


And now you should probably see yourself, which is odd.


But here we go.


OK, so here we have the website.


And we can see that it's quite long already.


And it does seem to be loading additional content as I scroll.


Let's see.


I tried to grab something from as far down here as possible.


OK, now that definitely loaded additional content.


So are we seeing NIU?


I'm not sure what that means.


My Russian is not very good.


But I'll try my best.


OK, now we go to the mobile friendly test.


In the rendered HTML, I'm looking for these characters.


And I'm not sure if that jump meant that it actually


saw something or not.


So we are not seeing this part here.


That's for sure.


Are we seeing something else?


Let's try this one.


Nope.


Wrong window.


Actually, wrong tab.


Yeah.


So we can definitely see up to here.


And it looked like we couldn't see the other string


that I was trying.


So you get a feeling for what we are having in the content


and what we are not having in the content.


If you are exposing this to other URLs,


then that's not a problem.


That's generally not an issue.


And it looks like some part of the lazy or infinite score


and lazy loading did work.


And some of it might not have worked.


You can also check here if you are using JavaScript


to load additional content.


Then you would see, well, this is all not relevant.


You would also see if we are doing the actual API call


to fetch the additional content.


I'm not sure that all of this looks like it's not


grabbing additional content.


But maybe I'm just missing something here.


So you want to make sure that we are loading all the content


and that the content is actually visible in the rendered HTML.


If it is in the rendered HTML, that's fine.


Having multiple H1s is not an indication for a problem.


And it's not an issue on our end either.


So that's a question of how you structure your content.


If I'm searching for H1 headlines,


then we see we have 1, 2, 3, 4, 5, 6, 7, 8, 9, 10-ish.


Oh, actually, no, hold on.


No, no, no, no.


We are jumping in between the things.


So we have two.


These are two independent headlines.


Saint Laurent and Cog Industry something, something.


So how the industry something, something.


So those are the two headlines that we definitely see.


Everything else is if you are looking


for more than these two H1 tags, then we're not seeing them.


Oops, no.


OK, I'm searching for a random Cyrillic phrase.


Let's try this again.


If I'm going in here, how many H1 tags do I see?


Say inspect.


Come on, inspection tool.


Come with me.


Be nice.


So if I'm now asking for all the H1 tags, we also get two.


So as far as I can tell, that seems to be fine.


There's two H1 tags.


We're seeing them in the browser.


We're seeing them in Google.


So we are seeing your H1 tags.


And again, having multiple H1 tags


is not a sign for a problem.


That's acceptable as far as we can tell.


OK, now stop presenting.


So you would have to check if everything from the content


that you care for is actually in the rendered HTML.


If it is, you have implemented this successfully.


You can use the IntersectionObserver


to implement infinite score and lazy loading.


You can make sure that you have endpoints where


your server creates different pages so that you give us


a pagination as well.


That's a way of implementing this.


You can submit different pages and sitemaps.


That also gives us a hint that, oh, there's


more content that we might not have seen on the home page.


So there's definitely ways of doing this.


And the fact that there's two H1 tags


does not mean that there's a problem per se.


All right.


Hope that answers the question, Maxime.


More questions?


What's the double dolly it did in the console line there?


Was that it?


Oh, that's a helper.


Yeah, so Chrome DevTools, it doesn't always work.


It stops working when the website actually


has jQuery installed or another library that overwrites this.


But by default, if you go to, I don't know,


any static HTML page, if you go to example.com,


then the DevTools give you a bunch of helpers,


which is super nice.


Actually, I'll show you real quick again.


I want to share this window.


So if I were to go to example.com,


let's say, because that's a really nice and simple website,


and I go to Inspect, then a few things


happen in the console automatically.


I have $, which is short for Document Query Selector.


It's just a lot to type.


So whenever I do a quick debug, I just try this first,


and then I can ask for H1 elements.


And then you see, as I do that, in the page,


I get the overview of what the page looks


like, which is quite cool.


Oops, that's not what I wanted.


You can also select an element here.


And then go to the console, and then you


can type $0, which is the element that you selected


in the Elements tab, which is also quite handy.


Because then you can do all sorts of things,


like text, content, hello.


So you can very quickly debug with these helpers in the


developer tools.


And this is just the shorthand for Document Query Selector


All, which also is a lot to type.


So this gives you all elements that match a certain selector.


So in this case, I think we have two paragraphs, for instance.


And then you can inspect what kind of paragraphs


you actually have here.


Oops.


Ah.


And then work with that.


So what I did is I basically just asked Document.Query


Selector All H1, so to see how many H1 elements


and which H1 elements we actually have on the page,


which is a cool way of quickly debugging that.


If you're interested, if you search for, I think,


it's called DevTools Tips or something like that.


Dev Tips, that's what it's called.


There's a GDE called Umar Hansa, who


has an entire course on developer tools, tips,


and tricks.


There's also other articles that explain all these helpers that


developer tools in Chrome provide you with.


It's quite nice.


You have a bunch of stuff that you can do in DevTools


if you ever need to debug things.


OK.


More questions.


It does.


Yeah, Maxime, it looks like two articles get rendered at once.


But it matches what the browser does.


When I opened it in the browser, I


got the same amount of H1 tags.


So to me, it looks like Googlebot


does exactly what the browser does when it comes to that.


Maybe we don't render as many articles,


because there might be something in the infinite scroll


implementation where we don't really


follow through with all the articles


that you get when you scroll in the actual browser.


Because Googlebot doesn't scroll.


But we definitely load a bunch of content.


So as far as I can tell, the content seemed to be there.


And the fact that you render two articles


is not a problem for Google.


Especially if these articles have links to their own page,


then we would probably consider their individual pages as well


and might rank that in search rather than the home page.


So that's not, per se, an issue.


I mean, for instance, e-commerce shops


might have an entire category page with lots of products.


That's not a problem.


You just don't have products.


You have articles.


That, per se, is not an issue.


OK, more questions?


Yeah.


So if nobody asks, I have a little question.


So we have the problem again with some little things.


So the Google search console said to us,


one such page is duplicate content but with another URL,


like another one.


But both page get separated content.


But this is loading by JavaScript Ajax.


So I don't know what we do wrong or is the problem


with the page is too slow and the Googlebot reads later.


It's not with all pages.


It's with some pages.


Have you tried rendering them through one of our testing


tools and looked at the rendered HTML?


Yes.


And they look completely different in the testing tools.


So yes, I can send you links if you like.


That would be fantastic to have a look,


because this sounds like a.


So this is the tool list?


So if we start on fuel machine.


OK.


So yeah.


So if I have a look at this, OK.


And let me go to mobile friendly text, for instance.


So I'm guessing the other search pages don't have that problem.


It's not that everything.


Not everything.


We have, let me not lie, over 500 pages and only 20 pages


have the problem also.


But with the same, with stop as canonical.


Right.


So what happened is somehow, and I don't know exactly how,


somehow we have seen similar content that's


put it that way and decided that it's like one thing is


a duplication of the other thing.


And then we have selected, I think, the stop one,


if you said that that's the canonical now.


And we have decided that that's the best match for this.


That can happen, especially because you're using JavaScript.


That is possible that there was a problem with fetching


the JavaScript at some point.


And that's why we decided, imagine that for some reason


we couldn't fetch the JavaScript,


we couldn't load the results, whatever,


or there were empty results for a while.


And then we're like, OK.


So this is an empty search results page.


This is an empty search results page.


And this is an empty search results page.


They're all the same.


So we're just picking one to keep and the other ones


we didn't begin.


But if that was the case, if there


was a glitch that led to that, eventually we


would probably recover.


You can also resubmit them for indexing.


And then we might revise the discussion.


Actually, we have John in the call.


I've seen you, John.


You did not escape my attention.


Is there anything else you would have to do to fix this D-Dup


issue, if it's an issue?


I didn't pay attention.


Sorry.


So sorry, I didn't want to call you out on this one.


I was like, this is a common problem people have.


So maybe John has something that I forgot about to mention.


Assuming that you have multiple pages loading search results,


not search results.


You have a catalog of different categories.


And for different search terms in the category,


sorry, for different categories, basically, in the catalog,


you're using JavaScript to load the products.


And we have decided that they are all


duplicates of each other.


And we have selected one as the Dup cluster lead.


So I think that sounds like we had an issue rendering


at some point, probably.


Maybe we couldn't fetch the JavaScript or something.


And now it works.


And now it gives you the different content.


So I guess, eventually, we would probably figure that one out.


Or you can request indexing to fix that, right?


Yeah.


Relatively small number.


I guess it depends on where the issue came from.


So what I've sometimes seen is that we


learn that particular URL parameters are irrelevant.


And if we learn something like that,


then it tends to take quite a long time for that


to be unlearned again.


So if you think that that's the case,


if you have multiple URLs that all use the same parameter


names, and you're noticing that different parameter values all


lead to the same canonical, then one thing you could do


is to kind of work around Google and change the parameter name.


So if you have something like, I don't know, Q equals,


just take like QU equals or S equals or some other letter.


And then, essentially, we'll look at that and see,


oh, it's a new type of URL.


We don't know about this parameter.


And we will need to analyze to see,


is there actually unique content per parameter value here?


And if we find that there is, then we


will index it normally.


Whereas if you keep that parameter


and we've kind of learned that the parameter value is


irrelevant, then it's going to take a really long time for us


to reconsider that.


OK.


And I noticed that you have a language parameter in the URL.


And I'm guessing the language parameter doesn't


make that much of a difference.


So maybe that's where we fell over.


That's a possibility.


We give Google the link without language parameter.


So Google get only text search question mark Q equals


stop, for example.


And I have copied this from my browser.


And so I'm setting to German so you get the language parameter.


But I have an extension question to this.


So we want to change something on our server


and add renderTron.


If we add renderTron and some of the URLs get recalled,


do we have the problem again?


Or it's not so important more if we add to the renderTron server


to redirect Google to renderTron rendered version?


I hope you know what I mean.


MARTIN SPLITTENBERG


And I say that as the renderTron maintainer.


So I would not do it if I don't really


have a really good reason to do that.


And I don't think that this is the reason why


you add renderTron.


I wouldn't.


Then I would do what John just said


and probably reconsider maybe renaming


the parameters or something.


But assuming that you set up renderTron correctly


and it doesn't fail and it generates static HTML


for Googlebot, then that won't incur the problem again.


It might incur different problems


depending on what the rendered output from renderTron


looks like.


But that specific problem that we can't load your JavaScript


would not happen again because renderTron strips


the JavaScript from the page.


So the main reason we add renderTron


is for social sharing.


Yeah, OK.


That's the point.


Yeah, I understand that.


I 100% understand that.


I've been in that position.


So if we use it for social bots, so we


can use it for Googlebot and maybe we have no problem.


Sure.


That's a possibility.


Just make sure to make renderTron use a cache


because renderTron makes things a lot slower if you are not


using caching.


OK.


You want to cache every now and then.


Cool.


Perfect.


Nice.


Thank you.


You're welcome.


Any other questions?


I think that's really weird.


I saw a question being submitted on YouTube


and now it's gone again.


Or at least I'm not seeing it anymore.


And I'm like, ah, here.


So someone submitted something to YouTube.


Hi, I have noticed a significant drop in the FAQ listings


in the two past months.


Here's a screenshot.


That's why I shouldn't just read things out


as I go along because that's not a JavaScript question.


That's a question for the regular office hours.


But basically saying, so they added FAQ markup to their page.


And they're seeing a significant drop in the FAQ listings


in the past two months.


The fact that you add the markup does not


guarantee that you get rich results in search.


So it can be that another page has better content


for these specific questions.


It might be that our algorithms have decided to just show less


FAQ markups for the queries that you have markup for.


It doesn't affect rankings.


It does affect visibility in the sense of you


don't get the rich results snippets


if you're not showing up with rich results snippets.


Why am I saying rich results snippets?


Which results?


And also the screenshot that you shared,


it doesn't look like a super unusual large drop.


It's just dropping.


It went up a little bit.


It went down a little bit.


That seems to me to be the regular change in algorithms


over time.


I wouldn't worry too much about that.


OK, now coming back to JavaScript questions, anyone?


Another 12 minutes.


SPEAKER 1 Could you explain a bit more about how


RenderTron works?


Is it a wrapper around Puppeteer type thing?


MARTIN SPLITTENBERG Yes.