But it's still worthwhile looking at these things.


English Google Webmaster Central office-hours from August 18, 2020

Yeah.


Sometimes they're weird quirks.


John, can I ask another one on internal linking really quick?


OK.


So assuming, let's say, you've changed something


like the CMS of your website or something like that,


and you have to use the 301 redirects,


and maybe because of some issues like the design or anything


like that, you kind of have to link using those 301 redirects.


You cannot link to the final version.


Your navigation and most of your internal linking


still goes to the old URL, which then 301 redirects the new one.


For users, you kind of add that extra step of a 301 redirect.


Does that affect Google in any way?


I know that Google kind of can pass over it and doesn't.


So.


Yeah.


I mean, it's not great.


But we work around that.


So essentially what happens there


is we try to understand what the canonical is.


So we'll follow that redirect.


And usually we'll pick the destination of the redirect


as the canonical URL.


And then we will treat that link as being between the source


page and the canonical destination.


So just because there is a redirect in between


doesn't mean that there's any value loss.


It's still linked between the source


and the canonical destination.


And just to be sure, that 301 step,


that 200 milliseconds, 300 milliseconds


that the user gets in addition, that's


not happening on Google's side because you kind of pick


the URL separately.


You don't follow it like a user.


Yeah.


So the one exception I think is if you


have more than five steps, five redirect steps in between,


then that's something where we'll


have to recrawl that in the second round.


But otherwise, we will follow those steps.


We will index the destination page.


Probably we'll pick the destination page


as the canonical.


And if we pick the destination page as the canonical,


then users, when they come from search,


they go directly to that canonical.


They don't even follow those redirects.


So even from a speed point of view,


users would be going directly to that kind of canonical URL.


And from that point of view, it's more


that you're adding a little bit inefficiency


and it's like you're keeping these things along,


where if you crawl the website on your own,


then suddenly you have all of this cruft that's


kind of collected over the years.


But it's not that it'll cause any problems with regards


to search.


OK, and one last thing is we have a weird case where it's


also related to pagination, where


the link to the next page is one URL


and it goes to the next page.


But within a few seconds, the link changes.


I think it's using the history API, the push state,


or replay.


I don't remember exactly when it was.


How does Google interpret that?


Does Google see this URL change even though there's


no refresh of any kind done?


Or what exactly is happening on Google site?


Not 100% sure.


Yeah, so we do watch out for this kind of when


you use a history API to change the URL,


because we do understand that some JavaScript-based site,


you click on a link and then it uses the history API


to change the URL.


And in that case, we should treat that as a link.


And similarly, on some sites, you load one URL


and it uses the history API to change the URL.


And that should be treated like a redirect.


So that's something where we try to figure out


what it is that you're trying to do there,


assuming that we can process the JavaScript that


is doing this.


So that could be that we would treat that as a redirect


to the URL that you're changing it to,


which I don't know if that makes sense in your particular case,


or if that simplifies the URL.


So that's it.


MIHAI APERGHISLAMOINNANO-MARTINETO


It removes some of the extra parameters that are just


for the CMS to understand you want the next page.


And it can remove only the page equals to something.


Yeah.


Probably you would see that if you use Inspect URL.


And you saw that we chose as a canonical the simpler URL.


So that's something where you can check both of those URLs


manually with Inspect URL.


If we've seen the simpler URL, perhaps we


have chosen that as a canonical for that kind of setup.


If you check the more complicated URL


and you see we picked the simpler one as a canonical,


it's definitely the case.


Cool.


Thanks.


I'll try that.


Hi, John, if I may?


Sure.


I don't know if you remember me.


We spoke a couple of weeks ago.


And I'm just really simple chasing up, really.


Sorry to do this.


On the outbound unnatural linking reconsideration saga,


which we're going through.


And I just wanted to see if you managed to take a look for us.


Which site was that?


Oh, I think you posted it as well.


The career site, right?


Yeah, that's correct.


Yeah.


I passed that on to the search spam folks.


But I didn't hear anything back.


So I don't know.


Is it still pending the reconsideration request?


That was a long time ago, right?


The reconsideration request?


It was initially from the 7th of May, the initial notification.


So from the last status we received,


which was the process notification,


it's been 14 weeks.


OK.


I'll see.


That seems too long.


I'll ping them again to see if they can double check


to see what is happening there.


All right, brilliant.


What's your recommendation for how long we should wait anyway?


Because it's been 14 weeks since we received the process


notification.


And we follow up a few times since.


I think it's been around four weeks now


since our last follow up.


But we spoke about this on the last one,


so I won't bring it up again.


But what's your recommendation for the wait period


at the moment?


I don't know.


I don't know what the current queue there is.


I mean, your site, I think, was in English, right?


So it's not.


Yeah, it's an English site, yeah.


Yeah.


Sometimes it's a bit different across different languages,


where someone from one of the local teams


has to double check.


But it feels like, especially with English,


we should be fairly up to date.


I'm kind of curious to see if those kind of extra submissions


that you did after the main one kind of block things


on our side.


But even that shouldn't be making it take that long.


All right, thanks for that.


No, it's just because you said that you feared last time it


could be stuck.


This is why we're wondering still what to do.


So yeah, if you could do something about it,


just to get us a response would be brilliant.


All right, WD40.


OK, cheers.


Cool.


OK, we're kind of at time.


I'm sure there are still more questions.


But feel free to maybe drop them into the next session


on Friday.


And hopefully, I'll see some of you all there,


or at least in the next Tuesday one,


depending on the time zone, whatever works better for you.


All right, thank you all for joining.


And I wish you all a great week in the meantime.


Thanks, John.


Cheers.


Bye, everyone.


Site Speed: What SEOs Need to Know #AskGoogleWebmasters

Welcome, everyone, to a special episode of Ask Google


Webmasters.


Here today is Martin.


Hi, everyone.


And this is John.


Hi.


Yeah, we're going to be answering questions


on the topic of speed that were submitted with the hashtag


AskGoogleWebmasters on Twitter.


So let's start off with a question from Rohit.


What is the ideal page speed of any content


for better ranking on search?


Oh, you're asking me that question.


OK.


So basically, we are categorizing pages more


or less as really good and pretty bad.


So there's not really a threshold in between.


It's just like we are more or less roughly categorizing


the speed experience for users.


And how are we actually doing that?


Where do we get the data from?


Yeah, that's important.


So we mostly get data from two places.


On the one hand, we try to calculate a theoretical speed


of a page using lab data.


And then we also use real field data


from users who have actually tried to use those pages.


And that field data is similar to the Chrome User Experience


Report data.


Cool.


So we are having hypothetical data and practical data.


So we don't really have a threshold to give away.


But basically, the recommendation I would say


is just make sites fast for users.


That's what it boils down to.


That sounds good.


Yeah.


The next question comes from Olga.


And Olga is wondering if a website's mobile speed is best


looked at using the Test My Site tool, the GTmetrix tool,


or PageSpeed Insights.


That's a really good question.


What's the most important tool for SEO?


We have multiple tools that measure multiple things.


And I can understand that that can


be kind of confusing at times.


In general, these tools measure things


in slightly different ways.


So what I usually recommend is taking these different tools,


getting the data that you get back from that,


and using them to discover kind of low hanging


fruit on your web pages.


So things that you can easily improve to really give


your page a speed bump.


How does that sound?


That sounds pretty good.


And also, the tools are differently aimed.


Some of these tools, like Test My Site,


is pretty high level.


So everyone understands roughly what's going on there.


Whereas GTmetrix is a lot more technical,


and PageSpeed Insights, I think, is kind of in the middle


of that.


So depending on who you are catering to,


who you are trying to give this report to to get things fixed,


you might use one or the other.


So figuring out what is the low hanging fruit,


and using the tool that gives you the best insight


into that for the audience that you are trying to convince.


Is it a C level?


Is it a other marketer?


Is it someone from the tech side, like is this a developer?


Then you probably pick a different tool.


Next question comes from Onur.


Onur is asking, I am testing an almost empty page on Dev Tool


audits.


And it usually gives me minimum results,


which are 0.8 milliseconds for everything,


and 20 milliseconds for FID.


What is FID?


First Input Delay.


First Input Delay, of course.


But sometimes, it gives worse results for TTI, FCI, and FID.


OK, let's talk about these metrics.


FID, we have covered first input delay.


TTI is time to interactive.


That's when you can first interact with a page.


And FCI is first CPU idle, which means


that there is no more JavaScript work or other work that


needs to be done by the CPU.


So it's the same page, same code, different numbers.


Why would that happen?


Well, first things first, these measurements aren't perfect.


So if it's between 0.8 milliseconds and 20


milliseconds, 20 milliseconds is a lot more than 0.8.


But it's still quite a short amount of time


if you think about it.


You roughly have 10 milliseconds for a single frame to draw.


So yeah, 20 milliseconds isn't too bad.


So you will always see some basically noise


in that measurement.


And also, don't get too hung up on these metrics specifically.


If you see that there is a perceptible problem,


and there's actually an issue that your site stays


working on the main thread and doing CPU work for a minute


or 20 seconds, that's what you want to investigate.


If it's 20 milliseconds, it's probably fine.


Our next question comes from Drew.


And Drew asks us, what is or are the best metric or metrics


to look at when you want to decide


if a page is fast or slow?


And why or why not would you just


look at things like FCP, which is first contentful paint, FMP,


which is first meaningful paint, instead of just the scores


that these tools give you?


Wow, I don't know, Martin, you need to tell me some more


about that.


Right, OK.


So I guess the question here really


boils down to what's the metric that you should look at?


And that's a really tricky one, because I


guess it depends on the site.


It's the typical, it depends answer.


If you have just a website where people are reading your content


and not interacting as much, then I


think first meaningful paint or first contentful paint


is probably more important than first interactive delay


or first input delay or time to interactive.


But if it's a really interactive web application


where you want people to immediately jump in and do


something, then probably that metric is more important.


So don't try to break it down.


And that brings us to the scores.


The problem with the scores is they are oversimplifying


things, aren't they?


Yeah, it sounds like it.


I mean, all of these measurements


sound like they're measuring different things


and ultimately trying to understand what a user would


perceive when they access the page.


So a score might be, I guess, a simple way


to look at it overall, but it's probably not all of the details


that you'd need.


It just gives you a ballpark, really.


It's like, how fast is this page?


Five.


What does that mean?


It doesn't really convey meaning, doesn't it?


So I would say use that to figure out


how you're roughly doing, and then use the specific insights


the different tools give you to figure out where you have


to improve or what isn't going so well.


Wow, yeah.


Sounds like speed is a tricky topic,


and you kind of have to know what you're measuring so


that you can take action on the right things.


So would that kind of explain why there is no simple number


that Google is just giving?


Yeah, so that definitely explains it.


Because if you think about it, you


can't break down speed into one simple number.


It is a bunch of factors.


If I'm painting really quickly, but then my app


is all about interaction.


It's a messenger.


So I show everything.


I show the message history.


But if I try to answer the message that I just got,


and it takes me 20 seconds until I actually


can tap on the input field and start typing, is that fast?


Not really.


But is it so important that I can use the contact form


on the bottom of a blog post within the first 10 seconds?


Not necessarily, is it?


So how would you put that into a number?


You don't.


So I guess speed's hard.


Speed?


Speed sounds hard.


What do you think?


Will this get easier?


I guess it will get easier, but it will never


go to a point where you just have a score that you optimize


for and be done with it.


It is such a broad topic that it's really hard to break that


down into one number.


OK, so you imagine the more advanced people


will continue to focus on the tiny metrics and counting


milliseconds, and others will look at a bigger overview


picture.


I guess so.


And together, we'll try to find ways


to improve the speed of the pages overall.


I think browsers are also doing a lot of work


to make things faster in general and easier to understand.


But generally speaking, you will still


need to go and do the work of figuring out


what matters to you, your audience, and your website.


Is it interactive?


Is it contentful panes?


Depends.


That sounds cool.


So I expect more questions on speed on the hashtag


AskGoogleWebMasters.


And as we get those questions, we'll


ask an expert like Martin, who knows all of these three letter


abbreviations, and who can help us figure out


which ones are the right ones.


So thanks for submitting all of these questions.


And hopefully, see you again on one of the future episodes.


Hopefully.


Thank you very much for having me.


And thanks for all the questions.


Duplicate content (and what to do with it) | Sustainable Monetized Websites

Welcome back to the series Sustainable Monetized Websites,


a series for online content creators and content managers


who also handle monetization.


I'm Aurora, and I work in policy education for publishers at Google.


In this episode, I will talk about duplicate content.


Organica monetized? Do not duplicate me.


Let me start by saying, make sure your site adds value.


It's common sense, you may think,


but unfortunately, not all content creators abide by this rule.


Think of yourself as a user.


Would you regularly visit a site with no original content?


Or would you prefer to go to the sources directly?


Your content, or a unique and valuable functionality


which will treat like content,


should be the focal point for users visiting your page.


If you have ads serving on pages with little to no value,


the ads may be limited or disabled


until you make the significant improvements.


As stated in the Google AdSense publisher policies,


Google Ads may not be placed on sites


with scraped or copyrighted content.


Some examples include sites that copy and republish content


from other sites without adding any original content or value.


Sites that copy content from other sites, modify it slightly,


for example, by manually rewriting it, substituting synonyms,


or using automated techniques, and republish it.


Sites dedicated to embedding content, such as video, images,


or other media from other sites


without substantial added value to the user.


It's important that you contribute your own original content,


whether it's specialist knowledge, improvement ideas, reviews,


or your own personal thoughts.


Check that the content on your site is unique


and that it's not copied from other sites.


Also, ensure that the same content is not duplicated in the same page


or across multiple pages of your site.


If you have many pages that are similar or have very similar content,


consider expanding each page or consolidating the pages into one.


Thank you, Monetized.


Organika, tell us a bit more about duplicate content.


Duplicate content refers to significant blocks of content


with or across domains that either completely match


or are pretty similar to other content.


This isn't always meant to be deceptive.


It's very common for sites to have some pages duplicated completely,


say, across the www and non-www versions.


This kind of technical duplication is fine.


The goal is not to have these versions indexed,


just to have them accessible


should users reach that version of the URL.


That said, having many copies of the exact same page


can make crawling a bit harder.


Another common use case is when parts of a page


are copied within a website.


For example, you may have printer-friendly versions of content


or you may have category or tag pages


that contain a snippet of the individual detail pages.


These are fine too, though it may result in Google


not being sure which of your pages to show for some searches.


If your site contains multiple copies of the same page,


we recommend telling Google what is your preferred URL.


This is called canonicalization.


To learn more about it,


check out this video on canonicalization by John Mueller.


In some cases, however,


content is deliberately duplicated across domains or different websites


to manipulate search engine rankings or win more traffic.


This adds no value and can result in a poor user experience


when a visitor sees the same content repeated on search results.


What happens if someone duplicates my content?


If you believe that another site is duplicating your content,


you can contact the site's host to request removal.


Depending on the situation,


you may be able to ask Google to remove the infringing page


from search results by filing a request


under the Digital Millennium Copyright Act, DMCA.


More on this here.


So remember, make sure your site adds value.


It's important that you create original content


and that you avoid scraping content from other sites.


Using duplicated content in your website


can make it harder to crawl and index pages from your site


and can make it harder for search to show the best-fitting page.


Duplicate content across domains


could negatively affect your performance on organic search


and your monetization strategy.


Instead of duplicating content from other domains or sites,


we recommend working on unique content instead.


The next episode will cover policies and best practices


around unique and relevant content.


Subscribe to this channel so that you don't miss anything.


Don't forget to like and share this video


and leave a comment if you have any cool topic suggestions


for future episodes.


JavaScript SEO office hours May 27th, 2020

Hello, and welcome to this week's JavaScript SEO Office


Hours.


My name is Martin Splitt.


I am in the Google Search Relations team,


and I am a little bit of an expert on the JavaScript


rendering and indexing side of things.


So I'm here to answer your questions regarding


JavaScript SEO problems.


And you can ask your questions either in the YouTube posts


that we do before these meetups, or you


can join these live recordings by joining the link that


will be posted shortly before the actual hangout in the YouTube


post comments.


Awesome.


Thank you very much for everyone who's joining.


I see that we are a small select group today.


There's not that many questions in YouTube,


so we'll see how much content we have today.


Let's start with the YouTube question.


Hi, Martin, from the quick look on the Frontify theme built


with ReactJS 2020.frontify.org.


Can you see any SEO issues there or recommend some improvement?


This is actually, I think this is a good opportunity


to give you an idea of how I would look into these things.


And let me just start a screen share,


because I actually haven't taken a look before.


So I'll just do this now.


And the question is if I can actually get that Chrome window


that I just opened.


No, it has to be.


OK, in that case, OK.


I just clicked on the wrong thing


and actually have to share an entire screen, which


has an entire window.


That's fine.


So I would start by looking at the theme itself.


So the theme itself is 2020.frontify.org.


No, was it not?


I'll just click on, actually, I'm


just going to copy the link, and then we'll


see what happens if I made a mistake.


OK.


It's a redirect thing.


Yes, I want to go to that site.


There we go.


I must have mistyped something somewhere.


Frontity, not frontify.


I don't know why I said frontify.


So a very quick litmus test would


be to just run a Lighthouse audit.


And in this case, I actually, you know what?


I'm just not going to run all of these.


Maybe not progressive web app.


We don't care that much.


To just get a rough feeling for how does this website generally


work, how well does it work, how fast is it,


that looks pretty good.


There are already, so the thing is,


in the SEO audits in Lighthouse, they are very, very basic,


and they are vendor agnostic.


So a lot of Google-specific stuff


we can't or don't want to put into Lighthouse.


But even here, we see some suggestions.


That's a mobile usability, potentially mobile usability


issue, where some of the tab targets are not large enough.


Most of the audits pass.


So we do have a viewport.


We do have a title.


We do have a meta description.


In this case, that would be service setup, kind of an issue,


not really a theming issue.


Links have descriptive tags.


Robots txt valid, out attributes and images, hreflang.


Eligible fonts, avoids plugins, so it's not


using Java or anything.


And the performance, it looks pretty good as well.


First contentful pane could be a little faster, but that's OK.


First meaningful pane, yeah, to view idle, sure.


So it could be better, but it's pretty solid.


I wouldn't worry about that.


So that's the very first stage of taking a look at this


that I would do.


And then the next thing that I like to do


is I like to ask the mobile-friendly test


if there's any surprises, any problems with actually getting


to the content.


Oh, great.


Yeah.


And I see a car there, and I see a car there,


and I don't see cars anymore.


Oh, OK.


Google doesn't trust my guest browsing window.


No more crosswalks.


Seriously?


I don't think I have more buses.


OK, this.


Oh, that was a cheap trick.


Is that a bus, or is that a truck?


No, it's a FedEx.


It's a truck, OK.


I wish I could machine learn these things


and not have to fill them out.


Are we good at some point, or are you showing me more buses?


No, that's the red light.


So that's what?


Ah, that is frustrating.


OK, sorry.


I should have just logged in, and then I


would have avoided all these funky.


What?


OK, I'm sorry.


I'm a robot, apparently.


Maybe it's because I spend too much time with Googlebot.


Who knows?


OK, nothing.


It's not a bus.


Good.


I think I'll cut that from the video,


because that was just stupid.


Wait, here we go.


Come on.


But yeah, so performance-wise, it looks good.


The screenshot looks all right to me in the very first get-go.


Mobile-friendly, no loading issues, no surprises.


There is something going on with some JSON issue,


but we don't care.


It doesn't matter for SEO.


Yeah, the HTML looks pretty complete.


There is.


Can I see if they have a description?


Is my computer getting slow right now?


Oof, come on.


Well, there was.


So we could figure out what this is, where the data RH comes


from, I'm not sure.


But it doesn't really matter.


It looks pretty solid from an SEO perspective.


It's fast.


Maybe look at the tap target sizes


where Lighthouse was complaining, because that could also


be a mobile usability issue.


But besides that, I think this is an OK theme.


And that's more or less what I look for.


So I look at the rendered HTML.


I do a smoke test of the screenshot,


and Lighthouse gives you a few pointers as well.


So good question, but I don't think there's anything


to look at specifically here.


I guess then the tricky bit is probably setting it up


correctly with your actual content.


But the theme itself seems to be fine.


Do we have questions from the audience?


Any of you having a question while we are at it?


I've got a quick one.


It's about how many resources and stuff


load in the page that if there's any kind of limits


that you recommend to make sure something's


going to render properly.


Good point.


I kind of got some arbitrary figures


when I look at it, trying to keep them under 50


and try and make sure they fire in 0.2 seconds.


And that seems to be a relatively sane target.


And under that, they tend to get reliably thin.


But that's not always the case.


You see pages with way more successfully always rendering.


Sometimes pages with less that don't.


So I know there's a lot of nuances to that,


but I don't know if there's any kind of guidance.


The general guidance is the fewer the better, I would say, with the asterisk that you want


to split reasonably so that caching is effective.


So basically, we don't really have a hard limit or anything.


It's just the more resources there are,


the more likely you are to experience that something does


not load.


Or if you're looking at a user's perspective,


from a user's perspective, there's


a chance that the network cuts out or some transmission error


happens.


So the fewer resources, the better.


But be reasonable.


It doesn't help to have everything in one huge file,


because then you can't effectively cache.


Because then if one thing in this large file changes,


the entire file needs to be downloaded.


So Googlebot generally tries to be smart about these things


by caching very aggressively.


So even if one render fails, we will see that it fails.


But then we have at least cached a bunch of resources already.


And then we would retry.


And then we would probably get the rest of the resources.


If crawl budget is a huge issue, and that is usually


true for websites with millions of pages,


then that could be a consideration,


like trying to keep the number of resources as low as possible.


But for websites that are relatively small,


that's not an issue.


And for us in general, we are trying


to fetch everything we can.


And we use aggressive caching to alleviate


the number of resources.


So that's why you see websites with lots of resources doing


well, as well as websites with small number of resources.


And sometimes things just go wrong.


And then it doesn't really matter how many resources


you have.


If it's that resource that is critical


and we are not fetching it, then we have to retry.


But we are retrying.


So that's at least good.


But that's a good question.


Yeah, it's a bit of a very subjective thing


and a very nuanced thing.


As you said, you want to make sure


that you are taking a look at your specific situation


and judge based on that.


There's no general formula or silver bullet


that can be applied to this question.


I could say it depends.


It depends.


Right.


One last YouTube question, and then I'll


ask you again for audience questions.


Thanks, Dave, for the question.


Hello, Martin.


I have a new site where in each article,


comments are enabled by users.


So users can comment on articles, OK?


These comments are grouped in the HTML code


within a JSON-LD called comment with different attributes


like ortho, text, and date.


I've seen at Googlebot renders these comments in the HTML view


from the mobile-friendly tool.


But I don't want to be indexed in search


because there are comments with no SEO value for the page.


How can I manage the situation?


Generally speaking, you don't have to.


The fact that we render something,


I mean, generally speaking, we index everything


that is on the page generally.


But if the comments are user-generated,


we are relatively good at figuring out


that this is user-generated content.


And B, it doesn't really matter.


It doesn't hurt to have these comments there.


It doesn't give you any benefit.


But I would not.


This is one of these cases where I think you're


looking at a non-problem.


And when there is a non-problem, then you


don't have to solve anything.


That's why I would advise against trying to come up


with a clever way of hiding.


There are definitely ways of hiding things


from Googlebot's view, but I just would not.


Because there is a chance that you are shooting yourself


in the foot by creating something


that is less stable, less robust,


or that you accidentally overshoot.


So for instance, to give you a very simple example for this,


one thing is if these comments are fetched by a script


on your website, let's say you have a comments.js file that


actually fetches these comments from the back end


and puts them in the content.


You could robot away the comments.js,


and then we can't fetch the comments.js,


and so we can't execute that JavaScript,


and so we can't actually fetch the comments.


But if you play around with your robots.txt,


you might end up accidentally maybe blocking more than just


that, or like, I don't know, block all the JavaScript,


and now your main content doesn't show up anymore.


So then in trying to fix something


that wasn't a problem in the first place,


you actually made the problem, or you


created an actual problem that wasn't there before.


So I would advise against worrying too much about this,


unless you have very, very good reasons to.


Like, I don't know.


If the content is very hard to distinguish


from the main content or something like that,


then we might not see that it's user-generated common content.


But if it is just comments, as you would normally


mark them up, and there is sufficient content on the page,


then don't worry about that.


It's more risky than useful.


All right.


We have exhausted our questions pool from YouTube, I think.


I will check the previous JavaScript SEO office hours,


because I know that sometimes people post after the actual


Hangout into these posts.


But do we have an audience question in the meantime?


Search News was playing.


That was loud.


Do we have an audience question that we could look at?


Not today.


Fair enough.


Let's have a look.


Maybe we have something in the previous one.


I think I answered that one previously,


but I can answer it again.


My crawl rate is very low.


I lost traffic.


Is it because of that?


No.


Crawl rate has no quality indicator or signal


of something positive.


It's just it's possible that your server was responding


with a 500 for some requests, or it's possible that we just


have a version of the continent, know that the change frequency


is low, in which case, why would we crawl again?


It doesn't hurt.


It's not a problem, per se.


If you're running a single page application


and it uses client-side rendering


and the HTML preview shows a blank page,


you want to figure out.


So I'm assuming that you mean the rendered HTML


and any of the testing tools that we provide,


like the URL inspection tool, the mobile-friendly test,


or the rich results test.


If you see blank HTML or an empty page


HTML that doesn't have the actual content in it,


check if you are roboting any of your resources,


like any of the JavaScript that fetches this stuff.


If anything wasn't loaded properly,


that's something that you want to look into.


And last but not least, just double-check


what could be the reason for this.


I mean, I can't answer that question,


because it's like saying my car doesn't start.


What's happening?


And it can be anything.


It can be the battery is dead.


It can be that you have no fuel.


It can be that it's still locked somehow, or whatever.


It can be so many different things.


And that's the same here.


You want to be a little careful with that,


but basically just try to make changes in the application


and simplify things in the application until you


get a positive render that includes the HTML


and rendered HTML.


And then try to figure out what breaks it.


I use a technique that's called bisecting.


So I have a situation here, or a version here, that doesn't work.


I know I had a version here that did work.


Let's say there's six months in between.


I jump three months back.


Does it work or not?


If it works at this point, then that


means that my search radius is now only the last three months.


Then I go one and a half months back.


Basically, I jump.


So now I'm three months back.


I jump one and a half months to the front.


Still works here.


So it must be in the middle here.


Then I basically just iteratively


jump closer to a version that will break.


And eventually, you find the version's like, works here.


Doesn't work here.


And then you know what the changes


are that were made between the working version


and the broken version.


And that way, you can figure out what could potentially


be the problem.


And this is why I am so happy that we have the newer tools.


Because previously, you had a screenshot


in the old-fashioned render.


And that didn't really tell you anything.


It's just like, yeah, it's broken.


And I see it's blank.


But that's pretty much it.


And then trying to figure out what specifically


broke that last bit that pushed it over the edge


was not that easy.


It's a lot easier now.


You can use tools like local tunnel or ngrok


to put your local development version


into a publicly reachable URL temporarily


so that you can try this out with the testing tools.


But yeah, that's what I would do.


All right.


If we don't have any further questions from the audience,


then I would say this was a short office hour, more


like an office half an hour.


Any questions from the audience?


So just a very quick one, if that's OK.


When you call something like an API,


if you have a noindex, XRobots header on that API,


that's not really an issue, is it, for rendering?


I don't think that's an issue.


I haven't tested this.


That's a really interesting question.


I should definitely have a look at that.


But normally, we would fetch the API call


because it's not roboted.


It's just noindex.


And then we would get the data back and use that in rendering.


But I don't think that noindex on an API


makes any difference, really.


That's what I would expect.


I would test that because I'm not 100% sure about this.


But from logic, that should be how it behaves.


Good question.


I like questions where I don't know the answer.


And I actually have to test.


It's good.


Awesome.


In which case, thank you very, very much for joining.


I'll post into the youtube.com slash Google Webmaster


slash community the thread for questions for the next office


hours, which will be on Wednesday the, ah, what day is


that, 3rd of June or something, I think it is.


I think it's already June, but I'm not sure.


Where's my calendar?


Next Wednesday is the 3rd of June.


Hey, I'm actually not bad.


So on 3rd of June, we'll have another JavaScript SEO office


hours.


I'll leave a thread on the community tab of our YouTube


channel where you can ask your questions.


And then I'll also post the link to the hangout


there if you want to join the recording.


Thank you so much for joining my little audience here


in the Hangouts call.


It was a pleasure.


Stay safe.


Stay healthy.


Have a great time.


Bye-bye.


Thank you.


Thanks for joining.


Common SEO Questions - Holiday Special #AskGoogleWebmasters

Here today is another round of Ask Google Webmasters,


the series where we answer short questions about Google


Search for webmasters everywhere.


We usually take questions from the hashtag


AskGoogleWebmasters on Twitter.


But since the end of the year is coming soon,


I thought we'd do something slightly different.


For today's episode, I'll do short, bite-sized tidbits


on questions we commonly hear.


What's the difference between 301 and 302 redirects?


Well, both send users onwards.


The difference is subtle but small.


A 301 is a permanent redirect, so the destination


is what we keep.


A 302 is a temporary redirect, so we'll come back


to the start to double check.


They're just different ways of redirecting.


Use the right kind when you can, but don't


worry about magical SEO dust.


Both work fine.


What's the difference between a 404 and a 410?


Yay, more HTTP result codes.


Both tell us that a page is now gone.


Theoretically, a 410 can be a tiny bit faster.


In practice, on a normal website,


they're both processed in the same way.


Don't sweat the difference.


How long should my page's description meta tag or title


tag be?


Well, there is no fixed length.


Make them unique, compelling, and useful.


Provide the information that people


are looking for so that we can highlight your pages


optimally.


Can I use symbols in my titles?


Sure.


We try not to show misleading symbols,


and there might not be that many people searching for them,


but you're welcome to use them.


What about keywords in URLs?


Do they matter?


Words in URLs play an insignificant role


for web pages, but most content management systems


make it easy to use them.


So I'd just use them if you already have them,


but don't worry if you don't.


That said, we do use them for image files, though.


So cute robot.gif is definitely better than image-1138.gif.


Does it matter if I use PHP or HTML endings for pages?


No, both are fine.


I'd also recommend not changing from one to the other


unless you're already making bigger changes elsewhere.


Don't sweat it seems to be the motto today, right?


Well, some things are easy, so let's focus on the bigger


picture.


What is the secret to ranking better, John?


Well, there's no secret, sorry.


Having a website that's on a technically strong foundation


with awesome content is a great way to start.


Compare it to, well, baking the best cookie.


You want something made from good ingredients


by trustworthy bakers, ending up with something


that's not just the same as you get elsewhere.


Stand out.


Use a good technique with great content.


Make something that's the best by far,


something you'll be proud of for the long run.


Our algorithms don't count the words


on a page similar to how you wouldn't judge a cookie just


by its weight.


If you want your pages to be recommended


as something awesome, then make something awesome.


Well, well, well, that's all for today.


I hope you found this video useful.


Is there something on your mind that you'd


like to send our way to get answered?


Post using the hashtag Ask Google Webmasters,


and we'll try to pick it up for a future episode.


Want to stay in the loop?


Make sure to subscribe to the channel,


and tune in next time for the next episode.


What to expect from Search Central in 2022?

Hi, everyone, and welcome to the next episode of the Search


Off the Record podcast.


Our plan is to talk a bit about what's


happening at Google Search, how things work behind the scenes,


and who knows, maybe have some fun along the way.


My name is John Mueller.


I'm a search advocate on the Search Relations team


here at Google in Switzerland.


I'm joined today by Martin.


My name.


What all do we have lined up?


I don't know.


We'll see.


Stay tuned.


So first off, voice search.


What do you think, Martin?


Is it going to be the year of voice search?


Yeah, totally.


2022 is both the year of voice search and Linux on the desktop,


I'm sure.


What are your thoughts, Gary?


Is it going to happen?


I don't have thoughts.


Oh, so you're like a voice search person then?


It's like you like to speak instead of think.


Why do you hate this horse?


Stop beating it.


Oh, man.


Yeah.


It feels like voice search used to be a really popular topic,


but it kind of has died down a little bit.


And I thought in the beginning, with the pandemic, everyone


working from home, like you'd be able to speak to your computer


because nobody is around to make you look foolish for that.


But I don't know how it is for you all.


But at least for me, I don't even


speak with my computer anymore.


No, me neither.


Also, if you're on a video call, you don't really


want to mute yourself just to speak to your computer


for something else.


It would be weird, wouldn't it?


But I got to admit, I am speaking to my car.


That does happen.


Is it like, who's a good car?


No, no, it's more like navigate to this and that place,


or turn on the windshield wipers or something like that.


Because then I don't have to take my hand away


from the steering wheel too much, which is nice.


You mean from the self-driving car steering wheel?


Yeah.


I should try that.


I wonder if I'll be as successful as you, Martin.


So one of the things I heard more towards,


I guess, the last part of last year


is that people are not really as happy with the way


that we're crawling and indexing content.


Do you think that's something that'll pick up this year,


Gary?


I mean, we are definitely thinking more


about the sustainability of crawling and indexing.


So probably we are going to focus even more


on those topics this year in 2022.


Sustainability, like is it gluten-free and vegan?


Yes, Martin.


We are feeding Googlebot with salad.


Salad.


Is that the only thing you think vegans eat?


Just got here.


I was trying to think of like non-brand vegan food,


and I couldn't come to anything.


And then I realized that, you idiot,


like all vegetables are vegan.


But I was like, oh, like what a burger and whatever.


Yeah, Martin.


We are feeding salad to Googlebot.


Sure.


No, what I mean is that computing in general


is not really sustainable.


And if you think of Bitcoin, for example,


Bitcoin mining has real impact on the environment


that you can actually measure, especially


if the electricity is coming from coal plants or other less


sustainable plants.


We are carbon-free since, I don't even know,


like 2007 or something, 2009.


But that doesn't mean that we can't reduce even more


our footprint on the environment.


And crawling is one of those things


that early on we could chop off some low-hanging fruits.


Like, for example, with HTTP2, we could stream better,


for example, from the net, open fewer connections.


And then that saved us resources,


but also for the servers that we were crawling from,


like the servers that were hosting the sites.


So we are thinking more about these things,


like how can we reduce even more Googlebots and other crawlers,


Google crawlers' footprint on the internet,


on the environment.


And then if you think about it, one thing


that we do and we might not need to do that much


is refresh crawls, which means that once we discovered


a document, a URL, then we go, we crawl it,


and then eventually we are going to go back and revisit


that URL.


That is a refresh crawl.


And then every single time we go back to that one URL,


that will always be a refresh crawl.


Now, how often do we need to go back to that URL?


You could say that, for example, if you


take the CNN or Wall Street Journal home page, which


is changing every five seconds, then we


do need to go back very often.


But then the About page of either of these news outlets,


they don't change too often.


So you don't have to go back there that much.


And often we can't estimate this well,


and we definitely have room for improvement


there on refresh crawls because sometimes it just


seems wasteful that we are hitting the same URL over


and over again.


Sometimes we are hitting 404 pages, for example,


for no good reason or no apparent reason.


And all these things are basically


stuff that we could improve on and then reduce


our footprint even more.


JOHN MUELLER


To a large extent, I think we can do it from our side.


Basically, we're rethinking how we issue refresh crawls,


but then there might be other things that we could try.


And for example, index now, which


I have a beef with that name because it's more like


crawl now, not index now.


It could be index maybe.


Martin, please don't start singing.


MARTIN SPLITT SANDBERG I was very close to doing that.


But index now could be something that might be useful


and we are running some experiments


to see if that's the case.


Probably it's not going to be in the form that people expect,


but we'll see.


But I can definitely see that it might prove useful,


in some cases at least.


JOHN MUELLER So I guess we had something similar with sitemaps


for a while.


Well, I guess we still do.


In the sense that people can also


specify the last modification date in the sitemap file.


Is that something that didn't work out so well?


Or are people using that wrong?


Are we, I don't know, just not able to use that


as helpfully as we could?


MARTIN SPLITT SANDBERG I think that's a combination of issues


with the last mod and also the sitemap format,


plus our communications documentation.


I don't know.


For one, what is last modification?


Is it the content changed?


Is it the page somewhere changed?


Did you update a footer link, or you bought some links


and then, or sold some links, and then you update the page


to reflect those links in the page?


We don't explain it well, what we are expecting


from the last mod date.


But the other thing is that even if we tell people what it is,


generally the largest portion of the internet


will not do it well.


And for example, I don't know, even


if there was a good WordPress plugin that


generates sitemaps, because the in-built sitemap generator


doesn't produce last mod tags.


I filed a feature request, but then they closed it.


So fine, I guess.


But for example, the Yoast plugin,


it is capable of generating last mod tags.


But then we also saw that some people are modifying


those last mod tags that the plugin is injecting


in the sitemap for reasons.


Basically, they think if they keep that tag very recent,


then we are going to, I mean, the date in the tag very recent,


then we are going to crawl more, which is not the case.


Or maybe it will be the case for like 10 crawls,


and then we figure it out that, wait, this tag is stupid,


and then we are just not going to use it.


But in general, we are not using it,


because it's just so unreliable that there's


no point using it.


So I guess that's kind of also a misconception


that people have in that they think


if a page gets crawled more, it'll get ranked more.


Is that correct, that that's a misconception?


Or is that actually true?


SPEAKER 1


I mean, it doesn't make sense, right?


If nothing has changed on the page,


then we already know everything there is to know about that


page.


Why would that be a thing that makes it more relevant or more


useful to users?


Doesn't make sense.


Well, I don't know.


You're talking about SEO, Martin.


SPEAKER 2 Fair enough.


But I'm also actually wondering how the Indexed Now


API solves that problem, because I can foresee people just


pushing the same URLs over and over and over and over


and over again into the API, and then you have a similar problem.


SPEAKER 1 I don't want to go into too much detail about how


we designed our experiment, but it will essentially not


allow that.


SPEAKER 2 Oh, OK.


And that's also something that I would


love to see in the sitemap protocol as well,


because we have that endpoint, google.com


slash sitemap slash ping or something like that.


And we sometimes see that people are pinging it


with every single update, which is like, do you have to?


Like, if you are publishing every second,


like you are the Daily Mail or the Telegraph or whatever,


and you are publishing every second,


do you need to ping every second?


Or it's enough if you ping once every 10th second or even


every minute.


I would love to see more guidelines on that,


for example.


SPEAKER 1 Yeah.


There should be some sort of rate limiting, I guess.


SPEAKER 2 I mean, on our end, we do have rate limits.


But then that's also not transparent,


because basically we are just dropping the request,


HTTP request, because, well, you are sending too many.


SPEAKER 1 I mean, that sounds like something we could do,


like be a little bit more transparent


about the guidelines and expectations


on both sides for things like sitemaps


and crawling in general, I guess.


SPEAKER 2 I mean, especially with crawling,


there is not much for us to hide or for search engines to hide.


Because it's a very basic process


using very basic internet architecture.


And there are some quirks that maybe are IP.


But even if you leave those out, you


are still presenting a very good picture


of how crawling works at Google or probably


any other search engine.


SPEAKER 1 Cool.


Cool, cool.


Sounds like we can do something there.


So another theme that came up, I don't know,


I guess towards the end of the last year,


was everything around outages, where something went wrong


in search or something got stuck in Search Console.


And it felt like there are lots of these happening.


Is that something that you think this search side will fix,


so there are no more outages and everyone will be happy?


SPEAKER 2 I mean, we have outages.


I mean, they are computers.


Computers have outages every now and then.


And we are trying to be as redundant as possible when


it comes to processing units.


But we are still going to have outages.


One thing to note is that we did start


to talk more about outages.


And previously, you could just not notice the outages


because we were not talking about them.


And it affected only a small portion of the internet,


for example.


And even Barry Schwartz from Search Engine Roundtable


wouldn't notice them.


Now we are actually talking more about them.


And we are preannouncing even those that were not yet.


Well, preannouncing an outage, that's great.


That's just great.


We are announcing outages even if they were not


noticed externally yet or they were


noticed by a certain geographic region, for example.


And at least the perception will be that we


are having more outages.


I think our SRE team, the site reliability engineers,


they would prefer to have fewer outages.


But it's up to them to make that happen.


So it's more that we've just been acknowledging them more


and talking about them more.


It's not actually that there are more outages.


Yeah, embracing them more.


Embracing the outages.


I mean, like you said, complex systems,


they have weird quirks that sometimes turn into failures.


And I think talking about them, explaining what's happening,


explaining what we're doing on our side to fix and prevent


them, I think that makes a lot of sense.


So maybe we should go more in that direction


or at least be a little bit more systematic in terms


of what we talk about when it comes to outages.


Yeah, indeed.


Cool.


Yeah.


I mean, if we can come up with a way to prevent the outages,


that might be neat as well.


But yeah.


I mean, that would be ideal, right?


I mean, it's kind of awkward trying


to figure out what we should say about outages when you don't


have anything to test it with.


It's almost like you need to have a test outage.


Don't do it, Gary.


Don't do it.


Well, too late.


No.


Uh-oh.


So one of the other things I noticed


that we've kept pretty stable for a really long time


is the documentation around our Webmaster guidelines,


for example.


It seems like that's something where we've kind of reached


a really good point, and people seem


to understand it more or less.


Do you think that's something we should


try to refresh a little bit?


I think it became very confusing.


OK.


More confusing, actually, not less.


So we used to have a very balanced Webmaster guidelines,


but then we started putting more stuff into it


whenever there was a push for something,


like, for example, HTTPS, then ended up in the Webmaster


guidelines when it was speed.


Then it ended up in the Webmaster guidelines


when there was something else, then it ended up


in the Webmaster guidelines.


And that's not necessarily good, because the Webmaster guidelines


is should be more about what are the basic requirements for you


to get into search.


Speed is not one of them.


HTTPS is not one of them.


So what are they doing there?


And the other thing is that many of the quality guidelines


were created, how many years is that?


Like 15 years ago, at least.


And some of the things that we have in the Webmaster


guidelines don't even exist anymore.


OK.


Like, when was the last time you saw a guest book, for example?


Except Martin's site.


Oh, come on.


My guest book is wonderful.


That's a blast from the past.


Yeah.


And then there's Web Position Gold, for example.


We still mentioned that somewhere in the Webmaster


guidelines.


When was the last time you used that or even encountered


that software?


Is that related to Search Console?


I am going to go with no.


I guess not.


Maybe it's like something JavaScript-based.


Here we are again.


Back in the days when Martin was 12-year-old


and he started playing with JavaScript,


then he created Web Position Gold that was his first project.


But I like the Webmaster guidelines.


I don't know.


You think they are overloaded with stuff?


And also not very clear.


Sometimes, for example, doorway pages,


if you take a look at that page, it


explains what doorway pages are, but it's very general.


And for example, if you have a site for every single state


in the US because you have shops in all those states,


then technically that's a doorway page.


But does it violate our guidelines?


It depends on other factors.


Are you trying to manipulate the search results or not?


How do you see it, Martin?


When you're talking with developers


who don't have that traditional SEO background where they know


about these Webmaster guidelines,


do you mention the guidelines to them?


Is that something that they take into account?


Or is it basically just if you get a manual action,


then you go read the guidelines?


MARTIN SPLITT


So developers usually don't reference them as much.


It does come up every now and then


because an SEO brought it up to them.


So they're like, oh, our SEO told us about this thing,


and apparently it's against the guidelines.


So they just seem to go with it.


And I sometimes reference it as well.


If they are doing things like doorway pages that


are manipulative, then I'm like, listen, here's the thing.


This is not a great way to approach this.


But I do understand that there is all this contextualized bits


and pieces where you need to be aware of the context


and if it makes sense or if it doesn't make sense,


if it's for manipulative reasons or if it's for good reasons,


that you have, for instance, a doorway page.


But then usually developers come to me


and don't bring up the Webmaster Guidelines


so they don't necessarily seem to be aware of them.


I guess it's more an SEO audience kind of thing.


Yeah.


I wonder if we should at some point


kind of split them off into more technical guidelines,


like what you need to do, and separate out


the kind of the spam, low quality, manipulative stuff


a little bit more.


Because then from a technical point of view,


the developers would still have enough to look at and be like,


oh, I'm fulfilling all of these requirements.


I mean, that's something that we are trying to do


or we are going to try to do this year in 2022.


I'm still thinking about how to do it


because every single time we touch the Webmaster Guidelines,


people freak out.


So maybe we want to pre-announce it at one point


that we are going to change a bunch of things


around the Webmaster Guidelines and then hope for the best.


And maybe people will not freak out.


Yeah, but I mean, the goal is to make


them clearer and more specific to help people


more with the guidelines.


So that should be good.


Sure.


I mean, yeah.


But if you just drop those changes,


then everyone and my grandmother will


start analyzing the changes.


And then they might read something into it


that's not there.


And that's what we want to preempt, ideally,


basically saying that, hey, we are


going to separate the technical guidelines from the quality


guidelines.


And we are going to be clearer in language,


plus give more examples, for example.


I don't know if we can give more examples because that depends


on the web spam folk.


Can we give more examples, for example?


Yeah.


I like that phrase, for example.


If you listen to our past podcasts,


I have at least 90 of them in every single episode,


for example.


You're quite exemplary.


So you're saying we should pre-announce it somewhere


where people will maybe listen and hear from us.


People never listen anywhere, do they?


They are listening to this podcast, I think.


That's what the.


Woo-hoo.


OK.


We pre-announced it.


I think that's a myth.


Cool.


The thing with these kind of guidelines


that I always run into is we try to make them clearer,


and then we get 500 people coming in and being like,


but what about this specific situation?


And what about this?


And look at these five things I found on the internet.


It's like, what about this?


Sometimes tricky to be both clear and cover enough


that people kind of know what to do.


But I guess there's also an aspect that people generally


know when they're being too sneaky.


So it's like, we don't really need


to cover all of the different ways to be sneaky.


I mean, right.


But we should still cover the main ones,


or the more prominent ones, and the ones


that people talk about.


Like, for example, oh, for example,


they talk a lot about cloaking, but not


in the context of hiding things from search engines


for manipulating search results, but more in the context of,


can I only serve pages to people from India, or people


from the UK, or people from Japan for legal reasons,


or whatever reasons, for policy reasons,


can I still allow Googlebot, even though Googlebot


comes from a US IP?


Is that considered cloaking?


And technically, yes, that's the literal definition of cloaking.


Will Google or other search engines penalize for it?


Highly unlikely.


I mean, that's something that developers come to me


with very often because of the dynamic rendering


recommendation that we used to give out years ago that is now


deprecated, by the way, where they're like, oh, we are


serving a static version.


It's the same content, just we are


serving the static version of the content to bots,


and then the dynamic rendered, like the JavaScripty fancy


version to users.


I'm like, it's the same content.


You're not manipulating the search results in any way


that way.


It's fine.


But yeah, it's these cases where people then get lost.


That's true.


JOHN MUELLER, MD, PhD I mean, especially cloaking


is super tricky because there's so many ways


to do it in a legitimate way, where people easily kind


of worry that, oh, they're breaking some rule on Google


side, and there'll be this giant penalty that they, I don't know,


won't be able to resolve anymore.


And it's something where we also kind of need to say,


well, here are some technical reasons why you might do this,


and here's what you shouldn't be doing.


GARY ILLYES, MD, PhD I will go back to some of the things


are very archaic in our guidelines,


and it would be great getting rid of those,


putting some examples in at least some of the guidelines,


quality guidelines, probably not in the technical guidelines


because the technical guidelines are more clear.


And that's it, probably.


JOHN MUELLER, MD, PhD OK.


So if you're listening, don't freak out


if we change the Webmaster Guidelines.


Or should we also change the name, or?


GARY ILLYES, MD, PhD Well, yeah.


JOHN MUELLER, MD, PhD Do freak out.


GARY ILLYES, MD, PhD So the name should be changed, right?


Because we moved away from the Webmaster Central name,


so we should come up with some new name for the guidelines.


Or it should have been like Webmasters Guidelines, right?


GARY ILLYES, MD, PhD Guidelines for Webmasters,


Guidelines for Content Owners, Content Producer,


Guidelines for Web People, Web Folk.


JOHN MUELLER, MD, PhD Search Central Guidelines, there we go.


Problem solved.


GARY ILLYES, MD, PhD Well, that's lame.


But it also has to be something that we


can translate to many languages, like the 18 languages


that we support.


So that also complicates things.


And something that you can easily say,


like Search Central Guidelines, that's


like even saying Search Central breaks my.


JOHN MUELLER, MD, PhD Oh, I have the perfect recommendation,


Gary.


What do you think of guideliney make guideline face?


GARY ILLYES, MD, PhD Yeah, let's do that.


Ship it.


I am surprised because I'm 100% on board.


I'm not sure.


Moving over to something else.


Back to you, Martin.


What do you think about events this year?


Is it the time for big events, traveling


to foreign countries, meeting lots of people


and hugging them?


What do you think?


Is it time now?


MARTIN SPLITT, MD, PhD Why do you have to hurt me like that?


I really, really miss in-person events.


But with the instability of the current pandemic situation,


I don't think we can already say we have definite plans


to be resuming in-person events.


But for sure, we're going to have some more virtual events.


If possible, we might try to figure out


how to do hybrid events where we can have local hubs


or something.


And if the situation allows over the summer,


I hope that we can have one or more small local events.


But it's definitely all up in the air.


It's very unclear how we are going to proceed.


What I do know is that we are definitely


going forward with virtual unconferences this year.


Really, really looking forward to that.


Are you thinking more of meetups or more like what


we had like the Webmaster Conference or WMCONF


that we had?


So I think to resume things, and also


because the situation might continue


to be very, very volatile, meetups make more sense for us


to resume more in-person events.


The Webmaster Conferences were amazing, and I loved them.


But I don't feel confident in organizing.


It also needs a longer timeline.


You need to start planning these things months ahead.


And then people usually travel across continents,


which is probably not the best idea right now.


And I don't see that happening.


I'd rather have us try to do things that we can,


if push comes to shove, cut short last minute,


if it has to happen.


We tried to have a pilot event last year in December


in Zurich with like 40 people and a small local meetup group


hosted at the Google offices.


And then our team would have joined.


But guess what?


That didn't happen.


And it proved that everyone was kind of fine with it


because no one had to invest much into it.


So when we canceled literally like the week


before it would have happened, everyone went with it.


Everyone was like, yeah, sure.


No, that makes perfect sense.


It doesn't make sense to run these events.


So I think local small meetups will be a thing


that we can probably do.


And I am seeing myself doing short-haul travel,


like maybe to Germany, maybe to Austria, France, Netherlands,


probably.


I think we can do something there.


Also, we have offices there.


So the venue issue is not really an issue.


We don't have to rent an external venue.


Because if you rent an external venue,


then canceling becomes trickier.


And yeah, I think it's going to be more meetups.


So I guess also like short-term organization.


It's like, by the way, in three weeks, we will have a meetup.


And if you're there, come and join us kind of thing.


Yeah, because then that makes planning a lot easier.


And I'm really, really, really worried


about people who are looking out for our events


and then going like, oh, there's something.


I live in the US, but there's something in Switzerland.


So I'll just book flights into hotel


and go there, because then I fear disappointing


these people when they come over.


And I have to pull the plug last minute.


But then again, that's the risk that people need to take.


So I do hope that we can organize lots of local events


across the globe, if possible.


But please, please, please, people out there,


be patient with us.


I'd rather not take too much of a risk here.


Can we do one in Japan?


Actually, we are planning a virtual conference for Japan.


Oh.


No, like a live one.


Maybe.


Do you want to do a, OK, you want to do Japanese spaghetti?


We can do a road trip with your car, Martin.


Yeah.


OK, Google, navigate to Tokyo.


Fantastic idea.


I'm not sure my range is large enough for that.


But yeah, Gary, let's definitely talk about it.


I think that's something that we might want to look into,


if the situation allows.


At this point, I don't think, anytime soon, though.


But I would love to go back to Japan.


Why Japan, Gary?


It's like, apart from the spaghettis.


Can you stop calling it spaghettis?


Maybe I should go at some point, and there'll be, John,


this is not spaghetti.


Yeah, definitely.


I mean, Japan is one of those countries


that we always loved, as in Google always loved.


And I think it was the first international office


that we opened that was in Japan, in Tokyo.


I think it was 2003, 2001, something a long time ago.


And there's a lively community there, both developer


community and SEO community.


Second most read language in our search documentation


is Japanese.


So it feels like it should be one of those countries


where we should invest more.


Unfortunately, we don't have a Japanese speaker on our team.


So we, for example, don't even know


if our docs make any sense, or we are using the wrong word


for certain things.


For all those reasons, it would be good to focus more on Japan,


just figure things out.


I think it's also one of those countries where people tend not


to speak English that much.


Or is that misconception?


No, that's true.


It's difficult. Especially if you go outside the city,


like Tokyo or Osaka, maybe, it would be very, very difficult.


Even if you go to Nagano City, for example,


buying a bus ticket was impossible in English.


You actually had to speak Japanese with a color.


Wow.


OK.


So I guess if I go, I should stay in Tokyo.


Unless I want to live there forever.


Just like, I want a bus ticket home.


So the interesting thing is that the Japanese people


are extremely helpful.


So they will go out of their way to try to understand you,


even if they don't speak English.


They will find you a map from the Edo period,


like from the 1600s, and point you out,


like, are you trying to go from this point to this point?


Or find ways to help you, even if they don't speak English


and you don't speak Japanese.


And for that, it's amazing.


I strongly recommend going out of Tokyo, if you ever there,


and just experience the non-big city Japan,


because it's absolutely worth it.


OK.


So it's like a terrible time to try


to encourage people to think about traveling


to other countries.


I don't know.


I mean, right now, you can't even get to Japan,


or get into Japan.


So it's not like I'm encouraging people,


and they will jump on a plane and get there,


because they just can't enter the country.


It's also that I think both we, as well as the community,


do miss the in-person events.


We have to be a little more patient, I think,


and large events will just take us longer to get back on track.


So yeah, we all have to be patient, which is something


that I'm absolutely not.


Yeah.


We're not patient.


We care about speed and page experience, right?


Yes, we do.


Yes, fast websites, fast travel.


I don't know, maybe we should just wait until there's


a rocket trip to Tokyo.


We can just go there for an afternoon.


Maybe.


But speaking of page speed and page experience,


aren't we rolling that out for desktop as well?


I think that's coming May, I think we announced,


or something like that, or April.


I forgot the actual date.


I should probably know.


Oh, well.


It's always, I don't know, it was


one of those things which, in my mind,


was perfectly obvious and clear.


We're doing this on mobile in the mobile search results,


and we're also doing it on desktop now.


But it seems that it kind of threw people off,


because while you're only indexing


the mobile version of a page, how can you do speed on desktop?


And understanding that difference between, well,


what we index and what we use to measure speed


are different things, that takes a bit.


And also, the search results are slightly different on desktop


and mobile.


And from our point of view, that's kind of expected.


But not everyone seemed to have been aware of that.


So we'll see what happens when that goes live,


if people understand it a little bit better,


or if it's confusing again.


Hopefully not too confusing.


So what are some other things that we should


be looking at for this year?


Should we start doing, I don't know,


something like short videos or, I don't know, a newsletter?


Newsletters seem to be all the rage.


On your blog.


A blog.


No, on your blog.


A newsletter on my blog.


Yeah, on johnmoo.com, we could put up a newsletter.


OK.


With all the updates that you post.


Maybe we should start a bulletin board.


A bulletin board.


Yeah.


I'm now thinking about actually guest books.


That seems like a really cool idea.


Maybe we should just go back to that.


Can we just stop this before you get more brilliant ideas?


I know.


We will mint NFTs with guest book entries.


Please, someone cut this.


Oh my god, you guys are not excited


about the future of the web.


I don't understand.


We are about the future of the web, not the.


Web 3.0, it's like all about guest books and NFTs.


Where's our producer?


Please cut.


Port mission abandon ship.


Oh my gosh.


I don't know, man.


Whatever.


Fine.


Well, thank you for joining us here, folks.


We've been having fun with these podcast episodes


most of the time.


I hope you, the listener, have found


them both entertaining and insightful as well.


But feel free to drop me a note on Twitter or on my guest book


or chat with us at one of our next virtual events


if you have any thoughts.


And of course, don't forget to like and subscribe.


Thank you and goodbye, everyone.


Requesting content removals from Google products for legal reasons

Sometimes you might see content on search that you think is illegal or unauthorized.


It could be a competitor that has copied your content onto a page on their website, or you


might see a web page infringing a law in your country, but you don't know how to remove


that content from Google search. Or even other Google products.


So how do you remove potentially illegal content on Google?


We have made it easier for you to report illegal content on any Google platform through our


intuitive troubleshooter at g.co.uk forward slash legal.


This will direct you to a page listing Google's products.


After selecting the product you want to remove content from, such as Google search, select


the reason for removal, such as an intellectual property violation like copyright.


When reporting a possible copyright violation, make sure that you are the holder of the content,


otherwise your request will not be successful.


Handy tip, you generally hold the copyright in content that you create. For example, the


copyright in a photo generally belongs to the photographer, even if it's a photo of


someone else. So unless we're talking about a selfie, you might not be the holder of the


copyright in a photo, even though you are in it.


Once you reach the form, fill out all information as required, so the team at Google can review


your request. Help us review your request more quickly and with the best chance of success


by providing specific information.


Please only send us the specific page URL. Do not send us the entire website.


We also recommend you clearly describe what content on the page belongs to you.


Once you have submitted your request, depending on the Google product at issue, you will either


be given a link to a dashboard or sent an email confirming receipt of your request and


containing a case number.


The length of time taken to respond will depend on the type of complaint you are making, but


rest assured, Google has teams of experts at work on content moderation across the globe


and your request will be handled with the required consideration.


Should we require more information from you, we will contact you by email. So remember


to check your email and help us process your request by providing the information we need.


You will receive confirmation on your dashboard or through email once your request has been


processed.


Do note that while we may be able to remove links to the potentially illegal content on


our search results page, only the host of the website can remove the content from their


website. You will need to contact the website owner if you want to remove the content at


its source.


For more information, check out the Legal Help Centre linked in the video description.


For an overview of the type of requests we receive, you can also check out our transparency


reports also linked in the video description below.


Canonical URLs: How Does Google Pick the One? #AskGoogleWebmasters

Today, we'll address Maria Chiara's question about canonicalization.


You can indicate your preference to Google using these techniques, but Google may choose


a different page as canonical than you do for various reasons.


So what are these reasons?


Let's take a brief step back.


It's quite common for a website to have multiple unique URLs that lead to the same content.


The really common one is when the www version of a site shows the same content as the non-www


version of the website.


Most websites are set up in this way.


Another common configuration is when the homepage is accessible as index.html or when upper


and lower case in URLs leads to the same pages.


Ideally, we wouldn't even run across any of these alternate versions, which is why we


recommend picking one URL format and using it consistently across your website.


In practice, life is rarely perfect and we discover all sorts of URLs leading to the


same content.


For search, it doesn't make much sense to index and show all of these versions, so we


try to pick one and focus on that.


But how does Google pick the chosen one, the canonical URL?


We try to pick the canonical URL by following two general guidelines.


First, which URL does it look like the site wants us to use?


So what's the site's preference?


And secondly, which URL would be more useful for the user?


When it comes to the site's preference, we look at a number of things.


That includes the link rel canonical annotation, which sites can use but also redirect, internal


linking, which URL is in the sitemap file.


We also prefer HTTPS URLs over HTTP URLs and try to pick nicer looking URLs when we can.


For all these, we consider the factors involved for each potential canonical URL and then


pick the one where more things come together.


So if you're a site owner and you have a strong preference regarding URLs that you want to


have shown to users in search, first of all, you should make sure that you use those preferences


consistently across your website.


Ideally, search engines wouldn't even be able to stumble across any of those alternatives.


If you have a preference, then stick to it.


Along that line, make sure that all of the canonicalization factors that we mentioned


are similarly aligned.


Make sure that internal links use your preferred URL format.


Make sure that sitemap files only list preferred URLs.


Make sure that the rel canonical link elements on these pages match.


The more you can be consistent, the more likely Google systems will follow your lead and use


those too.


But what happens when a different URL is chosen?


Simply put, it's just the URL that's shown in search.


If our systems pick a different URL as a canonical, it'll rank just the same in search.


In the end, it really just comes down to your preference.


If you have a preference regarding the URLs, then let search engines know unmistakably.


But if not, that's fine too.


And if a different URL happens to be chosen from time to time, that's not going to negatively


affect the site either.


Thank you for watching, and I'll see you again in one of the next episodes of Ask Google


Webmasters.


Googlebot: SEO Mythbusting

A lot of confusion revolves around SEO


Googlebot: SEO Mythbusting

because no one understands how the Googlebot actually works.


Googlebot: SEO Mythbusting

Hello and welcome to another episode of SEO Mythbusting. With me today is Suze Hinton from Microsoft. Suze, what do you do at work and what is your experience with front-end and SEO?


Googlebot: SEO Mythbusting

I'm doing less front-end these days. I focus more on IoT.


Googlebot: SEO Mythbusting

In the time you were a front-end developer?


Googlebot: SEO Mythbusting

I was a front-end developer for 12 or 13 years.


Googlebot: SEO Mythbusting

I got to work on lots of different contexts of front-end development, different websites, things like that.


Googlebot: SEO Mythbusting

Today I wanted to address a bunch of stuff about Googlebot specifically and nerd out about Googlebot.


Googlebot: SEO Mythbusting

That was the side of things that I was the most confused about at the time.


Googlebot: SEO Mythbusting

Googlebot is basically a program that we run that does three things.


Googlebot: SEO Mythbusting

The first thing is it crawls and it indexes.


Googlebot: SEO Mythbusting

And then last but not least, there's another thing that is not really Googlebot anymore.


Googlebot: SEO Mythbusting

That is the ranking bits.


Googlebot: SEO Mythbusting

So we have to basically grab the content from the internet and then we have to figure out what is this content about,


Googlebot: SEO Mythbusting

what is the stuff that we can put out to users looking for these things.


Googlebot: SEO Mythbusting

And then last but not least is which of the many things that we picked for the index is the best thing for this particular query in this particular time.


Googlebot: SEO Mythbusting

But the ranking bit, the last bit where we move things around, that is informed by Googlebot but it's not part of Googlebot.


Googlebot: SEO Mythbusting

Is that because there's this bit in the middle of the indexing, like the Googlebot is responsible for the indexing


Googlebot: SEO Mythbusting

and making sure that that content is useful for the ranking engine to kind of...


Googlebot: SEO Mythbusting

Absolutely.


Googlebot: SEO Mythbusting

You can imagine someone has to, in the library, someone has to figure out what the books are about


Googlebot: SEO Mythbusting

and get the index of the bits in a catalog, the catalog being our index really.


Googlebot: SEO Mythbusting

And then someone else is using that index to make informed decisions and going,


Googlebot: SEO Mythbusting

here, this book is what you're looking for.


Googlebot: SEO Mythbusting

I'm really glad you used that analogy because I worked in a library for four years.


Googlebot: SEO Mythbusting

Oh, so you know much better than I how that works.


Googlebot: SEO Mythbusting

And I was that person.


Googlebot: SEO Mythbusting

People would be like, I want Italian cookbooks and I'm like, well, it's 641.5495.


Googlebot: SEO Mythbusting

You were just like, give it to me.


Googlebot: SEO Mythbusting

So if I would come to you as a librarian and ask a very specific question like,


Googlebot: SEO Mythbusting

so what is the best book on making apple pies really quick,


Googlebot: SEO Mythbusting

would you be able to figure out from the index of books, you probably have lots of cookbooks.


Googlebot: SEO Mythbusting

We did. Yeah, we had a lot.


Googlebot: SEO Mythbusting

But given that I also put lots of books back on the shelf, I knew which ones were popular.


Googlebot: SEO Mythbusting

I have no idea if we can link this back to Googlebot.


Googlebot: SEO Mythbusting

Yeah, it's pretty much so you have the index that probably doesn't really change that much


Googlebot: SEO Mythbusting

unless you add new books to the right. Exactly.


Googlebot: SEO Mythbusting

So you have this index which Googlebot provides you with.


Googlebot: SEO Mythbusting

But then we have the second, the librarian, the second part that basically based on how the interactions


Googlebot: SEO Mythbusting

with the index work, figure out which books to recommend to someone asking for it.


Googlebot: SEO Mythbusting

So that's pretty much the exact same thing there.


Googlebot: SEO Mythbusting

Like someone figures out what goes into the catalog and then someone uses it.


Googlebot: SEO Mythbusting

I love this. This makes sort of sense to me.


Googlebot: SEO Mythbusting

But I guess that's still not necessarily all the answers you need, right?


Googlebot: SEO Mythbusting

Yeah, I just want to know what does it actually do?


Googlebot: SEO Mythbusting

How often does it crawl sites? What does it do when it gets there?


Googlebot: SEO Mythbusting

How is it generally behaving? Does it behave like a web browser?


Googlebot: SEO Mythbusting

That's a really good question.


Googlebot: SEO Mythbusting

Generally speaking, it behaves a little bit like a browser. At least part of it does.


Googlebot: SEO Mythbusting

So the very first step, the crawling bit, is pretty much a browser coming to your page.


Googlebot: SEO Mythbusting

Either because we found a link somewhere or you submitted a site map


Googlebot: SEO Mythbusting

or there's something else that basically fit that into our systems.


Googlebot: SEO Mythbusting

You can use Search Console to give us a hint and ask for reindexing and that triggers a crawl before.


Googlebot: SEO Mythbusting

I've done that before.


Googlebot: SEO Mythbusting

I have re-asked for it to be done again.


Googlebot: SEO Mythbusting

And that is perfectly fine.


Googlebot: SEO Mythbusting

But the problem then obviously is how often do you crawl things


Googlebot: SEO Mythbusting

and how much do you have to crawl and how much can the server bear?


Googlebot: SEO Mythbusting

If you're on the back end side, you know that you have a bunch of load


Googlebot: SEO Mythbusting

and that might not be always the same thing.


Googlebot: SEO Mythbusting

If it's like Black Friday, then the load is probably higher than on any other day.


Googlebot: SEO Mythbusting

So what Googlebot does is it tries to figure out from what we have in the index already


Googlebot: SEO Mythbusting

is that something that looks like we need to check it more often?


Googlebot: SEO Mythbusting

Does that probably change? Is it like a newspaper or something?


Googlebot: SEO Mythbusting

Or is that something like a retail site that does have offerings that change every couple of weeks?


Googlebot: SEO Mythbusting

Or even do not change at all because this is actually the site of a museum that changes very rarely,


Googlebot: SEO Mythbusting

like for the exhibitions maybe, but a few bits and pieces don't change that much.


Googlebot: SEO Mythbusting

So we try to segregate our index data into something that we call daily or fresh


Googlebot: SEO Mythbusting

and that gets crawled relatively frequently.


Googlebot: SEO Mythbusting

And then it becomes less and less frequent as we discover.


Googlebot: SEO Mythbusting

And if it's something that is super spammy or super broken, we might not crawl it as often.


Googlebot: SEO Mythbusting

Or if you specifically tell us, oh, no, do not index this, do not put this in the index,


Googlebot: SEO Mythbusting

this is something that I don't want to show up in the search results,


Googlebot: SEO Mythbusting

then we don't come back every day and check.


Googlebot: SEO Mythbusting

So you might want to use the reindex feature.


Googlebot: SEO Mythbusting

If that changes, you might have a page that you go like, no, this shouldn't be here.


Googlebot: SEO Mythbusting

And then once it has to be there, you want to make sure that we are coming back and indexing again.


Googlebot: SEO Mythbusting

So that's the browser bit. That's the crawler part.


Googlebot: SEO Mythbusting

But then a whole slew of stuff happens in between that happening,


Googlebot: SEO Mythbusting

us fetching the content from your server,


Googlebot: SEO Mythbusting

and the index having the data that is then being served and ranked.


Googlebot: SEO Mythbusting

So the first thing is we have to make sure that we discover if you have any other resources on your page.


Googlebot: SEO Mythbusting

The crawling cycle is very important.


Googlebot: SEO Mythbusting

So what we do is the moment we have some HTML from you,


Googlebot: SEO Mythbusting

we check if we have any links in there or images for that matter or video,


Googlebot: SEO Mythbusting

something that we want to crawl as well.


Googlebot: SEO Mythbusting

And that feeds right back into the crawling mechanism.


Googlebot: SEO Mythbusting

Now, if you have a gigantic retail site, let's say, just hypothetically speaking,


Googlebot: SEO Mythbusting

we can't just crawl all the pages at once, both for our resource constraints,


Googlebot: SEO Mythbusting

but also we don't want to overwhelm your service.


Googlebot: SEO Mythbusting

So we basically try to figure out how much we can put,


Googlebot: SEO Mythbusting

how much strain we can put on your service,


Googlebot: SEO Mythbusting

and how much resources we've got available as well.


Googlebot: SEO Mythbusting

And that's called the crawl budget oftentimes, but it's pretty tricky to determine.


Googlebot: SEO Mythbusting

So one thing that we do is we crawl a little bit and then basically ramp it up.


Googlebot: SEO Mythbusting

And when we start seeing errors, we ramp it down a little bit more.


Googlebot: SEO Mythbusting

So like, oops, sorry for that.


Googlebot: SEO Mythbusting

We are not, oh, ah.


Googlebot: SEO Mythbusting

So whenever your server serves us 500 errors,


Googlebot: SEO Mythbusting

there are certain tools in Search Console that allow you to say,


Googlebot: SEO Mythbusting

like, hey, can you maybe chill out a little bit?


Googlebot: SEO Mythbusting

But generally, we don't try to get all of it at once and then ramp down.


Googlebot: SEO Mythbusting

We are trying to carefully ramp up, ramp down again, ramp up again,


Googlebot: SEO Mythbusting

ramp down again, so it fluctuates a little bit.


Googlebot: SEO Mythbusting

There's a lot more detail in there than I was even expecting.


Googlebot: SEO Mythbusting

I didn't even know that, I guess I never considered that a Googlebot


Googlebot: SEO Mythbusting

sort of crawling event could put strain on somebody's website.


Googlebot: SEO Mythbusting

That sounds like it's a lot more common than I even thought it would be.


Googlebot: SEO Mythbusting

It does happen, especially if we discover, say,


Googlebot: SEO Mythbusting

a page that has lots of links to sub-pages, then all of these go into the crawling queue.


Googlebot: SEO Mythbusting

And then you might, like, these have links to,


Googlebot: SEO Mythbusting

let's say you have like 30 different categories of stuff,


Googlebot: SEO Mythbusting

and each of these have a few thousand products


Googlebot: SEO Mythbusting

and then a few thousand pages of products.


Googlebot: SEO Mythbusting

So we might go like, oh, cool.


Googlebot: SEO Mythbusting

Crawl, crawl, crawl, crawl, crawl, crawl, crawl, crawl.


Googlebot: SEO Mythbusting

And then we might crawl like a few hundred thousand pages.


Googlebot: SEO Mythbusting

And if we don't spread that out a little bit, so it's a weird balance, right?


Googlebot: SEO Mythbusting

On one hand, if you add a new product, you want that to be surfaced in search


Googlebot: SEO Mythbusting

as quickly as possible.


Googlebot: SEO Mythbusting

On the other hand, you don't want us to take all the bandwidth that your server offers.


Googlebot: SEO Mythbusting

I mean, cloud computing makes that a little less scary, I guess.


Googlebot: SEO Mythbusting

But I remember the days, I'm not sure if you remember the days


Googlebot: SEO Mythbusting

where you had to call someone and they ask you to send a form or fax a form,


Googlebot: SEO Mythbusting

and then two weeks later you get the confirmation letter that your server has been started.


Googlebot: SEO Mythbusting

Yes, I remember the days when we would have to call


Googlebot: SEO Mythbusting

and then we would basically pay $200 to have a human go down the aisles


Googlebot: SEO Mythbusting

and push the physical reset button on the server.


Googlebot: SEO Mythbusting

Those times were a lot trickier.


Googlebot: SEO Mythbusting

And then imagine you basically renting five servers somewhere in the data center


Googlebot: SEO Mythbusting

and that taking a week, and then we come and scoop up all your bandwidth


Googlebot: SEO Mythbusting

and you're like, great, we're all fine today because Google has its crawl day.


Googlebot: SEO Mythbusting

That's not what we want to have.


Googlebot: SEO Mythbusting

But nowadays it's more of a hack-a-do's kind of moment when you get hit.


Googlebot: SEO Mythbusting

Exactly.


Googlebot: SEO Mythbusting

So I feel like you're much more considerate.


Googlebot: SEO Mythbusting

Yeah, we try to not overwhelm anyone and we respect the robots' TXT,


Googlebot: SEO Mythbusting

so that works within the crawl step as well.


Googlebot: SEO Mythbusting

And once we have the content, we can't put strain on your infrastructure anymore,


Googlebot: SEO Mythbusting

so that's fantastic, but modern web apps being mostly JavaScript driven,


Googlebot: SEO Mythbusting

we then put that in a queue and then once we have the resources to render it,


Googlebot: SEO Mythbusting

we actually use another headless browser kind of thing,


Googlebot: SEO Mythbusting

we call that the web rendering service,


Googlebot: SEO Mythbusting

then there's other crawlers as well that might not have the capacity


Googlebot: SEO Mythbusting

or the need to run JavaScript.


Googlebot: SEO Mythbusting

It's like social media bots, for instance, they come and look for metadata.


Googlebot: SEO Mythbusting

If that meta tag is coming in with JavaScript, you usually have a bad time


Googlebot: SEO Mythbusting

and they're just like, sorry.


Googlebot: SEO Mythbusting

Yeah, so that's always been a big myth,


Googlebot: SEO Mythbusting

and I remember when single page applications or SPAs really came into vogue,


Googlebot: SEO Mythbusting

a lot of people were really concerned.


Googlebot: SEO Mythbusting

There's a lot of FUD around, well, if crawlers in general don't execute JavaScript,


Googlebot: SEO Mythbusting

then they're going to see a blank page and how do you get around that?


Googlebot: SEO Mythbusting

So contextually within Googlebot, it sounds like Googlebot executes JavaScript,


Googlebot: SEO Mythbusting

even if it does do it at a later point.


Googlebot: SEO Mythbusting

So that's good.


Googlebot: SEO Mythbusting

But is there anything that people need to be aware of beyond just,


Googlebot: SEO Mythbusting

oh, well, it'll just run it and then it'll see exactly the same thing as a human


Googlebot: SEO Mythbusting

with a phone or a desktop would see?


Googlebot: SEO Mythbusting

There's a bunch of things that you need to be aware of.


Googlebot: SEO Mythbusting

So the most important thing is, again, as you said, it's deferred.


Googlebot: SEO Mythbusting

It happens at a later point.


Googlebot: SEO Mythbusting

So if you want us to crawl your stuff as quickly as possible,


Googlebot: SEO Mythbusting

that also means we have to wait to find these links that JavaScript injects.


Googlebot: SEO Mythbusting

Basically we crawl, we have to wait until JavaScript is executed,


Googlebot: SEO Mythbusting

then we get the rendered HTML, and then we find the links.


Googlebot: SEO Mythbusting

So the nice little short loop that finds these links relatively quickly


Googlebot: SEO Mythbusting

right after crawling will not work.


Googlebot: SEO Mythbusting

So we will only see the links after we render,


Googlebot: SEO Mythbusting

and this rendering can take a while because the web is surprisingly big.


Googlebot: SEO Mythbusting

Yeah, just a little bit.


Googlebot: SEO Mythbusting

I was like 130 trillion docs in 2016.


Googlebot: SEO Mythbusting

Oh, so there's way more now.


Googlebot: SEO Mythbusting

There's way more than that.


Googlebot: SEO Mythbusting

So robots.txt is very effective at being able to tell bots how to do a certain thing.


Googlebot: SEO Mythbusting

But in this scenario, how do you tell that it's Googlebot visiting your site?


Googlebot: SEO Mythbusting

That's a good question.


Googlebot: SEO Mythbusting

So as we are basically using a browser in two steps,


Googlebot: SEO Mythbusting

one is the crawling and one is the actual rendering,


Googlebot: SEO Mythbusting

both of these moments we do give you the user agent header.


Googlebot: SEO Mythbusting

But basically there's the string list, literally the string Googlebot in there.


Googlebot: SEO Mythbusting

Oh, that's so straightforward.


Googlebot: SEO Mythbusting

Yes, and you can actually use that to help with your SPA performance as well.


Googlebot: SEO Mythbusting

So as you can detect on the server side, oh, this is a Googlebot user agent requesting,


Googlebot: SEO Mythbusting

you might consider sending us a pre-rendered static HTML version,


Googlebot: SEO Mythbusting

and you can do the same thing for the others.


Googlebot: SEO Mythbusting

Like all the other search engines and social media bots


Googlebot: SEO Mythbusting

have a specific string saying that they are robots.


Googlebot: SEO Mythbusting

So you can then basically go like, oh, in that case,


Googlebot: SEO Mythbusting

I'm not giving you the real deal, the single page app.


Googlebot: SEO Mythbusting

I'm giving you this HTML that we pre-rendered for you.


Googlebot: SEO Mythbusting

It's called dynamic rendering. We have docs on that as well.


Googlebot: SEO Mythbusting

The one thing that still doesn't quite make sense to me is


Googlebot: SEO Mythbusting

does the Googlebot kind of have different contexts?


Googlebot: SEO Mythbusting

Does it sometimes pretend that it's, like, I think of it as this little mythical creature


Googlebot: SEO Mythbusting

that's pretending to do certain things.


Googlebot: SEO Mythbusting

So does it pretend to be on a mobile and a desktop?


Googlebot: SEO Mythbusting

Are there different sort of, I guess, user agents even though it still says Googlebot?


Googlebot: SEO Mythbusting

And can you differentiate between them?


Googlebot: SEO Mythbusting

You are asking great questions because, yes, we have different user agents.


Googlebot: SEO Mythbusting

So I'm not sure if you heard about mobile-first indexing being rolled out and happening.


Googlebot: SEO Mythbusting

I've heard that it's going to affect how you're ranked potentially.


Googlebot: SEO Mythbusting

That as well.


Googlebot: SEO Mythbusting

I don't know if that's a rumor or not.


Googlebot: SEO Mythbusting

That's two different things that get conflated so often.


Googlebot: SEO Mythbusting

So mobile-first indexing is about us discovering your content


Googlebot: SEO Mythbusting

using a mobile user agent and a mobile viewport.


Googlebot: SEO Mythbusting

So we are using mobile user agents, and the user agent string says something about Android


Googlebot: SEO Mythbusting

and the name, and then you're like, aha, so this is the mobile Googlebot.


Googlebot: SEO Mythbusting

We have documentation on there.


Googlebot: SEO Mythbusting

There's literally a Help Center article that lists all these things.


Googlebot: SEO Mythbusting

So we try to index mobile content to make sure that we have something nice to serve


Googlebot: SEO Mythbusting

for people who are on mobile.


Googlebot: SEO Mythbusting

But we're not pretending like random user agents or anything.


Googlebot: SEO Mythbusting

We stick to the user agent strings that we have documented as well.


Googlebot: SEO Mythbusting

And that's mobile-first indexing where we try to get your mobile content into the index


Googlebot: SEO Mythbusting

rather than the desktop content.