across your whole site and apply that appropriately.


English Google SEO office-hours from October 16, 2020

I suspect we'll have a little bit more information on this


as we get closer to announcing or kind of closer to date


when we start using Core Web Vitals in search.


But it is something you can look at already


a little bit in Search Console.


There is a Core Web Vitals report there.


And if you drill down to individual issues,


you'll also see this URL affects so many similar URLs.


And based on that, you can already kind of tell,


oh, is Google able to figure out that my forum is grouped


together, or is it not able to figure out


that these belong together?


OK.


And now, like, a really long question from David,


who doesn't like Pinterest.


Is the indexing API going to open up to general URLs?


I don't know.


My guess is it'll be tricky.


Because we see a lot of abuse with all of the


submit to indexing features that we have,


and I don't know if it would make sense


to open up yet another channel that we


have to kind of figure out how to deal with more abuse on.


Is Discover going to be in the API?


I hope so.


I will take this question as a nudge and ask the team again.


Can we have fetch and render API that has a static IP


or a unique user agent?


I don't think so.


Mostly because the fetch and render feature or the inspect


URL feature in Search Console is meant


to reflect what Googlebot would actually see


and is not meant to be used kind of as a single request kind


of fine tuning for Googlebot in the sense


that you have a specific IP address or a specific user


agent there.


I don't think that's something we'd be able to do.


A suggested feature, Google parser breaker.


Is there something in the code you can


fetch that can break your parser?


For example, a few years ago, an image tag in the head


would break parsing and therefore terminate


the head prematurely.


We want to make sure that our terrible HTML isn't


causing issues.


I guess the easy solution is not to make terrible HTML.


It sounds like you're pretty advanced,


so you can figure that out.


But the part with individual elements in the head,


that still applies in the sense that when we render a page,


similar to a browser, if there are elements


in the head of the page that belong to the body,


then we will open up the body and treat


the rest of that section essentially


as part of the body.


And there are specific elements that we really


need to be able to find in the head of the page


so that we can take them seriously.


That includes things like the rel canonical, the robots meta


tags, and hreflang links, for example.


So if you have elements on top of the head of the page that


essentially break the head in the DOM,


that's something that could cause


problems for those elements.


And I don't think you explicitly see that in fetch and render


or inspect URL in Search Console because we just say, well, oh,


it looks like the webmaster wanted to start


the head of the page here.


We will just treat the rest like it is,


or treat the rest like it is part of the body.


So we wouldn't necessarily see that as a bug or something


that the site is doing wrong, but rather we're


just trying to deal with the broken HTML that


is out on the web.


Extend the API.


Some of us are using screen scraping techniques


to extract a lot of data that isn't available in the API.


Why not make it available and monetize it?


So I don't think we'd want to monetize the API.


That's kind of the first step there in the sense


that everything around money is really hard at Google,


especially when dealing with customers and all of that.


So I don't think that would be something that we'd monetize.


I do think it would be really nice to extend the API,


but it's really hard to encourage the product


leadership to start doing more on the API side.


So if you have any arguments that you think


would be really useful in terms of this


is how we can make search better if we were able to pull data


through the API, that would be really useful for us.


So feel free to send that our way,


and we'll pass that on to the team.


Bring back advanced search operators.


I don't really see this happening,


because it's something that really most sites or most


users don't actually use.


And there are really a lot of work


to maintain, especially across the ever-changing


infrastructure in Google Search.


Maintaining some of these features


that not a lot of people use is really expensive.


Bring back site removals.


Pinterest is a parking meter of the internet.


No one likes to see them.


I don't know.


Bring back site removal.


It sounds like you want to remove other people's sites.


I'm sure the SEO of Pinterest would be kind of against that.


I don't use Pinterest personally,


but I do see a lot of people getting value out of that.


There is a lot of good content there.


So I don't know.


I think, in general, with regards to site removals,


especially on a personal level where you could say,


I don't want to see the site anymore,


I think that's an interesting idea.


I think implementation is just really, really hard


with all of the different ways that sites can


be visible in search nowadays.


Is there any preferred time zone for the last modification date?


Yes.


Well, in the last modification date,


you should be specifying a time zone.


It doesn't matter which time zone you use,


but it should have a time zone.


I think that's part of the kind of date time standard that's


used in the XML file.


How sensitive is Google when it is about to ignore


the last modification date?


In general, we use it as a guide to understand


when pages have changed.


And it's not so much that it has to be exact,


but rather we have to be able to understand, oh, this page has


changed since the last time we looked at it,


so we should look at it again.


And from that point of view, it's


important for us to be able to roughly trust


the last modification date.


In particular, the thing that we sometimes see


is that people generate sitemap files,


and they just use the current date for the last modification


date for all URLs.


And that's something that's obviously wrong.


When we look at the sitemap file,


and there are 10,000 URLs in there,


and they were all updated in the last minute,


that's probably you're calculating


the last modification date wrong.


And from our point of view, it's not so much


that we want to penalize the website for doing that.


It's just, well, there isn't a lot of signal,


a lot of useful information in that sitemap file for us


to work on, other than perhaps there are some new URLs


that we haven't seen at all there before.


So in a case like that, if you just


need to always have the same last modification date,


or if you can't specify it at all,


we can at least pick up the new URLs,


but we wouldn't be able to know when the old URLs actually


changed.


Let's see.


Question goes on in, oops, I think


slightly different direction.


We update our meta robots frequently, index and no index,


and two months ago we implemented last modification


on product pages, which are back in stock in the last seven


days, and marked them as indexed.


But we didn't see any impact on submitted URL marked no index.


I manually checked some of the last modification URLs.


Google never seems to follow them.


In general, I think this fluctuation


between indexed and non-indexed is something that


can throw us off a little bit.


Because if we see a page that is no index for a longer


period of time, we will assume that this


is kind of like a 404 page, and we don't have


to crawl it that frequently.


So that's something where probably what


is happening there is that we see these pages as no index,


and we decide not to crawl them as frequently anymore,


regardless of what you submit in the sitemap file.


So that's something where kind of fluctuating


with the meta no index is probably counterproductive here.


If you really want those pages to be indexed every now


and then.


So, John, hello?


Yeah.


So last word that is not useful in that case,


where we've quickly changed our back-end rules?


What I would try to do in a case like that


is maybe set up a page that you can persistently


maintain and link from there to the individual products


that you want to kind of have listed or not listed.


And then maybe focus more on that persistent page


rather than on the individual products that


kind of come in and go out.


OK, and you mentioned that the time zone.


Where do we have to measure that what time zone we are using?


So I assume you're talking about the sitemap file, right?


Yes, yes, yes.


Yes, OK.


So in the sitemap file for the last modification date,


the date time uses a specific standard.


And in that standard, it has a time zone attached to the end.


If you have a z at the end of the date time,


then that means it's, I think, UTC time.


But you can also specify different time zones there.


OK, so at the end of every last board tag,


we have to measure the time zone.


Yes, well, I would make sure that you're doing it


in the right standard.


So I would check the sitemap file documentation


and look up the date time standard that we use there.


It's like RFC and some number.


And usually there is a Wikipedia page with a lot of examples


where you see with time zone, without with the UTC or the z


at the end so that you can compare that.


Thank you.


Sure.


OK, in light of the recent indexing issues,


any update on the last 45% affected canonical URLs


still hasn't been restored.


Is it OK to submit affected URLs manually in Search Console?


Can you shed some light on what happened here?


What is Google doing to prevent this in the future?


I don't have any big updates on this at the moment.


So I haven't been following up on that.


I think Danny has mostly been keeping track of that


and pushing things along.


One thing I can mention here is you


can't use the inspect URL and submit to indexing feature


in Search Console because at the moment that's


kind of in maintenance.


So that's kind of tricky there.


I don't know for individual sites


how strongly you would still see this because my understanding


is usually these kind of issues are


resolved for the visible URLs of pretty much all sites


fairly quickly within a day or so.


But if you're still seeing a really strong effect from this,


maybe drop me a note on Twitter and I


can take a look with the team to see if maybe there's


something else that's happening with your site in particular.


I was wondering how Google Search was evaluating the ad


experience as a ranking factor.


It's easy to estimate the readability of a font size,


but I have trouble wrapping my mind


around how much a huge sticky video on mobile


would impact ranking.


Let's say 30% to comply better ad standard,


but still highly deceptive, disruptive experience.


So I had to double check, but as far as I know,


we don't use the ad experience report as a ranking factor


in search.


But rather, this is something that is specifically in Chrome


where if we see that a site does not comply with this standard


and we can confirm that, then Chrome will automatically


not display those ads.


So it's not something that we would essentially really


need to use in search because Chrome would handle that


anyway.


But it is something where probably the effects


that you would see in search would be based on things


like the above-the-fold content or the, I forgot what we used,


not the intrusive interstitials, but kind of the,


if you have a lot of ads on top part of your page,


that's probably where you would see an effect from search.


I forgot what the name was we had for that,


but it's one of the, I don't know,


maybe two, three years old things


where we launched something around kind of making sure


that the top part of the page is actually content.


I don't know how the font readability would


fit into that.


My feeling is the font size would be more something


that would affect whether or not we would see a page as being


mobile-friendly.


And that's something that would apply for the mobile search


results, but not for the general search results.


So I don't think I really answered your question there,


but I'm not 100% sure what direction I should go there.


I wanted to know how we can stop paginated pages from ranking.


I want them to index, but not to rank,


as I always want the first page to rank and not


page two or page three.


But I can't even add no index, as the listings mentioned


in those second and third are important.


If anyone searches for a specific name,


I want the listings mentioned on page two and three also


to rank.


But I don't want other pages rather than page one to rank.


So that's, I think, a tricky situation


in terms of you can't really specify where you want


your pages to rank or whether or not


they should be shown in the search results


if you want them to be indexed.


So if they're indexed, then our systems


will try to figure out how to rank them appropriately.


So that's kind of, you're in that situation


where you kind of want both things, both have it indexed,


but not have it counted kind of thing.


And well, not have it indexed and have it counted.


But that's not really something that's fully supported there.


What we generally recommend when it comes to pagination


is to allow your paginated pages to be indexed.


Block your filtered pages from being indexed,


because these are essentially links to the same products


as you have before, unless there are individual kind


of filtered versions that you would consider something more


like a category page.


With regards to paginated pages, one way


you can make sure that we focus more on the first page


is to link incrementally between the pages


of your paginated set.


So instead of linking from page 1 to page 2, 3, 4, 5, 6, 7,


and all of the other ones, link from page 1 to page 2,


and from page 2 to page 3, page 3 to page 4,


so that when we look at that, we see


that the first page is highly mentioned within your website.


So clearly, this must be a very important page.


And the other ones are incrementally less important,


because they're further and further away


from your home page.


And by doing it like that, you can have those pages be indexed.


But we will understand that these are not really


that relevant for your site.


And usually, that does end up with us


focusing on the first page, especially


if someone is searching for a category.


Then we can focus on that first page


and say, this is clearly the best page for that category.


The site map we submitted from years ago


keeps updating the submitted date in Search Console.


It looks like it's getting resubmitted,


but no one on the account has resubmitted it.


This happened a couple of times a month.


What might be happening there?


So it's hard to say without knowing


which site you're looking at.


But one of the things that does happen is that, or can happen,


is that there is the anonymous submission feature for site map


files, where you can just ping a specific URL.


And Google will see that as a resubmission of the site map


file and try to reprocess it.


And usually, that is used by CMSs or site maps


plugins, those kind of things, where when you update something,


it automatically pings that URL and lets


Google and the other search engines know something


changed in the site map file.


And that would be treated as a resubmission from our side.


And we would probably track that or show that in Search Console


as well.


So it's not so much that someone is manually going in there


and clicking that button, but it could very well


be that your CMS is doing that for you


and kind of taking some of that work out of your hands


and saying, well, we can deal with this automatically.


Theoretically, it could also be that someone else is


resubmitting your site map file because it doesn't


need to be tied to your account to kind of ping that update


URL.


I imagine it's very unlikely that someone else is randomly


resubmitting your site map file like this


because there's really no reason to do that.


It doesn't help them.


It doesn't really change anything from your side.


So from that point of view, my guess


is it's probably just your CMS that is doing this for you.


Rendering check question.


We have a website single page application


where when we check the index HTML and the URL inspection


tool, it looks fine.


Rendering and mobile friendly test looks good as well.


But when we test the live URL, the HTML is minimal


and the screenshot is blank.


What might be the reason for this discrepancy?


So my guess, just based on this question


without looking at the URLs, is that we


have trouble rendering the page at a high speed.


So when it comes to Google Search, we're very patient.


And we render pages.


Essentially, if they take a little bit longer, that's fine.


If some of the embedded content isn't available yet,


we will fetch that individually and then


use that for rendering.


But with the testing tools, we want


to give you an answer as quickly as possible.


And when we see someone is using one of these testing tools,


we set the timeouts a little bit more aggressively just so that


we can give people an answer.


And that means that if you're using these tools to look


at the screenshot or look at how things are rendered,


it's very possible that some of the embedded content,


especially if you're pulling things from an API


or from your back end, that they're timing out


for that specific test.


On the one hand, if it works in Search,


then you're probably fine because Search,


like I mentioned, is a little bit more patient.


On the other hand, it makes it a little bit harder to debug.


So I would try to look into how much embedded content is


actually needed for your pages to load.


You can look at a waterfall diagram


in things like webpagetest.org or in Chrome,


developer tools directly, and make a rough estimation of,


is this something that I can improve?


Can I maybe fold some of these files together?


So instead of 300, 400 requests, it just


takes 100 requests to render this page.


And if so, then that can make it easier for you to debug.


And it does make it more likely that we won't accidentally


run into issues when we render for Search as well.


And of course, usually it makes things faster for users too,


which is a nice side effect.


So my recommendation there would be not


to panic if it's working in Search,


but still to kind of look into ways that you


can improve that over time.


Is it recommended to use keywords


as it is in the content to get the best results in Google?


Does a crawler use the fundamentals of AI


to make a combination of keywords


that are closely related with the content


and then rank them?


We want to know how a crawler picks up and makes


combinations of keywords to be ranked on Google.


So we don't kind of make up the keywords


and say this page will rank for these keywords,


but rather we take the queries and we


try to find the best matching pages for that.


So one kind of the simplest approach that we take


is we split a page into words and we


store all of the words in an inverted index


so that when we know that someone


is looking for this combination of words,


then we can find all of the documents


that have those words in them.


And then based on that list of documents,


we can kind of reorder that list and say,


well, this is the right order that we should


be showing these to people.


So it's not so much a matter of Google reading the content


and then deciding which keywords to use,


but rather we getting the keywords


and then deciding which document to actually show


in the search results.


And I think the fundamentals of AI


is something that always comes up because we talk about this


all the time as well, especially in kind of the more marketing


and user side of search where we use things like BERT


to improve the quality of our search results, which


is ultimately machine learning, artificial intelligence


algorithm.


And for a large part, these algorithms


are there to try to help deal with cases where we don't


really understand what the user is searching for.


So we still see about 15% of all searches


are completely new every day.


And these aren't things that we can prepare for.


And we have to kind of estimate what is the user actually


searching for.


Are there any acronyms in there?


Are there synonyms that we can use?


Is there like singular or plural that we can figure out?


And that's something we, for the longest time,


we used to do more algorithmically and kind


of make hard rules where we say, oh, if there's an S,


maybe it's a plural.


But that doesn't apply to all languages


and all kinds of words.


So for that, we use a lot of machine learning


to really understand better what is it


that people are actually looking for instead of just


matching individually the individual words in a query.


SPEAKER 2 Hey, John.


Sorry for interrupting.


Go for it.


Quick question.


Did you have the chance, like just following up


on our last discussion two weeks ago,


did you have the chance to look into the side


that they shared on the chat, the side that


had a drop of about 40% and that went recently


through a light redesign?


I'm not sure if you know who I'm referring to.


SPEAKER 1 I don't remember which one.


SPEAKER 2 It was a digital trends website.


Yeah, I think I passed that on to the team.


Yeah, I don't know if I heard anything back there, though.


So basically, because we are still working into that,


and one question that we have, because we are still,


and I think that Ed's experience question


that you had on the chat, and I think


you were trying to mention the page layout algorithm.


Oh, yeah.


Yeah.


So the page layout.


So how long?


You should answer these questions.


So how long does it take to recover from a page layout


algorithm if you've been impacted by it?


I think for a lot of these quality algorithms,


I don't know if we have a specific time where


we can say that it's been resolved now.


It depends a little bit on, so I don't know specifically


with the page layout algorithm, but with a lot of these,


we do look at it more on a broader level for a website,


where we say, well, we can't test the page layout algorithm


we can't test every page individually for this,


but we've seen a large part of the pages for the site


have this issue.


And in cases like that, it is something


where we have to first understand,


well, a large part of this site doesn't have this issue


anymore.


And that means we kind of have to reprocess


the larger part of the site, which


is something that can easily take a couple of months.


So if you're making changes like that


and you're essentially affecting the quality


of your pages overall, then that is something


that I would expect is not going to happen overnight,


is not going to happen within a week.


It's more a matter of, I don't know, two, three, four,


five months maybe, where we really understand,


well, the site has significantly changed.


AUDIENCE 1


Yeah.


So could it be related to having an ad that is about,


let's say, 30%, and it appears above default?


Could that trigger the page layout algorithm?


We don't have any specific percentages


that we would talk about there with the page layout algorithm.


I could imagine that if you're talking about something that


is really a bigger part of the page,


that it could be affecting there.


But we don't have specific numbers where we'd say,


oh, the ad has to be maximum this many pixels


or this many percent of the viewport.


That's not something we have.


One other question.


And this is just trying to find if we miss something.


What could be some reasons that would cause a drop of traffic,


35% to 40%, across subdomains, not just the main domain,


but across subdomains as well?


That could be that could cause this.


I don't know.


Lots of things.


It's really hard to say because on the one hand,


there are things like our understanding of the quality,


which can change.


And that is something that could affect


the larger part of the website.


But it's really hard to say in general.


Because on the one hand, we make algorithm changes.


The rest of the web changes all the time.


And all of that plays in together.


Even if you don't make any changes on your website,


then that can, at some point, cause either a subtle decline


or a really strong drop.


Yeah.


When you're talking about quality issues,


would it cause quality issues if, let's say, for example,


we are updating the content frequently


but not significantly?


Could that be an issue?


Could Google take some issues into doing that?


I think that's perfectly fine.


I don't think that that would be generally a problem.


Usually, what I see with regards to quality issues


is really when you look at the website overall,


and you can tell that users aren't trusting the content as


much anymore, or they're just not sure about the content


anymore, then that's something where


I'd say there are more clear-cut quality issues.


Just individual updates of a site,


or making a lot of changes on a site,


or making a few changes on a site,


that's usually more the usual noise


that is out there on the web.


And that's something that some sites make a lot of changes.


Some sites don't make a lot of changes.


That's all fine.


Yeah.


And last question.


Yahoo syndicates a lot of our content.


In the majority of cases, our content


ranks first because Google recognizes


as the source of the content.


But Yahoo doesn't canonicalize its content


that it's syndicating from us.


And in some rare cases, they manage


to rank above us for some content that is created by us.


Could that be a cause for a drop like this?


No.


I don't think so.


I think that's something where, ultimately, the issue that you


will see is that these syndication sites essentially


rank as well in the search results.


And sometimes they rank above your content


because just for whatever reasons.


But it's not a sign that we would


say your site is lower quality because it's syndicating


content to other sites.


I think, in general, that would almost


be the opposite, that if other sites are wanting


to syndicate your content, then that's kind of a good sign.


Yeah, we are circling back always to us being affected


by the page layout algorithm.


And we made changes.


And we now understand that it will


take some months for those changes to be recognized.


But how can we make sure that those


are the necessary changes?


What else are we missing?


Or how can we check to make sure that after two or three months,


we don't realize that, oh, we missed this thing


and we should have changed this as well?


I don't think you can, unfortunately.


I don't think there is a, I mean,


there definitely isn't a testing tool that will tell you


what specifically you need to change with regards


to your website's quality.


The best I would recommend is to get a lot of people's opinions


on it.


So that could be something where you could go to the Webmaster


Health Forum and say, from a quality point of view,


what else should we be taking into account?


And sometimes you get a lot of feedback


that is less relevant, maybe small details.


But sometimes you also get some feedback on issues


that you've been trying to push out of your mind for a while


and that you actually should focus on a bit more.


Thanks a lot.


Sure.


All right.


Maybe I'll just open it up to more questions from any of you.


Hi, John.


I have a question related to the merging of the websites


that we talked about earlier.


So we're merging two pretty big websites, not for SEO reasons,


for branding purposes.


And there are some checklists out there.


But I want to ask you, what is the most common error


that people do when they're merging big websites?


What do they forget?


What would you recommend me to really focus on?


Yeah.


I don't know if we really have a list of the common errors


that people make.


The one thing I would recommend doing


is really tracking all of the things that you're doing,


really making sure that you have a clear list of all of the URLs


ahead beforehand and where they should be going so that you can


check this afterwards as well, that all of the redirects


are in place, that all of the internal links


are working appropriately, and just making sure


that you have all of these technical details really nailed


down.


And it is something that, especially if you're


doing it all at one time, it's kind of nerve-wracking.


But I think the best you can do is really just make sure


that all of those small details are all covered properly.


OK, thank you.


And then last question.


Developer seems to not want to do a full 301, just a 302.


How big of a difference will it make to only do a 302,


not a full permanent redirect?


So my guess is it will take a little bit longer


for the canonicalization to kick in


and to focus on that new URL that you have there.


But ultimately, I think in the long run,


it won't play a big difference.


But if you're moving permanently,


then it feels like a 301 would be the right thing


to do in any case.


I was hoping that you would give me more of a, yes, you


should do that, so I can go and tell the developers


that that was the thing.


Yeah.


I mean, it is what we recommend.


It's something like, if you don't do it,


and a lot of sites get it wrong, and we


have to deal with that as well.


But if you really want to make sure


that everything is perfect, then a 301 is the right way to go.


Thank you.


That was all.


Hey, John.


Can I ask a question?


Sure.


OK.


So we have a software application website.


And basically, it's a product website.


So recently, we decided that we should


create a news portal kind of section


so that we can cover industry news.


And we would be writing like four to five articles


every day.


Now, the situation is something like that.


Because our website is a product website,


and the team is confused whether we


should create a separate section for news on the existing


website, or should we just create a new domain


for the news section?


Because the confusion is that if we


go with the existing website, what are the chances


that Google might consider our product website


as a news portal only after a certain period of time?


Or vice versa, like it's a product website,


so Google might consider that it's not a news website.


So why to rank the news in the news column?


So what do you say that we should do here?


I would recommend keeping it within your existing website.


By keeping it within your existing website,


you're automatically kind of making sure


that it has all of the weight that it can get


from the rest of your website.


So that's something where if you split it out


into a separate domain, then essentially you're


starting with a new website, and you have to kind of work


to get that well-known out on the web.


Whereas if you start with your existing site,


then you already have kind of something


to work with with regards to Google seeing it


as a news site or a product site.


In general, that's not something I would worry about,


especially if you can structure the URLs in a way that


is clear, this is news, and this is like the rest of your site,


for example.


So if you have a clear URL structure there,


I definitely don't see any problem there.


Yeah, with existing approach, we decided


to create a separate category for the news section.


And we would be using news schema


so that Google has no doubt.


But it's like, Google should not be confused.


It's a news website or a product website.


So that was confusing.


Yeah, thank you for the giving.


Cool.


Now, any other questions?


What can I help with?


Nothing.


Wow, OK.


OK.


I think we're pretty much on time, so that's also fine.


If everything otherwise is kind of covered.


We have news publishers for four years.


And every week, we do our news coverage,


which is linked back by other publishers.


But we never appear in the top stories.


But they appear.


What should we do?


It's hard to say, because for the top stories,


it's not so much that there is a specific meta


tag or something kind of technical


that you need to do there.


But rather, it's an organic search feature.


And depending on how our search algorithms kind of look


at that for your site, that can go one way


or it can go the other way.


So that's really kind of tricky to say.


One thing you could do is maybe start a thread in the Webmaster


Help Forum so that someone can take a look at your details


and see if there's something specific


that they can point out.


But in many cases, there is nothing specific, kind


of technical to point out and say,


you need to do this specifically to be visible in the top story


section.


OK.


Cool.


Maybe we can take a break here.


I need to jump off and do some other things this time.


Thank you all for joining.


It's been great having you all here.


Thanks for all of your questions.


And like I mentioned, if there's something


that's still kind of stuck or that you need some help on,


feel free to ping me on Twitter and let me know about that.


And otherwise, yeah, thanks for all your questions.


And have a great weekend, everyone.


Thanks, John.


Cheers.


Bye, everyone.


Thank you.


Bye.


Bye, John.


Bye-bye.


Googlebot and Web Hosting (WMConf MTV '19)

Hi, everyone.


My name is Jin, and I'm going to talk a little bit


about Googlebot and web hosting.


As many of you probably know, before Google


can index a page and serve it to the users,


we had to crawl the page and render it.


This is done by a system known as Googlebot.


Googlebot does a lot of things.


For example, we follow out links and discover new content.


We predict a duplicate content behind different URLs


so we can save crawl bandwidth.


We also re-crawl URLs to keep our index fresh.


But for today, I'm going to focus on some specific things.


First, I will share some of our observations


how web hosting has evolved over the years.


Next, I will talk a little bit about robots.txt fetch


and also our crawling rate.


This is a graph that shows the HTTP server


popularity over the years.


Popularity here is defined as the percentage


of websites that are served by a particular HTTP server.


We can see that Apache started out being very popular.


About 10 years ago, more than 70% of the websites


are served by Apache.


But this number has been dropping over time.


Now Apache is serving about 35% of the websites.


On the other hand, Nginx has become really popular.


10 years ago, it was serving very few websites.


Now it's serving about 40% of the websites.


Microsoft IaaS started about 22% of the sites.


Now the number is about 15%.


There are a few other HTTP servers.


I'm not going into the details.


I should note that this kind of data


is actually externally available.


There are sources of data externally you


can find about HTTP server market share or something.


So the numbers will differ a little bit


with what we see here.


But we find that the general trend is consistent.


This graph shows from Google point of view


what's the relative traffic between HTTP and HTTPS.


10 years ago, almost all of our crawling traffic is HTTP.


But in the last several years, HTTPS has really picked up.


Today, about 75% of our crawling traffic is HTTPS.


HTTP only accounts for about 25% of the traffic.


Now on the one hand, we see that the web


has become really more secure.


But on the other hand, we can see


there's also still quite some room for HTTPS to grow.


This graph shows the average URL download time


seen by the Googlebot.


So we can see about 10 years ago,


to download a web page on average,


it takes about 800 milliseconds.


Today, the number is about 500 milliseconds.


Of course, these numbers are specific to Googlebot.


But I think the general trend seems


to indicate that networks are getting faster,


servers are getting faster.


That's good news because in the same amount of time,


Googlebot can download more web pages.


Now let's switch gears and talk a little bit


about robots.txt, which has been mentioned


in previous talks.


So many of you probably know that robots.txt


is a way for webmasters to specify access to their site.


Here you see an example robots.txt snippet.


This one actually shows what URLs


Googlebot can crawl from this site


and what URLs we are not allowed to crawl.


Although robots.txt has been used in industry


for more than 25 years, there has never


been really a standard for it.


This creates some issues because the same robots.txt


may be interpreted differently by different search engines.


That's why in the past year, my team and the webmasters team


here at Google, we worked together


and we also worked with Bing to propose a draft standard


to IETF.


Once this becomes a real standard,


hopefully there's no more ambiguity


in terms of how robots.txt can be interpreted.


Now when Googlebot is crawling the web,


before we crawl every single URL,


we have to check it against the robots.txt.


In order to do this, we have to fetch the robots.txt itself,


and this fetch can fail.


When we try to fetch robots.txt, if we get 200 OK,


that's great because we have the robots.txt.


We know how to check if a URL is allowed.


If we get 404 not found, that's also good


because it means there is no restriction about crawling


the URLs on that site.


Sometimes a web server will be temporarily overloaded


and they return five exact responses.


That's fine as long as this is a transient response


because we will retry the fetch later.


But if the website starts returning five exacts


every single time, or for some other reasons,


we cannot fetch the robots.txt at all,


then it becomes more serious because we


don't know the robots.txt.


We don't have it, so we don't know whether our URL is allowed


or not.


So to be safe, we don't crawl any URL from the website.


This graph shows the distribution of robots.txt fetch


status on a typical day.


We can see about 69% of the time,


we get either 200 OK or 404 not found.


This is great.


About 5% of the time, we got five exacts.


This is fine as long as it's transient.


For 26% of the time, we get some other kind of errors.


This could be that the site is just down,


or sometimes a website will timeout


when we try to download robots.txt,


or they just close the connection


before sending anything back.


So these are bad.


So if the website is already returning 200 OK or 404,


that's good.


If it's a five exacts, we recommend


that you only return this temporarily


when the server is overloaded.


For the other cases, we recommend


that the webmaster either create robots.txt


if you want to control access to different parts of your site,


or just return 404.


This will make the life of a crawler much simpler.


So lastly, let's talk a little bit about Google's crawling


rate.


When Googlebot is crawling the internet,


we try very hard not to overload your server.


That's why we maintain our crawl rate to every website


out there.


And we have sophisticated ways to adjust our crawl rate


in order to meet both the internal demand and also


the server's capacity needs.


But our crawl rate is not perfect, right?


And the notion of being overloaded


can be different for different websites.


That's why in our Search Console,


there is a tool called the Customer Rate Tool.


You can go there and set a crawl rate that


will limit how fast we can crawl.


There is actually a webmaster article


explaining how you can do this.


But we should caution that just because you set a customer


crawl rate, that does not trigger more crawls.


Sometimes the webmaster may have a misunderstanding.


They want Googlebot to crawl more,


so they try to set a higher number.


This does not work.


It will not trigger Googlebot to crawl more URLs


because the actual crawl rate is determined by many things.


Sometimes when you set a customer crawl rate,


it may actually accidentally reduce your crawl.


Recently, we have a news website in some Asian country.


They contacted one of our webmaster team members


at midnight, and they complained about their crawl rate drop.


Turns out they set a customer crawl rate a few days ago.


So for these kind of reasons, our recommendation


is in terms of Googlebot crawling rate,


please leave it to Google unless your website


is being overloaded.


That's all I have today.


I hope you find this interesting.


If you have any questions, feel free to reach us


through the Webmaster Forum's channel.


Thank you.


Google and the SEO community: SEO Mythbusting

It depends, really, is the answer to a lot of things.


So what does it depend on?


It depends on so many different things.


It depends on the question you're asking.


What it depends on is like the weird meta answer here.


It's like, is this a new site?


Is this a site that is literally doing a site move?


Is it an actual site move?


Or are you just changing the URL structure or something?


What does your service setup look like?


How fast is your website?


What is the content looking like?


Is there a lot of competition around this piece of content?


Is it a duplication of content that is somewhere else?


And it depends on so many different things,


because as you might figure out, the entire process


on our side, the entire infrastructure on our side


is very large and vast and complex.


And so we're going to talk about the whole process.


Hello, and welcome to another episode of SEO Mythbusting.


Today with me is Barry Schwartz.


You might know him from Twitter and Search Engine Roundtable.


We're happy to have you here.


Welcome to SEO Mythbusting.


And we're going to talk a little bit about community.


Awesome.


Thanks for having me in your awesome office.


All the secret blueprints and stuff.


Don't tell anybody.


They're not that secret anymore.


So when I reached out to you to check


if you're available to do this, you


said you wanted to talk about the relationship between Google


and the SEO community.


So one thing that does puzzle me a little bit


is that we are trying to be as transparent as possible


to people.


And we are balancing between being transparent and being


confusing.


But we are oftentimes met with this idea


that we are withholding truth, or we're


having a different intention than what we are saying we do,


which is not the case.


Where does that come from?


You have to understand that these are people building out


content in order to get traffic to their websites.


And in some examples where either you


have feature snippets, where these publishers believe


you're taking their content and putting it


on your own website, the Google search results,


and people don't have to click on from the Google search


results to their website because you answer the query right


in that on the Google search results page.


Because of that, people feel like, why am I doing this?


Why should I go ahead and write content


that I'm getting zero traffic from,


which I'm not going to convert on?


But yes, I see that.


I see that.


I understand that.


We hear that a lot.


But we already know from studies and from experience reports


from the community that you oftentimes get


a lot more qualified traffic.


You get better traffic to your site from featured snippets.


Well, maybe.


I mean, there's been studies going both ways on that.


Fair enough.


And it would be great if you guys could go ahead and publish


your own internal data or share in Google Search Console


to show what this specific publisher is getting


from featured snippets.


A little filter, say, filter my performance reports


by featured snippets.


And you can see your click-through rate,


your impressions, all that data.


And I think if it's positive, I think the community


will be very happy.


All right.


Well, I'm not so sure about that because there's always


a little bit of a agenda behind it.


And yeah, different people have different angles to this.


And I think it's important to understand that fundamentally,


if you have content that is really useful for the users


and you add more value around it because your site is


full of good content, then I think


these can drive a lot of traffic.


But I think the complaints might be coming from people


where the content is maybe not as great.


But I mean, there's bits and pieces


that we can pull out for the user.


But the user has no incentive to actually go there.


So it's like a, uh.


A lot of this stems from, I'm in the US.


So I've been seeing on the TV a lot of governmental,


like the Congress and Senate, talking about these studies


that were produced by different people in the industry,


or outside the industry, even, data providers saying,


Google's taking the lion's share of the clicks.


Meaning people are going to Google,


and Google's sending less and less traffic day


by day to publishers.


And that's a trend that scares the SEO webmaster developer


publishing community.


But that's not something that we want to do.


I mean, what we want to do is we want


to bring people and the content publishers together.


That's literally like the fundamental idea


of the search engine.


And we're trying our best to make that happen.


But we also know that sometimes the intention is not


to do more on the publisher side.


Like if the user wants to follow through,


that's great for both of us, for the publisher for us.


But if it's traffic that doesn't lead anywhere,


if it's like the zombie traffic, then, huh,


that's a tricky one to deal with.


But I see where you're coming from.


And I hope that we can get more transparency and visibility


to these issues in the hands of developers.


Yeah, and maybe that will backfire


if you give more transparency.


Maybe it will be better for everybody.


Who knows?


That's the thing that I find very interesting


that you're saying that, because that's something that we keep


experiencing, that when we are trying


to be as transparent as possible,


that sometimes people are taking us out of context


or misrepresenting what we were saying.


And then we are in this weird lose-lose situation.


We provide more transparency, we lose.


We provide less transparency, we lose.


That's a tricky one to deal with.


I mean, obviously, you know I write about this transparency.


And it's just funny, because the more transparent you are,


it's like a lose-lose situation.


And I've been covering this for so long.


Sometimes over the history, you've


pulled back some transparency, and then you've increased it.


And now it's like, I think it's the highest it's ever


been in the community.


I don't know.


I don't know the answer to it.


I mean, I think being as transparent as possible


is ultimately going to be the best.


Honesty is the best thing you can do without stabbing yourself


in the foot.


Yeah, we're not trying to misdirect you,


or we're not lying to you.


So how can we make this into a win-win situation?


I think we are doubling down on being transparent.


It's definitely the way that we want to go.


And we just hope that the community picks that up.


I mean, again, I think what you're doing today


at these videos, what you've been doing on the Webmaster


YouTube channel for a long time, the activity that you


and your whole team at the Webmaster Trends team


is putting out on Twitter, on the forums, and everywhere.


And I think that is what you need


to keep doing because otherwise, it's just proof.


It's actual proof and truth.


You're on the field doing what you need to do.


Yeah, even just not so long ago, you


held the Google Webmaster Conference,


and you're having one around the world,


but you had a big one in Mountain View.


And people are Google engineers sitting there, taking notes,


seeing what they could actually learn


from the SEO community and the publishing community,


and trying to make it.


And they're trying to say, oh, that's


an interesting point of view that I didn't have.


You're taking that feedback.


You're going to hold it.


All Google Search Console is based on feedback


from the community.


Absolutely.


So many things.


I wish more people would use the Submit Feedback button.


That is so helpful because if you're reporting it


over Twitter, it doesn't really go anywhere.


If you're reporting it over our Twitter accounts


or talk to us in person, then you're like, OK, sure.


I bring that to the Search Console team,


and they're like, that's one opinion out of many opinions


that we hear every day.


But if you're using the Submit button,


like the Submit Feedback button, we actually


get quantitative data of what the people want.


And that's actually, I know that we're not talking back.


We can't answer everyone submitting feedback.


But we are taking this in, and we


are trying to make the things happen that we see


are important.


That's how the Speed Report happened.


Yeah.


I mean, constantly.


It's amazing how much you push out with Search Console.


And again, it's not really making you guys a dime.


It's just more about transparency.


It's about transparency.


So let's hope that that gets us all together


as one community further in building


good stuff for the web.


I hope so.


And then you have all the people thinking, hey, oh, you


have Chrome.


You have Android.


And you're tracking all these users.


And you're saying, hey, why aren't you


using this data for search?


It's a great way to improve search.


But then you looked at Direct Hit, which was a big search


engine back in the day.


And they used click data.


And they were manipulated by a bunch of people


clicking on results and so forth.


It is very noisy as a data source.


It's so useless.


Nobody believes you.


Why don't people believe you?


I wish to know this.


I don't understand that either.


And I think it's partly confirmation bias.


People like to hear the things that confirm their hypothesis.


That's just psychology.


And on the other hand, it is maybe this feeling


or perception that we are trying to hide the truth.


I mean, conspiracy theories are quite well off these days.


Well, it's easy for people to think that, of course,


because you can't tell the full truth in all situations


because potentially people can manipulate the search results.


That's one thing.


The other thing is also just like when I say,


no, we're not using it for ranking,


then I mean exactly that.


And we might use it for A-B testing of different ways


of presenting things in the front end,


or we might be using it for I don't know what.


But people tend to only hear the bits they want to hear.


And then you get misrepresented.


And then we have to clean up that rather than doing


other good things for the community.


I mean, so one example is with AMP.


When you released AMP, when you initially released it,


there were people on different sides of the coin, Google Ads


team, Google Search team, and other teams in the AMP


community as well saying different things


and communicating different things, which


confused the larger community.


Some were saying it is a ranking factor.


Some were saying it's not.


And AMP, while it is not a ranking factor in the core


rankings, the only way to get to that top carousel


is to have AMP pages.


And people look at that top carousel


and say that's the top position.


If you want to be over here, you don't need AMP.


If you want to be over here, you need AMP.


And to be number one, that's kind of like a ranking factor.


MARTIN SPLITTINER


Yeah, right, but as you say, it's not really a ranking factor.


The AMP team obviously believes in what they're doing.


And they're doing a fantastic job at what they're doing.


They solve really, really hard problems.


They also give developers at publishing companies


leverage to basically say, oh, no, we cannot have this very


slow tracking thing here.


We cannot have this terrible piece of JavaScript


on our side because then it would not be conformed to AMP.


So that's a really, really helpful tool.


There's a lot of privacy and security implications


around the entire distribution of other people's content.


And we want to catch that and make that quick.


You can't preload for many interesting reasons that easily.


And that's where the signed exchange bundle came from.


But that's not really being looked at much.


And that's not really being investigated or considered


much.


So AMP is being seen as what it was a couple of years ago,


I would say.


And people are misunderstanding what the idea is.


The idea is not to break the web.


The idea is not to create a Google-centric web.


The idea is to create a web that is fast for the user


and successful for the user.


And the matter of fact simply is, in a bunch of countries


in what we call the next billion users countries,


the web as it is seen is either like a bunch of mobile apps


or like a bunch of very proprietary walled gardens.


And that's what we are trying to alleviate and fix


by making the web fast and accessible for these users.


And AMP is the vehicle to get there.


Yet people are like, oh, what's happening here?


I mean, Google is always going to be held to a higher standard.


And that gives you guys a lot more responsibility.


We understand that, yeah.


That's why you guys are producing all that.


You have AMP developer conferences all over the world


all the time.


You have crazy amounts of videos and documentation around that.


You're doing these things.


So you obviously are stepping up.


And that's the point of this conversation


is SEO myth busting.


So SEO myth busting session is that you


want to make sure that even though you have all


these theories out there, all these myths or whatever


it might be, you're doing whatever you can because you


know of the responsibility Google


has to kind of give as much information as you possibly


can to the SEO community and the publishing community


because of that.


Yeah.


And we are aware of this responsibility.


And we're taking it very, very seriously.


You can also see that by, if you look


at when we launched the evergreen Googlebot,


we didn't just like throw it out there.


We could have just say, like, we've tested production.


Let's see what happens.


And we made sure that we are not causing adverse effects


to other people first.


And that's what we're trying to do.


And I would love the community to understand that we are not


an antagonist.


We're trying to do what we can to bring good, qualified


traffic to people.


And we're trying to put out as much information as possible.


That's why we're filming videos like this.


And that's why we're writing all this documentation.


And I would love for people to not have these snap moments


where they just jump on whatever it is at the time,


but be more considerate and trying


to understand what's happening and getting data for themselves


and thinking for themselves, rather than just getting out


the pitchforks.


Right.


I think people don't realize there's real human beings


at Google.


And it was fun.


I was having a conversation with Danny Sullivan, who's


the search liaison at Google now.


And he was basically saying, yeah, Googlers don't have skin.


We're robots.


Because people just talk to Google like they're a company.


And there's real people there with real feelings.


And just because something, you know,


agree with something at Google or whatever,


it doesn't mean you should really just yell your head off


at whoever you're talking to.


Because people have feelings.


And there are real people there.


Sometimes I have a bad day.


And then when I read certain things on Twitter,


I'm like, why am I doing this?


I apologize for those tweets.


No, it's fine.


Not you.


It's OK.


Mostly.


That just happens.


And we have to deal with that, I guess.


That's part of the deal of working at Google


is also the side of where people are just disgruntled and angry.


That's OK.


It's just sometimes when it's what I don't like


is when it gets unfair, when people are twisting your words


and are basically like calling you out for things


that you didn't do or didn't say.


That's like a really weird one.


And when I get asked a question, I say no.


And people are like, oh, so look at the exact wording


that he has used.


That must have a secret hidden meaning.


I'm literally not sitting there in my computer or at my phone


thinking 20 minutes about a sinister way to.


I'm not a lawyer.


I don't.


You have to go through some training, I guess.


Yeah.


I do go through some training, but mostly


to make sure that I'm not confusing people.


Right.


That's the biggest risk.


It's not that we are trying to hide something.


It's more like, especially when I don't know something,


I'd rather say I don't know.


And that doesn't mean that people are like, ah,


so you can't say.


It's like, no, no, no.


It's not that I can't say.


I literally don't know.


Well, John Mueller said I don't know.


And I asked him about what does I don't know mean.


Either I can't say or I don't know.


OK, so that's John's way of doing it.


That's John's way of doing it.


When I can't say, I'm like, I can't really say.


Or maybe I misunderstood what John said.


Who knows, right?


But basically, I'll say when I can't say something,


because I think that's just fair to say I can't say that.


For instance, if someone's like, so how do you


rate something for spam?


And I'm like, I can't say that.


Because if I would give this information away,


how we are doing spam detection stuff,


then spammers could use this information to game the system


again.


So I won't discuss that.


I won't discuss ranking simply because I


don't know much about it.


And I deliberately keep it that way.


So Martin, why don't you want to know about rankings?


I mean, it's a secret sauce.


You go home, you just go ahead and build your own websites


and make so much money on the side.


So first things first, I think we should not


know about ranking because we are trying


to represent the community as much as we can internally


as we do externally as well.


The other thing is I'm just really bad at keeping secrets.


So that's why I don't want to know about the ranking


bits and pieces.


Because if I accidentally blab them,


that's not very helpful.


And also, it keeps changing, and there's so much going on.


And we have hundreds of ranking factors.


And I don't think there is that much actionable bits


and pieces in there.


So I don't feel like we should be talking about it as much.


We should be talking about what do our users want,


what do our users need, how can we better understand it,


and how can we better deliver good web experiences.


And we have a lot of work to do performance-wise,


content-strategy-wise, literally just copywriting-wise.


There's so much that we need to look into


that I don't think that that's a fruitful discussion to have.


But when you tell the community, just


show that's a good idea.


There are a few technical things that you can influence


and they should do, like HTTPS, making your website fast.


But these stem from, again, thinking user-first.


And I would love to see more companies doing user testing,


as in trying to understand who are your users


and then actually having a conversation with them.


Because what I think is the best content for my users


might not be what the users think


is the best content for them.


And that's a reality that I've seen in all the companies


that I worked at.


The ones that did user tests learned a bunch


from just having a five-minute conversation


with some of the users.


And Google has a product, Google Consumer Survey feature,


I think.


I remember I got hit by that Panda update once,


a long, long time ago.


And John's like, why don't you ask your users


what they think of your content?


I'm like, OK.


So I embedded one of those Google survey things,


and then I published the results,


and they liked my content.


So Google went in there, changed the algorithm,


made my website look really high.


I'm just like, well, you know, you now know how it all works.


Yeah, that's interesting.


I didn't know that we had that product.


That's a good one.


Yeah, and it takes literally a second to embed.


I don't know if it's exactly called that,


but it's basically a little embed,


and then it pops up, rate the site,


and then based on how you rate it and what page you're on,


it gives you dynamic information to ask you more questions,


which goes into a Google spreadsheet, which you then


use to publish, if you want.


That's awesome.


That's really cool.


Awesome.


Thank you so much for being here, Barry.


I think this was a really, really interesting conversation


about community and how Google and the SEO community


work together or can work together.


And I'd like to thank you all for watching the season


two of SEO Mythbusting and listening to us


and the other guests talking.


And thank you very much, and bye-bye.


Hey.


Hello and welcome to SEO Mythbusters.


Today I have been, what?


Testing and debugging JavaScript sites for Search - JavaScript SEO

When you are building websites or web apps with JavaScript,


you don't want to end up accidentally making your content inaccessible to crawlers.


Crawlers, like Google's Googlebot,


help you make your content available and discoverable for users searching for it.


As part of your development process,


you are probably already using tools to make sure your code works with different browsers and devices.


Using the tools we will cover in this video,


you can also make sure that your content is discoverable for crawlers.


A good first step to check your website or web app is the mobile-friendly test.


The main purpose of this tool is to show you potential issues with your content on mobile devices,


but it can do a lot more than just that.


Enter any URL and run the test to see if your page is considered mobile-friendly.


If you want to run this for your local development server,


use a tool such as ngrok to get a publicly available URL.


Once the test is completed, you will not only see if your page is mobile-friendly,


but also a screenshot of your page, as a Googlebot smartphone crawler would see it,


along with the page's rendered source code.


If you see that some of your content is missing or the page doesn't render the way you expect,


you can check if there are any recommendations for you,


or dive in deeper by looking at the detailed report that also shows you which resources have been loaded,


and all JavaScript logs.


But there's one more tool for you to make sure your site is working well with search.


The Search Console.


It is a great way to find out how your website or web app is doing in terms of being discoverable in search.


We will only briefly cover it here, but make sure to check out our in-depth videos and documentation.


Among many other things, Search Console shows you which of your pages are indexed by Google Search.


The URL inspection feature in Search Console lets you see the status of a URL as well.


It shows you the most recent index data about the page, if it is indexed at all,


when it was last crawled, if it's mobile-friendly, and many other things.


You can also run a live test version if needed.


Search Console also offers many more reports to help you identify and solve problems with your website.


Search Console will even proactively message you if there are any known issues.


To get started, verify your site to get reports and insights.


It is also definitely worth checking out the documentation and videos about Search Console.


If you use structured data, you can use the Rich Results Test to check if your pages are eligible for rich results,


and even get a preview of such a result in the search pages.


The test will also show you errors and provide guidance on how to fix them.


Search Console provides a Rich Results report that gives you a site-wide overview of your structured data status as well,


and it shows information in the URL inspection tool.


You can also use the Structured Data Testing tool to check for a broader range of structured data features.


Some structured data is not supported as rich results,


but still important to provide useful information to Google and others, like breadcrumbs.


The Structured Data Testing tool can either be used as a development sandbox,


where you can enter some markup and see how Google would pick it up,


or you can use it as a way to test your structured data implementation by entering a URL


and getting a report on the kind of data that we detected, along with errors and warnings.


Past just content and structured data, speed is important too.


Your goal, as well as ours, is to provide users with useful results and a great experience.


Performance is a big part of that.


If you are not using performance auditing tools like Lighthouse, you should definitely start doing so.


On top of performance audits, Lighthouse also provides a growing number of accessibility and SEO audits.


In combination with the other tools, such as the Mobile-Friendly Test and Search Console,


they all provide different levels of insight into your web content.


Lighthouse also provides insights into the speed and performance of your pages.


Among many other elements, speed is one factor for ranking your page in Search.


So following the guidelines and recommendations that Lighthouse provides


is great for happier users and better ranking.


Last but not least, you can and should check if your pages show up in Search.


For instance, if you look for some of your content, say the title of a blog post or one of your specific products,


within Search you should find some results.


If you found this video useful, consider sharing it.


There are many more JavaScript SEO episodes coming out soon,


discussing topics such as JavaScript frameworks and dynamic rendering.


But in the meantime, check out our Webmasters Help Center and our Webmaster Forum.


English Google Webmaster Central office-hours from June 12, 2020

OK.


Let's start.


All right.


Welcome, everyone, to today's Webmaster Central Office Hours


Hangouts.


My name is John Mueller.


I'm a webmaster trends analyst at Google in Switzerland.


And part of what we do are these office hour hangouts,


where folks can join in and ask their questions


around their website and web search.


A whole bunch of things were submitted already.


But if any of you want to get started


with the first question, jump on in now.


Yeah, hi, John.


Hi.


I have a question regarding my eight-month-old website,


actually.


It was ranking very good till one month before.


It was ranking very well.


And it has few backlinks also from nice websites,


good websites.


And that's why it was ranking well.


But all of a sudden, my website got spammy backlinks


from my competitor.


My competitor created phone-related keyword


backlinks all over the internet.


And after that, all of a sudden, my website


gone out from the search results.


So what preventive measures I can take,


and how can I recover my rankings,


and how to proceed from the current situation?


OK.


So I think if your website is just, let's say,


a couple of months old, maybe eight months, maybe a year,


then that's still very, very fresh with regards


to the rest of the internet.


So that's kind of a time where our algorithms are still


trying to figure out how and where


we should show your website in the search results overall.


So that's something where it would


be normal to see some kind of fluctuations


around how it's being shown in search.


Sometimes things go up for a while.


Sometimes they go down for a while.


And then I'd say over the course of a year,


things settle down in kind of a stable state.


So that's something where I wouldn't necessarily


worry too much about this particular situation.


I'd continue working on your website.


And over time, that's something that should


be reflected in search overall.


Actually, my question is regarding spammy backlinks.


My competitor has created 1,500 websites, backlink websites,


created backlinks from 1,500 websites, different websites.


And those keywords on those links


are spammy, like porn-related keywords.


And I think that's the case.


That's something that we see all the time.


It's really easy for people to run a script


and to drop thousands and millions of links


on the internet.


And we ignore that.


That's something where people get really busy,


and they do all of this crazy stuff.


But our algorithms have seen it so often,


they're used to just ignoring it.


So that's not something I would worry about.


But the anchor text, does it matter on those spammy


backlinks or not?


If we ignore those links, it doesn't matter at all.


It doesn't matter which website.


Whether it is follow or no follow.


Whether it is do follow or no follow, doesn't matter.


Yeah, no.


No.


This kind of thing is really easy,


where you can take $5 and send it to someone,


and they'll put thousands of links up on the web.


And people have been doing this for years and years and years.


Should I pick a preventative measure


like this, all those URLs?


I don't think you need to.


I don't think that would change anything.


I would just completely ignore it.


If your competitor is focused on building bad links


for your website, then at least they're not making a better


website for themselves.


OK.


Thank you, John.


Sure.


Hi, John.


I have a question.


Hi, I'm Chris.


So thank you for your all the help with the Webmaster


community.


So I think my colleague has joined the German Webmaster


yesterday.


And thank you.


You have been there too.


Yeah.


So my colleagues and I are facing some display issue


for some of our pages in Google search.


So basically, for many of our US.com product subscribe pages,


Google is putting the metadata for such as the meta title,


meta description, and structured data for product pricing.


But it's not putting it on the URL.


So you can see the URL is.com.


But we are seeing German and Finnish metadata


on our US.com pages.


So when we subscribe, we check the URL for the US.com pages


in the Google Search Console.


We just saw that the self-deplet canonical is the German


and also sometimes the Finnish pages.


But when we look at our source code,


it's actually for all the US, German, and Finnish pages.


It's actually pointing to our own respective pages.


So there's no issue with our HHFing tech implementation.


And I just have some quick questions here.


So one question is, why is Google


showing the German and Finnish metadata on our pages?


And also, is it possible that the Googlebot


is visiting our US site from non-US countries?


If so, could the automatic redirect of the search box


be causing the incorrect caching?


And finally, would it be acceptable?


I mean, would it incur any penalty


if you were to exclude the search box for these Joe IP


browser language redirect, as our intention is


to serve the best page and language


to the correct country users?


OK.


So I think that adds a little bit more flavor


to what I heard yesterday.


Yesterday, it really sounded like we


were mixing things up across different languages


and countries.


Usually, when we mix things up with regards


to internationalization, it's content


that's in the same language for different countries,


such as if you have the same German content for Germany


and for Austria, then our systems might say,


well, it's the same content.


We can treat it as one page.


But that wouldn't apply if it's German and Finnish and English.


This kind of makes up.


But what you mentioned there is that you're


doing some kind of an automatic redirect based


on the location of the user.


Yes.


And sometimes that can be problematic,


because Googlebot crawls from one primary location


for a website.


And for the most part, that's probably


going to be from the US.


That means when we crawl your German pages, your Finnish


pages, or whatever, we will crawl them from a US address.


So if you're automatically redirecting US-based users


from the German page to the English page,


then we would think that you're trying


to fold these pages together and that we should treat them


as one page.


So that might be something that's happening there.


I think from a practical point of view,


it's kind of hard to test unless you have the ability to test


from different locations.


One thing you could do specifically for Googlebot


is to use the Inspect URL tool in Search Console


to see which version Googlebot sees


when it tries to access the German version


or the Finnish version.


Maybe that makes it a little bit clearer


with regards to what is happening on Google's side.


The issue is for US site, when the search engine robots are


visiting the German.e and.finish pages,


they are not redirected.


Only when they are in Germany, when they visit the.com pages,


they are redirected.


So I'm just curious whether we are searching it from the US


bot's perspective.


There shouldn't be any redirect to the German.finish pages


for our.com.


OK.


That sounds better.


Yeah.


So like I mentioned, we generally


crawl just from one location for a website.


And usually that's US.


So it wouldn't be the case that we would, from time to time,


crawl from Germany or crawl from other IP addresses.


We do it all from the same region


for each individual website.


And for most websites, that's from the US.


What I would do in a case like this


is maybe send me some details of what exactly you're seeing,


which URLs you're seeing, and which kind of queries


you're seeing the wrong language and country versions.


I can send the site search for you in the chat.


So this is the site that we are seeing.


So we are seeing actually in the query,


there's different versions.


But you can forget about the parameters.


But we're seeing the English version


and also Finnish and German language.


But it's getting better.


For now, sometimes when we search an English query,


we actually get the Finnish or German metadata.


But for now, today, when we search, it's back to English.


But it keeps alternating.


So it actually affects the click-through rate


for our English users.


So what we have thought is, is it all right


if we exclude Googlebot from the redirect?


Is there any penalty from our end if we do that?


Because what we did is we just want


the user to go to the correct version for maybe in Germany,


but maybe for the US bot, if they're having problem


to display correctly, we can just


exclude them from the redirect.


I don't think we would see the redirect.


Because if you're not redirecting users from the US,


then we wouldn't see what you're doing with German users anyway.


So that should be something where I don't think it would


change anything if you explicitly excluded


Googlebot from that redirect.


But I'll take a look at the sample that you have there


and see if there is anything specific that you can change.


Or maybe there's something on our side


that we're picking up wrong.


Because sometimes it also happens


that we pick something up wrong.


AUDIENCE 1 Sure.


Thanks.


Thanks for that.


OK.


Let me run through some of the submitted questions.


And if you all have any comments or questions along the way,


feel free to jump in as well.


And probably towards the end, we'll


have some more time to chat as well.


So the first one is about mobile-first indexing.


There are websites out there that are mainly


used from desktop devices.


And the question has a little bit more details,


some ideas about that.


And the question is, will mobile-first indexing


hurt those websites?


They might still be responsive and everything.


But maybe they're seeing by Google


as not really working great on mobile devices as on desktop.


So from a practical point of view,


with mobile-first indexing, it's about indexing for us


primarily.


That means we need to be able to see


the content on mobile devices.


And if the functionality is slightly different,


but the content is the same or the content


is what you want to have indexed, then that's all OK.


Then you're pretty much set for that.


So with that in mind, even if your website is primarily


used by desktop users, once we switch to mobile-first


indexing, we will index it like a mobile user.


And it's not that it will be shown less frequently


to desktop users or less frequently to mobile users.


It's just we use the mobile version


for indexing, for determining what you want to have


shown in the search results.


So that's something where websites, where I would say


they're essentially desktop only,


like the really old-school websites that


are built with a tables layout that are not responsive design,


where when you access them with a mobile device,


you have to kind of zoom in and try to find


all of the content that way.


Those are websites that work really well


on mobile-first indexing because all of the content


is exactly the same on desktop and on mobile.


And the display is slightly different,


but the content is essentially the same.


So those work really well for mobile-first indexing.


What we'll kind of play into this a little bit, I think,


is once we start using page experience as a ranking factor,


within page experience, we do have mobile friendliness


as a ranking factor as well.


So if a website is not mobile-friendly,


then maybe that's something that will


play a role in kind of the ranking of that website


overall.


But page experience is something that we've essentially just


announced our intent to go down that direction,


and we'll let you know at least six months


before we start using it as a ranking factor.


So that's something where you still have a bit of time


to try to figure out what really makes sense for your website.


The next one is also about mobile-first indexing.


We're trying to be prepared as possible


for mobile-first indexing switch that's happening in September.


How much can we rely on Lighthouse's SEO mobile results


as an indication of what to expect


when the switchover happens?


So this is kind of a common point of confusion


with regards to mobile-first indexing.


It's not about mobile friendliness.


It's really about mobile indexing.


So if you're using Lighthouse to test


how mobile-friendly a website is,


then that wouldn't necessarily reflect


what we would use for indexing.


If it's a responsive website where the content is


essentially the same, the presentation is very similar,


then that's something where probably we


will just be able to work with that right away.


If you show different content from desktop and mobile,


or if the responsive design is very different on desktop


and mobile, then that's something


where maybe you would see changes with regards


to mobile-first indexing.


So with regards to kind of the way the content is presented,


one of the things that I feel people haven't really


focused on too much, or where we still see problems at least,


is that if you have images or videos on your pages


and they're very prominent on the desktop version,


and on mobile they're very, very small, very tiny in a corner


or on the bottom of the page, then with regards


to image search and video search,


those are less likely to be good landing pages for images


and videos, right?


Because the user is trying to search for an image.


They find the image in the search results.


They click on the page.


And then they can't find the image on the page.


And that's a really bad experience.


So if you care about image search or about video search


and you have a website that's switching over


to mobile-first indexing, then it's


really worthwhile to make sure that your image and video


landing pages are really such that the content that you


want to be found for is very prominent on those pages.


And then a question about Core Web Vitals.


There are custom fonts on a page that need to be loaded.


It's recommended to use font display swap


because it will show the text to the user as fast as possible.


Now with Core Web Vitals, it's recommended


to use font display optional because we


don't want to shake our page when swapping the fonts,


or we will get a bad CLS score.


To be honest, I don't have any experience with regard to that.


So I don't really feel qualified to say


which exact attribute you should be using for your custom fonts


there.


What I would recommend doing is maybe pinging us on Twitter.


And we'll try to find someone for you that


can help you to figure out what the optimal approach here is.


Does Google use links as a part of geotargeting


and determining the location of a website?


From what I've been seeing, geotargeting


takes a bit of time before we start


ranking in major countries, such as the US or the UK


or even Russia.


We start ranking in little or non-spam countries


with a low population.


It's very confusing.


So we use links in lots of places around search.


I don't know if we would particularly


use that for geotargeting.


But in general, with regards to geotargeting,


you can tell us the preferred geotargeting of your website,


either by using a country code topical domain, where


you can essentially tell us right away what


the geotargeting is that you want,


or by using the geotargeting setting in Search Console.


And those are really the primary ways


of letting us know which countries you want to target.


And all of these more exotic signals that we have to pick up


are more for the situations where


we don't have good information about a website.


So that also includes things like the IP


location of the server.


If nobody tells us where this website should be geotargeted,


and we think it should probably be geotargeted,


then sometimes the IP address of the server helps us.


But if we already have other information for geotargeting,


then we don't need those small details as well.


With regards to the time it takes for ranking,


I think it's completely normal that if you


start with a website, then you probably


end up ranking first in areas where maybe there's


a little bit less competition, where it's a little bit easier


to get started.


Whoops.


Let's see.


Bit of noise in the background.


OK, cool.


We have a website version for the US.


However, we're not able to display the same content


for all US states.


Is there a way that we can tell Google to send us visitors


or just targeting only for certain states?


If not, how does Google handle the fact


that the users from some states are very


satisfied with our content, but others from the states


were not able to display the content


and certify their queries or not?


Is Google able to see which states users come from?


And depending on that, adjust our ranking


for only certain states.


So I think, first of all, we kind of have that situation


that I mentioned before with the other international sites


in that we primarily call from one location.


So if you're changing the content for different regions


within a country, then probably we would not ever see that.


So if you're doing that on a state-by-state basis


or on a state-by-state basis, then


state-by-state basis or on a city basis,


we would never really see that.


We would probably see the content


that is mapped to, I don't know, California, Mountain View,


or wherever.


It's a bit weird with IP addresses that


come from data centers because they end up


being shared across the different data centers,


and you just assign them to one arbitrary location.


So that's something where we would probably just


see the version from one location.


So that makes it a little bit tricky


with regards to swapping out the content


for different locations.


Usually, what I recommend in a case like this


is to have a home page where you have some part that


is customized by location and some part that is just general


for your home page so that when we crawl that home page,


we have enough general content to rank your website for


and the more customized content for individual location


links out to different versions.


So you would have a link to maybe a version for California


or a link for Alabama or whatever,


and we'd be able to find those links anyway.


So we can crawl all of those city or state-specific pieces


of content, and we can index them individually


so that if someone is searching for whatever your business


does plus that state, then it would be easy for us to map


that and say, well, actually, this


is the piece of content that probably matches


what you're looking for.


So that's kind of the direction I would head there.


With regards to Google automatically figuring that out,


sometimes we can figure it out.


Sometimes that's a bit trickier.


It depends a lot on how easily we can understand


what the user's intent is.


Is it that they really need something local?


And if so, then maybe their locations


should be a factor that plays a strong role in the query.


Maybe it's something where we don't


know if the user is looking for something really local.


Then maybe we wouldn't know to pick


that local version of your page, and rather, we


would pick a generic version or maybe a different location


version of that page.


Because this is something that is kind of, I don't know.


I think we're getting pretty good at it,


but it's still kind of hit and miss.


It's something where my general recommendation there


for international sites, but especially for sites


where you're customizing the content by subparts


of a country, is to try to recognize


the location of the user.


And if you can recognize that they're


accessing a page that is not the best one for their location,


then show a little banner on top so that the user can


go to the better version, but they can still


access that individual version.


By letting the user still access that individual version,


we don't have any problems with indexing,


because we can access all of these different country


versions or city or state versions


from our crawler in California, because we can still


access that content.


But for users, when they go to that page


and they go to the wrong version,


they have a little banner on top telling them,


hey, actually, there is a better version for you, probably.


Check it out here.


So that's kind of the recommendations


that I would have there.


When people use Google Gmail Instant Login or sign up


and it redirects users away from our website


to Google login interface and then redirects back


to our website after logging in, does Google


understand that they're not bouncing,


or does it look like people are bouncing away


and in turn damage our SEO ranking?


The same for Facebook login redirects.


So I think there's a bit of misconception


here that we're looking at things like the analytics


bounce rate when it comes to ranking websites,


and that's definitely not the case.


So if you have content on your website that


is accessible to users once they log in,


then using these kind of login providers, I think,


is a really good thing.


From a security point of view, from what


I hear from the security teams, this


is actually a lot better than if you implement your own login


scheme for most websites, because creating


a good and secure login system is surprisingly hard.


So my recommendation would be to continue


using things like this.


And if you're using analytics to track it yourself,


then obviously you need to figure out


how you want to track that.


But it's not something that I would worry about


from an SEO point of view.


The only aspect from an SEO point of view


that kind of comes into play with these kinds of sites


sometimes is that obviously Googlebot is not going


to be logging in to your website.


So if there's content that's only accessible after login,


we would not be able to index it.


So that's kind of the main aspect with regards


to these kind of sites.


That's worth keeping in mind.


My rank was number one for most words,


and the impressions on my site increased from 200,000


to 300,000 a day, but there was no change in the clicks.


And the next day, my impressions dropped back to 200,000,


and my clicks were the same.


What could be the reason for that?


It's really hard to say.


So this is the kind of thing where


I would recommend going to the Webmaster Help Forum


and giving some of the details of what it is that you've


been seeing, because probably there are very


simple explanations for this.


But it's worthwhile to kind of talk with other people who


have seen similar things.


It might be something as simple as, I don't know,


maybe an image was being shown from your website instead


of a normal text listing.


These things can sometimes result in the impressions


going up really high.


It might be that suddenly people are searching


a lot more for your website.


Or it could be that your website kind of went


from the first entry on page two to the last entry on page one,


and suddenly the number of impressions


went up significantly.


But maybe it's still the case that people are not


clicking all the way down to see your site in the search


results.


So there are lots of these subtle things.


But it's worth chatting with other people who have looked


at this kind of data before and showing a bit more


about the things that you've been seeing,


like which URLs, which kind of queries,


what numbers you're actually looking at there.


I'm doing a 301 redirect from a subdirectory on my site


to a subdirectory on a new domain.


Once they're redirected, I will be displaying a pop-up


to inform users that they have been redirected


to this new page.


And once they read it, they can click Continue to move on.


Would that affect anything?


No.


That would not affect anything from an SEO point of view.


Because from an SEO point of view, we would see that


redirect.


We would follow that redirect.


But we would not be sending any refer information


for all of the requests that we make to those pages.


So purely from an indexing point of view,


we would probably never see that pop-up.


So we would be able to index the new content right away.


We'd be able to follow the redirect,


forward the signals to the new page.


And all of that would essentially just


continue to work well.


I don't know how well this works for your users.


But ultimately, that's something between you and your users


where you probably want to make sure that you're not


getting too much in the way.


But that you're informing them appropriately


if that's something that you're trying to do.


I work on a site with separate mobile URLs.


And recently, it switched to mobile-first indexing.


How would a page benefit in any way


from getting external links to m.example.com


rather than the desktop URL?


Have you ever tested or can you speculate


what would happen if the canonical became the m.version


and both versions and the rel alternate


added to the mobile version pointing to the desktop


version?


So with separate mobile URLs, or when


we switch to mobile-first indexing, we, well, in general,


with separate mobile URLs, even before mobile-first indexing,


we take the m.version and the desktop version


and we fold them together as one page,


essentially for indexing purposes.


So any signals that go to either of these versions


would be combined and focused on one version.


Before mobile-first indexing, they


would be combined on the desktop version.


With mobile-first indexing, they would


be combined on the mobile version.


And what will happen from a practical point of view


is, internally, we will pick the m.version as the canonical URL.


That's the one we'll use for indexing.


And we'll see the desktop version


as kind of an alternate URL for this web page.


So when someone on desktop is searching,


we will know, instead of ranking this mobile URL,


we should be showing the desktop URL, that kind of thing.


So internally, we would be picking the m.version


as a canonical and kind of working from there.


However, externally, from your side,


you should be treating things as they were before.


So that's kind of a bit confusing there,


in the sense that you should continue


to have the link rel canonical from the mobile version


to the desktop version.


You should have the link rel alternate from the desktop


to the mobile version.


The main reason we decided to keep that


is because we're using this set of connections


to recognize that these pages belong together


and to recognize which one is the mobile, which one is


the desktop version, and to keep it such that you don't need


to watch out for when we switch to mobile-first indexing.


And then suddenly, you have to change your whole website.


So we prefer that you keep kind of this normal connection


between mobile and desktop with the kind of canonical pointing


to the desktop version and the alternate to the mobile


version, even if internally, we treat it the other way


around for indexing purposes.


So that's sometimes a bit confusing


if you look at the details a bit too far.


But essentially, our goal here is


to make it so that you don't have to make any changes,


but rather that we try to work out the changes on our side.


And I guess just to be really clear with the first question


with regards to getting external links to the m-dot version,


that wouldn't change anything at all.


So with mobile-first indexing, without mobile-first indexing,


we combine all of those signals.


It doesn't make sense to do any kind of separate work


to promote your m-dot version over your desktop version


or vice versa.


It's essentially we combine all of those signals.


An author on a prominent industry news site


recently wrote that the rel canonical tag doesn't


consolidate external link signals to the canonicalized


page among a group of duplicates in a situation,


this is complicated, in a situation where


external links point at each of the pages


in that group of duplicates.


So I haven't taken a look at this article,


so it's really hard for me to say what exactly


was written in that article.


But with the rel canonical, what happens there


is we essentially use this as a signal for canonicalization.


And for that, we need two steps.


So on the one hand, we need to be


able to recognize that these pages should


be treated as one group.


And we can do that by looking at the content,


by looking at the pages.


Sometimes we can recognize it in the URL


if there are URL parameters that we can drop.


Those kind of things, we also look at the rel canonical.


So all of these things kind of come together.


And then as a first step, we say this set of pages


should be treated as a group.


We should pick one of these pages from that group


and treat it as the canonical URL.


And then the second step is picking the canonical.


And for picking the canonical, we use a number of signals


as well.


I think we have these documented in the Help


Center somewhere, so things like 301 redirects.


The rel canonical definitely helps us here.


Internal and external links, things like scipen files,


hreflang links, all of these kind of extra annotations


where you tell us a little bit more which of these URLs


you actually want to have indexed.


And we take all of these signals together,


and we try to weigh them appropriately.


And we try to pick the right URL to show


from this group of URLs that we think are essentially


the same thing.


So that's essentially what happens there.


And in the situation where we put all of these pages


into the same group, and the rel canonical


is one of the signals that aligns with the preferred URL


that you want to have shown, then essentially all


of the signals that go to any of the pages in this group,


they're concentrated and focused on that one canonical URL.


So that includes things like external links.


That includes any other signals that we


have for that page, also internal links.


Everything is kind of combined into that one single URL.


But depending on the situation, us


recognizing that specific pages should


be a part of this set of pages that we group together,


that's something that might be kind of tricky


if the content is very different, for example.


And also with regards to selecting the canonical URL


from that group of pages, that's also sometimes a bit tricky.


That sounds easy, but it is not that easy to do in practice.


And it's something where we, I would say every couple of days,


we email a team that works on canonicalization


and ask them, why did you pick this one instead of that one?


Because people just have so many unique configurations


on the web where our systems have


to deal with the whole web.


And sometimes we pick things in ways


that don't make a lot of sense to humans


when they look at these pages.


Let's see.


Question about content length and schema structures.


Imagine a website that's presenting a feature


about a specific topic.


Along with the feature, there is a schema markup FAQ,


or how-to section.


Is the word count of these additional sections


taken into account for determining


the word count of the feature?


Or is the length of the feature just


determined by the feature itself without the FAQ or how-to


sections?


And so I think maybe as a general comment,