and crawlable, and that will rank instead.


English Google SEO office-hours from October 1, 2021

So if we have the choice between a roboted page


that we don't know about, kind of what's on there,


and a reasonable page on your website,


and that reasonable page has those keywords on it,


then we'll just show your normal pages.


So from that point of view, it's probably extremely rare


that they would show up in the search,


and it's more a warning in case you weren't aware


that these pages were blocked.


SPEAKER 1 OK.


Yeah, our main concern is more about them reflecting


on the quality of our website.


So if you're saying that it doesn't, I guess, no harm.


SPEAKER 2 No.


Yeah, that's perfectly fine.


I think what you could do, if you wanted to make sure,


look in the performance report in Search Console


and see what kind of queries they show up for.


And if you can't find any, then it's like, well,


we're not showing them for any queries that you care about.


So it's not that we're going to take them


into account for that kind of query.


While I'm here, a question about one of those URLs in robots


is not really accessible to the bot


because it's behind authentication.


So you have to be a logged in user.


Is there a preference for Googlebot


on those types of pages on the status code?


Because I think the convention is kind of like,


I've seen a lot of 302s to a login page.


Google, I assume, pretty quickly figures out that something,


it doesn't need to be like a 401 unauthorized or something


like that, right?


No.


No, that's usually fine.


If it's blocked by robots.txt, we


don't see the status code anyway.


So from that point of view, we wouldn't see it.


The one situation where I would look into login pages


a little bit more is if your primary content is


behind a login page and you want to have at least that login


placeholder kind of appearing in the search results.


So we have that, for example, with Search Console.


I just, I don't know, happen to see that more often there.


A lot of the tools are essentially behind a login.


And if you search for one of those tool names,


we know it's behind this URL, but we


can't find any content there.


So we kind of need to at least be


able to index some kind of a templated login


page for that query.


But if in your case, this is actual content that's


not meant to be findable in search anyway,


then it doesn't matter.


SPEAKER 1 OK, perfect.


And sorry, one more quick one.


No indexing pages.


So I've heard you say in past office hours


that for a large site, processing a lot of pages


that are now no index could take up to a month or so.


And we've had to do that in order to address some site


quality issues.


And as we monitor console, is it normal to see the index pages


kind of gradually decline?


I guess that would be my guess as they're crawled.


Or is it more like one day there's a sharp drop off?


Usually it's a gradual decline.


So yeah, I mean, that's kind of the normal crawling


and indexing process.


Sometimes when we recognize large site-wide changes,


we will crawl a little bit faster.


So it won't be as slow as of a decline.


But it's still a gradual decline.


SPEAKER 2 OK, cool.


That makes sense.


Thank you so much.


SPEAKER 1 Cool, thanks.


Brandon.


SPEAKER 3 Hey, John.


So I'm going to set the stage with some context.


And this is actually relevant to the no index discussion.


So let's say you consider a site with 100 million pages.


And there may be 10 million pages


that we believe are low quality.


Maybe 5 million of those are actually indexed.


And let's suppose that we want to add a no index


tag to these 10 million, as well as reduce


the amount of internal links that they receive.


So over a hypothetical four months,


let's say Google crawls and removes 2 million


from the index, while the other 3 million remain.


And maybe a few months down the road,


we determine that a few hundred thousand


are actually of decent quality.


And we want to reinstate them, essentially.


So we remove the no index tag.


And we start adding meaningful internal links back.


Do you foresee or think the history here


of being indexed and applying a no index tag


and then trying to get it indexed again at a later time


would be detrimental?


Would Google be reluctant to crawl these URLs again,


knowing that they previously were no indexed?


And do you think that there would


be any issues with getting them indexed again


after they were crawled?


JOHN MUELLER I think in the long run,


there definitely wouldn't be any kind of long-term effect


from that.


I think what you might see initially


is that if we've seen a no index for a longer period of time,


that we might not crawl that URL as frequently as otherwise.


But if you're saying this is kind of in the process


of we added a no index, it dropped out,


and then we changed our mind, we added or we removed it again,


then that's not going to be a long-term no index situation.


That's probably going to be something where we still


crawl that URL within the normal crawl rate anyway.


And when you're talking about millions of pages,


we have to spread that out anyway.


So that's something where you'll probably see, I don't know,


depending on these pages, I would


imagine you might see every couple of weeks


or every month or so, you might see us try to crawl


those pages individually.


So I don't see that kind of crawl rate dropping off


to completely zero.


Maybe it'll go from one month to two months


if we see a no index for a longer period of time,


but I don't see it going completely to zero.


So that kind of, like, we added no index,


and we changed our mind at some point later on.


Usually, that's perfectly fine.


And this is especially the case if you're


working on internal linking at the same time, where


you add more internal links to those pages, then, of course,


we're going to crawl those pages a little bit more because we


see, like, oh, it's freshly linked


from this and this place within your website.


Therefore, maybe there's something we actually


do need to index.


Gotcha, do you think it might actually be better to reinstate


these URLs under a different URL so that way there


is none of that historical context


and potentially the likelihood that it gets crawled


would be higher?


Because it seems like if over the long run,


let's say after six months, Google knows, oh, this URL,


we no indexed it before, that it may be less likely to crawl


that URL despite increased internal linking as opposed


to a brand new URL that has the same amount of internal links.


I don't think you would see a noticeable difference.


At least from my point of view, it's


not something where I would spend a lot of time trying


to figure that level out because if you're talking


about a really small website where one page index is


a significant portion of the web page,


then that kind of optimization, I think, makes sense.


But if you're talking about a website with millions of pages,


then I think any optimization you do there


is such a small level that you actually can't measure it.


I see.


And when it comes to the no indexing of content,


should we anticipate that impacting


the overall assessment of the site, where


if the majority of the content actually is low quality,


accelerating its de-indexing might fast track us


to improve our site quality in the eyes of Google?


Does that sound about right?


Sometimes.


Yeah, sometimes that can help.


I think with a large website, it sometimes


makes sense to make bigger changes like this.


But I think the thing to keep in mind


is that our quality evaluations, they


tend to take quite a bit of time.


So if you go off and just de-index,


I don't know, 10% of your website,


then it's not going to be the case that next week we


will pick that up and re-evaluate the quality


of your website overall.


But rather, we need to do that over time.


And that's something that'll take a couple of months anyway.


And with regards to just removing a lot of content


or not, I think that's something that depends quite a bit


on the website itself.


And oftentimes, just removing content


doesn't make the website's quality overall much better.


It's more like, well, you're removing


some of the bad stuff, but essentially,


the whole framework for being seen as a low quality website,


that's still kind of there.


So you need to take care of that part as well.


So it's not just blindly removing things,


but actually, you kind of need to make sure that the rest


also levels up a little bit in terms of quality.


SPEAKER 2 OK.


Thank you.


I appreciate the responses.


Thanks.


SPEAKER 1 Cool.


Gregory.


SPEAKER 3 Yeah, hey, John.


I've been working with a client for like two months now.


And I found in my research that they


were cloaking internal links.


SPEAKER 1 OK.


SPEAKER 3 Yeah.


But then I checked in with the Wayback Machine,


and I found out they have been doing this for,


they made a certain template change.


And it's about footer links.


And these footer links were there back in January.


But when I checked Search Console, I don't really


see anything happening with a penalty or such.


So I wonder, how long does it take?


Because I want to advise them, of course, hey,


you need to take action before you get a penalty.


But I can imagine a response, hey,


we've been doing this for apparently nine months.


Should we be afraid?


SPEAKER 1 Yeah.


So what kind of cloaking do you mean?


Or what are you doing?


SPEAKER 3 This play is not on a, with CSS.


And there is no way to make these links appear


with a link or such.


I think it's something where, theoretically, we


don't like that.


But I don't see the Web Spam Team taking action on that.


Because especially when it comes to internal linking like that,


it's something that has quite a subtle effect


within the website.


And you're essentially just shuffling things around


within your own website.


I think it would be trickier if they were, I don't know,


buying links somewhere else and then hiding them.


That would be problematic.


That might be something that our algorithms pick up on


or that even the Web Spam Team at some point


might manually look at.


But if it's within the same website,


if it's set to display none, then I


don't think it's a great practice.


If you think it's an important link, then


kind of make it visible to people.


But it's not going to be something where the Web Spam


Team is going to take action and remove the site


or do anything crazy with that.


SPEAKER 3 OK.


So in this case, you would say leave it as it is.


Well, I wouldn't leave it as it is.


I would see it as something to try


to improve for the long run in the sense of,


if you think this is an important link


to an important page, then just be straightforward about it.


Because users are going to use it too.


Or maybe if users don't care about it,


maybe it isn't actually an important link.


But I wouldn't see it as something


where I drop everything, we need to fix this this week kind


of thing.


OK.


Right, clear.


Thank you.


SPEAKER 1 Cool.


Let me go through some of the submitted questions.


And then I'll get back to all of you with your raised hands


as well.


Submitted questions are always useful along the way


as well, because not everyone can join these sessions live.


Let's see.


The first one I have here is a longer question


about product pages.


And I'll just try to paraphrase it a little bit.


There are two aspects there.


On the one hand, they have kind of promotional products


where they have samples, essentially,


that are also shown.


So individual companies who are using


these promotional products.


And the other part is that some of these


are shown behind a View More button.


And I took a look at the example and at the website


a little bit to try to figure out what is actually


happening there.


And one of the things that I see there is,


so I think a part of the question is,


well, how do we get all of these sample pages indexed as well?


And the thing there, I think, is it's probably not


necessary to get all of these sample pages indexed.


Because essentially, these samples


are promotional material for other companies.


And it's not that you need to have those pages indexed


so that your page is findable.


Because essentially, those pages would


be findable for some other person's company plus folder


or plus cups or whatever.


And from that point of view, it's


not that these are pages that would be driving traffic


to your website.


Because nobody would be looking for, I don't know,


some other company plus folder and hope to find your website.


They probably want to find that other website


or that other business name directly.


So from that point of view, I wouldn't


focus on these pages as being critical for your website.


Obviously, there might be individual situations


where it does make sense to have those indexable and findable.


But at least the ones that I looked at


seemed like the kind of pages where you don't really need


to worry about them too much.


The other part is the JavaScript View More button


that you have on some of these pages with collections


that you're showing there too.


I took a quick look at how you have this implemented.


And essentially, you load a small subset of your collection.


And then the JavaScript View More button


would load more of these items onto the page.


And Googlebot doesn't use View More buttons.


So we would not be able to see the additional content


that you're loading there.


So instead of using View More, usually we


recommend using some form of pagination.


In our e-commerce stocks, which we just published this week,


we have, I think, a whole article on pagination.


So I will take a look at that to see what kind of makes


sense in your specific case.


And like I mentioned for the first part,


maybe it's also something where you don't actually


need to worry about this too much because those pages are


not critical for your site's traffic.


Let's see.


Official movie sites we oversee were recently redesigned.


Some content was just reorganized.


But the new site is a better user experience.


The ranking dropped afterwards.


Reverting the sites brought back the ranking.


We concluded that no JavaScript technical issues.


This started just after the June spam updates.


Same findings for multiple movies and domains.


Could the new sites be getting hit by a spam penalty


and not the old site?


We can provide some examples.


I mean, it is possible that you're


doing something spammy on the new sites


or something our systems pick up as being spammy.


But I doubt that this is the case.


Usually, when you do a bigger reorg of a website,


you do see some fluctuations.


So especially in the short term, you


will see things shuffle around a little bit


until they settle down again.


So those kind of fluctuations, I think, are kind of expected.


And the longer term changes, it's


something where it almost depends a little bit more


on what you're actually doing on the site.


And I think you're here.


Andrew.


Yeah.


I'm here.


And so, yeah, this issue has been happening since the June


updates.


And we reverted back a few sites a few weeks afterwards.


But the current site we're looking at


has been that case since the June update.


And before the June update, it was active since December


of last year.


Constant top five ranks.


And then site redesign ranks dropped.


And they've been dropped ever since.


And we know, based on the experience


of the other domains, if we revert back the site,


they'll just pop right back in place.


So there's something going on with the new template.


And these movie sites are really just the home page.


We have sub pages.


But the synopsis page and even the gallery page,


those have been moved to the home page.


And that's really the only change.


Now, I wouldn't expect the spam update to have any effect there


because usually the type of spam that we pick up


is more along the lines of really bad spammy content


or really bad spammy linking.


And if you have a site and you do a redesign,


even if it's seen as like, oh, it's


like maybe there's keyword stuff in here or something like that,


then that's not something that these spammy updates would


be looking for.


So to me, it sounds a little bit more like maybe just


the on-site setup of those new pages is, I don't know,


not the way that we can process them as easily


as the other ones.


JOSHUA BERGISLEHORE-MARTIN.


They're almost the same.


I can look at the Google Search Console render,


and it sees the content all as is it should be.


Google's HTML output from Google Console is there.


And the render looks great.


So we have no idea why this is happening.


And the spam is the only reason because it really


happened at those spam updates.


Like the tool we use for ranking gives us little footnotes.


And all the Fire sites we review all


fell in line with that update.


And the only reason why we're thinking this spam is just


because this site hosts very similar content


to a lot of other sites.


The synopsis content is pretty much


boilerplate across other entertainment sites


across the industry.


But we're dealing with the official site.


So we're trying to figure out why this is happening.


JOHN MUELLER.


Yeah.


If you can drop some examples in the chat,


I can take a look at that afterwards.


So I think the one thing that I could imagine that maybe


our algorithms are running into is


we see a lot of sites that essentially offer,


I don't know, illegal downloads of movie content


and things like that.


And it might be that combination of lots of movie content


and maybe a different template is throwing us off


in that direction, where we're like, oh, is this actually


a real site or not?


But.


JOSHUA WILKINS.


The odd thing about that aspect is


that we lost ranks for the core movie site ranks,


just the movie name and just some simpler variations.


But we actually retain all those piracy ranks.


So it doesn't make any sense why this is happening.


And yeah, if we revert the site, we know we'll come back.


So I'll throw a few examples in there


that you can take a look at.


JOHN MUELLER.


OK.


Cool.


I'll go to your team to kind of double check.


Thanks.


Then what can you say about the effect


of question-oriented titles, such as what, how, which,


and who are in the content in terms


of being comparable with semantic search engines?


I don't know exactly what direction


you're headed with this question.


So it's hard to say.


In general, I would recommend focusing


on things like keyword research to try


to understand what people are actually searching for.


And if you want to match that one to one,


then that's kind of up to you.


I always find it a little bit risky


to try to match these queries one to one


because those queries can change fairly quickly.


So that's something where I wouldn't focus so much on,


well, Google is understanding this content right away,


therefore it's good, but more thinking about,


what is a long-term thing that you


want to achieve with your pages, and how do you


want them to show up in search?


And sometimes that means having some question-and-answer


content, but not always.


How does Google evaluate interactive content,


like a quiz or a questionnaire that


helps users figure out which product or thing that they


need?


Would rankings still be based on a static content that


appears on the page?


Yes.


The ranking essentially is based on the static content


on these pages.


So if you have something like a full page of questions,


then we will essentially rank that page


based on those questions.


And that means if you have products


that are kind of findable after going through those questions,


you should also make sure that you have internal linking


to those products without needing


to go through that questionnaire.


So something along the lines of having a normal category set up


on the website, as well as something like, I don't know,


a product wizard or whatever you have to help the users make


those decisions, I think that's really important.


The questionnaire pages can still be useful in search.


If you recognize that people are searching in a way


that they're not sure which particular product matches


their needs, and you can address that in the questionnaire


and the text on the questionnaire,


then those questionnaires can also appear in search as well.


But it's not the case that Googlebot


goes in and tries to fill out a questionnaire


and sees what's kind of happening there.


Does giving a dofollow link to a trusted authoritative site,


is that good for SEO?


I think this is something that people used


to do way in the beginning, where they would create


a spammy website, and on the bottom,


they'd have a link to Wikipedia and CNN,


and then hope that search engines look at that and say,


like, oh, this must be a legitimate website.


But like I said, people did this way in the beginning,


and it was a really kind of traditional spam technique


almost.


And I don't know if this ever actually worked.


So from that point of view, I would say,


no, this doesn't make any sense.


Obviously, if you have good content within your website,


and part of that references existing other content,


then kind of that whole structure,


that makes a little bit more sense


and means that your website overall is a good thing.


But just having a link to some authoritative page,


that doesn't change anything from our point of view.


I'd like to do basic research regarding


what do foreigners Google the most with Taiwanese culture.


What are the most relevant keywords


with Taiwan on Google search?


It would be great if I could generate a ranking list of it


to acquire that information.


I could further designate a campaign for certain products.


I don't have that information, so I can't give that to you.


But essentially, what you're looking for


is probably everything around the lines of keyword research.


And there's lots of content written up


on how to do keyword research.


There's some tools from Google that you can use


to help you figure that out.


There are a lot of third-party tools as well.


And I have no insight into what all is involved there.


So I can't really help with that.


And I definitely can't give you a list of good queries


that people in Taiwan do.


I was having two languages, like Hindi and English,


on the same page.


And I was ranking good on Google.


But after the December core update,


I lost ranking for Hindi keywords mostly.


What should I do to get it back?


I don't know.


So on the one hand, we don't recommend


having multiple languages per page,


because it makes it really hard for us


to understand which language is the primary language


for this page.


So from that point of view, I think


that configuration of having Hindi on one side


and English on the other side on a single page


is something that can be problematic on its own.


So I would try to avoid that setup.


And instead, make pages that are clearly in Hindi


and clearly in English.


And by having separate pages like that,


it's a lot easier for us to say, oh, so when


it's searching in Hindi for this keyword,


here's a page on the specific topic.


Whereas if we can't recognize the language properly,


then we might say, well, we have an English page,


but the user is searching in Hindi,


so we probably shouldn't show it to the user.


And if we're not sure about the language of a page,


then that's also kind of tricky, especially


when there are other comparative pages out there


that are clearly in Hindi.


So that's kind of the one thing.


The other thing is with regards to core updates,


we have a lot of blog posts around core updates,


and I would go through those as well.


Because if you're seeing this kind of a change happening


together with a core update, it might


be due to kind of two languages on page,


but probably it's more likely due to just general core update


changes that we've made.


So I would take a look at those blog posts


and think about what you might want


to do to kind of make sure that your site is still


very relevant to modern users.


My question is whether it's OK from an SEO perspective


to create doorway pages when they actually help users.


Take the URL below, for example.


This page leads users who have searched


for a non-scientific name of a cactus to the original page.


Now, I don't know about this specific situation.


Usually, we would call things doorway pages


if they essentially lead to the same funnel afterwards,


where essentially you're taking a lot of different keywords


and you're guiding people to exactly the same content


in the end.


In the case of something like an encyclopedia,


it's not the same content.


It's essentially, I don't know, very unique pieces of content


on there.


And just because it covers a lot of keywords


doesn't necessarily mean that it's a doorway page.


So without digging into this specific site in detail,


my guess is that that would not be considered a doorway page.


But a doorway page might be something where, I don't know,


if you have a cactus page on your website


and you're saying, like, cactuses in all cities nearby,


you make individual city pages where all of the traffic


is essentially funneled to the same direction on your website,


then that would be considered a doorway page, where


you're kind of creating all of these small doorways,


but they lead all to the same house.


A question related to classified websites.


If I have ad listings on search results,


I allow to crawl and index.


If I have no ad listings for some time,


should I disallow to index or should I let Google decide?


If search results don't have ad listings


and excluding those pages from the site map


would also be a good practice.


So I think, just for sake of clarity,


I think the search results that this person means


are the search results within their own website.


So if someone is searching for a specific kind of content,


then the website pulls together all the ads that it knows,


and it's those search results, not Google search results.


And essentially, the direction here


is what you should do with empty internal search results pages.


And our preference is essentially


to be able to recognize these empty internal search results


pages, which could be by just adding


no index to those pages.


That's kind of the ideal situation,


because what we want to avoid is to have


a page like that in our index, where it's basically


like saying, oh, someone is searching


for a blue car of this model and make,


and you have this page on your website,


but it says, I don't know of any people selling


this kind of a car.


Then sending people to your website


for that kind of a query would be a really bad user


experience.


So we will try to recognize those pages and say,


these are either soft 404, in that we recognize


they're an empty search results page,


or you put a no index on them, and you tell us that it's


an empty search results page.


So essentially, that's kind of the direction to go there.


If you can recognize it ahead of time,


we would generally prefer having a no index directly


from your side.


If you can't recognize it ahead of time,


then using JavaScript to add a no index might be an option.


With regards to sitemap or not, the sitemap file


only helps us with additional crawling within a website.


It doesn't prevent us from crawling these pages.


So removing these pages from the sitemap file


would not result in us dropping them from search


door and would not result in us recognizing that actually they


don't have any content there.


So removing something in a sitemap file


wouldn't negatively affect the natural crawling and indexing


that we do for individual pages.


So I think those are kind of the two aspects.


If you can recognize it's an empty search results page,


put a no index on it, removing it from a sitemap file


is not going to remove it from our index.


Let's see.


Starting from the 21st of September,


a number of our product pages across a few of our sites


has been labeled as soft 404.


We made sure that all critical resources and main content


are available to the bot and get properly rendered.


It seems like we are missing something


as we can't pin down exactly why this is happening.


Any suggestions?


I would probably need to take a look at some examples.


So I don't know if you're here.


Oh, you are.


Oh, fantastic.


Cool.


Hi, John.


Hi.


Pleasure to meet you, first of all.


Yeah.


Could I share with you a URL maybe?


Yeah, if you can drop them in the chat,


I can pick them up afterwards.


In the chat, in this chat, right?


It's always tricky to debug them live,


but it sends me a chat log afterwards,


and I always go through that for examples.


Yeah, I'll send you a couple of them.


It just that it's been happening since I mentioned my question,


right, the 21st of September.


And we just can wrap our head around what's happening.


I use the inspection URL tool in Search Console,


and all the main resources or sources are there.


So it's difficult to understand what's happening.


Yeah, OK.


Should I wait for an answer for you?


Usually what happens with these kind of things


is I double check them briefly, and then I forward them


onto the team so that they can take whatever action


that they need to there.


If these are really just normal pages


and we're recognizing them properly,


it sounds like something we have to fix,


not something you have to fix.


OK, fantastic.


OK, thanks a lot.


Really appreciate your help.


Cool.


Let's see, some of these, I think we already looked at.


My question is that Google is not indexing websites, even


fresh sites, and also not updating data in Search Console.


Is there any hidden update going on?


I mean, there are always updates going on,


so that's kind of hard to say.


I don't think there's anything explicitly hidden going on.


What I do sometimes see is that because it's so much easier


to create websites nowadays, people


create websites with a large number of pages,


and then they focus more on the technical aspects of getting,


I don't know, a million pages up,


and they disregard a lot of the quality aspects.


And then because of the way that Search Console tries


to provide more and more information about the indexing


process, it's a lot easier to recognize that, well,


Google is not actually indexing everything from this website.


And then the assumption is often there


that, well, perhaps this is a technical issue


that I just need to tweak.


And usually, it's more of a quality issue,


where when we look at the website overall


and we're not convinced about the quality of this website,


then our systems are not going to take the time


to invest in more crawling and indexing of the website.


So if you give us a million pages and the pages


that we end up showing initially don't convince us,


then we're not going to spend time


to actually get all of those millions of pages indexed.


We're going to hold off and keep a small subset.


And if over time we see that this subset is doing really


well and has all of the signs that we look at with regards


to quality, then we will go off and try to crawl more.


But just because there are a lot of pages on a website


does not mean that we're going to crawl and index


a lot of pages from that website.


Cool.


OK.


We still have a bunch of hands raised.


Wow.


That list is getting longer and longer.


I have a bit more time afterwards,


so we can continue on as well.


But maybe I'll just go through some of these that


have their hands raised now.


Let's see.


I think I trend Greece.


VASILIS GOULASENIAKOULOUIS-MARTINIOUKOSAKIS


Hi.


My name is Vassilis.


I am from Greece.


Hi, everybody.


We have a problem with our main site.


The DNS for a region are down, and we create another domain,


iTrendMobile.gr.


iTrend.gr now is dead.


So is there a way to say to crawl that move iTrend SEO


to iTrendMobile.gr SEO?


If the DNS for the old domain is not accessible, then we won't.


No, it's not accessible.


We're not right now because of that matter.


Yeah.


If we can't access the old site, we can't process and redirect.


So we need to be able to see the old site


and to see that there is a redirect from the old one


to the new one.


We can't just process the site move without looking at that.


OK.


One more question.


For how much time your boards, if you know,


I try to access, let's say, iTrend.gr, even if it's dead?


How long?


I don't think we have a fixed time.


So it depends a little bit on how often we try to crawl.


And if, for example, the home page


is something that we will try to crawl every couple of days,


and probably that will disappear fairly quickly because we


try a lot and we can't get to it.


But the lower level pages of the website,


maybe we just recrawl them once a month.


And then it takes at least that one month


before we can drop them from the search results.


And something very last.


If this court law has two stages, one short,


so you can have an answer, let's say,


in 20 days about the restore of the DNS, after that period,


is a way to, say, Google, recrawl the, let's say,


dead pages?


That happens automatically.


So if we see that a website transitions


from being broken to being fixed again,


we will try to crawl it a lot faster.


OK.


Let's say an alternative way to call Google about that


is a way through, let's say, Google Analytics or Google


Webmaster, the tools pages.


In Search Console, yes, you can submit them


for recrawling as well.


But usually, if the website goes from not being accessible


at all to accessible again, then we will crawl that.


So you can help a little bit, but it's not critical.


Thank you for your time.


Sure.


Thanks.


All right.


Mert.


Hi, John.


Thank you very much for doing this.


So basically, I did a small website for my mom's business


using a CMS tool called Squarespace.


I know that they're automatically submitted a site


map once you create a new page.


And now we've decided to add the e-commerce functionality


about two weeks ago.


And the site was password protected.


So my first question would be if Google penalizes you


in a way, if the user can't access the page,


if it's basically just password protected.


And the second would be, yeah, the site


was basically indexed and shown really nicely


on the pages before.


But after editing all those products and different pages,


I looked it up on Google Search Console for crawling.


But basically, my mom was giving me a hard time now


when these pages are going to be shown again.


So I'm kind of looking for answers.


Yeah.


I think so there's no penalty for having password protection.


But it means that we can't access


the content behind the password.


So probably that is the initial step


that happened with regards to kind of turning on almost


like an e-commerce site or shop section on your website.


We actually have a whole article on that


now in our search documentation specifically


for e-commerce sites.


So I would take a look at that.


There might be some tricks that you missed out


on that can help to speed things up there.


SPEAKER 2 OK.


Perfect.


Yeah, thank you.


That's it.


Cool.


Fantastic.


Let's see, Nisha.


Hi, John.


Hi.


Thanks for the question, too.


My question is regarding the performance measurement


of Google Discover.


So basically, I have a client whose site is up for, I think,


seven or eight years.


And they have a great content over their website.


And what I am seeing is for a particular day


and for a particular page, if Search Console is showing you,


let's say, 100 clicks.


And if you check that particular page and particular date


on Analytics, so it shows you the discrepancy in the sessions


and the clicks.


So there it is showing around 20 sessions in Analytics.


So there is a lot of discrepancy I saw.


So what is the good way to measure the Google Discover


performance?


I think the only way to measure it is in Search Console.


Because in particular, in Analytics,


the traffic from Discover is almost always


folded into Google Search.


And then you can't separate that out.


So it's only in Search Console do you


see the bigger picture.


All right.


OK, so there is another question.


OK, we should look at Search Console basically, as you say.


But if we are going to, so a lot of people


say this traffic goes to the direct channel medium.


So I mean, why is that?


Is there any particular reason for that?


I don't know.


I think it's something that the product team just


decided to do it like that in the beginning.


So in particular, Discover we see as a part of Google Search.


So we don't separate it out.


And in Analytics, it might be that I


don't know exactly how they would even track that there


in which category.


But at least the Discover team, they basically


decided, well, Discover is a part of Search.


So we're not going to separate it out.


OK, thank you.


Sure.


Amos.


Hey, John, good morning.


Yes, I had a question in regards to internal search pages.


So we're allowing indexation of on-site searches.


So when someone does a search on our site,


we create a page for that.


And now that's gone out a bit of control.


So we have hundreds of millions of these pages.


So how would you recommend we sort that out?


And if there is actually any benefits to cleaning that up


or if we shouldn't worry about it?


I think for the most part, it does make sense


to clean that up because it makes crawling a lot harder.


So that's kind of the, I don't know,


the direction I would look at there


is to think about which pages you actually


do want to have crawled and indexed


and to help our systems to focus on that.


Not so much that you should get rid of all internal search


pages.


Some of these might be perfectly fine to show in search.


But really try to avoid the situation where anyone can just


go off and create a million new pages on your website


by linking to random URLs or words


that you might have on your pages.


So to kind of take it and say, well,


you have control of which pages you


want to have crawled and indexed rather than whatever


randomly happens on the internet.


And those pages then that, so we control this,


and the pages then aren't any longer links.


They're kind of these orphan URLs,


but they're still going to be crawl and probably indexed.


What do we do with them?


Do we throw up for them, noindex them, or?


I mean, if they're still usable by users,


I would keep them accessible, but I


might add a noindex to them.


So one thing you could do is create a list of search queries


where you think we have reasonable content


and just double check that list.


And if it's on the list, then let it be indexed.


If it's not on the list, then add a noindex to those pages,


something along those lines.


And finding out which of these pages are actually reasonable


is sometimes a bit tricky because it's


easy to take simple metrics and just say,


oh, I will just take the top 10% pages from traffic.


But I don't know if those are actually the good pages.


If you would create category pages for those,


are those the ones that you would pick or not?


So fine.


SPEAKER 2 And it's continually changing, too.


So you might get pages that suddenly have demands


and then don't have demands.


So it's a tricky thing to manage.


Is there any kind of benefit, then, do you think,


a longer term?


So if we do this cleanup, obviously,


the business would argue, if we're going to do this work,


it's investment from our side.


What is the return on this?


Is there an SEO benefit to that?


So on the one hand, you have the crawling side,


which will be affected.


So if we go off and don't have to crawl all of those pages,


then we can focus more on the actual content


you do care about.


And especially with an e-commerce site,


you have things like price changes all the time.


And if we can't pick up the price changes quickly enough,


then that's kind of a bad thing.


Whereas if we can concentrate our crawling more on the content


that you do want us to care about,


then that helps to improve those pages.


So it's not so much that there's an immediate ranking


boost for your website overall.


So if you take care of that, it's more, well,


you're focusing the energy a little bit better.


The other aspect that sometimes plays a role


is that these pages might be seen as being low quality


pages, and almost like doorway pages,


and that you just have different variations of the same keywords


over and over again.


And that is something that could pull down


the overall quality of a website.


But it really depends on the specific situation.


And it's not the case that you can say,


well, I'll clean up my internal search results,


and then Google will think my website is suddenly


significantly higher quality.


It can be a small change.


And sometimes a small change is relatively large for a website.


But I wouldn't rely on that aspect.


I would really focus more on the crawling aspect.


SPEAKER 1 Great, thanks.


One final question is, so these pages


are linked from the footer of the site


across most of the pages of the site.


And is there a, in Google's eyes,


is there any benefit to have these links in the navigation?


So they're more seen from a user's benefit


to navigate through the site than having them


in the footer.


I think there might be a subtle difference.


But I don't think you'd see a measurable change in that.


Because we do try to figure out what the primary content is


of a page and focus on that.


But it's not that we will ignore everything


that is out of that area.


I think that's something where you


can work on that to make the site's UX better overall.


But I don't think you'd see an immediate change in SEO


from changing, kind of like moving links from the footer


to the primary content.


OK, thanks.


That's it.


OK, let me take a break here with regards to the recording


so that we have a reasonably long session.


And thank you all for all of your questions so far.


It's been great having you all here.


And if you're watching this on YouTube,


you're, of course, welcome to join these sessions live


or at least drop your questions as well into the community


section in YouTube, where we always


have the next one signed up.


Should my web page show multiple languages? #AskGooglebot

Today's question comes from Yusef, who asks if it's OK to have two language versions on the same page.


Well, for search, it's ideal when our systems can recognize the primary language of a page.


This makes it much easier to recommend the page to users who are searching in that language.


Having some content in other languages is fine, though.


In the case of a website that's teaching another language,


my recommendation would be to clearly have one primary language for each page.


You can create pages for other languages on the same website, too.


There's no meta-tag that you need to use to tell search engines your language choice,


but the HTML lang attribute is recommended for screen readers.


Making websites in multiple languages adds complexity.


We have a lot of information on the options in our developer documentation linked in the description below.


Et voila! I hope this helps you on your journey to creating a great multilingual website.


English Google SEO office-hours from October 2022

Hello, hello, and welcome to the October Google Search SEO office hours, where we'll be taking


a look at some of the questions that you all submitted to us. My name's Lizzie, and I'll be


answering some questions today together with some of the other folks on the Search Relations team.


We're still experimenting with the audio-only format, and so you may still see some changes


in the way that we're doing these sessions. All right, let's get started.


Bruno asks, what is your advice for websites following best practices in international SEO,


but where Google still shows the wrong folder? So in general, you can't guarantee that a user will


only ever visit the correct language or country version of a page. That's just from the way that


the web works. It's impossible to guarantee, even if Google got everything right. So my recommendation


is always to make it as easy as possible for users to find their way to the right version.


One way to do this is to do some kind of a banner on your pages. When you recognize that the user is


from the wrong country or from a wrong language version, and they're going to your pages,


show them a banner. This banner can be implemented with JavaScript if you want something that's a


little bit outside of your server. It can also be something that you run directly on your server


yourself. The advantage of using a banner versus just redirecting the user to the so-called right


location is that you can never be 100% sure that the location which you have in mind is actually


the one that is correct for the user. And additionally, by redirecting the user automatically,


you kind of run into that situation where perhaps one of Google's crawlers tries to access your


pages. And if they're always redirected, for example, to the English version of a page,


we would never know about the other language versions that you have there.


So ideally avoid automatic redirects and instead show some kind of a banner to users so that they


can find their way to the preferred version. Matthew is asking for single page apps or SPAs


with server rendered HTML. Is there a way to tell Googlebot it doesn't need to run those SPAs


JavaScript bundles? Hi, Matthew. The short answer to your question is no. The slightly longer


version is from the server-side rendering, you get the benefit of having a very robust setup,


and the content is in the HTML as early as possible. So Googlebot benefits from the


server rendered HTML to begin with. However, avoiding running the JavaScript bundles through


Googlebot would increase complexity dramatically, and complexity usually attracts bugs. There's no


benefit of avoiding running the SPA bundles in Googlebot, so I would not worry too much about it,


to be honest. Anonymous is asking, I teach people foreign languages and I give them example


sentences with translations. What would be the best HTML markup for a sentence or a word in


a foreign language? Thanks for your question. From a Google search perspective, there's nothing


particular you need to do in the HTML itself, as long as the content is primarily in one language.


You can use hreflang as usual to specify the language of that page, and as for the specific


words and example sentences that are throughout the page in another language, the W3 HTML


guidelines do say that you can tag the language with the span language equals whatever the


language code is to flag these things in line. And it's always a best practice to follow HTML


standards. It's not always about SEO, and these things can also be used by screen readers to


change pronunciation, so it can be helpful to add for other reasons. Tech Swang from Twitter is


asking, is it duplicate content if I post the same article on my e-commerce site and on my blog?


Well, yes, of course it's the same content, but maybe that's fine. So taking a step back,


when it comes to duplicate content, we try to index one version of a piece of content,


and we try to show that in the search results. If you publish the same content on multiple sites,


like your e-commerce site and your blog, we will try to pick up one of those and show that.


It can happen that we pick up both of these and show them in the search results.


If you want to fix that on your side and you really want to say, well, I would prefer to have


my blog indexed instead of my e-commerce site, you can do that. You can use the rel canonical link


annotation on these pages to let us know which of these is your preferred version. The rel canonical


is not a guaranteed way to set the canonicalization. We pick up lots of different signals to make that


work out, but it's a great way to let us know about this information, and ideally we would try


to follow it and index that canonical version. That said, if you do end up just publishing this


without anything extra on both of these places, that's also fine. It's not that we have some kind


of a duplicate content penalty that would suddenly make your site look less valuable in search,


just because you have the same content in two locations. Oftentimes, that's just how things


work out in reality. You can't always set a rel canonical. Sometimes you have multiple country


versions with the same content as well. All of that is essentially fine. Bilal is asking,


is it good to add FAQs in a blog post? Do you think Google won't consider it for ranking


purposes? Hi Bilal. It's totally fine to add FAQs to a blog post. It's not inherently good or bad


that it's just a format. And if you're thinking about your audience and answering what's top of


mind for them, then you're on the right track. It's important to remember the format itself is


not inherently good or bad. It's more about how you answer those questions and making sure that


those FAQs are actually the questions that your readers want to know more about. Bilal asked,


Google is displaying the wrong pages to its search visitors. As an example, in the US,


Google recently selected the main domain as a canonical instead of domain.com slash


English-US, for example. How can I fix this? The appropriate approach to handling these kind of


international issues with regards to pages where you have the same content on multiple pages,


maybe for different countries or for different languages is to use hreflang. Hreflang is an


annotation that you can add to the header of your pages or to a sitemap file or the HTTP header


that essentially links different country and language versions of pages together and lets us


know which of these pages belong into one set. For this to work well, you also need to keep in mind


that things like redirecting home pages also need to be a part of that. Because if you don't include


the redirecting home page, then we will see that page as being separate from that set of hreflang


pages and we might show it separately in the search results. The correct approach if you have


an automatically redirecting home page is to use the x-default hreflang annotation for that


redirecting home page and then the individual country and language versions for the appropriate


sub URLs that you have there. My guess is that this might be missing there. The other thing to


keep in mind is everything around internationalization is never guaranteed.


It's always possible for users to reach the wrong version of a page. And as I mentioned before,


make sure that you have this backup banner available as well, where if you can recognize


that users are going to the wrong version, you can guide them to the right place or at least


the place that you think is correct for them. Martin is asking, we have an Angular app that is


deployed, but is faulty in the web cache. Is that problematic? Hi, Martin. Nice to meet another


Martin here. No, that's not problematic. The web cache is a convenience feature to work with pages


which are no longer available. Your Angular app will be fine.


Hiba asks, using a foreign language in search while being in a different country,


why do the search results positions differ? This is a great question. And I think anyone


who's worked for websites that are not in their own country runs into this as well.


In practice, this comes down to things like geo-targeting, where we try to figure out which


country is relevant for which kind of queries and also localization when it comes to things like


different languages. A lot of times the results do vary from country to country, and they do vary


from language to language, and that's expected. For some kinds of queries, it's obvious that a


user wants to find something local. So if you're looking for, I don't know, washing machine repair,


then you probably want something local. Whereas if you're looking for a washing machine repair


manual, then you probably don't care if it's local or not. You just want the manual. So with that in


mind, it's expected for things to vary across different countries and languages. If you'd like


to try things out in a little bit of a more advanced way, one thing you can do is use the


advanced search settings within Google search, which you can access with one of the menus on the


side. There you can specify individual countries or individual languages. And with that, you can


often drill down and see, well, what would be relevant in this particular country or in this


particular language? Amit Saha is asking, in a UGC content site, we run a weekly no index tag


edition program over pages with no traffic. Does it affect SEO due to continuous increase of no


index pages over time? My goal is to get the new contents crawled fast and indexed rather than


wasting crawl budget on zero traffic pages. Hi Amit. So no index won't really help you here if


you're primarily wanting to optimize crawl budget. In order to find the no index tag, Google needs to


crawl the page and that's using crawl budget. The only way to control the crawling of your site is


with robots text. Also having more or fewer no index pages doesn't affect the overall SEO or


ranking of the website. If this is more about removing low quality content, then sure,


no index can do that, but you can also just remove the low quality page directly.


Does hyphenation matter for keywords when optimizing titles and headings?


I've seen differences in search volume on the Google ads keyword planner. In Portuguese,


for example, there are keywords that need a hyphen. Well, Rafael, thank you for your question,


but unfortunately I don't know about the Google ads keyword planner. So that's not something


I'd be able to really help you with. I'm not sure how that search volume there would be compiled and


how things like hyphenation play a role there. In practice, when it comes to search, my


recommendation is to think about what users would be doing when searching and try to express the


right approach on your pages and ideally match what users are doing as well. So some words,


for example, in German are just really long words and that's just a really long word. You can also


do the same thing with a hyphen in between, but sometimes that also changes the meaning slightly.


So from our point of view, it's not something where you can just randomly add a hyphen or


remove a hyphen or separate things out into separate words or not. Sometimes these are seen


as different words. And again, I don't know how the keywords planner would handle these


kinds of differences or if that's something that maybe is smoothed out overall or depends


on the individual languages. I honestly don't know.


Rajeev is asking, I have a finance blog that's ranking well, and I covered some topics related


to new startups and human resources. Does this affect anything with the helpful content update?


Hi, Rajeev. Sure, you can totally cover multiple topics on a website or a blog,


as long as they're relevant to your audience and you've got something valuable to add there.


One question to ask yourself is, would your audience find the content useful


if they came directly to you? Or are you writing this new topic because you thought you'd get


more search traffic because of it? The thing to watch out for with the helpful content update


is that you're creating this content for your users first and not primarily for search engines.


Hope that helps. All right, Roland is asking, is it a problem if an editorial page contains too


many links to external pages? Is there an upper limit here? Well, taking a step back, yes, there


is a theoretical limit to the number of links on a page overall, but it's way above what's practical


for a web page. Even if you have a mega menu on your pages, you're unlikely to reach any of those


limits. That said, it's useful to be reasonable when it comes to the web. I think overall,


your users aren't going to try something like 100 links before they can find what they're looking


for. So ideally, you should make it as easy as possible for them to recognize important parts


of your pages and the important links there. So I would really try to limit it to something that's


reasonable. Another person asks us if the cached version of a page does play a role for search and


top stories. The answer is no. The web cache really is just a convenience feature. If you


don't see something in the cache that is not a problem, please use the testing tools such as the


URL inspection tool or the mobile-friendly test to see if the content is in the rendered HTML.


If it is in the rendered HTML, you will be fine. Search Console still shows tons of same site


non-secure pages as referrals to old HTTPS pages. How do I flush them out?


Bringing some context in here, this is specific to the URL inspection tool in Search Console,


where we show the referring page in the indexing section. For this, you don't have to do anything.


It's perfectly fine to keep those pages there. The referring URL is the place where we first found


a link to that particular page. If this page has been indexed for quite some time now, then that


initial link might be quite old as well, and it might be an HTTP page. From our point of view,


that's perfectly fine. This is more about giving you information about where did Google ever pick


up this page, and it's not something that we will take into account when it comes to


search ranking, for example. All right, we have Sudip asking,


is 150 words enough for a news article? Well, there's no minimum or maximum word count for


a given article. Word count alone is not a ranking signal. Instead, we recommend that


you focus on creating high-quality content that's relevant to your audience. In the case of a news


article, does it cover all of the relevant details, facts, and information about the story?


There's no magic number for how many words it takes to answer that question.


All right, Roland asks, if all editorial pages of a website start with the same introductory text,


for example, a text block that encourages newsletter registration, can this have a


negative effect on the SEO visibility of the individual article pages? All right, yes, you can


reuse text blocks across a wide part of your website, and that's fairly common. For example,


most websites have a shared header and a shared footer across a large number of pages, and that's


perfectly fine. In practice, what happens is that one of these pages may show up in the search


results if someone searches for something specifically related to that shared piece of text,


which I suspect would be rare for something like a newsletter registration, unless people are


searching for your company newsletter registration, in which case it would make sense to show these


pages. But in most cases, you probably wouldn't be seeing that. It doesn't negatively affect the


search results for the other parts of the page. That said, a page should have sufficient unique


and compelling content of its own, so it should be more than just, well, a header menu, the


newsletter pop-up, and one or two sentence piece of context followed by a shared footer. It should


really be something where there's sufficient unique content on the page as well. Sam Gooch is


asking, a site uses machine translations to offer posts in other languages. The content is reviewed


by human translators, and they're often happy with the quality after minor tweaks. Is this okay for


Google? Hi, Sam. Well, that's good to hear that the human translators are happy, and this is


totally fine for Google as long as there's a human involved in the review process. That's the key.


The thing you want to watch out for is making sure that the quality continues to be good


and working well for the humans that are reading the content.


All right. So I have hundreds of URLs that are indexed but blocked by robots because of SID


parameters in my URL. If I do a site query for domain with in URL SID, I see a lot of


omitted results. Does Google know that these are duplicates? Is there anything I have to do?


Well, it's kind of a complicated question, but in practice, I wouldn't worry about this. If these


URLs are blocked by robots.txt, then we don't know what's on these pages. We might still index those


pages because we see a reference to those URLs somewhere, but in practice, that's not something


that would show up in the search results. And like you notice, you really have to dig for those


by doing a specific site query plus an in URL parameter, and then you have to expand the


omitted results. So it's really not something that users would see. And with that, it's really not


something that would affect your website in a positive or negative way. The one time I would


watch out for robot URLs is when they're ranking instead of your actual content, because that to me


is a sign that, well, probably your actual content is not relevant or is not clear enough for users.


So that's something where I would look into what is happening with robot URLs,


but if these robot URLs are just showing up in a site query, that's perfectly fine.


So, LaLinda asks, does it make sense to add organization and review schema to every page?


Well, you don't need to add organization schema to every page. You only need to add it one time.


And which page you pick is completely up to you, although the homepage or an about this


organization type page may make more sense depending on how your site is set up.


And as for review rating, you want to add this markup to pages with reviews about a particular


item. If the page doesn't have reviews on it, don't add the markup. You want the markup to


match the content that's on that page. It's also important to note that for local businesses or


organizations, any review that's self-serving, meaning the entity hosting the review is also


in control of the review that's about itself, is it eligible for those review stars? Make sure to


check the documentation for details on specific features. It covers edge cases like this,


requirements, that kind of thing. And also keep in mind that schema is used for other things. It's


not just for SEO. But if you're curious about what does Google search need in order to show


your content in a particular feature, check out the search gallery for more information about that.


All right. Roman asked, what are the implications of merging and redirecting a website B to website


A if website B is actually a hundred times bigger than website A? I'm concerned about diluting the


overall topicality of the website. The two websites are in topically close niches, but not


really having the same intent. All right. So I think this kind of topic is really complicated


because there is no really clear answer with regards to what happens when you merge or split


a website. In practice, if you do a site migration from one website to another one, and you're just


changing their domain name, then that's something that is very easy to process and fairly


straightforward. And that's something where you have clear expectations on the old state and what


should happen with the new state. But if you're merging two websites, what would the situation be


afterwards? These are two independent websites. And for these kinds of things, there is no absolute


answer. So that's something where I would watch out for setting expectations if you're doing this


with clients. If you're doing this for your own websites, then keep in mind that you're not


just moving one website to another. And by creating something new, essentially Google systems


have to look at the new overall website before they can really understand what is happening and


what is the context of this new website within the scope of the entire web. So that's something


where I would be cautious. In practice, sometimes you just have to do these kind of things. We've


had to migrate a part of our documentation from various blogs to our main developer site,


which is also a part of a bigger developer site. And these kinds of migrations happen.


It's just have to keep in mind that it's sometimes a bit tricky. And when things get tricky,


one thing I recommend is really trying to follow our recommendations as much as possible,


ideally tracking down all of those details. And so I would recommend doing that.


So I would recommend doing that. And secondly, keeping a log of everything that you're doing


so that afterwards, should something weird happen, you can trace back and see where that change is


coming from. So if you're merging two websites, keep a track of the individual URLs that you're


merging, the old source and the new destination, so that afterwards, when you look at your new


website, you can follow back and see, oh, well, this specific page actually comes from that


page. And the sooner you can keep track of this kind of migration and follow along, the more


likely you can catch any issues before they cause something bigger to go wrong.


So I've sure asked us, Hey, john, we want to use more modern image formats like WebP,


of concern. Hi there, I'm not john, but I would still try my best to answer your question here.


There's different strategies to do this. One would be to use some sort of image proxy. The other one


is to basically use source set or the picture element to provide fallbacks for different users


and devices and browsers, then the browsers will figure out which format to show that has the


upside of being absolutely backwards compatible, and relatively easy to do. There are a bunch of


services and software that allows you to convert an image into all the different formats. All right,


that's it from my side for this round. All right. And that about wraps it up for this month's


rendition of Google search SEO office hours. We hope that you're finding it useful or at least


interesting while we're in experimental mode with the format and the name and how we're publishing


these. Let us know what you think. You can leave us a comment on the YouTube video or on Twitter


Why do Search Console graphs fluctuate?

Today, we have a question from Michael on Twitter.


Why do Search Console graphs fluctuate?

Michael asks, why the graphs in Search Console


Why do Search Console graphs fluctuate?

sometimes just go up and down on their own?


Why do Search Console graphs fluctuate?

Well, that's a great question, and one


Why do Search Console graphs fluctuate?

I'm a bit familiar with because I often


Why do Search Console graphs fluctuate?

leave my website alone for months,


Why do Search Console graphs fluctuate?

and the numbers also change.


Why do Search Console graphs fluctuate?

What's up with that anyway?


Why do Search Console graphs fluctuate?

Well, Google Search includes many incredibly complex


Why do Search Console graphs fluctuate?

and interconnected systems, even small changes on one side


Why do Search Console graphs fluctuate?

can have a surprisingly visible effect.


Why do Search Console graphs fluctuate?

These changes tend to even themselves out overall,


Why do Search Console graphs fluctuate?

but when looking at individual parts,


Why do Search Console graphs fluctuate?

they can still be noticeable.


Why do Search Console graphs fluctuate?

For example, if crawling from one data center


Why do Search Console graphs fluctuate?

is minutely faster than usual, then that


Why do Search Console graphs fluctuate?

can cause changes to the content we have available for indexing


Why do Search Console graphs fluctuate?

and, in turn, the content shown in search results.


Why do Search Console graphs fluctuate?

Or perhaps you write an insightful social media post


Why do Search Console graphs fluctuate?

that suddenly becomes important, and our systems


Why do Search Console graphs fluctuate?

focus on that a bit more.


Why do Search Console graphs fluctuate?

Either way, any of this can cause fluctuations


Why do Search Console graphs fluctuate?

in the charts.


Why do Search Console graphs fluctuate?

So what can you do when you see the graphs fluctuate like this?


Why do Search Console graphs fluctuate?

First of all, don't worry about the little changes.


Why do Search Console graphs fluctuate?

These are normal for any website.


Why do Search Console graphs fluctuate?

Things go up a bit.


Why do Search Console graphs fluctuate?

Things go down a bit.


Why do Search Console graphs fluctuate?

That's all fine.


Why do Search Console graphs fluctuate?

Second, look out for trends.


Why do Search Console graphs fluctuate?

Has the graph been continuously headed in one direction


Why do Search Console graphs fluctuate?

over a few weeks?


Why do Search Console graphs fluctuate?

Then that's often a sign of broader changes overall.


Why do Search Console graphs fluctuate?

And finally, do watch out for big spikes.


Why do Search Console graphs fluctuate?

When the graphs change significantly


Why do Search Console graphs fluctuate?

over a short period, then that can


Why do Search Console graphs fluctuate?

be a sign of a serious change.


Why do Search Console graphs fluctuate?

It might be a good change, but I'd recommend you double check.


Why do Search Console graphs fluctuate?

So overall, small fluctuations over time


Why do Search Console graphs fluctuate?

are completely normal and not a sign of issues.


Why do Search Console graphs fluctuate?

Watch out instead for broader trends and big, sudden changes.


Why do Search Console graphs fluctuate?

I hope that helps.


Why do Search Console graphs fluctuate?

See you next time here on Ask Googlebot.


Why do Search Console graphs fluctuate?


SEOs & Devs: Harmony in Tech - Official Trailer

Why don't SEOs listen to developers?


Why can't SEOs give us technical guidance?


Do SEOs really keep up with the most recent search engine changes?


Don't developers want people to use what they build?


Sometimes I'm looking at job offers for devs and I'm like, what does it even mean?


Basically one job offer is an entire IT department.


I'm very result-driven and I know like my team, the teams I've been in have been result-driven


and that's in a place of the love for the user.


I like that statement.


It's tough for engineers to not think of SEO as a bunch of smoke and mirrors.


I mean I've been told that statement a million times.


The entropy of a website, like if you don't look at it, it's like CSS.


Like CSS will naturally decay.


There's sometimes people who do loads of marketing and loads of stuff on a website


and do lots of work and they don't necessarily call it SEO.


I'll tell you this, no SEO is trying to ruin your website.


Hello, I'm Martin Schmidt, host of SEOs and Devs, Harmony in Tech.


English Google Webmaster Central office-hours from December 10, 2019

Mm.


Hold on.


Now.


Wow, that was recorded.


OK, welcome, everyone.


Today is Webmaster Central Office Hours Hangout.


We have a bunch of special guests today


that have chosen to join us here in Zurich.


I don't know, do you want to introduce yourself?


OK, I can start.


So I'm Ulrika Veburg.


I come from Stockholm, Sweden.


I do technical SEO and digital marketing


and digital strategies.


I run my own agency called Unicorn.


Cool.


Interesting.


My name is Peter Nikolov.


I come here from Russe, Bulgaria.


I built the stuff for my own company


called Mobile Development.


So as the name shows, it's mostly for development.


But last days, we were working a lot of SEO stuff


or things like that in those sections mostly.


Yeah, I mean, most of you probably know me by now.


I'm Martin Splitt.


I work with John here in the Zurich office.


And as we have a few guests, I thought


I'd just pop in again as well.


So if you have JavaScript questions, fire away.


And I'm John.


I do these Office Hours Hangouts from time to time.


I don't know, since we have a bunch of people here


in the room, do any of you want to get started with a question?


Or?


I think you should start with someone and then.


I could jump in, if that's OK.


All right.


Someone from the Hangout.


Go for it.


Hey, hi, John.


Hi, everybody.


I have kind of like a technical question.


So we have a customer who bought a domain name which


used to be hacked.


So the site used to be hacked.


We know that.


And the issue is that there are millions of URLs, unique URLs,


that are indexed in Google at the moment on this old hacked


website.


So currently, the site is about 100 pages.


It's just a regular site and too fancy with some regular URLs.


But we are having issues on our server


because Googlebot is crawling all these old URLs.


And they get redirected, or they even give it 200 status code,


or 404.


But my question is, what is the best approach here


to, in the first place, get all these URLs out of Google?


And in the second phase, make it so


that Googlebot doesn't overload our server with requests.


Because the past 14 days, I think,


we had about 15 million unique requests from Googlebot


trying to get these pages.


But they are old hacked pages.


So I'm wondering what the best approach is here.


I can help you.


It's impossible.


So I have similar situation like you,


like years ago or two years ago with different customers.


So probably best way is to make in HD access things


a write so that hacked URL to return 404 or 410 gone.


And this will stop processing the, for example, PHP


making calls to the database or stuff like that.


This will help because a patch will send that URL


and return direct status code to the Googlebot.


And this will help you with, for example, when Googlebot go


and make the CPU hammering to your website.


Another way, but probably you need to be dev up things.


You can make, for example, get a virtual machine


from someone, for example, like Google Cloud,


and make some kind of redirector to the right site.


But if URL is infected, that site


will return 404 or 410 directed to the Googlebot.


So it's not easy.


But if you have do that, it actually


can take a few hours of work.


I think that kind of makes sense.


I mean, essentially what you want


is Googlebot to stop processing these URLs.


And it primarily does that when it sees


that these are always 404.


So there will be a period of time


where we try to crawl all of these like crazy.


And then after we see that it really


doesn't make sense to crawl these anymore,


then we stop crawling them or drastically reduce them.


So some kind of a way to bridge that time until that happens,


I think, makes sense.


And removing them from search is something.


So I would first see if it's actually


a real problem in that if for normal queries,


these pages even show up.


Because if they only show up for site queries


where someone actually knows that there


is hacked content on this site, then


that's probably not so much of a problem.


And they'll drop out over time.


But if you really need them removed urgently,


the URL removal tool is kind of what you would need to do.


The tricky part there is you need


to do that probably on a per directory level


to say those hacked directories are things that should not


be string-owned in search.


And the removal tool doesn't remove it from the index,


despite the name.


It just hides them in search.


So it will still crawl.


It needs to see the 404s.


And then it just won't show them in search.


SPEAKER 1 OK, thanks.


Just a small follow-up question.


We are currently rate limiting all bots


that are crawling the server because our server was going


down because of the requests.


Do you think that's the idea?


Do we have to turn off the rate limiting?


SPEAKER 2 I think that's fine.


So what will happen in practice is


Googlebot will crawl less.


And we'll try to prioritize better.


So we'll try to focus on the URLs


that we think are actually important, which


should be the normal content.


SPEAKER 1 May I extend you?


SPEAKER 2 Sure.


SPEAKER 1 So maybe I support that John said.


But if you have a budget for that,


my idea is to get a VPS to extend that VPS


and to handle everything there to remove that crawling


limit.


So let Googlebot see and remove them faster.


And later, can be month or two or three later,


you can shrink down that VPS things to be normal,


or even move to the shared hosting again, if possible.


Yeah, I think that also makes sense.


I mean, if you can do that, that cleans it up


a little bit faster than if you slow things down.


You can also slow things down in Search Console


if you really need to with the crawl rate setting there.


SPEAKER 2 OK, maybe one more follow-up question.


Would it be a good idea to, because as I said,


the current site is only a couple hundred URLs,


would it be a good idea to use the URL removal tool


and just temporarily hide the entire website,


to just use slash at the end and temporarily hide everything


and start from a fresh index, so to speak?


So de-index everything?


SPEAKER 1 It doesn't remove it from the index.


It just hides it in the search results.


So that wouldn't fix anything.


SPEAKER 2 OK.


SPEAKER 3 Another way, just to extend it,


is if site is very small, for example,


I have a client with 10 pages that adds a few million pages.


So if site is very small, but is built on WordPress or WooCommerce


or things like that, maybe it's a good idea


to make a static website to put it on the server,


disable all PHP processing or Node.js or whatever


it is in background, and let Googlebot


access in that static files, because this will make zero CPU


usage to the hosting.


And you can clean up much faster.


OK, but this is for small or medium sites


where you have, for example, thousands of pages.


That's OK.


But if you have a million pages, that is virtually impossible


at large scale.


But for small or medium sites, it's OK.


Let's work around.


SPEAKER 1 With Googlebot at that specific IP of the website,


for the particular time saying, OK, this is hacked,


is there any ranking issue with that?


So I mean, we try to recognize when it's hacked content,


and we'll flag that in search.


But it kind of depends on what you're actually seeing there.


If you're seeing that it's flagged as hacked,


then you can go through the reconsideration flow, which


kind of helps with that, too.


But it sounds like we just know that there


used to be hacked content there and tried to crawl


as much as possible.


But then the rankings will flow back to where it was, right?


SPEAKER 2 Well, if you're taking over an existing domain,


then you would have a new website there.


It wouldn't be ranking like before.


SPEAKER 3 It's different content,


so you would have different signals and thus different


rankings.


SPEAKER 1 No, I mean, if you were in the same industry,


for instance, if you were in one specific industry


and the site did not change.


Well, if it literally did not change and the contents did,


because being in the same industry does not mean


that the content is the same, right?


If literally nothing changes when you take over that website,


then sure.


Maybe.


Yeah.


Even then, other websites can pop up,


and ranking is volatile.


I am even commenting on this.


It's ranking, I know.


SPEAKER 2 OK, thanks, guys.


Sure.


All right.


Any other live questions before I


jump into the submitted ones?


SPEAKER 1 Hi, John, I have a question.


SPEAKER 2 OK.


SPEAKER 1 Hi, my name is Matt.


I work for a well-known image and multimedia


brand with hundreds of millions of images in our catalog.


We also have tens of thousands of contributors


to the site that add their own tags for those images.


And the site automatically creates a combination


of the category of tags, which has unfortunately


made for some difficult pairings.


In the last year, the site has become much more well-known


for not-safe-for-work content, adult terminologies,


because of these pairings.


And it's ranking in the top three results


with a pretty good click-through rate.


My general question is, does a site's high rankings


and relevancy in a category of terminology


that's not safe for work preclude it


from ranking well for other more innocent or professional


terms?


It doesn't prevent it completely.


But what could happen is that we would see it


as something that would need to be under the SafeSearch


filter, which means if someone has SafeSearch turned on,


it might be that we wouldn't show it.


So that's something where usually our recommendation is


if you have adult-oriented content


and kind of general content, you would


have adult-oriented content and kind of general content,


then separate those out as cleanly as possible


by directories or subdomains so that we can kind of clearly


identify this part of the site should be under a SafeSearch,


and this part of the site doesn't


need to be under SafeSearch.


And then it's a lot easier for us


to make sure that we're showing it at the right time.


Otherwise, if it's a mix of different things on one site,


then it could happen that we think the whole site needs


to be filtered with SafeSearch.


And then it's not so much a ranking issue.


It's more that, well, it's not visible to people


who have SafeSearch activated.


All right.


Thank you so much.


Sure.


Can I type in?


Am I allowed to ask ranking questions?


Sure, we can try.


OK.


If a site is about books and then expands to CDs,


would it have equal good chances of ranking in the new vertical


given the same quality of content?


And would the first category say books suffer


from the expansion in verticals?


Not necessarily.


I think that seems like a fine move.


And that's kind of like a natural evolution of a website


anyway.


You grow, you have more different types of content.


That seems completely fine.


Thank you.


Sure.


All right.


I'll jump into some of the submitted ones.


And if you all have any comments or questions along the way,


feel free to jump in.


The first one is about international sites.


All of our English market products and recipe pages


are being canonicalized to their global English market


equivalents.


Regardless of hreflang and Search Console


targeting for our subfolders, local markets,


Search Console show duplicate Google


chose another URL as canonical pointing


to the English global version.


We believe this is due to duplicate content.


So we ran some tests between the global English and the English


UK markets.


And only a full copy change was effective.


As there are limited ways to fully rewrite the same product


and recipe content, what would you recommend to solve this?


How widespread do you see this canonicalization


effect on the web?


You've probably seen this in the forums a few times as well.


Yes.


That question is asked multiple times.


So one of the best solution is to try to check,


recheck the things.


Because many times, we have seen very specific, very small


issues in implementation.


But they totally broke your hreflang issues


and make strange, weird things like that.


So even if you have seen this, check, recheck,


and then check another time that is.


Because most of times, there is something


is burnt into a lot of code, a lot of piece of things.


And something is left, like probably wrong canonicals


or redirects or hreflang for Australian language


is redirected to South Africa or things like that.


And this, if Googlebot is seeing this,


Googlebot then start to not trust to the hreflang things.


And we have seen weird things that


can be explained original in some easy way.


I experienced that as well.


Keeping stuff in order is the easy answer here.


Yes.


I think with hreflang, it's really tricky.


Because it feels like something easy to connect.


But in practice, it really gets complicated quickly.


I'm sorry to interrupt you.


And one of best answer for us is try


to make hreflang things in different domains.


Because if you have see country-specific TOD,


Googlebot know that, for example,


this is for German people.


This is for Switzerland people.


This is for Australian people.


And he know that.


But if you have put everything into same domain, most of time


where things happens inside.


Where is possible?


I think the different domains help,


especially for geotargeting.


Yes.


That makes sense.


Yeah.


One of the tricky things with, I think,


this question is that when we see the same content,


like you have English for UK and English for Australia,


and it's really the same content,


and we think these are duplicates,


and we try to help the webmaster by folding them together.


And what happens then is in the reporting in Search Console,


we only show one version.


And in the search results, if hreflang is set up properly,


like you mentioned, it's sometimes tricky,


then we will show the different URL in the search results.


So we will show the different versions,


but we will report on just one version.


And that makes it really kind of confusing sometimes.


But that's the thing.


From our point of view, that's expected and not necessarily


a problem.


Like, you wouldn't rank lower.


It's just very confusing in Search Console.


Exactly.


So John, when we've tried this sort of thing,


regardless of speed, backlinks, correct hreflang,


correct hreflang, it doesn't seem to matter.


So when we see this as a business,


if you were operating from a supply chain model


and you were being folded up algorithmic,


from the site from Googlebot, can you


imagine the cost of that to a business if,


because Googlebot might not know what we're trying to achieve.


So you're saying, OK, maybe try subdomains.


But even with Search Console targeting,


hreflang is designed to bring people to the right site.


So where we're saying, and plenty of sites around the web


have English content for different sites,


where we have English content for different locales.


So how do you know when to roll up that site


if it's duplicate content?


Because if you're on a supply chain basis,


that would affect the business.


And if there's no way out of that other than fully rewriting


content, what do you do?


So it shouldn't affect the search results.


So in the search results, we would


show the different language versions,


like the UK and the Australian versions.


It's just in the reporting in Search Console.


We would only report on the canonical data,


which would be one of those.


So in practice, the traffic you get is still the same.


The rankings are the same.


It's just the reporting in Search Console


is focused just on one of those versions.


So the reporting side is the confusing part.


And at the moment, I don't really have an answer


for how you can separate those things out properly


from a reporting point of view.


Because the indexing report will be on the canonical URL, which


will be one of those versions.


And the performance report is on the canonical URL.


In the performance report, you can separate by country,


which gives you that information again.


But it's still on the canonical URL,


not on the individual country URLs that are actually


shown in the search results.


Yeah, OK.


It's just very, very hard to get out of that situation.


Because I can either say to the business, OK,


you're going to fully rewrite all of the content


across the site.


But it's the same situation if will that work?


How much of that works?


Well, again, it's a reporting problem, not


a performance problem.


Right, so we're saying there shouldn't


be traffic loss as a result of this duplication, this folding


up.


Exactly, yeah.


Right, OK.


Thanks.


Sure.


But it's trick.


I mean, in English, it's pretty common.


In German, it's really common.


Oh, we have it in German all the time.


So we have something where we constantly need your feedback


and people complaining about these kind of things


so that we can bring it back to the team.


It would be good to see something


like that in Search Console as more of a clearer report that


says, yes, we're folding these up because of this.


You know what I mean?


It's not too easy when you go there, just


see a canonical report.


Right, I know what that means.


But how do I then explain to the business


this is going to happen because Google sees


your content as duplicate?


Yeah.


And explaining the difference between the reporting side


and what actually happens in Search


is really tricky sometimes.


People should just have different languages.


But that's what I'm hoping is the magic bullet.


OK.


Another international question.


Suppose I have a website, domain.com,


which is set to USA using the geotargeting tool.


Can I start domain.com slash India and set it to India?


The short answer is yes.


You can definitely do that.


That's fine with a generic top-level domain.


You can set subdirectories or subdomains


to different countries however you want.


So that's kind of an easy one.


Nice.


You don't have many of those.


No hand-waving needed.


Now we have a complicated one.


It seems as if Google still has major problems


with subdomain leasing.


So it goes into kind of the different things


where basically things like product review or coupon sites


are hosted within an existing domain.


And it's like, why can't Google get its act together


and fix this?


I don't know.


I'm happy to pass this feedback on to your team.


I know they've been working on this for a while


to try to improve this.


In general, the other side that I see here


is also that a lot of these more established sites, when you


start adding more and more unrelated content


to your site, you're kind of making it hard for search


engines to understand what your site is about.


So if you take a news website and you add a big subdomain


or subdirectory for coupons or product reviews or whatever,


then suddenly it becomes a lot harder for us


to understand that this is actually a news website.


And then for your core content, it's


a lot harder for us to say, well, this


is something that we should show prominently and rank and kind


of crawl quickly because it's news content,


because we have this mix of different things in there.


So it's, on the one hand, something


that I think we need to improve on to better understand


the differences within the site, but also where


sites kind of are playing with the fire a little bit in that


it can result in us having trouble understanding


their core business, which is something they probably


want to avoid.


So that's kind of my answer there.


The question also went into, well, Bing


said they will apply manual actions and penalties


to these sites.


And I have no idea what Bing is actually doing.


I know they did this blog post about it,


so it's hard for me to compare what other people are


doing there.


Can I have a follow-up question?


So what would you recommend instead of subdomain


with the coupons, creating a new website,


totally a new domain totally for that instead?


That would be better for everyone, right?


I think that's usually the approach that we would take.


And it's something where if you want


to host other content on your site for monetization reasons,


then I would try to make it as clear as possible


that it's really separate, and maybe even consider


something as drastic as blocking it from crawling or indexing.


So that the content is there.


You can refer to it if you want, but it's not


going to get in the way of your normal search results.


Yeah.


OK.


Thank you.


But I suspect people have different experiences


with this, so I'm sure it will be tricky.


But I know that the team is looking into this.


And as we get feedback around these kind of issues,


then that's something that we'll definitely review more.


You would say that maybe a subdomain isn't really


the main domain, and then hence it would be something different.


It would be not seen as, but it is.


Yeah, I think the general idea is that people


try to fold it into the main site


so that they feel like, well, if this is some big website


and a part of it is suddenly coupons,


then maybe those coupons are really important too.


And you're kind of playing tricks there


in that it's harder for us to separate these two,


and that means that it'll also be much harder for us


to recognize that this is really important content,


and this is kind of just used for monetization,


not really what you want to be known for.


If you're a big news website, you


don't want to be known for your coupons.


You want to be known for your news.


So like a holiday special, whatever,


like coupons, whatever, if it's a brand, just robots.txt,


that page?


I mean, that's an option.


I mean, that's kind of like an extreme option


if you were hoping for search traffic, because search traffic


won't happen then.


But if you want to host something on your site


that you want to use for monetization,


but you don't care for it to be indexed,


then that's definitely an approach.


SPEAKER 2


More from people, then we'll try to find


ways to make that a little bit better on our side


as well to really understand these differences.


SPEAKER 3


Yeah, but this make another problem.


For example, if I have site for medical news, for example,


suddenly some content for car repairs or things like that


is stuffing into the main site, this is making that problem.


For example, Googlebot can't understand


what is the main content for that website


and why two are interleaving.


And it's tricky.


I think it will continue to be tricky.


So having specific feedback on issues


that you're seeing around this is always useful.


Sometimes it's very theoretical in that, well,


other sites are doing well like this.


It's like, should I do it as well?


It's very hard to use that as an argument with the engineering


team internally.


But if you're really seeing issues


where we're showing things in the wrong way


for specific search results, then


that's always really useful to have.


OK, something completely different.


A few weeks ago, we set a large number of pages to 404.


Over the last few weeks, the number


of pages visible in Search Console


has decreased continuously.


Currently, this number is stagnating


at quite a high number with a slightly increasing trend.


That doesn't fit together.


What could be the reason?


Any guesses?


For me, probably Googlebot has seen that 404 arising.


And it suddenly started to crawl more and more just


to see what part of content is staying there


and what kind of content, pages, for example, has been removed.


So this is probably tricky if you


have to do removal at large scale.


That's why probably it's good to remove


the content in small, small batches of pieces


just to check how it's going on.


But if you have removed, for example, half of site suddenly,


this can went to weird edge situation in bot.


There's also the sheer statistics of it.


If you have, I don't know, 1,000 pages


and you have just removed 200 of them


and in the first crawl that we're


going to do after the removal, we're


probably going to hit more of these removed pages.


So they're going to fall out of the index after a while.


And then eventually, we're hitting less and less


because we're just maybe crawling


other parts of the site depending


on your crawl budget.


And then as you said, it depends on


if we think that the structure of the site


has fundamentally changed, then we


have to crawl a lot more anyways.


And then we might spend more time on pages


that are no longer.


As in like on pages that you haven't removed,


we might spend our time there as well.


And eventually, we remove a lot in one place maybe.


And then eventually, we have less likelihood


of hitting the remaining bits as we crawl the entire site.


And eventually, they're going to drop off as well.


And if you set them to 404, it doesn't matter that much.


They might still show up.


But eventually, they're just going to drop off.


There's not that much that you need to take care of.


Can I come in then?


So what's the difference between a 404 and a 410,


which is the bots?


40410 is like nuke that page.


It's like throwing a nuclear bomb against it.


Googlebot, when it see 404, it's regularly


returned to that page to see maybe that URL.


So it comes back.


It's coming back.


But if you throw a 401, it's crawl a less and less more


that page.


And on my tests, after first crawling, that page is gone.


So if you want to remove a full section,


is it better to put a 410 on the pages than a 404?


In practice, the difference is relatively small.


But basically, what you're doing is, as Peter explained,


you're giving us a different signal.


404 means not found.


410 means gone, which means I made this go away versus I


don't know what you're trying to do here.


So you give us a more clear signal.


But in practice, unfortunately, a bunch of people


are giving us incorrect signals when they do not want to.


They give us a 410 when what they meant was a 404,


but the other way around.


So it's not like we're literally just going to banhammer this.


But it is a hint for us to not do that.


And you might see better results with 410,


but it's not a guarantee.


And there is another problem that Martin probably


explained it better.


What if you throw 401, 410 to some page,


and after two months, you change your mind,


I want to return content to the same URL?


Bummer.


It's banned from the things, and it's harder.