Search Status Dashboard

and then they blame the ranking update.


Search Status Dashboard

But then when you actually look at the site,


Search Status Dashboard

it's like they put up a no index on every single page,


Search Status Dashboard

and that's why they disappeared from search.


Search Status Dashboard

Ouch.


Search Status Dashboard

So it's not like everyone is affected by an incident,


Search Status Dashboard

unless we say so, of course, but probably a subset of sites.


Search Status Dashboard

And then you have to make a judgment still.


Search Status Dashboard

Like you as a site owner would have


Search Status Dashboard

to make a judgment about whether you


Search Status Dashboard

are affected by an incident or not.


Search Status Dashboard

Cool.


Search Status Dashboard

So from a process point of view, we


Search Status Dashboard

would get alerted by an internal team or external people


Search Status Dashboard

around an issue.


Search Status Dashboard

And then what happens there?


Search Status Dashboard

We basically decide is it a big issue or not,


Search Status Dashboard

or who makes that kind of a call?


Search Status Dashboard

I don't know how familiar the listeners are with SREs,


Search Status Dashboard

like Search Site Reliability Engineers.


Search Status Dashboard

You can search for it on your favorite search engine.


Search Status Dashboard

And they have, or at least the Google SREs,


Search Status Dashboard

have a mini site where they describe what they do


Search Status Dashboard

and how they do it.


Search Status Dashboard

So they set up these dashboards where


Search Status Dashboard

they monitor system health.


Search Status Dashboard

And they define thresholds in those dashboards.


Search Status Dashboard

And if one metric, one signal passes that threshold,


Search Status Dashboard

then it sends an email or pages the SREs or something happens.


Search Status Dashboard

And we are subscribed to those alerts.


Search Status Dashboard

So when something happens, then it ends up in our inbox.


Search Status Dashboard

But of course, it's virtually impossible


Search Status Dashboard

to create alerts for everything.


Search Status Dashboard

So we are also looking on Twitter


Search Status Dashboard

for clues about more widespread issues with search.


Search Status Dashboard

OK, that was a lie.


Search Status Dashboard

We are just looking at Barry Schwartz's blog.


Search Status Dashboard

No, OK, that was a joke.


Search Status Dashboard

But we are looking on Twitter.


Search Status Dashboard

And if we see that, let's say, more than five people


Search Status Dashboard

post about a failure that seems to be related,


Search Status Dashboard

then we would start investigating internally


Search Status Dashboard

whether we have a failure or not.


Search Status Dashboard

And the investigation can be John or me or whoever.


Search Status Dashboard

Or we ping the team that we think


Search Status Dashboard

is responsible for the system that's potentially failing


Search Status Dashboard

and ask them to look into it.


Search Status Dashboard

So we basically have two sources where we source incidents from


Search Status Dashboard

and then make a judgment call about whether we should


Search Status Dashboard

externalize or not.


Search Status Dashboard

And then you would, I don't know, hit a button


Search Status Dashboard

and it adds an entry into the dashboard or something


Search Status Dashboard

along those lines.


Search Status Dashboard

Sort of.


Search Status Dashboard

So internally, we have an admin interface for the dashboard.


Search Status Dashboard

And we have basically canned or templates


Search Status Dashboard

that we can use for the messages.


Search Status Dashboard

Well, I mean, the idea is to be able to move quickly


Search Status Dashboard

when these things happen.


Search Status Dashboard

So if it's already approved, then we


Search Status Dashboard

can post without having to get approval


Search Status Dashboard

because they signed off on the process and the messages


Search Status Dashboard

that we want to say.


Search Status Dashboard

Sure.


Search Status Dashboard

That was the last fun way to say it.


Search Status Dashboard

Sorry.


Search Status Dashboard

And then we can add some stuff to it, to the messages.


Search Status Dashboard

Like, for example, if we want to say that one particular DNS


Search Status Dashboard

provider is refusing Googlebot connections,


Search Status Dashboard

then we could add that, switch DNS providers or whatever,


Search Status Dashboard

and then it will suddenly work.


Search Status Dashboard

Or not DNS provider, name server provider.


Search Status Dashboard

And then it would start working, perhaps.


Search Status Dashboard

But generally, it's just template messages


Search Status Dashboard

that we put up and try to explain as briefly as possible


Search Status Dashboard

what's going on and when will be the next update


Search Status Dashboard

to the incident on the dashboard.


Search Status Dashboard

Because that's the other thing.


Search Status Dashboard

On Twitter, it was kind of weird to keep posting


Search Status Dashboard

about the updates because then you


Search Status Dashboard

had to find the thread that you started,


Search Status Dashboard

and then you post an update, and then maybe people


Search Status Dashboard

will not notice it.


Search Status Dashboard

Maybe they will.


Search Status Dashboard

On the dashboard, it's very clear when we post an update,


Search Status Dashboard

it will be very visible.


Search Status Dashboard

I think that was an issue in the 2019 series of incidents


Search Status Dashboard

where we would post something and then not set expectation


Search Status Dashboard

for when you would hear back from us.


Search Status Dashboard

So we're still looking into the thing.


Search Status Dashboard

Three days have gone by.


Search Status Dashboard

People are replying to the tweet, like, what's happening?


Search Status Dashboard

Why haven't you said anything?


Search Status Dashboard

And we're still looking into it.


Search Status Dashboard

We don't have anything more to say.


Search Status Dashboard

So the idea with the 12 hours thing


Search Status Dashboard

that we will provide another update within a certain time


Search Status Dashboard

period so you kind of know what to expect.


Search Status Dashboard

And it might be the next update is


Search Status Dashboard

we're still looking into it, but at least you


Search Status Dashboard

know when you're going to hear from us.


Search Status Dashboard

Cool, so what kind of issues would you cover?


Search Status Dashboard

I imagine it's more about technical issues in Google


Search Status Dashboard

Search, or is it crawling, indexing?


Search Status Dashboard

Would it also include ranking?


Search Status Dashboard

Or I guess that might fall under serving?


Search Status Dashboard

Or how do you have that split up?


Search Status Dashboard

So we mapped the dashboard to the major search systems, which


Search Status Dashboard

is crawling, indexing, serving.


Search Status Dashboard

And the incidents would go into those buckets.


Search Status Dashboard

So for example, if for whatever reason,


Search Status Dashboard

Googlebot cannot crawl the whole internet, that would end up


Search Status Dashboard

in the crawling bucket.


Search Status Dashboard

If there is some kind of an equalization problem that's


Search Status Dashboard

affecting lots of sites, then that


Search Status Dashboard

would end up in the indexing bucket.


Search Status Dashboard

And then if google.com is not accessible to lots of users,


Search Status Dashboard

then that would end up in the serving bucket.


Search Status Dashboard

But what happens when a dashboard goes down?


Search Status Dashboard

Do you have a bucket for that?


Search Status Dashboard

Good question.


Search Status Dashboard

It's extremely unlikely that the dashboard would go down.


Search Status Dashboard

And I was geeking about this for a long time.


Search Status Dashboard

Basically, I was looking at the serving system that's


Search Status Dashboard

servicing the dashboard.


Search Status Dashboard

Specifically, it's uptime.


Search Status Dashboard

And it's been up constantly since 2006.


Search Status Dashboard

Like they have not had an incident this whole time?


Search Status Dashboard

They had not had an incident since 2006.


Search Status Dashboard

So if it went down, that would be a very different kind


Search Status Dashboard

of problem for Google.


Search Status Dashboard

Yeah, I just don't see how it would go down.


Search Status Dashboard

I guess then we would post on Twitter that, oops,


Search Status Dashboard

our dashboard is actually down.


Search Status Dashboard

But maybe in that case, Twitter is also down,


Search Status Dashboard

or we're in some kind of apocalyptic scenario.


Search Status Dashboard

That would be cool.


Search Status Dashboard

Hey.


Search Status Dashboard

That's his line.


Search Status Dashboard

That's my line.


Search Status Dashboard

We can put posters up.


Search Status Dashboard

Flyers.


Search Status Dashboard

So have they built it so that it's


Search Status Dashboard

decoupled with google.com?


Search Status Dashboard

So if there's an incident on google.com,


Search Status Dashboard

then we can still post about it on various status dashboard


Search Status Dashboard

places?


Search Status Dashboard

Yeah, all the dashboards that Google has,


Search Status Dashboard

like status cloud google.com and status search google.com,


Search Status Dashboard

they are all decoupled from each other and from google.com.


Search Status Dashboard

Technically, if the DNS server went down,


Search Status Dashboard

that would be the one point of failure.


Search Status Dashboard

But the Google DNS service is quite reliable.


Search Status Dashboard

And usually, the failures that we have,


Search Status Dashboard

they are not related to DNS.


Search Status Dashboard

They are related to the systems that are actually


Search Status Dashboard

serving the content.


Search Status Dashboard

And the serving system that we use


Search Status Dashboard

for the dashboard, that's not the same


Search Status Dashboard

that we use for google.com or most other services.


Search Status Dashboard

And it's isolated.


Search Status Dashboard

It is the same service that we use for, I think,


Search Status Dashboard

serving the thumbnails on Google Images.


Search Status Dashboard

But it is isolated from that part.


Search Status Dashboard

It's a different instance.


Search Status Dashboard

And if something in Search Console were to break,


Search Status Dashboard

which I think is also on search.google.com,


Search Status Dashboard

would that be listed here?


Search Status Dashboard

Would that affect the dashboard at all?


Search Status Dashboard

Is there a connection there?


Search Status Dashboard

I thought Search Console has its own dashboard sort of thing,


Search Status Dashboard

don't they?


Search Status Dashboard

They have a page that's talking about data anomalies


Search Status Dashboard

within Search Console.


Search Status Dashboard

But I don't know if they would post there


Search Status Dashboard

if Search Console is completely inaccessible,


Search Status Dashboard

if it's down for a period of time.


Search Status Dashboard

But I guess that is considered an anomaly.


Search Status Dashboard

You could not access the dashboard from this date.


Search Status Dashboard

Because it's always day to day.


Search Status Dashboard

For five days, Search Console is not, I don't know.


Search Status Dashboard

I'm imagining a scenario like that.


Search Status Dashboard

I don't know if that is a real scenario


Search Status Dashboard

that they would post there.


Search Status Dashboard

Yeah, I don't remember them being down.


Search Status Dashboard

But I guess usually they have more issues with regards


Search Status Dashboard

to the various data pipelines that lead up into Search


Search Status Dashboard

Console.


Search Status Dashboard

And then, I don't know, the crawling report


Search Status Dashboard

is delayed by a couple of days or something like that.


Search Status Dashboard

Right.


Search Status Dashboard

I only remember incidents like some users


Search Status Dashboard

are experiencing 500 statuses, or the API


Search Status Dashboard

being down because reasons, but not complete Search Console


Search Status Dashboard

outage.


Search Status Dashboard

OK.


Search Status Dashboard

And what if there is just one specific feature in search


Search Status Dashboard

that is kind of broken?


Search Status Dashboard

I don't know.


Search Status Dashboard

Let's say the featured snippets are not showing anymore.


Search Status Dashboard

Is that something that we would show here?


Search Status Dashboard

Or would this really be all of search is broken,


Search Status Dashboard

or no snippets are shown at all?


Search Status Dashboard

Good question.


Search Status Dashboard

In this version, we are only focusing


Search Status Dashboard

on major incidents affecting the systems that we mentioned,


Search Status Dashboard

like crawling, indexing, serving.


Search Status Dashboard

In a next iteration, we are thinking


Search Status Dashboard

about exposing search features.


Search Status Dashboard

So for example, top stories, or feature snippets, or whatever,


Search Status Dashboard

if they would go down for whatever reason,


Search Status Dashboard

then we might communicate something


Search Status Dashboard

about those incidents.


Search Status Dashboard

But in the current iteration, I don't


Search Status Dashboard

think that we would externalize those issues.


Search Status Dashboard

Would that be if top stories just stops appearing


Search Status Dashboard

altogether, like a serving issue, or certain sites saying,


Search Status Dashboard

I'm not appearing as a top story.


Search Status Dashboard

There's a problem.


Search Status Dashboard

I mean, either, depending on how many sites are affected.


Search Status Dashboard

Cool.


Search Status Dashboard

And I don't know, related services to search,


Search Status Dashboard

like maybe Discover, or Google News, or Google Reader,


Search Status Dashboard

if those went down, would that also be listed here?


Search Status Dashboard

Or is this really focused on web search?


Search Status Dashboard

Did you say Google Reader?


Search Status Dashboard

He's checking if you're listening.


Search Status Dashboard

So sorry.


Search Status Dashboard

What year is this?


Search Status Dashboard

So yeah, if Google Reader goes down,


Search Status Dashboard

then I will definitely put that on the dashboard.


Search Status Dashboard

Oh, wait.


Search Status Dashboard

No, that would be yet another version.


Search Status Dashboard

We know that some services want to appear on the dashboard,


Search Status Dashboard

but we have to think about how to present


Search Status Dashboard

other, basically, services.


Search Status Dashboard

And I just didn't have either time or nerves


Search Status Dashboard

to think about that just yet.


Search Status Dashboard

Plus, I think it would be valuable to just launch V1


Search Status Dashboard

and then see how it goes, whether we can learn something


Search Status Dashboard

from it, and then see if we can improve it by including


Search Status Dashboard

other services.


Search Status Dashboard

Are there plans for translations,


Search Status Dashboard

or is that already happening?


Search Status Dashboard

No.


Search Status Dashboard

No translation.


Search Status Dashboard

It's just not supported.


Search Status Dashboard

Like in the current setup, it's practically impossible


Search Status Dashboard

to have translations.


Search Status Dashboard

And they are not even thinking about it,


Search Status Dashboard

mostly because they, as in the developers of the dashboard,


Search Status Dashboard

because the dashboard is kind of like a CMS


Search Status Dashboard

that we share with other Google products or Alphabet products.


Search Status Dashboard

And it is developed to be as simple


Search Status Dashboard

as it needs to be to serve the dashboard itself,


Search Status Dashboard

and no complexity to it whatsoever.


Search Status Dashboard

And translations would be major complexity,


Search Status Dashboard

because then you have to load the different translations


Search Status Dashboard

from different rows in the database that's


Search Status Dashboard

serving the content.


Search Status Dashboard

And I don't think that it's worth


Search Status Dashboard

it to add the complexity for something like this.


Search Status Dashboard

Basically, if you are using Google Translate, for example,


Search Status Dashboard

or any other online translation software,


Search Status Dashboard

then that will give you enough clue about what's happening.


Search Status Dashboard

I think with this, it's also particularly time sensitive.


Search Status Dashboard

So if there was a delay to translate the thing,


Search Status Dashboard

then that language would be behind or not


Search Status Dashboard

having the same timeline of events, which


Search Status Dashboard

could be an issue.


Search Status Dashboard

And then people might think, oh, is this not affecting search


Search Status Dashboard

in French or something, because it's not


Search Status Dashboard

been translated in French?


Search Status Dashboard

Or it didn't happen for three days.


Search Status Dashboard

Does that mean anything when, no, it just


Search Status Dashboard

means that the translation is behind?


Search Status Dashboard

Yeah.


Search Status Dashboard

OK.


Search Status Dashboard

That makes a lot of sense.


Search Status Dashboard

So what would you recommend for the average site owner?


Search Status Dashboard

Should they keep this page open in their browser


Search Status Dashboard

and just refresh it every couple of minutes?


Search Status Dashboard

To boost our numbers?


Search Status Dashboard

No.


Search Status Dashboard

What?


Search Status Dashboard

So we are successful.


Search Status Dashboard

We are having many page views to this dashboard.


Search Status Dashboard

I mean, they can subscribe.


Search Status Dashboard

Are you going for promo?


Search Status Dashboard

Or why do you need the numbers?


Search Status Dashboard

They could subscribe.


Search Status Dashboard

There's a RSS feed functionality.


Search Status Dashboard

So you don't need to refresh.


Search Status Dashboard

You could get notified if you want to,


Search Status Dashboard

if you're still using RSS.


Search Status Dashboard

Oh, like in Google Reader, you could add that.


Search Status Dashboard

Cool.


Search Status Dashboard

Yes.


Search Status Dashboard

Wow.


Search Status Dashboard

OK.


Search Status Dashboard

Maybe not Google Reader.


Search Status Dashboard

I'm so sorry.


Search Status Dashboard

Today, I noticed that the Twitter account for Google


Search Status Dashboard

Reader was still alive and still had a blue checkmark.


Search Status Dashboard

Really?


Search Status Dashboard

It just brought back all kinds of memories.


Search Status Dashboard

Yeah.


Search Status Dashboard

I mean, they're not posting.


Search Status Dashboard

So it's not that something is happening.


Search Status Dashboard

That would have been weird.


Search Status Dashboard

OK, so subscribe with whatever your favorite RSS reader is,


Search Status Dashboard

and you can stay up to date there.


Search Status Dashboard

And I imagine folks externally who are really curious


Search Status Dashboard

will subscribe anyway and let the rest of the search world


Search Status Dashboard

know if anything is posted there anyway.


Search Status Dashboard

I was about to suggest you just have to follow Barry Schwartz,


Search Status Dashboard

and you're all set.


Search Status Dashboard

We are promoting him a lot in this episode.


Search Status Dashboard

I mean, Twitter, and especially a few particular users


Search Status Dashboard

of Twitter, they are a very good source


Search Status Dashboard

for noticing weirdnesses in search.


Search Status Dashboard

And Barry specifically is a very, very, very good source


Search Status Dashboard

for bugs, like search bugs.


Search Status Dashboard

And it's very valuable and nice.


Search Status Dashboard

So do you think for the first incident that we have,


Search Status Dashboard

would we also publish on Twitter just to extra promote,


Search Status Dashboard

like this is what's happening in case people don't know?


Search Status Dashboard

I think so.


Search Status Dashboard

I think the first one we would definitely post on Twitter


Search Status Dashboard

as well, and basically just tell people


Search Status Dashboard

that now we have this dashboard, we


Search Status Dashboard

are going to post updates there.


Search Status Dashboard

Goodbye.


Search Status Dashboard

Good talk.


Search Status Dashboard

That's it.


Search Status Dashboard

Cool.


Search Status Dashboard

Well, thanks a lot.


Search Status Dashboard

Really nice to have your insights, Gary and Lizzie.


Search Status Dashboard

You're welcome.


Search Status Dashboard

Sorry.


Search Status Dashboard

So sorry.


Search Status Dashboard

Well, that's it for this episode.


Search Status Dashboard

In the next episode of Search Off the Record,


Search Status Dashboard

a podcast interview is coming about how the year was


Search Status Dashboard

and what might be happening next year.


Search Status Dashboard

So stay tuned for that.


Search Status Dashboard

We've been having fun with these podcast episodes.


Search Status Dashboard

I hope you, the listener, have been


Search Status Dashboard

finding them both entertaining and insightful as well.


Search Status Dashboard

Feel free to drop me a note on Twitter


Search Status Dashboard

or chat with us at one of the next events that we go to.


Search Status Dashboard

If you have any thoughts.


Search Status Dashboard

And of course, don't forget to like and subscribe.


Search Status Dashboard

Thank you and goodbye.


Search Status Dashboard

Goodbye.


Search Status Dashboard

I do.


Search Status Dashboard


Crawl Budget and the Crawl Stats report - Google Search Console Training

Hi, I'm Daniel Weisberg, Search Advocate at Google.


And today I'll provide a short introduction to how Google crawls pages


and provide definitions for terms such as crawl rate,


crawl demand, and crawl budget.


Then I'll dive into the Search Console Crawl Stats report,


which provides data on crawl requests, average response time, and more.


The report contains information for several Google crawlers,


but I'll focus on the search aspect of it.


As a disclaimer, this presentation will be more relevant


if you work with a large website.


If you have a site with fewer than a few thousand pages,


you do not need to worry about it.


But if you ask me, it never hurts to learn something new.


Who knows? Your site may become the next big thing.


The Crawling Process


The crawling process begins with a list of web addresses


from past crawls and sitemaps provided by site owners.


Google uses web crawlers to visit these addresses,


read the information in them, and follow links on those pages.


The crawlers will revisit pages already in the list


to check if they have changed, and also crawl new pages they discover.


During this process, the crawlers have to make important decisions,


such as prioritizing when and what to crawl,


while making sure the website can handle the server requests made by Google.


How much is the content required by Google?


Crawl demand is how much the content is desired by Google.


It is affected by URLs that haven't been crawled by Google before


and by Google's estimation on how often content changes on the known URLs.


The successfully crawled pages are processed and passed to Google Indexing


to prepare their content for serving on Google search results.


Google wants to make sure that it doesn't overload your servers,


as a good citizen would do.


With that in mind, Google computes your site's crawl rate,


which represents the maximum number of concurrent connections


a crawler may use to crawl your site.


This value is calculated by Google periodically,


based on your site's responsiveness,


or in other words, how much crawling traffic it can handle.


If the site is quick and consistent to respond to crawlers,


the rate goes up if there is demand from indexing.


If the site slows down or responds with server errors,


the rate goes down and Google crawls less.


In rare cases where Google crawlers overload your servers,


you can set a crawl rate limit


using the Crawl Rate Settings Report in Search Console.


Taking crawl rate and crawl demand together,


we can define crawl budget as the number of URLs Google can


and wants to crawl.


My colleague, The Gary, wrote a detailed blog post


about crawl budget in the Search Central blog.


Check it out.


To find out how often Google crawls your site


and what the responses were,


you should use the Search Console Crawl Stats Report,


which provides statistics about Google's crawling behavior.


Here are some questions you can answer with the data provided.


What's your site's general availability?


What's the average page response for a crawl request?


And how many requests were made by Google to your site in the last 90 days?


Let's dive in to learn more about the report.


Log into Search Console and find the Settings page,


where you can open the Crawl Stats Report.


Note that this report is only available for properties at the domain level.


It is not available for properties that include a path.


In the summary page, you'll find a lot of information.


The main elements are the Crawling Trends Chart,


the whole status details, and the Crawl Request Breakdown.


The chart shows the trends for three metrics.


The total crawl request for URLs on your site,


whether successful or not.


Requests to resources hosted outside of your site are not counted.


So if your images are served on another domain,


such as a content delivery network or CDN, they will not appear here.


The total download size from your site during crawling.


If Google cached a page resource that is used by multiple pages,


the resource is only requested the first time.


And the average page response time for a crawl request to retrieve the page content.


This metric does not include retrieving page resources,


such as scripts, images, and other linked or embedded content.


The response time does not account for page rendering time.


Look for major spikes, drops, and trends over time in your data.


For example, if you see a significant drop in total crawl requests,


make sure no one added a new robots.txt rule to your site.


Or maybe your site is responding slowly to Googlebot.


This might be understood as a sign your server cannot handle all the requests.


Another example would be a consistent increase in average response time.


This might not affect your crawl rate immediately,


but it's a good indicator that your servers might not be handling all the load.


Ultimately, this may affect user experience too.


The whole status is an easy way for you to check your site's general availability


in the last 90 days.


If you verified your website as a domain property,


you will see all child hosts separately here.


This can be very handy to evaluate all your host performance in one place.


When you click to get the whole status details, you'll find three categories.


Errors in this section mean Google cannot crawl your site for a technical reason.


Robots.txt fetch tells you the failure rate when crawling your robots.txt file.


Your site is not required to have a robots.txt file,


but it must return a successful response, 200 or 404, when asked for this file.


If Googlebot has a connection issue, for example, a 503,


it will stop crawling your site.


DNS resolution tells you when your DNS server didn't recognize your host name


or didn't respond during crawling.


If you see an issue here, check with your registrar


to make sure your site is correctly set up


and that your server is properly connected to the internet.


Server connectivity tells you when your server was unresponsive


or did not provide a full response for your URL during a crawl.


If you're seeing spikes or consistent connectivity issues here,


you might need to talk to your provider about increasing your capacity


or fixing availability issues.


If Google found at least one of those errors on your site in the last week,


you will find a warning alerting you of problems last week.


Dig deeper into the error and fix it.


If Google found at least one error on your site in the last 90 days,


but it occurred more than a week ago,


you will see a warning that you had problems in the past.


You should check your server logs


or contact a developer to review what the problems were


and decide whether you need to take any action.


If Google didn't find any crawl availability issues on your site


in the past 90 days, you're all green.


Good job.


The crawl request cards show several breakdowns


to help you understand what Google crawlers found in your website.


There are four available breakdowns.


Crawl response shows the responses that Google received


when crawling your site.


Common response types would be 200, 301, 404, or server errors.


Crawl file type shows file types returned by the request.


Common file types would be HTML, image, video, or JavaScript.


Crawl purpose could be discovery when a URL is new to Google


or refresh for a recrawl of a known page.


And Google bot type shows the user agent used by Google


to make the crawl request.


For example, smartphone, desktop, image, and others.


Click any row to drill down to that value


and review its trends and specific sample URLs.


Check out the Help Center article about the Crawl stats report


to learn more details about it.


To recap, crawl budget is the number of URLs Google can


and wants to crawl on websites every day.


It is relevant to large websites where Google needs to prioritize


what to crawl first, how much to crawl, and how frequently to recrawl.


To help you understand and optimize Google's crawling,


you should use the Search Console Crawl stats report.


Use the summary page chart to analyze crawling volume and trends.


Use the whole status details to check your site's general availability.


And use the Crawl Request breakdowns to understand what Google bot


is finding when crawling your website.


And make sure to check my Search Console Training YouTube series


to learn more about how to use the product to succeed on search.


Googlebot Crawl breakdowns


That's all the crawling I'm doing today.


Add a sitemap for more than 50,000 URLs - SEO Snippets

Today's question is about sitemap files.


Mike from India asks, how do we add a sitemap for more than 50,000 URLs?


Sitemap files are a great way to make your content known to Google and to other search


engines.


However, they're limited to 50,000 URLs per file.


What do you do if you have more URLs?


You can generate more than one sitemap file per website.


You can either submit these individually, for example, through Search Console, or you


can create a sitemap index file.


A sitemap index file is like a sitemap file for sitemaps.


You can list multiple sitemap files in it.


If you use a sitemap index file, you can just submit that file for your website in Search


Console.


Even if you have fewer than 50,000 URLs, you can submit multiple sitemap files.


For example, you might want to do that to keep track of different sections of your website,


or just in general, to make maintenance of your sitemap files a little bit easier.


When it comes to creating sitemap files, we strongly recommend that you have these made


automatically through your server directly.


That's the best way to make sure that your new and updated content is highlighted to


search engines as quickly as possible.


Most modern content management systems will take care of this for you.


Often it's just a matter of flipping a switch in your control panel to turn these on.


SEO Mythbusting with Google & Bing

Especially in backlinks, there's one angle on myths, right?


SEO Mythbusting with Google & Bing

If I have a lot of backlinks, I'll probably rank higher.


SEO Mythbusting with Google & Bing

So there's a myth.


SEO Mythbusting with Google & Bing

Yes, it's one of the signals, but that's not the only signal.


SEO Mythbusting with Google & Bing

You have hundreds and hundreds of signals,


SEO Mythbusting with Google & Bing

and people actually spam.


SEO Mythbusting with Google & Bing

People actually spam with bad URLs.


SEO Mythbusting with Google & Bing

It's easy to game that one.


SEO Mythbusting with Google & Bing

If that would be the signal, it's so easy to game.


SEO Mythbusting with Google & Bing

You just buy a bunch of links, really, and then you're like, woo.


SEO Mythbusting with Google & Bing

Hello and welcome back to SEO Mythbusting.


SEO Mythbusting with Google & Bing

This is the beginning of season two,


SEO Mythbusting with Google & Bing

and we have listened to your comments and feedback last time.


SEO Mythbusting with Google & Bing

So in the first season, we were basically


SEO Mythbusting with Google & Bing

taking questions from developers and busting myths


SEO Mythbusting with Google & Bing

that are coming from developers.


SEO Mythbusting with Google & Bing

This season is more on the SEO side.


SEO Mythbusting with Google & Bing

And the first one is really exciting.


SEO Mythbusting with Google & Bing

It's Sandhya from Microsoft, and you


SEO Mythbusting with Google & Bing

are on the team that makes the Bing Webmaster Tools.


SEO Mythbusting with Google & Bing

Absolutely.


SEO Mythbusting with Google & Bing

Thank you, Martin, for inviting me.


SEO Mythbusting with Google & Bing

And it's amazing to be here and talking to you.


SEO Mythbusting with Google & Bing

It's two search engines.


SEO Mythbusting with Google & Bing

Yes.


SEO Mythbusting with Google & Bing

And we are not isolated from each other.


SEO Mythbusting with Google & Bing

We are working together to make search better


SEO Mythbusting with Google & Bing

for all the users on the web and for all the webmasters there.


SEO Mythbusting with Google & Bing

Wherever we speak today is going to be my personal opinion.


SEO Mythbusting with Google & Bing

I'm not a Microsoft spokesman person, so yeah.


SEO Mythbusting with Google & Bing

Yeah, pretty much the same on my side.


SEO Mythbusting with Google & Bing

So what we are doing here is exchanging


SEO Mythbusting with Google & Bing

our personal experience, working in this field,


SEO Mythbusting with Google & Bing

working in this community.


SEO Mythbusting with Google & Bing

What would you say is actually the most frequent myth


SEO Mythbusting with Google & Bing

or question that people are coming with to you?


SEO Mythbusting with Google & Bing

All right.


SEO Mythbusting with Google & Bing

Yeah, one of the top questions we ever


SEO Mythbusting with Google & Bing

hear in all forms of communication


SEO Mythbusting with Google & Bing

is, how can I rank on the top, right?


SEO Mythbusting with Google & Bing

That's like number one question everywhere.


SEO Mythbusting with Google & Bing

That's number one question from everyone, literally.


SEO Mythbusting with Google & Bing

Everyone, literally.


SEO Mythbusting with Google & Bing

And it's interesting that they're very passionate


SEO Mythbusting with Google & Bing

about it, but we can't help it.


SEO Mythbusting with Google & Bing

Because at the end of the day, we're trying to meet the users'


SEO Mythbusting with Google & Bing

needs and what they really need.


SEO Mythbusting with Google & Bing

And we try to match the need and what they have.


SEO Mythbusting with Google & Bing

So we try to educate them.


SEO Mythbusting with Google & Bing

And there's hundreds of factors.


SEO Mythbusting with Google & Bing

Yes.


SEO Mythbusting with Google & Bing

It's not that we can turn a knob or something,


SEO Mythbusting with Google & Bing

or we go into a database and go like, change that number


SEO Mythbusting with Google & Bing

from five to two, and boom.


SEO Mythbusting with Google & Bing

No, that never happens.


SEO Mythbusting with Google & Bing

There is a myth.


SEO Mythbusting with Google & Bing

There is a myth that we could, because some people actually


SEO Mythbusting with Google & Bing

reach out to us and say, hey, can you help?


SEO Mythbusting with Google & Bing

Can you tell us exact steps what to do?


SEO Mythbusting with Google & Bing

And we try to help as much as possible


SEO Mythbusting with Google & Bing

in terms of giving them what are the fundamentals they


SEO Mythbusting with Google & Bing

have to do on the website.


SEO Mythbusting with Google & Bing

And if they have the right intention.


SEO Mythbusting with Google & Bing

It's all in a public documentation.


SEO Mythbusting with Google & Bing

You don't get private support from the UOB.


SEO Mythbusting with Google & Bing

And they're all there in the webmaster guidelines,


SEO Mythbusting with Google & Bing

both on Google's side as well as in Bing as well.


SEO Mythbusting with Google & Bing

So I think if they follow that, I'm sure for the right intent,


SEO Mythbusting with Google & Bing

they will rank on top.


SEO Mythbusting with Google & Bing

Exactly.


SEO Mythbusting with Google & Bing

Just put the user first, and you'll be doing fine mostly.


SEO Mythbusting with Google & Bing

Right?


SEO Mythbusting with Google & Bing

That's kind of the get the basics right,


SEO Mythbusting with Google & Bing

and you'll be doing all right.


SEO Mythbusting with Google & Bing

And I find it interesting.


SEO Mythbusting with Google & Bing

Like sometimes people get really creative.


SEO Mythbusting with Google & Bing

Like I got a message on Instagram


SEO Mythbusting with Google & Bing

asking about ranking, and I'm like, what's happening here?


SEO Mythbusting with Google & Bing

Yeah, I know.


SEO Mythbusting with Google & Bing

We get direct messages on Twitter sometimes,


SEO Mythbusting with Google & Bing

especially if you're an actor out there.


SEO Mythbusting with Google & Bing

And sometimes even on emails, support.


SEO Mythbusting with Google & Bing

Support is understandable.


SEO Mythbusting with Google & Bing

Sometimes they're trying to figure out who knows you,


SEO Mythbusting with Google & Bing

and then they reach out through that as well.


SEO Mythbusting with Google & Bing

The second hand kind of thing.


SEO Mythbusting with Google & Bing

Yeah, they do that.


SEO Mythbusting with Google & Bing

This is interesting.


SEO Mythbusting with Google & Bing

Because we get all these requests coming in,


SEO Mythbusting with Google & Bing

and people ask us questions pretty much all the time


SEO Mythbusting with Google & Bing

in all sorts of channels.


SEO Mythbusting with Google & Bing

And I know that we are not one person facing these questions


SEO Mythbusting with Google & Bing

in the public eye, and you are not one person on the team


SEO Mythbusting with Google & Bing

either.


SEO Mythbusting with Google & Bing

So you have Fabrice Carnel, and you have Frédéric Douart,


SEO Mythbusting with Google & Bing

and it's you and it's others probably in the future as well.


SEO Mythbusting with Google & Bing

What we see happening is that sometimes we


SEO Mythbusting with Google & Bing

get asked the same question, each of us


SEO Mythbusting with Google & Bing

individually, to figure out, are we responding the same way?


SEO Mythbusting with Google & Bing

Do you get that?


SEO Mythbusting with Google & Bing

Yes, absolutely.


SEO Mythbusting with Google & Bing

We hear that a lot, because we see the similar pattern coming


SEO Mythbusting with Google & Bing

through, so we try to give the same responses at the end


SEO Mythbusting with Google & Bing

of the day.


SEO Mythbusting with Google & Bing

Otherwise they would get taken in different contexts


SEO Mythbusting with Google & Bing

a little bit.


SEO Mythbusting with Google & Bing

That's the thing, the context.


SEO Mythbusting with Google & Bing

Sometimes these questions depend on so much context.


SEO Mythbusting with Google & Bing

And then someone asks you a question given this context,


SEO Mythbusting with Google & Bing

and the answer is A. But then the same question given


SEO Mythbusting with Google & Bing

a different context, the answer is B.


SEO Mythbusting with Google & Bing

And then people like to ditch the context,


SEO Mythbusting with Google & Bing

and then just go like, but Martin said A.


SEO Mythbusting with Google & Bing

Yeah, and they would take it literally.


SEO Mythbusting with Google & Bing

They take it literally.


SEO Mythbusting with Google & Bing

So we're very conscious about it.


SEO Mythbusting with Google & Bing

What we do is when we say it, we try


SEO Mythbusting with Google & Bing

to do a little longer, bigger sentences so that it's


SEO Mythbusting with Google & Bing

not, yes, that's due.


SEO Mythbusting with Google & Bing

That's probably we'll take it in a different context.


SEO Mythbusting with Google & Bing

We try to cover that fully.


SEO Mythbusting with Google & Bing

We try to share as much and be transparent as much.


SEO Mythbusting with Google & Bing

If it's taken out of context, it's unfortunate.


SEO Mythbusting with Google & Bing

There would be some incidents like that.


SEO Mythbusting with Google & Bing

It's a challenge.


SEO Mythbusting with Google & Bing

Sometimes it's a challenge.


SEO Mythbusting with Google & Bing

I'm sure Yandex and Baidu probably


SEO Mythbusting with Google & Bing

do the same thing as well.


SEO Mythbusting with Google & Bing

Do you also have the feeling that certain things are


SEO Mythbusting with Google & Bing

getting a lot better, and we're getting


SEO Mythbusting with Google & Bing

a better understanding, and also the tone


SEO Mythbusting with Google & Bing

is nicer and more constructive?


SEO Mythbusting with Google & Bing

Or what do you think the trend is in the community?


SEO Mythbusting with Google & Bing

I think people are much more aware.


SEO Mythbusting with Google & Bing

I think there is a lot of evangelism, what


SEO Mythbusting with Google & Bing

happens behind the scenes.


SEO Mythbusting with Google & Bing

I think if you think about people


SEO Mythbusting with Google & Bing

understand how search works.


SEO Mythbusting with Google & Bing

Earlier, there was a concept in terms of,


SEO Mythbusting with Google & Bing

hey, they're probably doing some manual actions and things


SEO Mythbusting with Google & Bing

like that.


SEO Mythbusting with Google & Bing

But now people know it's not true.


SEO Mythbusting with Google & Bing

The other thing which we also see now,


SEO Mythbusting with Google & Bing

I don't know if you've seen from Google, is, hey,


SEO Mythbusting with Google & Bing

I'm advertising out here.


SEO Mythbusting with Google & Bing

Will I get ranked higher?


SEO Mythbusting with Google & Bing

Oh, yeah.


SEO Mythbusting with Google & Bing

Do you hear that?


SEO Mythbusting with Google & Bing

We hear that.


SEO Mythbusting with Google & Bing

We hear that, yes.


SEO Mythbusting with Google & Bing

Do you have that as well?


SEO Mythbusting with Google & Bing

Yes, we do that.


SEO Mythbusting with Google & Bing

So we're saying, no, ads is very different ranking,


SEO Mythbusting with Google & Bing

and organics very different.


SEO Mythbusting with Google & Bing

They're very separate.


SEO Mythbusting with Google & Bing

You don't want to mix it.


SEO Mythbusting with Google & Bing

Absolutely.


SEO Mythbusting with Google & Bing

There's a very strong separation between it,


SEO Mythbusting with Google & Bing

and I'm sure you're doing the same thing.


SEO Mythbusting with Google & Bing

So a thing that I wonder, and maybe that's


SEO Mythbusting with Google & Bing

just my personal lens to it, because I am very technical


SEO Mythbusting with Google & Bing

and coming from a technical side of things,


SEO Mythbusting with Google & Bing

I think the community is picking up more and more technical SEO


SEO Mythbusting with Google & Bing

and getting more technical and looking


SEO Mythbusting with Google & Bing

at the technical details.


SEO Mythbusting with Google & Bing

Would you say the same, or is it different for you,


SEO Mythbusting with Google & Bing

what you're seeing?


SEO Mythbusting with Google & Bing

No, they are definitely getting technical.


SEO Mythbusting with Google & Bing

And they're doing a lot of work, especially trying


SEO Mythbusting with Google & Bing

to make sure that we understand the content behind it, right?


SEO Mythbusting with Google & Bing

And what's the intent?


SEO Mythbusting with Google & Bing

What is the website?


SEO Mythbusting with Google & Bing

They do a lot of work.


SEO Mythbusting with Google & Bing

But one thing we also notice is that they


SEO Mythbusting with Google & Bing

forget the fundamentals in the grand scheme of things


SEO Mythbusting with Google & Bing

and why they're doing it.


SEO Mythbusting with Google & Bing

And a classic example is the blockers and robots text.


SEO Mythbusting with Google & Bing

And you're not able to crawl them.


SEO Mythbusting with Google & Bing

Hey, why are you not crawling me?


SEO Mythbusting with Google & Bing

Hey, you blocked me.


SEO Mythbusting with Google & Bing

You're telling us we shall not crawl,


SEO Mythbusting with Google & Bing

but you're not crawling me.


SEO Mythbusting with Google & Bing

Yeah.


SEO Mythbusting with Google & Bing

So we actually developed a couple of tools for that,


SEO Mythbusting with Google & Bing

and we have a verified Bing bot.


SEO Mythbusting with Google & Bing

The reason we did that is because when


SEO Mythbusting with Google & Bing

we talk to all these people, hey, why are you blocking us?


SEO Mythbusting with Google & Bing

They say, hey, you're crawling me too much.


SEO Mythbusting with Google & Bing

And then we say, hey, did we really?


SEO Mythbusting with Google & Bing

Because we have blogs and stuff like that.


SEO Mythbusting with Google & Bing

And we found that it's actually somebody masking as Bing bot,


SEO Mythbusting with Google & Bing

and they were crawling.


SEO Mythbusting with Google & Bing

That happens a lot, doesn't it?


SEO Mythbusting with Google & Bing

Yes.


SEO Mythbusting with Google & Bing

And then we're basically telling them, hey,


SEO Mythbusting with Google & Bing

check out the IP range.


SEO Mythbusting with Google & Bing

And we said, this is the IP range from which Bing bot will


SEO Mythbusting with Google & Bing

crawl you.


SEO Mythbusting with Google & Bing

If it's different, you can just block it at your server level


SEO Mythbusting with Google & Bing

or let us know, and then we'll figure it out.


SEO Mythbusting with Google & Bing

We're also having the indexing API that you can basically


SEO Mythbusting with Google & Bing

submit content.


SEO Mythbusting with Google & Bing

Yes, URL submission.


SEO Mythbusting with Google & Bing

We call it URL submission API.


SEO Mythbusting with Google & Bing

Websites can actually submit URLs, especially


SEO Mythbusting with Google & Bing

if it's new or updated.


SEO Mythbusting with Google & Bing

And we allow them instant, very fast indexing,


SEO Mythbusting with Google & Bing

instant indexing of that.


SEO Mythbusting with Google & Bing

Yeah, it's a push mechanism rather than a pull mechanism


SEO Mythbusting with Google & Bing

as robots and crawling have been.


SEO Mythbusting with Google & Bing

We've also done something called crawl control.


SEO Mythbusting with Google & Bing

You tell us if it's peak business hours,


SEO Mythbusting with Google & Bing

then we won't crawl.


SEO Mythbusting with Google & Bing

You tell us off peak, and then we'll crawl you during the time


SEO Mythbusting with Google & Bing

and tell us the rate.


SEO Mythbusting with Google & Bing

And that's a good signal to return.


SEO Mythbusting with Google & Bing

That's a really, really cool tool.


SEO Mythbusting with Google & Bing

So after that, it's got better.


SEO Mythbusting with Google & Bing

But it's still out there.


SEO Mythbusting with Google & Bing

A lot of people still do that.


SEO Mythbusting with Google & Bing

We do get all sorts of support requests,


SEO Mythbusting with Google & Bing

like, oh, I have this problem that every now and then,


SEO Mythbusting with Google & Bing

or not every now and then.


SEO Mythbusting with Google & Bing

Every day, Googlebot crawls all these pages


SEO Mythbusting with Google & Bing

that I set to no index and forbidden robots txt.


SEO Mythbusting with Google & Bing

And we're like, are you sure that it's Googlebot or not?


SEO Mythbusting with Google & Bing

It's like someone masking it Googlebot.


SEO Mythbusting with Google & Bing

And then it usually turns out you do the reverse GNS look up.


SEO Mythbusting with Google & Bing

It's someone else masking as Googlebot.


SEO Mythbusting with Google & Bing

There are so many crawlers out there.


SEO Mythbusting with Google & Bing

We're seeing a bunch of stuffs happening.


SEO Mythbusting with Google & Bing

So the robots txt is a fantastic example


SEO Mythbusting with Google & Bing

because it has been a basically de facto standard.


SEO Mythbusting with Google & Bing

But it is not really a standard.


SEO Mythbusting with Google & Bing

But I'm really happy to see that we are all working together


SEO Mythbusting with Google & Bing

and getting it into an actual standard


SEO Mythbusting with Google & Bing

and making that more explicit these days.


SEO Mythbusting with Google & Bing

That's right.


SEO Mythbusting with Google & Bing

That's really, really cool.


SEO Mythbusting with Google & Bing

And I saw the articles.


SEO Mythbusting with Google & Bing

Very nice, and we've been getting.


SEO Mythbusting with Google & Bing

Yeah.


SEO Mythbusting with Google & Bing

Gary is working with y'all.


SEO Mythbusting with Google & Bing

And it's so nice to just get everyone together.


SEO Mythbusting with Google & Bing

And as you say, we are all trying


SEO Mythbusting with Google & Bing

to service the same intention and the same customer.


SEO Mythbusting with Google & Bing

Exactly.


SEO Mythbusting with Google & Bing

And we are not plotting each other against each other.


SEO Mythbusting with Google & Bing

It's more like trying to figure out and learn from each other.


SEO Mythbusting with Google & Bing

I really, really like the Crawl Control tool.


SEO Mythbusting with Google & Bing

And I wish we had that.


SEO Mythbusting with Google & Bing

But yeah.


SEO Mythbusting with Google & Bing

Yeah, there are so many tools out there which you have.


SEO Mythbusting with Google & Bing

I wish we had some of them as well.


SEO Mythbusting with Google & Bing

You're integrating with them, right?


SEO Mythbusting with Google & Bing

Yes.


SEO Mythbusting with Google & Bing

I am verified in Search Console.


SEO Mythbusting with Google & Bing

I am automatically verified in Webmaster Tools.


SEO Mythbusting with Google & Bing

Yes, we just released that.


SEO Mythbusting with Google & Bing

One of the things, hey, it's so tough.


SEO Mythbusting with Google & Bing

I mean, when we talked to webmasters, they said, hey.


SEO Mythbusting with Google & Bing

So it's very time consuming to verify.


SEO Mythbusting with Google & Bing

And you notice that the same verification mechanisms


SEO Mythbusting with Google & Bing

are also there in Search Console.


SEO Mythbusting with Google & Bing

So we said, hey, if the same verification mechanisms,


SEO Mythbusting with Google & Bing

they will honor that.


SEO Mythbusting with Google & Bing

And you've done all the work, so we'll honor it.


SEO Mythbusting with Google & Bing

You can actually keep both of them in sync together, right?


SEO Mythbusting with Google & Bing

You verify.


SEO Mythbusting with Google & Bing

You can import your site maps.


SEO Mythbusting with Google & Bing

And you're all set in sync, making it so easier.


SEO Mythbusting with Google & Bing

So I have an interesting challenge


SEO Mythbusting with Google & Bing

that I wonder how you are dealing about this.


SEO Mythbusting with Google & Bing

Because it is very hard to get people


SEO Mythbusting with Google & Bing

the right guidance on this.


SEO Mythbusting with Google & Bing

And when I'm saying that, I mean JavaScript, right?


SEO Mythbusting with Google & Bing

So we both know JavaScript is probably


SEO Mythbusting with Google & Bing

the most expensive resource you have on your website


SEO Mythbusting with Google & Bing

because images are not that expensive.


SEO Mythbusting with Google & Bing

Even video, you can parse more or less as it comes in.


SEO Mythbusting with Google & Bing

But JavaScript, you can't.


SEO Mythbusting with Google & Bing

On one hand, people are building really cool things


SEO Mythbusting with Google & Bing

on JavaScript.


SEO Mythbusting with Google & Bing

On the other hand, I don't want them to entirely rely


SEO Mythbusting with Google & Bing

It's a really narrow path to navigate.


SEO Mythbusting with Google & Bing

What are you doing on these things?


SEO Mythbusting with Google & Bing

So it's very similar, right?


SEO Mythbusting with Google & Bing

I know that the challenges are very expensive.


SEO Mythbusting with Google & Bing

And the first thing we have to figure out


SEO Mythbusting with Google & Bing

is the intent, if the website or the page has the right intent.


SEO Mythbusting with Google & Bing

And if you're able to figure out the intent behind it,


SEO Mythbusting with Google & Bing

it's easier to prioritize.


SEO Mythbusting with Google & Bing

Sometimes you would just have, like,


SEO Mythbusting with Google & Bing

just execute a couple of JavaScripts


SEO Mythbusting with Google & Bing

and no intent, no title, no description, no meta


SEO Mythbusting with Google & Bing

description, none of that.


SEO Mythbusting with Google & Bing

And we don't even know what it is.


SEO Mythbusting with Google & Bing

Do we really want to execute?


SEO Mythbusting with Google & Bing

So that time, it's a challenge.


SEO Mythbusting with Google & Bing

Because at the end of the day, you


SEO Mythbusting with Google & Bing

have a certain set of resources.


SEO Mythbusting with Google & Bing

And you figure out.


SEO Mythbusting with Google & Bing

MARTIN SPLITT


SEO Mythbusting with Google & Bing

So what do you say?


SEO Mythbusting with Google & Bing

I mean, a client-side rendered application in JavaScript


SEO Mythbusting with Google & Bing

is not a problem by itself.


SEO Mythbusting with Google & Bing

It's just more expensive.


SEO Mythbusting with Google & Bing

And you have no hints up front.


SEO Mythbusting with Google & Bing

So you can't figure out the intention.


SEO Mythbusting with Google & Bing

You do not have the title.


SEO Mythbusting with Google & Bing

None of this exists until the JavaScript is executed.


SEO Mythbusting with Google & Bing

Would you say that server-side rendering and hydration


SEO Mythbusting with Google & Bing

is a good way to fix that, as in when


SEO Mythbusting with Google & Bing

you ship a bunch of initial content to the browser


SEO Mythbusting with Google & Bing

and then use JavaScript to enhance it for the user,


SEO Mythbusting with Google & Bing

giving you a balance?


SEO Mythbusting with Google & Bing

Yeah, the right balance, yes.


SEO Mythbusting with Google & Bing

You need to do both of them.


SEO Mythbusting with Google & Bing

Because at the end of the day, you


SEO Mythbusting with Google & Bing

also want to make sure it's a lively page, right?


SEO Mythbusting with Google & Bing

I mean, you want rich results as well to serve your customers.


SEO Mythbusting with Google & Bing

And especially in this world, people want richer results.


SEO Mythbusting with Google & Bing

So you need to balance that.


SEO Mythbusting with Google & Bing

So some server-side and some client-side.


SEO Mythbusting with Google & Bing

So that's lovely to hear that we are aligning


SEO Mythbusting with Google & Bing

on recommendations again.


SEO Mythbusting with Google & Bing

This is really cool.


SEO Mythbusting with Google & Bing

And do you also sometimes get these weird questions


SEO Mythbusting with Google & Bing

where people are leading with a random third-party tool?


SEO Mythbusting with Google & Bing

Like, oh, we have this tool.


SEO Mythbusting with Google & Bing

And the score in this tool is 75.


SEO Mythbusting with Google & Bing

And I don't understand why we're not ranking higher.


SEO Mythbusting with Google & Bing

Yeah, that's the DAs.


SEO Mythbusting with Google & Bing

All of these lovely metrics.


SEO Mythbusting with Google & Bing

I'm like, hmm, fantastic.


SEO Mythbusting with Google & Bing

That's good.


SEO Mythbusting with Google & Bing

They need some number and metric.


SEO Mythbusting with Google & Bing

And they try to ask those questions,


SEO Mythbusting with Google & Bing

but you don't have an answer because that's not


SEO Mythbusting with Google & Bing

the only signal.


SEO Mythbusting with Google & Bing

There are multiple things.


SEO Mythbusting with Google & Bing

I think a lot of people really don't understand


SEO Mythbusting with Google & Bing

that there's a query intent.


SEO Mythbusting with Google & Bing

People really think about the serving side


SEO Mythbusting with Google & Bing

and what they have in the index.


SEO Mythbusting with Google & Bing

People forget there is a query intent.


SEO Mythbusting with Google & Bing

There's a user intent behind it, and we have to match.


SEO Mythbusting with Google & Bing

I think that education is not yet there with the audience.


SEO Mythbusting with Google & Bing

I think if we do a little more in that,


SEO Mythbusting with Google & Bing

people would understand, hey, there is a query intent,


SEO Mythbusting with Google & Bing

and then there is the document intent.


SEO Mythbusting with Google & Bing

So we can kind of sum it up into rather than just looking


SEO Mythbusting with Google & Bing

at a number that a random tool gives you,


SEO Mythbusting with Google & Bing

try to understand what your user needs


SEO Mythbusting with Google & Bing

and what the intention is, and then serve that intention.


SEO Mythbusting with Google & Bing

Yeah.


SEO Mythbusting with Google & Bing

I mean, it's just an education.


SEO Mythbusting with Google & Bing

People need to get to know that.


SEO Mythbusting with Google & Bing

That is true.


SEO Mythbusting with Google & Bing

But we keep saying this for years and years on end.


SEO Mythbusting with Google & Bing

It's not yet landed this way, I would feel.


SEO Mythbusting with Google & Bing

It's unfortunate.


SEO Mythbusting with Google & Bing

But I think we have landed a success


SEO Mythbusting with Google & Bing

in getting an overview of what the community brings


SEO Mythbusting with Google & Bing

to our tables and to share that we are not


SEO Mythbusting with Google & Bing

different in that sense.


SEO Mythbusting with Google & Bing

We are basically seeing the same kind of challenges,


SEO Mythbusting with Google & Bing

and we're seeing the same kind of recommendations.


SEO Mythbusting with Google & Bing

And if you do the same stuff, all the search engines


SEO Mythbusting with Google & Bing

would pick up exactly the same stuff.


SEO Mythbusting with Google & Bing

The fundamentals are very, very similar.


SEO Mythbusting with Google & Bing

There could be some differences, but that's expected.


SEO Mythbusting with Google & Bing

Exactly.


SEO Mythbusting with Google & Bing

The secret sauce is a little different.


SEO Mythbusting with Google & Bing

But the fundamentals are the same across all search engines.


SEO Mythbusting with Google & Bing

So hopefully people will get to the point


SEO Mythbusting with Google & Bing

of serving their users and have very nicely clean intent pages.


SEO Mythbusting with Google & Bing

And yeah, we are seeing more interesting things


SEO Mythbusting with Google & Bing

coming in the future.


SEO Mythbusting with Google & Bing

Yes.


SEO Mythbusting with Google & Bing

Thank you so much for being here and talking


SEO Mythbusting with Google & Bing

about all of this with me.


SEO Mythbusting with Google & Bing

Yeah, thank you so much, Martin, for inviting me.


SEO Mythbusting with Google & Bing

This is wonderful.


SEO Mythbusting with Google & Bing

Absolutely.


SEO Mythbusting with Google & Bing

All right, in that case, thank you very much for joining us.


SEO Mythbusting with Google & Bing

And enjoy.


SEO Mythbusting with Google & Bing

Hey there.


SEO Mythbusting with Google & Bing

I hope you liked the video.


SEO Mythbusting with Google & Bing

This is my next guest, Alexis Sanders.


SEO Mythbusting with Google & Bing

And in the next episode, we will be talking about?


SEO Mythbusting with Google & Bing

Crawl budget.


SEO Mythbusting with Google & Bing

So stay tuned and check out the next one


SEO Mythbusting with Google & Bing

to learn more about what we have to say on the topic.


SEO Mythbusting with Google & Bing

See you there, everyone.


SEO Mythbusting with Google & Bing


English Google Webmaster Central office-hours from August 4, 2020

Maybe it'll start recording.


It shows recording to me.


Oh.


There we go.


OK.


Woo-hoo.


All right, welcome, everyone, to today's Webmaster Central


Office Hours Hangout.


My name is John Mueller.


I am a webmaster trends analyst at Google in Switzerland.


And part of what we do are these office hour


hangouts, where people can join in and ask their questions


around web search.


Bunch of stuff was submitted already,


but we probably have time.


So if any of you want to go ahead and ask


a first question, feel free to jump on in.


Can I put that in?


An easy one for you, John.


How does a site get in the Chrome User Experience Report


if they're not there, that data set?


Over 4 million sites, and this is the only one


I've ever seen not be in it.


I don't know for sure.


I think it's based on a sample of the Chrome traffic.


But I don't know how Chrome does that.


So it's not something you have to sign up for or do manually.


It's something that Chrome figures out automatically


that this is one of those sites that needs to be included there.


OK.


Yeah.


Make it more popular.


I guess so.


I mean, it is a local business site,


but it gets thousands from searches there.


Click something to search.


It's not tiny, but it's not massive.


I mean, the data that you see there,


you can also pick up either through doing kind of a lab


test, where you test it yourself live,


or you can use the JavaScript code, the JavaScript library


for Core Web Vitals, and pull that data directly


into analytics, for example.


So that might be a way to kind of get that data as well.


You don't see it in Search Console in those cases,


but you at least have a rough understanding


of what it is with the speed with regard to that site.


Yeah, it's hard for me to gauge the progress


as I'm trying to make changes.


And sure, I can use the lab data and page speed tools


or whatever it may be.


But it's nice to see that green graph spike up,


and I don't get that reward with this site.


I have no idea.


So you said I can use a JavaScript library to bring it


into Google Analytics?


Yeah, I'm pretty sure.


At least that's what I remember.


So I can double check and find the link for you afterwards.


OK, sure.


Thank you.


John, I have a follow-up question


from a couple of weeks ago, what you said.


So about negative reviews not hurting.


So if you have a bad reputation online


and you see a lot of negative stuff about your company,


you search Google, and Google has a lot of negative reviews


and a lot of bad stuff comes up, would that


hurt potentially from Google ranking for keywords?


Could Google look at that and say, oh, this


is a bad company.


We're not going to rank it as well because they


have a lot of negative reviews.


You said, I don't think that would hurt the overall rankings


for a website if there's a bad reputation around the site.


But if you go back to 2010, there


was some type of algorithm out there


that looked at people manipulating their customers


so that they would write bad stuff about them


so that they would get links and so forth.


And Google said, yeah, we will go ahead and take that


into account.


Has that changed?


Or you're just saying in general, it probably won't impact.


You don't have to worry about it in general.


But if you do a crazy amount of damage to your customers


where everything is bad about you, it might actually hurt you.


Yeah, I don't remember offhand exactly what the context was


a couple of weeks ago.


But that was, I think, that eyeglasses company


that you mentioned, right?


Yeah, it's a core company.


Yeah.


I mean, that's something where if all of the signals


point in that direction, I could imagine


that we might pick that up.


But if you're talking about there


are a handful of people that are upset


and they're writing these random things online,


and there are lots of people that are happy with your site


and everything is normal, then that's not something


where I would really worry about.


I think those situations where it's


like there are a lot of people that are really upset


about your site, those are probably pretty rare,


not something that most normal sites would run into.


OK, but as far as you know, whatever


that was from 2010 is still around,


but it's probably hard to kick in, or you don't even know.


I don't know if that specific thing from 2010


is still around, because things change quite a bit over time.


But that is something that we would try to pick up on,


if it's something where we see that everything is really


bad about this site.


And that might be something that our algorithms


try to pick up on.


But that's something where, in general, these kind of things,


because they're so vague sometimes,


really need to be really strong.


It really needs to be a strong signal for us to say, OK,


we really can trust this problematic information


and apply it appropriately for the site's ranking.


But I don't remember in detail what


we talked about a couple of weeks ago there.


My feeling was that was more a case of kind


of like a normal website, and there


are a bunch of bad reviews out there,


but they're kind of embedded in the web in a normal way.


And that feels like something where


it's easy to get obsessed about a handful of bad reviews,


and it's probably not something that would drastically


affect the outcome in search.


I think that's always tricky with these kind of things,


to kind of infer from in a small situation where you are in,


like, this applies, and then to take that and say,


well, that applies to everything.


But yeah.


No, the reason I'm not, I'm just curious if that specific thing


that the Google blog wrote about in 2010


is still something that's active in general.


Obviously, I would think they didn't really


it would have been cool if you said, yeah, if 80% of what's


written about your company on the web is negative,


then that's the threshold.


But obviously, you guys are not going


to talk about thresholds.


But I just want to know in general


if that's still around that specific thing,


but you don't know what it seems like.


So that's OK.


Yeah.


I mean, things have changed quite a bit in 10 years.


So it's like that specific thing is almost certainly not there


in the same way as it is in the Google blog.


It's probably not there in the same way as it was back then.


But we probably take something similar into account.


And threshold, I think that's something that


is almost impossible to say.


OK.


Let's take a look at some of the submitted questions.


I was wondering if I need to submit separate XML sitemaps


for images and URLs, or can I have one sitemap that


includes both URLs and images?


Will that have any impact on crawling and indexing?


You can use one sitemap file for everything, essentially.


There are limits with regards to the number of URLs


you can put into a sitemap file and with regards


to the size of the sitemap file.


But you can split that up however you want.


I don't think, like in most normal cases,


that you would see any effect on crawling and indexing


depending on how you split things up in a sitemap file.


It might be if you take a 1 million URL website


and you turn it into 1 million sitemap files


that you might see an effect.


But if you're talking about reasonably sized sitemap files,


that is something where I wouldn't


expect to see any effect with regards to differences


how you split that up in a sitemap file.


When we have no content in our new sitemap,


we answer requests with a 304, not modified status code.


But we're also using other solutions on other domains


where we always answer with a 200 status code,


no matter if the sitemap was updated or not.


What do you recommend regarding Googlebot?


Is 304 a problem?


So 304 is a HTTP status code that you


can return when a request comes in that says,


has this file been modified?


If so, send me the file.


And in those cases, returning 304 basically says,


no, it hasn't been modified.


The thing to watch out for with a 304


is that really only makes sense when the request is


one of these conditional requests,


not for all requests that are done to a site.


So if Googlebot is just saying, give me your sitemap file,


then returning 304 essentially doesn't really help there.


Because it's essentially, from our point of view,


just a sign, well, this file is currently not available.


And there's no content essentially


that you can return for that.


If you don't want that sitemap file


to be processed at the moment, returning a 304 is fine.


Returning a 404 is fine.


If you do want the file to be processed,


then you should return the sitemap file.


I don't know how, for sitemap requests,


if we always do conditional requests


or we never do conditional requests,


I know for normal web crawling, we do a mix.


We do some requests with this conditional.


Has this been modified since the last date that we know?


And some requests are essentially just give us


the current version.


And for web search in general, I would not


return a 304 status code blindly unless it's really


a conditional request.


Because if you return a 304 status


code when the request is a normal unconditional request,


then essentially, you're not returning any content.


And we don't have anything to index.


So for indexing, that would be bad.


For sitemap file, if you just don't return anything,


then we can basically say, well, we


don't have a sitemap file at the moment, but that's OK.


We can still work with the rest of your site.


So for normal crawling, only return a 304


if the request is really conditional.


And for sitemap file, if you want


to just return an empty file or a 404 or a 304


for cases where the sitemap file hasn't been changed,


that's kind of up to you.


Why some of my pages show in the Google web cache


while others don't?


What could be the reason?


There is generally no specific reason


for some pages being shown in the cache


and some pages not being shown in the cache.


That's essentially just a side effect of our internal systems.


So it's not a bad sign if a page is not shown in the cache.


It's not a sign of quality if it is shown or if it isn't shown.


It's essentially just our internal system saying,


well, we currently don't have a cached page


that we can show you.


And that's completely normal.


That cached page is not reflective of anything


when it comes to ranking.


So that's something where I wouldn't necessarily


worry too much about the cached page being shown or not shown.


If you want to test what Googlebot would see,


then I would use the Inspect URL tool and not the cached page.


I changed the niche of my blog.


Earlier, it was on how to blog.


And now we're only writing content about gadgets.


We removed all of the old niche content.


And now it's been more than a year.


And we're still not getting any traffic.


We have around 100 quality blog posts.


Why are we not getting any traffic?


We run a Google Search Engine ad campaign.


And we found that users spend more than eight minutes


on a page.


So it's really hard to say why you're not getting traffic.


There are lots of reasons why you might or you might not


get traffic from Google Search.


So that's something where there's no absolute answer


or you can just say, well, you need to do this.


And then you will suddenly get traffic from search.


Because obviously, everyone wants that and wants


more traffic from search.


So what I would recommend doing here


is going to maybe the Google Webmaster Help


forums or some other webmaster SEO forum,


mentioning your site, mentioning some


of the details about your site, specific URLs and queries


on your site that you kind of would like to rank for,


and getting some honest advice from other people


about your website.


And sometimes there are small technical things


that you can change.


Sometimes it's more a matter of, well,


you're writing about this topic where


there are millions of other people writing


about the same topic.


And it's a lot of competition.


It'll be hard to kind of get a foothold there.


But all of this advice can be really useful for you


to figure out what you can do next.


Where can you go from that situation where you are now?


Is it something where you can tweak the content?


Is it something where you need to reconsider your targeting?


Is it something where maybe from a technical point of view,


something is not working as well as it should for your website?


So that's kind of the direction I would go there.


I have a website with user-generated content,


a listing site.


Users submit their website URL.


But after some period, some of the website domains


expire or they're not working.


What should we do with those kind of users?


Really hard to say here.


I think, first of all, from a very high-level point of view,


if this is a website where people can submit their URL


and you just show a link to their site on your website,


then it feels like one of those kind of website models


that's a little bit, I don't know, out of date,


in that it's really hard for us to understand what we should


be showing your pages for.


So in particular, if you're just linking


to other people's websites, what kind of queries


would make sense for us to show your website instead


of their website directly?


So that's kind of from a very high-level point of view,


looking at this kind of directory website model,


that's something where I might reconsider,


maybe it would make sense to kind of move


into some more modern website model, I guess.


With regards to the links on your site


no longer being valid at some point,


404s that you link to are unfortunately part of the web.


It's not something that we would take a look at and say,


this is a sign of low quality or it's a sign of a bad website.


Users might be kind of unhappy if that's


the primary reason for your website


and that doesn't actually work.


But that's more of a thing between your users and you.


That said, if you're providing user-generated content


and you know this user-generated content is kind of low quality,


if you will, then that might be something


where you might want to step in and say,


well, I need to make sure that everything


that my website provides is of the highest quality possible.


And then things like maybe moderating the content


that people submit to your website,


double-checking the links, double-checking


that it's functional, that it's useful for other people,


that's something that you could do.


And especially checking links is something


that's usually pretty easy to automate as well.


So that's something I would kind of go into that direction.


That said, like I mentioned in the beginning,


this kind of model of having a directory where people can drop


links and you just post those links on the web,


that feels like something where at some point


you might want to move on to something more modern.


We have two pages with almost the same content,


but in different languages and different subdomains,


like an article in Hindi, and the same translated slightly


modified version in Tamil or Punjabi.


Is there any effect in search ranking and traffic


on the original Hindi page in this case?


Does Google do some internal translation


and compare the article content for ranking in any way?


So it's cool to see translated content on the web.


In general, we see these pages as being completely separate.


Essentially, they stand on their own.


If you translate the content, it might be the same thing,


but they're very different pages.


So just purely from a technical point of view,


we look at the words on the page,


and they're very different words.


One is in English, the other is in Hindi,


or in Tamil or Punjabi.


So from that point of view, these


are different pages that we would rank independently.


We would index them independently.


We don't see them as duplicate content.


So that's perfectly fine.


It's not something that you need to kind of resolve or fix


or change in any way.


What you could do is use hreflang


to link between those pages so that we


know which version is in which language or for which country.


If the traffic that you're getting to these pages


is already pretty clear in that people in Hindi


are landing on the Hindi page, people


searching in Tamil are landing on a Tamil page,


then you don't really need to use hreflang.


So that's kind of something where


there is a lot of work involved with making hreflang work well.


And if you're already seeing that things are working out


well, you don't really need to implement that.


With regards to ranking, if there


is kind of a positive or negative effect there,


there's no real effect there.


These are separate pages in separate languages.


We will rank them individually.


If someone is searching in Hindi,


we'll try to show the Hindi page.


We don't try to compare those pages


to see if they're equivalent, if they're exactly the same.


Sometimes translations will be significantly different,


which might be that a certain subset of functionality


is available in one language and a smaller set


is available in another language.


It's still a translated version of the same page,


but the content itself might be very different.


That's perfectly fine.


So overall, I'd kind of continue going that direction.


That sounds good.


Why does Google keep selecting the wrong canonical?


For example, in Search Console, it


says user declared canonical is this one,


and Google selected canonical is a different one.


So in general, that's really hard to say without looking


at the actual URLs.


So looking at the sample URLs you have there,


the domain name is not mentioned,


so I can't actually check to see what is being shown there.


But within the URL, you have slash en-se,


which I imagine is for English for Sweden or something


like that, and then slash en-za, which sounds like English


for South Africa perhaps.


And the cases where I have seen something like this happen,


where we kind of mix up the canonicals,


are usually situations.


Well, not usually, but it sounds like this


would be one of those situations where


you have different versions for different countries,


but the content itself is actually the same.


So in this case, probably the English version for Sweden


is the same version as the English version


for South Africa.


It's a different URL for different countries,


but if the content itself is essentially the same,


it doesn't have to be exactly the same.


But if it's primarily the same, then what will happen


is we'll say, well, we can save you some trouble.


We'll just crawl one version, and we'll


index that one version, and we'll use that as a canonical,


even though you've specified that these are separate pages.


So from my point of view, that's probably


what's happening there.


I don't know without looking at the exact URLs.


If that's the case, on the one hand,


you can just kind of take that and kind of work with that.


And in many cases, that doesn't change anything.


If you do need to make sure that you have separate pages shown


in those two countries, then you need


to make sure that the content itself is really significantly


different across those pages so that when our algorithms look


at those two URLs, we will say, well, this


is really different content.


We need to make sure that we index it individually.


And in cases like that, we will index it individually.


Usually, switching from kind of us picking one as a canonical


to us indexing both of these versions separately


is something that takes a bit of time.


So I wouldn't expect that to switch from one week


to the next.


But maybe over a couple of weeks or a couple of months,


that would settle down and we'd index both of those versions.


For one site I'm helping, the crawl stats in Search Console


show about 600,000 URLs crawled per day.


But the site only has 100,000 crawlable URLs in total.


About 30,000 of those are indexed.


I know crawl stats take all URLs into account, not just


HTML pages.


But that's a massive difference.


Most larger scale sites I'm helping


are not showing a six times difference


crawl URLs per day versus total URLs on the site.


Let's see.


Do you know why they would be showing such a large number?


It is really hard to say without looking at the site itself.


The thing that's sometimes a bit confusing


is the count that we show in Search Console


includes, on the one hand, all URLs


that we pick up for crawling, which


means it also includes everything


that we use for rendering.


So things like images, CSS files, JavaScript, server


responses, all of those are URLs as well from our point of view.


So we would show those there.


And the other thing that's sometimes a bit confusing


is that this includes all requests that


go through the Googlebot infrastructure.


And that also includes things like the ads landing page


checks.


I think it also includes, for shopping and e-commerce,


the product landing page checks, all of those things.


And depending on the type of website


and how they're active on Google,


those numbers can sometimes be quite significant


in that we request a large number of URLs


from a site to do these ads landing page checks,


depending on how you have Google Ads set up.


And those would also fall into this graph


of crawled URLs per day.


Technically, it's not crawled.


And technically, it's not pages.


But essentially, everything that goes through that infrastructure


is shown there.


So that might be something that you might see there.


But again, it's really hard to say without knowing more.


And that's something where, if you look at the server log


files, then just looking at the user agent alone


would not show that.


But usually, if you take a rough look at the IP address


and take a chunk from the Googlebot crawled IP addresses,


then you should be able to see some overlap there.


The other thing to keep in mind is that a lot of these requests


can be at essentially different priorities


in that they might have different importance.


So if you're seeing that we're crawling too much,


if this is crossing a load on your server overall,


you can limit the total number of requests.


And that will limit the total number of requests


that we make to the site.


But that doesn't mean that the crawled for web search


essentially goes proportionally down.


It could be that we continue to crawl everything that we need


for web search and the URLs that we use for double checking


404 pages, all of those random things as well,


that those requests go down.


So that might be something to take a look at to see if we're


crawling too much for your website.


Why would my website.com slash English slash SE website


get a majority of impressions and clicks


from India and the US?


How can I make sure that the site only shows up


in the Google index for Sweden?


This is only a site that's giving me problems.


All 12 other countries have sites


that seem to get the correct local traffic,


except for this one.


So I think this is the same kind of question,


or similar as before, in that you have English for Sweden.


And probably what is happening, I mean,


I don't know your website to be sure,


but probably what our systems are seeing


is that we're seeing the same English


content from multiple countries.


And we're picking maybe the Sweden version


as the canonical for all of that English content.


And that's the one that we would be showing in search,


because that's what we picked as a canonical.


So in a case like that, it can happen


that we show your Sweden page in India and in the US,


because we think these are all the same pages.


In general, the hreflang annotations


help us a little bit.


But what's really important is to make sure


that the content is different so that we


don't fold these together if you care about this difference.


And another thing to keep in mind


is that all of this geotargeting is not guaranteed to happen.


So it can still happen that we recognize


that the English for Sweden page is


different than the English for India page, for example.


But if we think the English for Sweden page is really


the right English version to show to other users,


then maybe we would show that English for Sweden page even


in India.


So geotargeting hreflang is not something that is guaranteed.


It's not a restrict that we would only


show this in those countries.


But rather, it's a signal that goes into our systems


and says, well, this is really my preference.


I'd appreciate if you did this.


But it's something where our systems will still


need to look at that and say, well, that's good to know.


But we still really think it's different.


So what you can do in cases like that


is try to catch that on your side.


So assume that users from all around the world


will be accessing your site from everywhere


or all parts of your website.


And try to catch those.


Try to show a banner on top, for example, saying, hey,


it looks like you're coming from India,


and this is the page for Sweden.


We have a page specifically for India.


You can click here to go there.


By showing a banner on top, you're


kind of sure that it won't affect search.


Whereas if you automatically redirect,


then any time Googlebot crawls the page from a location


that you're automatically redirecting away from,


then we would not be able to see that content at all.


So showing a banner would be the right thing


to kind of catch these users that


are going to the wrong version of your website.


Yeah, that's kind of the direction I would go.


I mean, the other approach for all


of these international sites is to try to simplify it


a little bit more and to say, well, actually, I


just care about the individual language versions,


and maybe just have one English version for multiple countries.


And by doing that, you can often reduce the number of URLs


that you need on your website, which makes things


a lot easier overall.


So that might be another approach to take there.


But again, it's really hard to say


without knowing your website and understanding


why you have different English versions


for different countries.


Yeah?


Sure.


I wonder if Google was thinking about changing


the way it treats the hreflang tag in this respect.


Because actually, when a webmaster


like the friend who posted this question specifically


wants to tell Google, in the end effect,


which users should be shown which website,


then I think it's a good idea for Google


to actually follow this guideline.


Wouldn't it be?


I think it would be good.


But it's always tricky in practice.


So I don't know, for example, in a case like this.


But I see this a lot with German websites.


I imagine that's also where you've seen it,


especially the set of German, Austrian, Switzerland,


where often the content is exactly the same.


And they're targeting for multiple countries.


And that's something where we do try


to recognize when there is something


kind of country specific on a page so that we don't


accidentally deduplifying these URLs into kind of one


canonical.


But it is something where if you really


want to be sure that this doesn't happen,


then you need to make sure that nothing on our side


would assume that these pages are actually the same.


The tricky part with hreflang that I've seen


is that some sites essentially implement all variations.


So it's like number of countries times number of languages.


You can generate all of those pairs.


And that's something where we just


end up getting lost when we try to crawl


and index sites like that.


So that's where our systems kind of have these protections


built in, where if we can recognize that you're serving


the exact same content for all of these versions,


we can simplify that for you so that we


don't have to crawl all of these versions


and we don't have to index all of these versions.


But obviously, there's middle ground in between there,


too, where if you just have two or three different versions,


we should be able to say you have hreflang set up properly


here.


We should treat those as separate URLs.


SPEAKER 2 I was just wondering, because you can easily say,


OK, you don't give the, let's stick to the German, Austria,


Switzerland example.


So when I take the hreflang tag, I could just say it's DE.


So that means all German speaking countries in the world.


Or I can say DE minus DE for specifically Germany.


And thinking about the URL structure


and what kind of an effect that has,


or a psychological effect, if you like, to the users,


then it might be well worth it, especially


thinking of all the English speaking countries in the world


or I'm aiming at Russian speaking countries in the world.


And then you have a conflict probably,


which is a bit racist, I admit.


But some Ukraine people probably won't


like to see a.ru in their URL.


But I'm not able to address them in the way it works currently.


So we do also have some protections in there,


which are, I think, almost a bit more confusing


in that in the situations where we recognize the hreflang


and we recognize that these pages are the same,


then we will fold them together.


We will pick one canonical.


But because we know of the hreflang,


we will show the appropriate version in the search results.


So for Germany, Austria, Switzerland, for example,


we might pick one as a canonical,


maybe the one for Germany.


And in the search results, we would show the Swiss URL


or the Austrian URL.


However, because we have chosen one as a canonical,


when you look in Search Console, you


will see all of the reporting on that canonical URL,


so on the Germany URL.


So it's something where if you just look at Search Console,


it might be, well, Google has everything


on the German URL, the German Germany URL.


But actually, the ones that we show are the local versions.


So that's something where if you just look at Search Console,


you could think, well, it's totally wrong.


But if you look at the search results,


you'll often see that actually the correct URL


is being shown.


So it's sometimes pretty complicated.


And that's kind of why I tend to point


into the direction of if you want


them to be treated separately, then


make sure that they're really separate content.


Because then you don't have the situation where


we'll try to fix it for you, and then we


show you confusing data in Search Console,


and then you're confused.


And then you look at it yourself,


and it's like, it's not how Search Console is showing.


And it's like, Search Console is broken.


It's like this endless cycle.


So if you clearly make them separate,


then you don't have to worry about all of that.


But I got that point.


It's not just so many times on the web


you're offering one service.


So well, exactly the same service in different countries.


Thank you very much.


John, I have a follow up on that.


So your friends at Microsoft, Bing Search,


or whatever they're calling it today,


said that I actually spoke with them on Friday.


And I'm asking them about HRF Lang.


And they weren't so into it.


And they said, potentially, they're thinking about new ways


of going about doing the same thing with you guys.


Have they been in touch with you guys?


Probably not.


I don't know.


You can't disclose that.


I mean, on the one hand, Gary is the one


who is generally in touch with them with regards


to a lot of these search things.


And I have no idea.


I think, in general, HRF Lang can be really, really tricky.


And it's something where, at first, when you look at it,


it's like, oh, you have two different versions of the URL.


And you just link them together.


It's so easy.


But with a larger website, with more than just two or three


languages, it gets really hard.


And we've looked into different ways of making that easier.


I came up with a really neat scheme


that I thought was really easy until it turned


into using regular expressions to map out


how your website works.


And at that point, it was like, well,


I don't know if this is really an improvement.


But if anyone, like if that's from Microsoft or anyone


out there, has an idea for a good and robust and easy


to understand way to make international websites so


that they work well, then by all means, come send it to us.


And we'll try to see what we can do.


SPEAKER 2


So in your example with the three German pages


with very similar content for three separate audiences,


basically, would it help?


It makes total sense how you treat that in Search Console


and how you treat that overall from a search point of view.


And I think that's helpful because there are situations


where you might want to have those three


different German versions, even if you can monitor them


properly.


Does it make any difference in how the webmaster sets


canonicals for those three pages?


So you mentioned you're selecting one canonical version


for those three pages.


Should the webmaster also do that?


Does that make any difference?


No.


I mean, we kind of expect the pages to be self-canonical.


So the German for Germany page is


canonical for German for Germany.


German for Austria page is canonical for German


for Austria.


If you canonicalize all of those to one page,


then we will probably follow that.


And we will only index the German for Germany page


or whatever you pick.


And then we won't be able to verify the hreflang


links between those pages.


You'll miss the hreflang.


We would kind of miss that, yeah.


Good point.


Yeah, OK.


And it's something where it's sometimes


tricky with regards to the content on the page.


But we do try to watch out for things like currencies,


like addresses, phone numbers, to make sure


that we're not missing kind of a country-specific version


of a page.


So if you have all of those same currency,


all of the same phone numbers and addresses


on all of these pages, then that's easier for us to say,


well, we should just pick one.


Whereas if you have an address in Germany,


an address in Austria, an address in Switzerland,


then it would at least be something


where I would take that to the indexing team and say,


hey, we should be able to figure this out.


Even if the rest of the content on the page is the same,


we should be able to recognize this is an address in Germany


and this is an address in Switzerland,


we should not mix these pages.


What if it's a question of just the currency or something


very minimal, something like that, just the price?


We should still pick that up.


I think that's something we should pick up.


It's trickier if you have something like a currency


dropdown and one currency is selected on one page


and the other currency is selected on the other page,


then kind of the same currencies are


mentioned on both of these pages.


It's just one of them is default here, one of them


is default there.


But if that currency is really on that product page,


then we should be able to pick that up and say that's


like Swiss francs and that's euros.


Those are different things.


We should not fold those together.


OK, so in Search Console, you would see them separately then.


But I would definitely bring up to the team


that you would advise on trying to build more local signals


around these websites as well or to enable


and things like that.


The easier you can make it for us


to understand that these pages are specific to one country,


the more we can kind of follow that.


So things like local phone numbers, local addresses,


all of that can play a role there.


So I think the price itself might be tricky.