on the same domain and tagged as alternate language


English Google SEO office-hours from December 17, 2021

versions of the US content.


Is it OK to translate only some of the website's content,


or should we translate everything and exactly mirror


the English website to stand the best chance of ranking


in other locations?


Will Google perceive the website as being less comprehensive


or authoritative to Spanish users


if not all content is translated?


Is it an issue if the Spanish version of a page


links internally to content that is in English?


So first of all, when we look at language,


we look at that on a per-page basis.


So it's not so much that we try to understand


this part of the website is Spanish


and this part is in English.


We essentially look at an individual page and say,


well, it looks like this page is in Spanish.


And then when someone searches for something Spanish,


we will be able to show that to them.


So from that point of view, it doesn't matter.


If only a part of your website is


translated into a different language, that happens.


Usually, you start somewhere and you expand from there.


I think the aspect of internal linking


could be a bit tricky in that it could provide a bad user


experience if your internal linking is all


focused on the English version.


But if you have individual pages which are in English only


and you link to them from your Spanish version, that happens.


I think that's something that's pretty common across a lot


of different websites.


It's just for the most part, if you


have Spanish versions of your pages,


then I would try to make sure that the Spanish version is


also properly interlinked.


But if they're individual pages that are not in Spanish,


then link to the English one, at least.


I found the search volume of a brand, say, Tomato Mask,


and tomato mask as one word varies a lot.


In Japan, most customers search without a space.


But for other countries, the customers


search with a space between the brand name.


Is it necessary to craft the content for Japan


without a space in the brand name to rank higher?


So it's not so much that we look at the search volume and say,


oh, there's a different search volume here,


and we should treat them differently.


But rather, what happens in cases like this is,


on the one hand, we try to figure out


what the tokens are that are involved


in the query and on the page.


And by tokens, we usually look at things like a word level.


And we try to match those.


And we also take into account things like synonyms,


where if we can tell that this is a common synonym that people


use for different versions of a brand name,


then we will take that into account as well.


And oftentimes, words that always come together,


so if you have a brand that is always


like one word and the next word, then we


will try to treat that as an entity.


And then it'll be similar to someone


searching for things with one word, for example.


So all of these are different ways for essentially


our systems to try to look at that situation and say,


oh, this is very similar to what you had before.


And we will rank them in a similar way.


Depending on the actual situation,


that kind of very similar might still be different enough


that you can tell.


So my recommendation here would be


to really look at the search results.


And based on that, decide, does it make sense


maybe to mention the version of my brand name without a space?


Because a lot of people look for it


without a space in maybe a specific language


or in a specific country.


Or is Google already figuring out


that these things are the same thing


and the search results are similar enough that I


don't have to do it myself?


So that's something where I kind of say it


depends to frustrate everyone.


But it really is something where you need to test it.


And you have the best knowledge of that problem space.


And you need to try it out and see what happens.


And you won't be penalized if you


have a version of your brand name


with a space and without a space on your pages.


So it's not so much that you're taking a big risk by also


including one version of your brand name on your pages.


So from that point of view, I would try it out,


see what shows up in the search results.


And if you don't speak Japanese and you only see this in Japan,


I would try to get a local speaker to help you with that.


Because especially with, I found with Japanese and some


of these other languages, it's sometimes


tricky to understand how people actually search locally.


And just taking something and putting it


into Google Translate will give you some ideas.


But it doesn't really give you that nuance


of what do people actually do or what do they


think when they search.


And especially things like word barriers


can be very different if you're not


used to a specific language.


Let's see.


Does Google treat URLs with a question mark at the end


as duplicates of the same URL without a question


mark at the end?


The question arises in connection


with redirects that will eliminate outdated parameters,


but not the question mark or the ampersand at the end.


I don't actually know.


So for the most part, if you have parameters at the end


or no parameters at the end, we will treat those


as clearly separate URLs.


But we do have some systems in place


that try to do some almost like lightweight canonicalization


for you in that they try to figure out


what simpler version of this URL could we actually be showing,


even if the website itself doesn't provide a rel canonical


or doesn't redirect to a simpler version of a URL.


So a really common one is if you have a page called index.html


and you link to that, then that's


often the same as just linking to a page that's called slash.


So if that's on your home page, if it's


website.com slash index.html, if we


were to see a link like that, we could say, well, index.html


essentially is irrelevant here.


We can just drop that automatically.


And that kind of canonicalization


happens essentially very early in our systems.


And it takes place without things


like the rel canonical or sitemaps or redirects,


all of those other things.


And I don't know offhand if just like a plain question


mark at the end would also fall into this category.


So that's kind of the place where I'm not super sure.


But if you already have this set up on your website,


then you can probably tell fairly quickly


if that extra question mark at the end


is actually being used by Google.


Because if you look at your server logs,


you'll see is that question mark there or not.


If you look at things like Search Console


and look at the URLs that are actually shown in the performance


report, you'll see is that question mark there or not.


So from that point of view, you can probably


test that fairly quickly.


I don't know offhand if we will just drop it automatically.


I don't think it would cause any issues even if we


didn't drop it automatically.


But if we don't drop it automatically


and you want it cleaner, then I would definitely


make sure that at least you have the rel canonical set up


to kind of remove that for you.


My website has recently been dropped


for a few very important keywords in our primary niche.


Adult toys get for some reason online pharmacies and health


blogs have been thriving on the traffic.


And my competitors, me and my competitors lost in this purge.


Does Google penalize adult sites for certain queries?


And if so, is this the reason that my own website and all


my competitors have been given the digital boots


from these important queries?


I don't think we penalize adult websites in that regard.


But we do, like I mentioned before,


have systems in place that try to figure out


is the intent of the query actually


to find something that would fall into the category of maybe


adult content.


And if the intent is clearly not for someone


to find adult content with that kind of a query


or for the most part not to find adult content,


then it is something where we would try


to filter those things out.


And that's something that usually makes a lot of sense


because sometimes there are adult websites that


are named very similarly or different types


of adult content that's named very similarly to things that


are maybe child's toys or things like that.


And you wouldn't want someone who


is looking for a child's toy to actually run


into an adult toy website just because it's


ranking for the same term.


So that's the kind of thing where our systems try to almost


silently figure out what the intent is


behind certain queries and then to kind of adjust that so that


we show something that matches a little bit more what


the perceived intent is.


And understanding what the intent is behind a query


is really hard sometimes.


And we also get it wrong sometimes.


So if you see certain queries where you're like,


well, Google totally messed up with this query


because the intent was clearly to find this kind of content


and Google is not showing any of that content at all


and it looks really weird, then those


are the kind of things that we would love to have examples for.


So it's not so much that Google has something


against adult websites.


There are lots of them out there and people find them


and search for them explicitly as well.


That's all perfectly fine.


But we really just want to make sure


that our search results are such that they kind of match


what people expect.


And the algorithms on our side evolve over time.


The expectations of users evolve over time.


All of this essentially means that these things


can change over time.


And that's kind of expected.


Let's see.


Two questions about content.


Do sponsored links with a sponsored link


help with SEO rankings?


So I assume this is with the rel="sponsored link attribute


attached to a link.


And no, they do not help with SEO rankings.


The idea here is essentially you're


paying someone for that specific link.


It's essentially a kind of advertising.


People can click on that link, go to your website


if they like it.


And if they really like it, they can also


recommend it to other people.


But the reason for that link being on that other website


is because there's some kind of financial exchange that


took place.


There's some kind of other exchange that took place.


And it's not essentially a natural link


that we would take into account for things like search.


So from that point of view, it's fine to have sponsored posts


and to have links in sponsored posts


if they're flagged with the rel="sponsored."


That's essentially the right way to do it.


It's just that these don't have any effect on SEO initially.


And again, if people go to your website


because they found this link and then they recommend it


themselves, then essentially that indirect effect


is something that can still be very valuable.


And oftentimes, especially new businesses,


will take kind of an approach to using advertising


to initially drive traffic to their website.


And if they have something really good on their website,


they kind of hope that, well, by driving all of this traffic


to our website, we get some awareness for the cool things


that we have.


And then those people share that further,


and we get some value from that.


Do brand mentions without a link help with SEO rankings?


And again, from my point of view,


I don't think we use those at all for things like page rank


or understanding the link graph of a website.


And just the plain mention is sometimes


kind of tricky to figure out anyway.


But it can be something that makes


people aware of your brand.


And from that point of view, could be something


where indirectly you might have some kind of an effect


from that in that they search for your brand.


And then obviously, if they're searching for your brand,


then hopefully they find you right away.


And then they can go to your website.


And if they like what they see there, then again,


they can go off and recommend that to other people as well.


How are Core Web Vitals calculated in single page apps?


Only on the initial first load or on subsequent page


use where only the body content is updated?


I don't actually know.


Offhand, I believe we changed something in the way


that things are calculated for the Core Web Vitals,


specifically for things like single page apps,


where otherwise you would have a situation that you


have a really long session on a single web page.


And I don't know the details offhand.


But they're definitely on the web.dev website.


My guess is if you ask on Twitter, for example,


then I can also point you at someone


who would be able to help you with the details there.


Let's see.


I don't know if any of you have any questions on your mind.


Nobody's raising hands anymore.


Everyone's already drinking eggnog, which is a good thing.


But I can continue with the submitted ones


or if we can go to more live questions.


All right, Sumit, go for it.


Hi, John, actually I have a React website


of five millions of pages.


And this is client-side rendering.


So we are facing issue for indexing and crawling


on the website.


As of now, only 0.4 million pages


only indexed in Google Search Console.


So my question is that can Google


read client-side rendering properly of a page?


Or do we move to the server-side rendering?


Yeah.


So we can do client-side rendering.


We do render essentially every page that we crawl nowadays.


So from that point of view, it's usually not so much a matter


that we can't do it.


But there are certain situations and setups


that don't work for search.


So in particular, if you use something


like the number sign, the hashtag symbol,


as a way of signaling the different types of content


that you have.


So instead of something like, I don't know, a path,


you have a hash symbol and then the path


after the hash symbol, then that's


the kind of thing that probably would not work for search.


But if you're seeing already hundreds of thousands


of pages indexed, then probably we can index your page as well.


What I would do is, first of all,


maybe look up the JavaScript-based websites


documentation we have on Search Central


in the documentation site.


There are some tips there for what you need to do


and what you should avoid.


And there are also some links to different testing tools


that you can use to test.


And maybe, first of all, look at that


and think about how you set up your website.


Would it work like that?


And then what I would also do is try


to find a sample of the pages that are not being indexed


and run them through the testing tools.


Ideally, find different templates


that are not being indexed.


So if you have things like product detail pages and


category pages, and you're seeing the category pages


are not being indexed so well, then I


would take a sample of the category pages


and really test them with the testing tools to see,


is there something small that I can


do to improve the indexability of my website or not?


And often that means you can find different things


that you can improve for a JavaScript-based website.


It's something where you could also


look at the server-side rendering


and say, well, maybe I can skip all of these extra tests


and just go directly to pure HTML


and send that to Google with server-side rendering.


From my point of view, I would, for the most part,


try to avoid a situation where you're just


setting up a lot of infrastructure just for Google.


And instead, if you're going to move to server-side rendering,


do it in a way that makes sense for your users


as well, because sometimes you will see also


a significant speed improvement by going


to server-side rendering.


And if you can use that speed improvement and say,


well, also Google will have an improvement,


then that seems to me like a really good scenario.


The other thing I think also to keep in mind


is that we almost never index all pages of a website.


And it might be that the number of pages


that you're seeing indexed from your website now


are essentially just the number of pages


that we want to index for your website at the moment.


And we will improve over time as we


see that the website itself merits


being shown more in search.


Then we might expand that and say,


we'll go off and index maybe, I don't know, a million more


or something like that.


But that is something that is almost


independent of the infrastructure


that you use for the website.


And it's really hard to evaluate,


because we don't have any kind of a meter or something


in Search Console that says, oh, we have enough of your website.


But it's something where if you run the different testing


tools and you see everything looks good,


Google can index it, and we're just not indexing that much,


then it might just be a sign that, well, this


is currently the limit that Google wants to do.


And to get more, I need to prove to Google that actually I


have fantastic content that should be indexed more.


And then proving that to Google usually


means proving that to your users.


So that's kind of like a non-technical approach then.


And we have checked from multiple tools.


I have read most of the blogs and find out


if you want to check the rendering of the page,


then you can use a structured data testing tool.


So I have tried that tool as well.


But when I fetched through structured data testing tool,


then I saw page content did not come on the HTML page


when we test the page.


Then the content is missing on the page that time.


So Google indexed the page in the same way


that we have seen on the structured data testing tool.


It's possible, yes.


And if you see things like that, that is definitely something


that you should fix on the site.


Because if Google can't render the full page


with the full content there, then we


won't be able to index it.


And we won't be able to show the website for that content


in the search results.


So that's definitely something I would look into.


And sometimes there are simple things,


like maybe the server speed is just too slow


or the number of requests required is too high.


And you can do things like JavaScript or CSS bundling


and make everything into fewer files.


And then everything is easier to render.


But I would definitely look into it then.


AUDIENCE MEMBER 1 Yeah, thanks.


And website navigation has also helped


to index multiple pages.


Yeah, I mean, the internal linking


is very important for us.


To me, it sounds like if you have a couple of hundred


thousand pages indexed already, then the internal linking


is probably OK.


Maybe it's not perfect.


I mean, no website is perfect.


But it sounds like it's not critically broken.


Actually, developers used the JS to interlink


multiple of the pages.


And they don't use the pagination.


They call the data dynamically on the page.


When people click on a 1, 2, 3 sign of pagination,


but the page is not changed that time.


So I thought this is the issue in terms of crawler.


Because crawler is not crawled the JS linked pages,


they need anchor tag to crawl the page


or to move to the next page.


So we are working on it, trying to fix that issue.


That sounds like something to look into too.


Yeah, I think it's sometimes tricky with pagination.


But it's also something you would probably


notice when you start to look at the patterns of the pages that


are indexed and those that are not indexed.


Because in the patterns, you might see, oh,


everything page 2 or page 3 in a set is not indexed.


Then maybe that step from page 1 to page 2 is causing problems.


Yeah, sure.


Thanks.


Thanks, John.


Sure.


Cool.


All right.


Any other last questions?


Last words.


No.


Wow.


Well, I'll say something.


Not for the sake of saying something.


But I thought it was an interesting thread yesterday.


And I see that Barry's here as well about one thing


being the cause of a problem.


I saw that the thread and basic gist


was that unless there's a big technical issue,


such as no index in your site, generally you


can't pick one thing that's the cause or the catalyst


for improving ranking.


And it sort of made me think about this quite a bit,


actually, because I and many others


have been in this situation.


And one thing, I don't know if it's helpful to others,


but to think of your website as your house, right?


And you're inviting someone into your home,


obviously not during a pandemic.


But you wouldn't just clean up one room


and leave the rest of your house in disarray.


And so if someone came to your kitchen


and the sink ran fast, your page speed, that's nice.


But if the rest of the kitchen is a mess, your UX is a mess.


And then the other stuff I was thinking


was like, well, more analogies like food.


OK, so if you're still in the kitchen,


your food is your content.


And you wouldn't want it bland or bad, like low.


So I really liked that thread because I saw a lot of people


discussing it.


But I was thinking, you have to think of it,


your website really as a home, maybe the bathroom


is plumbing or something, the technical part.


Your attic is your storage.


You can't get to everything right away.


But over time, people are going to wind their way


through your home, through your site,


and the more you clean house, obviously the better


your website will be.


But it was something I've been thinking about quite a lot,


that you just can't really look at one thing.


I think Glenn Gay often talks about also the kitchen sink,


as he calls it.


You have to do everything.


You can't just sweep it under the carpet.


Eventually, you have to get to it.


Yeah, yeah.


I think it's worthwhile to think of websites


as more than just this unnatural thing, where you just


tweak some things or set some screws,


and then suddenly everything runs better.


It's like if you have a race car, or even just a normal car,


and you wanted to drive a little bit faster,


you can do a lot of different things.


You can make sure the tire pressure is OK,


that you have good tires on, that your motor is oiled,


all of these things.


And which one do you do first?


That's really hard to figure out.


If your car ends up using less fuel at some point,


you're like, well, which of these things


cause it to use less fuel?


It's all kind of interconnected.


Yeah.


Well, you've got to make sure those brakes are good too,


right?


At some point, you have to make sure that everything


that the chassis of the car, heaven forbid,


hits a wall, that it's secure, that it's built right.


Yeah, yeah.


And I mean, I certainly went through it, where it was UX,


it was content, it was page speed,


it was ad monetization.


We eventually got to everything.


So I felt the pain when I read that.


Cool.


Yeah, John, I just remember, actually, I


do have a question, if you have a couple minutes.


We were talking a few minutes ago about brand mentions.


And while you said that, yeah, it's not something


that Google directly takes into account,


if most of the people mention your brand


and say that your brand is a scam,


then maybe there's some issue there, no?


But now, let me flip that.


What if, also, a lot of pages that are mentioning you


and saying you're a scam actually


spam pages created by people intending to hurt you?


Then does Google know how to handle that?


Because for backlinks, we can sort of disavow the backlinks.


But you cannot disavow brand mentions.


Yeah, I think that's also one of the reasons why


it's really hard to use anything like that with regards


to rankings.


Kind of understanding the almost subjective context


of the mention is really hard.


Is it a positive mention, a negative mention?


Is it a sarcastic positive mention


or a sarcastic negative mention?


How can you even tell?


And all of that, together with the fact


that there are lots of spammy sites out there,


and sometimes they just spin content,


sometimes they're malicious with regards to the content


that they create, all of that, I think,


makes it really hard to say we can just


use that as the same as a link.


So from that point of view, I feel


it's something where, for the most part,


we don't mention it as something that positively affects


your website or negatively affects your website.


It's just, I think, too confusing


to use as a clear signal.


Yeah, thanks.


John, I just wanted to thank you for everything over the past,


almost two decades.


Thanks.


Have a great holiday.


It's almost been two decades, right,


since you've been in this industry.


I don't know.


Like John, last two now.


Well, thank you.


Happy holidays.


You too.


John, one more question I have related to like DOM.


Like a developer created the DOM and insert some SEO tag in it.


So I just want to know like Google crawler,


crawl the content from the DOM for AngularJS kind of websites.


Because if we not insert content in the DOM,


then Google cannot read that content of the page.


Exactly.


Yeah.


So we render the page, and we take the rendered DOM,


and we use that for indexing.


OK.


So I have checked the DOM, and I have


found like content is missing in the DOM for category


and subcategory pages.


So that is a big issue on my website.


Sounds like something to fix.


Yes, yes.


Actually, developer told me like if you


add content in the DOM for millions of pages,


then it will affect the server.


Like server will go down.


So they have to like they told me like that.


Find the balance.


Or get a bigger server sometimes.


Yes.


Cool.


Well, let's maybe we'll take a break here.


Thank you all for joining in.


I have another one lined up for next week,


which I don't know if anyone is actually


interested in thinking about SEO next week.


But if you are, you're welcome to join in.


If you're watching this on YouTube,


you're also welcome to just submit questions and maybe


see the recording to see what happens.


So anyway, thanks again for all the questions


and for coming by.


And hopefully, I'll see some of you again next year.


Or next week.


Bye, everyone.


Thanks, John.


Bye.


Thanks, everyone.


Mobile-first Indexing: Is Google Planning Opt-in/Opt-out? #AskGoogleWebmasters

Today's question is from Dan. He's asking, is there any plan to eventually let webmasters


tell Google they want to be included in the mobile-first index or at least be considered


for inclusion? Some nonresponsive sites might believe their mobile site is more SEO-friendly


than their desktop site. Mobile-first indexing is a process of indexing


websites using a smartphone-like user agent. We started working on this a few years ago


because we noticed most users are now using smartphones.


Mobile-first indexing is separate from mobile friendliness. Even sites that don't have a


mobile version at all can be indexed fine with a mobile Google bot.


Our goal is to use the mobile-first indexing for all websites in our search results. We're


taking our time to make sure that users will continue to be able to find what they're looking


for and to help site owners to adapt should there be something they need to do.


We found that for the most part, sites can be indexed fine with a mobile device. At the


moment, we're algorithmically testing to see when sites are ready and switching them over


once that's the case. To determine when a site is ready, we compare


the mobile and the desktop versions to make sure that we're still able to find all of


the content, including structured data and images.


With the longer-term goal of moving all sites over to mobile-first indexing, we don't plan


to provide an opt-in or opt-out for this kind of indexing.


If you're curious about mobile-first indexing, we have a bunch of documentation and blog


posts that I'll link in the description below. See you in the next episode of Ask Google


How to remove a page #AskGooglebot

We recently had a question about removing old pages


How to remove a page #AskGooglebot

from Google.


How to remove a page #AskGooglebot

So first of all, if you remove a page from a website,


How to remove a page #AskGooglebot

it will automatically drop out of Google systems over time,


How to remove a page #AskGooglebot

too.


How to remove a page #AskGooglebot

There's nothing additional that needs


How to remove a page #AskGooglebot

to be done in these cases.


How to remove a page #AskGooglebot

If you're curious about the details,


How to remove a page #AskGooglebot

here's a little bit more, though.


How to remove a page #AskGooglebot

First, make sure that the server returns the proper HTTP result


How to remove a page #AskGooglebot

code for a removed page.


How to remove a page #AskGooglebot

Most websites do this automatically.


How to remove a page #AskGooglebot

Second, remove the references to the page on your website.


How to remove a page #AskGooglebot

This includes links and sitemap files.


How to remove a page #AskGooglebot

Finally, keep in mind that it can take a few weeks for Google


How to remove a page #AskGooglebot

systems to update.


How to remove a page #AskGooglebot

If you need to have it hidden faster,


How to remove a page #AskGooglebot

you can use the urgent URL removal tools linked below.


How to remove a page #AskGooglebot

In most cases, you won't need to use those tools,


How to remove a page #AskGooglebot

but it's good to know about them.


How to remove a page #AskGooglebot

This covers most of the details when


How to remove a page #AskGooglebot

it comes to removing old pages from your website.


How to remove a page #AskGooglebot

For more specific help, I'd recommend


How to remove a page #AskGooglebot

checking out our help forums.


How to remove a page #AskGooglebot

And that's it for today.


How to remove a page #AskGooglebot

Make sure to subscribe if you'd like


How to remove a page #AskGooglebot

to stay in the loop for our future episodes.


How to remove a page #AskGooglebot


How to hide a website in search #AskGooglebot

How to hide a website in search


Amman asks whether it's possible to hide a website in search


while still letting users access it.


In short, yes, you can.


There are three ways that you can do that.


Use a password, block crawling, or block indexing.


Let's take a quick look at them.


Locking a website down with a password is often the best approach


if you want to keep your site private.


Neither search engines nor random web users


would be able to see your content, so it's quite secure.


Blocking crawling is another approach.


This is done with the robots.txt file.


Random users of the web would still be able to use your website


while well-behaved search engines would know not to access it.


This isn't an ideal approach,


since search engines might still index the address of the website


without accessing the content.


It's rare, but it can happen.


Blocking indexing is the other option.


For this, you add a noindex robots.meta tag to your pages.


This tells search engines not to index that page after they crawl it.


Users don't see the meta tag and can still access the page normally.


Overall, for private content,


our recommendation is to use password protection.


It's easy to check that it's working,


and it prevents anyone from accessing your content.


Blocking crawling or indexing are good options


when the content isn't private or if they're just parts of a website


which you'd like to prevent from appearing in search.


I hope this helps answer your question, Aman.


Stay tuned for the next episode of Ask Google Live.


How do removals work in Google Search, really?

Hello, and welcome to another episode of Search Off the Record, a podcast coming to you from


the Google Search team discussing all things search and maybe having some fun along the


way. My name is John, and I'm joined today by Lizzie and Gary from the Search Relations


team of which I'm also a part of. So how are you coping with the Swiss summer so far Lizzie?


I am not doing well. I don't think that I know how to cool the apartment appropriately.


It just seems to get very hot even though I am closing the blinds. And I don't know


if there's a trick to it or I just picked the wrong apartment or what is happening.


I don't know if you also have the same problem.


It's the same here. I don't know. In Switzerland, nobody has air conditioners or almost nobody.


So you have to cope with it. I think this might be one of the reasons why people in


Europe just go on vacation for several months in the summer.


It's not hot in Europe.


Maybe you have to go to someplace where there are glaciers, I guess, like live in an ice


cage.


Gary is making a face.


How are you doing, Gary?


I actually love it. Hence the face. Also in my apartment is like 23 degrees Celsius. So


it's perfectly fine.


I thought you were saying it was humid.


Outside it's humid.


Now it's not humid anymore. I mean, inside it's not humid anymore. So that's great. And


I love the heat.


So you are having a great time then in the heatwave.


I'm having a great time. It's one of the reasons why I go to Singapore to be in a humid, extremely


hot environment. Well, we used to go to Singapore. And now apparently the Singapore weather just


came here. And that's very, very nice. I appreciate it.


So ideal Gary climate.


Correct. I just moved to Singapore.


I usually enjoy the weather in Singapore because then I feel a lot more happy going back to


Switzerland where it's like, oh, it's only like so many degrees here. This is nothing.


I can tolerate it. But they have lots of AC there. So could you just avoid the heat if


you wanted to?


Well, you do have to go outside sometimes. Like for a commute, for example, you have


to go outside.


So talking about blocking. Last time we talked about blocking sites in search kind of from


an owner point of view. And now it's our turn to ask Gary how that actually works in reality


from a search engine point of view.


What?


Yeah.


Tell us how it really works, Gary. So I know we have like the crawling, indexing, and serving


side of things like those three big buckets. Are they buckets? I don't know.


Bins.


Bins. Bins. Systems. Like the engineers definitely want to not have them be called buckets, I


guess, because that sounds kind of awkward.


I think bucket is a good name for them.


I like it. Yeah, it's fun to say bucket, you know.


I work on the crawling bucket.


Yeah, the bucket team bucket of crawling. It's a bucket of fun.


Okay.


No. Too far.


So looking at those three bucket systems, how does the blocking side of things kind


of flow in there? Is it just like a flag that someone sets and then everything gets processed


normally or how does that actually work?


As you said, we have those three buckets and each of the things that we talked about in


the last episode map to one of those buckets. I think we talked about robot.cxt and then


we talked about some meta tags, specifically robots meta tags. And then we talked about


maybe about the removals tool a little. And each of these things actually map to something


within search, like one of the bigger systems that you mentioned, crawling, indexing, serving.


And I think that's why removals in general are so chaotic when we talk about them because


we have all these systems that have their own way to control what they ingest and how


they process things.


Right. So certain things, certain methods can apply to various buckets or stages of


search. So you sort of need to know how the systems work in order to know which method


to pick. Would that be a right way to think about it?


I think so. I don't think it's necessary, but if you are dealing with removals often,


then you probably want to understand those systems because if you are like, you can combine


certain things in ways that will not work. For example, like you take the robots.txt


disallow rule as a way to block something from search or remove something from search.


And then let's say that that page is very, very popular on the internet. So we still


want to index the URL of that page. And then your expectations are not met because you


are still going to see that URL in search results, not the content, like the URL will


be without a snippet. But the title link will still be there. And that's because sometimes


that can happen. It's very rare compared to the rest of indexing. It's like the robot.urls


in our index is tiny, tiny, tiny minority, but it can still happen nonetheless. And then


if you want to remove that URL, then you would think that you can just put up a meta no index,


meta robots no index in the HTML or in the HTTP header. But then we would never see that


if the robots.txt disallow is there.


I see. So knowing when these things happen can help you troubleshoot or you're now aware


that like, hey, Google may not see this thing that I'm putting up on my page because I have


this first thing that's blocking earlier on in the process.


So robots.txt is basically all about crawling now that I think about it. It has a sitemap


file and where you can and can't crawl. So it's basically for that.


That's also one thing that we worked quite hard when we were open sourcing our robots.txt


parser and matcher. We've removed certain things from the parser and we also blocked


about it that didn't make sense from crawling perspective. So for example, it used to be


that, and I know that externally this is still a pain point, but no index was there. Like


you could use a no index colon and then path. But we really needed, like search engines


really needed to have robots.txt just for crawling. And then you can control with other


things indexing and serving, like generating search results, but robots.txt should just


be about crawling and nothing else.


So cool. Now I never thought about it that way, but it makes total sense. Also the sitemap


file, kind of mentioning that there then kind of seems like the right fit. So maybe it should


be renamed into crawling.txt.


No, no, no, no, no renaming. No. You know that for the past, I think three or four years


we've been working with Lizzie and Henner, one of the engineers who was working on the


Google bot team on standardizing the robots.txt protocol together with the original author


Martin. And it feels like it's kind of a pain.


Any change suggested, I think the more changes were like, oh no.


Yeah, exactly. Especially now that we are in the finish line of that process. And every


single time I see a comment coming in that, oh, you need to update this. It's like, no,


just no. What? No.


But then if you don't get comments, you're like suspicious. Like, did you look at the


thing? Because there have been so many comments before. If there's none, it's like, something's


not right here potentially.


I think that's just you.


That's just me? You wouldn't, if I just, like the change list that you sent me, if I just


LGTM'd it, no comments, you would be like, yeah, good. Or would you be suspicious?


Gary would just publish it. And it's like, okay, cool.


I mean, I'm sending it to you for comments. I know that you are going to leave comments.


Like I could have sent it to Edu, one of the engineers who we worked with on the parser,


and then he would have approved it. But then I would have gotten no improvement comments.


We would have gotten improvement comments by these other people, the committee. What's


the term for them? The group that's reviewing the stuff, IETF


Directorates.


Directorates, yes. They pay close attention to everything. And I try to anticipate and


still we get comments.


I think it's a good time for me to bring up the suggestion to rename it to crawling.txt


then right?


Oh no. In the open. No, we're closed. We're past that stage, I think.


Yeah. So at this point we only get comments from the directorates, like from the security


directorate or the ARIA review team or whatever. And there are no more community contributions,


which is great because fewer comments, I guess. But yeah, we've been dealing with comments


for a long time and I would rather not introduce a change into the protocol.


Would that mean that people would have to change all of their files? Like old named


robots text files, everything would break. Like it would be chaos.


Okay. No chaos. No chaos. We like chaos.


We thrive in chaos. We love chaos. No, but so technically it wouldn't change anything


because it's down to the implementers to support the things that we mentioned in the specification.


So for example, if they still want to follow the old naming like robots.txt, then they


could just do that. Like it's, it's not like it's set in stone. And if you look at the


HTTP standard, for example, you can find things that browsers in general don't support, even


though it's in the, in the specification. So it's, it's not like you must absolutely


do this. It's like, here are the considerations that we thought of. We think that you must


follow this and this. You should follow this and this, but ultimately it's up to you what


you do with it. We already talked a lot about robots.txt. So I'm, I would rather not make


this an episode about robots.txt again. Okay. So let's return to the crawling bucket.


What other things do we notice in the crawling stage that's related to a removal? Like how,


what other things could we notice at that point right away that, Hey, somebody has requested


that this thing be removed. So there's robots.txt. What other things would be noticed at the


crawling point?


I guess the URL removals tool used to be a thing there where if you flagged a URL parameter


as something that we should skip, then that would also be dropped. But that tool doesn't


exist anymore. So probably nothing. Or can you think of anything, Gary?


I'm thinking about status codes. Oh, cause that's the other thing is if there, if like


the robots.txt file is like a 404 or something, or like we can't access it, then we don't


know what to do. So that page itself, it could be.


Well, not 404, because that means that it's missing like 503 or 505 or whatever. But not


even just that, like for example, if you return a URL with a 404 status code, then that will


be removed from search. It's not a removal tool. Like we are not saying that this is


a tool for removing URLs. I don't think we do. But technically it works and it's just


working as intended. Basically you instruct the client with something. Well, you don't


instruct you inform the client.


Well we do say like one way to remove URL would be to like delete it. So then you would


serve the status code. So then that would be working as intended as like, Hey, it's


gotten like, cool, good job. It's out. So we would have to crawl it and then decide


not to index it.


I mean if it's 404 or 410 or 403 maybe, then we just wouldn't even pass it on to indexing.


We would pass on a signal to indexing that like this URL doesn't exist anymore. And then


indexing will decide that then we are going to drop it.


Okay.


So what else happens in the indexing bucket?


Not much.


This is when we would find like all those other tags, like no index, meta tag, nofollow,


like things that are on the page. Like don't go here, don't go over here, don't stop looking


at me.


So I think in an early episode of Search of the Record, we talked about indexing a little


bit and one of the things that we mentioned was the content parser and rendering. And


during those processes we do extract the meta tags that are relevant to indexing. So for


example, rel canonical is extracted, meta robots is extracted and so on. And if we see


something like meta robots no index, and if we see meta robots no index, then that will


mean that we want to drop that URL from the index.


What would happen if you've got other languages, like you've got the no index on one language,


would we then do something about the other languages? Like if I guess if you've set it


as the rel canonical then...


So it depends on how the setup is because it can happen that you like for example, if


we have some sort of duplication across languages and Google detects that as a duplication,


then it might mean that one no index can affect the whole cluster, but it's hard to trigger


it, but nonetheless you can.


Okay, so it's something that could happen, but I'm wondering like is it safe to rely


on that or you should put it on every language page just to be sure like if you wanted it


for sure not indexed or removed, like you should be thinking about all the things or


can you just add it to the canonical page and then we will know to get rid of it like


from just an efficiency perspective I guess like I don't want this page so I therefore


also wouldn't want the other pages that I've linked up.


You probably want to add it to every page that you want to remove. I think our setup


is more special because we do use adroflang so we are instructing, well not instructing,


but hinting to indexing that these are clusters of pages and then they might influence each


other.


Might, I mean oh no it sounds like maybe and if you wanted to be 100% sure then you would


put it on all.


That's why I'm saying that we are more special like pages with adroflang clusters are more


special and you have to keep in mind that they might affect each other in the cluster.


So for pages that are not in a cluster not in an adroflang cluster for example there


you can just noindex the pages that you want to get removed and don't think about anything


else just remove the page or noindex the pages that you want removed. With adroflang clusters


it gets more complicated.


Also recurring topic that adroflang is complicated. Yes it is.


So how does rendering fit into this? Like if you add the noindex robots meta tag with


rendering in JavaScript would that still work or is that a bad thing?


We are trying to figure out what happens with rendering but I think in theory it like at


least in most of the cases it should work but we definitely need confirmation from Martin


about what's actually happening.


And the reason you bring this up is because it's are we going to see it in time? Like


if it's injected with JavaScript would we know to not show it because we haven't seen


it?


Of course kind of like the timing difference there might be like a window in between or


if we don't render a page for whatever reason once then what happens?


Right like we didn't know that hey this is actually noindex because we didn't see it


when Googlebot is rendering it.


Okay so basically be as consistent as possible just like if you wanted noindex just noindex


everywhere.


Yes I'm trying to think of like corner cases but I can't come up with anything useful.


How could you put the meta tag on things like PDFs?


Well you can't.


Oh that sucks. What can you do there? Like if you have a PDF and you're like I just don't


want this indexed.


Or you want it removed basically like you're it's being served in search results and you


want it removed how would you go about I guess you would just delete it?


We do support x-robots-tag HTTP header and then for binary files you can just add that


with a noindex thingy and then that will work like a metanoindex.


Okay so this method the header method is this fall which bucket does this fall into would


we notice it at the crawling or the indexing stage?


Technically we notice it in crawling and then we forward it to indexing and then indexing


will drop it so that's the signal that we sent with like the 404.


So basically we create a wireframe of whatever we collected about the URL like the timings


and the different meta information that we could collect including the headers and then


based on that indexing can decide what to do with the page.


So basically in crawling we just notice it but we don't actually do anything it will


be indexing that will has to do something about it and same goes for status codes as


well.


Okay and when you say do something about it what does that mean?


Keep it or drop it.


So it's basically a blockchain from crawling to indexing.


No don't quote me.


No bet John.


You can't just join up any words together.


I'm German it's like in German you can just take random words and put them together and


make a new word.


What if they're existing words?


This is a term already.


Then you created a pun.


Okay so moving on to a third bucket we can call it Web3.


What happens then?


It's like he's not hearing us on purpose selective listening.


It's a pleasure working with you John.


I have actually a question regarding the crawling bucket.


So in the docs we have the unavailable after I don't know if we talked about this last


episode but there's a part in there that says if you add that then crawling will slow down


over time which is like interesting I guess.


Is it though?


Yeah I mean so you put it there like you want it removed but like we're just slowing down


the crawling we're not stopping completely or like we will still check back in on it.


We don't see it as a hey remove it.


It's sort of a soft exit like a opposite of a soft launch.


So I don't actually know how exactly it works or how it's implemented but my understanding


was always that it's a soft exit because we still need to ensure that you are not bullshitting.


Wait but I put it there like or whoever it is who put that tag like I guess.


No not the tag.


Like you don't believe them that the content is actually unavailable after?


Like with the last mod tag in sitemaps.


Oh I see.


That's also something that we might use it if we can detect that it's consistently accurate.


My understanding is the same goes for unavailable after.


And is this something that people would use to sort of like schedule a removal or something


like they eventually want it to be removed from search results so they put it there to


sort of like preemptively like hey this thing like won't be relevant but like it could still


hang out in search just crawl me less and if it falls out eventually like that's fine


with me?


I think so.


John is aggressively nodding there.


Yeah but he's also sort of making a face I'm not sure like do you concur or?


Yeah I mean my understanding was we understand from the page that they want to have it go


away at some point and then like either we can just blindly trust it and start hiding


that page in search or we double check it at some point and then it's more like a no


index and then whatever happens there.


Like maybe a decade ago some auction site like big auction site was using this and they


were using it accurately and consistently and you could actually see the auction pages


disappearing from search when the auction expired.


It might have taken like a day or two because we actually needed to crawl the page again


but it actually worked if nothing else for scheduling the crawl to that date.


So going back to the last bucket I won't mention its name what kind of blocking could you do


there?


So it's basically indexed in Google systems.


I mean you can do what Lizzie said basically removing all together but you can also control


the snippets that we show.


Would this be also like removing the cache like the cache gets removed at the serving


stage or no like we still have it but then?


That's kind of weird.


I think it's good that we have the cache page because when the pandemic hit then we could


serve some pages from there when the site went down due to heavy load but I usually


don't even think about the cache page.


But yeah you can remove from the cache page for example whatever we have there.


You are not removing from search results anything you are just removing the cache page.


So you are controlling the cached page not the content that we might have indexed.


So that's the no archive meta tag right?


There's also the cache removal tool.


Right.


Where we used to have one.


And that would be for what purpose like you would use what do you mean we used to have


it we don't have it anymore?


I don't know I haven't taken a look for a while so I'm kind of like do we still have


that maybe like whatever you do.


Can you yeah I'm wondering can you use the removals tool to remove the cache page or


like what's in the cache or would that stomp on the full page too like oh whoops we removed


the whole thing like you just wanted us to remove the cache and there's an updated version


but if you put the cache page in there would we?


Let me assume that this tool still exists.


Okay.


Because probably it does but otherwise someone would have told us hopefully.


Or we would get complaints about it.


So my understanding is there are two ways that you can remove a page with these tools.


One is you remove it completely if you're like I don't want this page at all to be visible


in search.


And the other is it removes the snippet that is shown and the cache page which is like


I changed something on this page and I want Google to like refresh it.


And definitely don't show the old one for the time being kind of thing.


Like you I don't know you put a wrong phone number on a page and people are calling your


neighbor instead of you you could say like fix this as quickly as possible or at least


hide it from people seeing it.


I guess what most people think like recrawl the URL like not remove the cache is sort


of a like a way to think about the same thing but maybe not typical.


I think the removing the snippet and the cache page is more for people who are not the site


owner.


So for them they can't go to search console and say like hey recrawl this random person's


web page for me please.


But they can go and say like actually the information on this page has changed and I


need you to reflect that in search as quickly as possible.


And it's quite an important aspect.


I think that like if someone wrote something about you and it was not accurate then you


definitely want it to be gone from search once they updated the article or whatever.


And is this happening in the rendering bucket?


No this we are already in serving bucket.


We're already in serving.


Oh wait we left you behind.


Oh whoops.


And then we have the normal removal tool which you can actually use to remove a whole result


from search results.


And from our perspective is that like going through all the buckets again or does it like


jumpstart you?


So that's removing from search results like the phrasing was important there because it


is just hiding from search results in the first step.


And if you're a site owner and you want something gone from search completely like from the


index and from search results then you also have to take one of the previous actions like


for example add a noindex or well just add a noindex that's the best one or remove completely


like 404 return 404 for 10.


But otherwise the removal tool that will just remove the search result itself.


Ah and is that why it's so fast or it's the faster method?


Yes that's correct.


Yes.


And that's also why we have it because in some cases at least we cannot refresh the


index that fast.


It may sound really weird but sometimes it may take months to remove a result from our


actual index.


Then does it just get removed from the data center that the serving is happening at or


does it drop out when you search?


So it drops out when you search.


So basically that's like if you if the removal tool was used on a result and the result is


still in the index while the URL is still in the index then we would normally do the


the retrieval that we do for any other result but then we would not present that result


to the user to any user.


But it doesn't necessarily flag the page to not be crawled again so we could go visit


it and.


Well you want to crawl it again.


Right but say the person used the removals tool that doesn't prevent it from like from


us finding it again they would have to do some other thing because they're still going


to attempt it.


That's correct that's why you want to use noindex or 404 or whatever and then that would


permanently remove well not permanently because if it comes back or the noindex is dropped


then it would come back but it would remove page that has a noindex or returns with a


404 from our search index.


So that sounds like a lot of work every time to do like when someone searches because like


you search for something and it comes back and like oh this search was done in 0.005


seconds I guess it has to look through the whole removals file every time that's like


pretty crazy wow.


Well but you load it in memory and then if you load it in memory then like you don't


have seek times you don't have anything so it's not not that bad.


Is that why regular expressions can't be done there?


No, stop it.


Regex.


That's how I say it.


I'm surrounded by regexers.


So basically Google systems are just like face palming whenever I try to submit something


like that.


Well I'm not sure if they are face palming I think they are just assuming that that's


part of the URL.


Very interesting URL structure you have going on.


Yeah very weird URL.


I think like theoretically we could support some form of regex to some extent but I'm not


sure that would do any good.


People would wipe out their sites in very or new kinds of imaginative ways.


I've heard regular expressions are hard I don't know what they mean with that.


Oh so next time I have a regex problem I will just go to you because you're a regex expert?


Yes.


Next time on Search Off The Record we'll be taking a look at accessibility which is kind


of the opposite of blocking specifically for your site.


We've been having fun with these podcast episodes.


I hope you the listener have been finding them both entertaining and insightful as well.


Feel free to drop me a note on Twitter or chat with us at one of the next events that


we go to if you have any thoughts.


And of course don't forget to like and subscribe.


Thank you and goodbye.


Bye bye.


English Google SEO office-hours from November 19, 2021

All right.


Welcome, everyone, to today's Google Search Central SEO


Office Hours Hangout.


My name is John Mueller.


I'm a search advocate here at Google in Switzerland.


And part of what we do are these Office Hours sessions,


where people can join in and ask their questions


around their website and web search.


And I wonder what we'll be talking about today.


So we have a bunch of questions that were submitted already.


But I see some of you already have your hands raised.


So maybe we'll just start off with you.


Let's see.


Akash, I think you're first.


Hi, John.


Hi.


So my question is, the past few days, a few of my web pages


have been considered they are failing for mobile usability


tips due to text size is too small.


So could this be possible if you fail for one


of the mobile usability errors, or your pages won't be indexed?


Is there something related to that?


They should definitely still be indexed.


Even if they're not completely mobile-friendly,


we would still index them.


The mobile-friendliness criteria is something


that we use as a small factor within the mobile search


results.


But we definitely still index those pages.


And sometimes, this kind of issue


can come up temporarily, where maybe we


can't crawl one of the CSS files for a brief time.


And then we don't see the full layout.


If you test your pages manually in the testing tool


and they look OK, then I would just let it be.


It'll kind of pop back into normal again.


Is it different than how mobile usability tool work


and a browser when they start entering the page?


Because the problem is just that that tool shows


mobile usability test fail due to the small text.


But on the mobile or the browser, the Chrome browser,


I see the page is properly loaded.


There are no such issues.


Yes, so in particular, we have to follow the robots.txt.


And in the robots.txt file, it can


happen that you block something from being crawled, which


could be something like the CSS file, which kind of tells us


how the page should look like.


So in a browser, the browser doesn't


look at the robots.txt.


But when we crawl the page, we do look at the robots.txt.


So that could definitely be something


where it's different when we look at it


than when you look at it.


Thank you so much for coming.


Sure.


All right.


Amir, SEO.


Hi.


How are you?


I have actually three short questions.


I work in news industry.


So we are publishing like 20 articles each day.


So when I was optimizing the SEO,


I told them we need to fix the title to 65 letter only.


And they were very upset about it,


because news should be like a long title and a good title.


So after I was struggling with them, I searched the web.


And I didn't get a right good answer.


Is that a really good factor of ranking,


that the title long should be like something


with a limitation?


No.


We don't have any recommendation for the length of a title.


So I think kind of picking a number from your side


and saying, on mobile, this much room is available.


So as an editorial guideline, we'll


say 65 or whatever you want to choose.


That's perfectly fine.


But from Google, from the search quality, the ranking side,


we don't have any guideline that says it should be this long


or not.


Even if the whole title didn't show up in the search page?


That sounds like something different.


How do you mean?


I mean, when I search on Google on something


and our website is shown, if the title is longer than 65 letter,


it will not show all in the search page.


So is that like a ranking factor?


That's my question.


No.


The length doesn't matter.


If we show something shorter, if we show something slightly


different, that's just kind of how we display it


in the search results.


It doesn't mean the ranking changes.


Good.


Good point.


Another thing.


Is the length of the URL important?


Because we used to put all the title in the URL,


and now we removed it.


Is it?


No different at all?


No difference.


I wouldn't.


My team will hate me for after that.


No.


No, no.


I mean, it's good to ask these questions.


I think it's good practice to have


some of the words in the URL.


So it's more like a readable URL.


But it's not a requirement from an SEO point of view.


So if it's only the ID to the article,


it's good for Google.


That's OK, too.


Yeah.


I mean, I think it's better to have some words in there,


but it's essentially fine.


For the most part, that's something that just users see.


And when they copy and paste the URL, they might see,


oh, I know what this article is about based on what I see here.


Whereas if they just see the number,


then they might pick the wrong one


or might not be sure about that.


But that's a user question almost, not an SEO question.


The last thing is about the coverage in the Google Console.


There is something called not indexed and requested


like in a gray something.


We have like more than 1 million page on that.


When I clicked it, it's redirect pages.


So I fixed the redirect issues in the website.


But is there any like validate the fix,


like the coverage error in this tab?


Or what time should I wait?


I fixed it like over the month.


What did you fix?


The redirection.


It was like redirecting with the wrong thing.


If you go to the page, it will redirect you to another page.


OK.


I mean, that's something that just settles down automatically.


So it's not even something where I


would say you would need to submit like a validate


fix or anything special.


As we recrawl those pages, we'll see the redirect works


and goes to a different place.


And we can just follow that.


So that's perfectly fine.


Thank you very much.


Cool.


Baruch, over to you.


Yes.


I just wanted to know the new design for PageSpeed Insights.


I wish it could be reverted back because I


have to keep on scrolling down.


See, you're laughing because you knew


this was going to happen.


But I mean, my main concern is a lot of sites,


I remember when HTTPS came, when that was introduced.


And there's a bunch of other algorithms


inside everything that makes the algorithm what it is today.


So this is not something, again, we


can't just continue to focus on making sure


that this site is going to be 98 out of 100 and so on.


But in February, when the ranking, I guess,


when PageSpeed Insights will count, what do we do?


Because at this point in time, I mean,


there's certain sites that I'm having issues with, LCP issues


and CLS.


And a lot of sites right now are, some sites, I guess,


are on the very popular CDN platform.


I'm not going to name them.


But they told me, look, it's servers within your area that


are creating a specific score.


And you're seeing, sometimes I would see the LCP and CLS


at a perfect score, which would be almost 99 out of 100.


And then it goes back into another score,


which has nothing to do with the site.


So I'm kind of, I mean, I'm getting worried


because Google sometimes crawls the site.


And then I get the error in Search Console saying,


hey, you have CLS issues and LCP.


And then I go and scan it.


And it's fine.


And it's like 99 out of 100.


And then some pages on the blog are 100 out of 100.


So then I'm like, what is going on here?


And then the new design that just got introduced.


And then I'm seeing some areas where it has no data found.


And I'm like, OK, I guess you guys need to crawl it.


And then data will come.


John, this is just, I don't know.


JOHN MUELLER-ROSENZWEIGER-MARTIN


I think there's one big aspect that plays a role there, which


is important, in that on the one hand,


we have these lab testing tools like PageSpeed Insights


and Lighthouse and things like that,


where essentially you test it either from your browser


with your connection or with some server.


Web page test, I think, does similar things.


Some of the various other tools do that as well.


That's kind of the lab testing that you can do.


And that's more something that is


useful for incremental improvements.


So if you need to fix something, you


need to see if it worked or not, you can just test it again.


We don't use the lab testing for search.


Instead, what we use is the field data,


which is basically what users actually saw.


And that's the data that you see in Search Console.


So essentially, what you're probably seeing there


is users saw that something was slow.


On an aggregate level, we look at that.


It's not done on a per-URL level.


And we give you that information.


Some users saw this kind of an issue in Search Console.


And then you can essentially use the individual lab testing


tools to see where could this be coming from.


Is this something that I can fix?


Is it something that was, I don't know,


some quirk that happened in the network overall that you can't


really help with?


That kind of helps you to narrow things down.


AUDIENCE 1 The thing is in GTMetrix,


I know webpagetest.org, it's a favorite of yours.


Because in a lot of Hangouts, you do have that shown.


It's perfect there.


So when I rescan it, of course, you got to wait 25.


There's 25 people ahead of you, all that stuff.


And it's not easy to rescan as page speed insights.


But when I do that, it's fine.


It's perfectly fine.


So what should I do then?


Who do I trust?


Well, essentially, what is happening in a case like that


is with the lab testing tools, they're


saying things are OK from a theoretical point of view.


But in practice, people see something different.


So all of the lab testing tools, they


make assumptions on what users might see.


So if you're running in a data center somewhere,


you can't just act like, oh, my, I don't know,


gigabit connection through the internet


is what every user has.


But rather, they slow things down artificially.


And they try to act like a normal user.


But of course, they can only go so far and make assumptions.


And what your actual users see might be different.


It might be that your actual users are


using something faster than the assumption


or using something slower.


And essentially, the Search Console part of that,


the search ranking side, is based


on what users actually see and not


based on some artificial number that some data centers.


So as 5G develops and gets better and better,


this, of course, will change.


And then it's going to be easier for everyone, right?


So this will be a thing of the past.


Well, I mean, speed is more than just the network connection.


I don't know.


No, it's just that I've gone and I've done my research out


there.


And if you just go and just stand on a mall,


I don't see anyone that has a 3G network anymore.


And yeah.


I mean, what you can also do, what some sites do,


is try to instrument the pages themselves.


So basically, you add JavaScript to the individual pages


on your site or the pages that are being flagged as being slow.


And you collect the actual data that users saw in analytics.


And that gives you a little bit of a faster feedback loop.


Where you can try things out and say, OK,


what happens if I make this image even smaller?


Or what happens if I remove, I don't know, AdSense


or analytics or whatever?


Then you can see fairly quickly, OK,


it was like the number is still 99 for me in the lab test.


But users actually saw it faster.


So maybe I should go in that direction.


OK, I see.


All right, so yeah.


It's hard.


Hard and it's fun at the same time.


As you're getting the hang of it,


and then, yeah, with, yeah.


So it's not something we should worry about.


I mean, I mean, I would dig into it


to try to figure out what it is.


But the thing to also keep in mind


is the core web vitals, the page experience,


is a ranking factor.


But it's not the biggest ranking factor.


So if you have critical things that you need to do,


if it's a matter of kind of restructuring your website


first, then I would kind of do the critical things first.


And then as you find time, think about what


you can do to improve speed.


OK, perfect.


Thank you so much.


Sure.


Adrian.


Hello.


Just check.


Everybody can hear me now?


Yes.


Good.


OK.


A couple of things.


First one up, I did write a question in there


with regards to concern with doorway pages.


This is kind of linked in with Google Ads.


I know this is search and all that kind of stuff.


But I came across an article that basically said,


explained a web person's experience, where


they created a bunch of landing pages and stuff,


exactly what I just did.


They were being flagged as doorway pages.


And not only were they delisted, their Google Ads account


was actually shut down.


I can't have either of those done.


I can't afford that.


I'm a small, local service-based business.


Don't have thousands of pages.


I've got like seven keyword-targeted landing pages.


In the Google discussion forums, somebody suggested,


well, just no index those landing pages, which I did.


But I'm also thinking is those seven pages,


they are targeted for specific keywords.


They could be very beneficial for organic search.


I don't know.


What's the risk level of being labeled as, oh, this is doorway?


So I think with seven pages, you don't have any problems.


That's kind of at that level, even


if someone from the web spam team were to manually look


at that, they would see, oh, it's seven pages.


It's not like thousands of pages.


It would be different if you were, for example,


a nationally active company.


And for every city in the country,


you had a separate page.


Then that's something where we say, well, this


is way past acceptable.


It's something that is really problematic, where


we need to take some action to preserve the quality


of the search results.


But when you're talking about, I don't know,


a handful of pages, like the seven that you mentioned,


I don't think that would be a problem.


Just purely from the web spam point of view,


even if someone were to manually look at that,


they'd be like, it's seven pages.


It's not going to change the search results completely.


Even though the content is really duplicated,


would I do any kind of canonical to the home page


for those landing pages?


I mean, I don't think it's ideal.


So that's kind of the other aspect there.


It's probably not an ideal situation


when you have pages that are very similar like that.


But again, it's not something from a web spam point of view


they would take action on.


So they would not, I mean, I can't promise this,


because I don't know what your site is.


And I can't tell them, don't do anything with this site.


But just purely what I know from them,


they would not take action on this.


So your search listing and probably also the ad site,


I don't know the details of the ad site,


but probably also that part would be perfectly fine


with seven pages.


I think overall, it's not optimal.


So I would think about ways that you


can make those pages actually useful


for those individual niches that you're targeting.


And really go past your-


AUDIENCE 1 I want to have one niche.


That's the problem.


That's the thing.


Yeah, I mean, just going past, I don't know,


swapping out the city name, for example, in the title


or something like that, where you have, I don't know,


maybe directions from that city to your place of business


or some additional information that gives people a little bit


more context there.


And that-


AUDIENCE 1 Well, they're not city keywords.


It's basically, I'm a photographer.


I shoot headshots specifically.


So my keywords are like Toronto headshot, headshots Toronto,


professional headshots, stuff like that.


All the content is pretty much the same.


I shoot headshots.


I don't know how many other different ways


I can describe what it is I do.


So that's why the content is very identical.


OK.


In that case, I would pick one page and make it canonical.


Because essentially, what you have there


is just variations of the same keywords.


Like when you're saying like photography Toronto, Toronto


photography, for example, those are just


variations of the same keyword.


And for the most part, we figured that out.


And even within the natural text on a page,


you can mention both of those.


It's not going to blow the page up


if you mentioned those seven variations.


Yeah.


I went down this route because, honestly, I've


been fighting with Google Ads.


Is there something like this for Google Ads


where we can really ask people that no?


I have no idea from the ad side.


Right.


OK.


But clearly from the search side, from my point of view,


you could combine all of that into one page.


The other thing that I think is also worth maybe looking into


is you can block pages from search and keep them for ads.


Kind of like that noindex that was mentioned in the forum.


Well, if you keep those pages and you're saying,


well, I don't want them shown in search,


you can choose a canonical or use a noindex on those pages.


But within ads, you can still use them as landing pages.


And that's something that's something that a lot of people


do where they say, well, these are my ad pages


and these are my SEO pages.


And that's perfectly fine.


Also, Adrian, I suggest maybe you


check your SSL certificate.


For some reason, it's not like I'm getting


blocked coming to the site.


That's interesting.


Yeah.


I say it's interesting because I just implemented something


with HTTP auth on my WP admin because I'm getting


hammered with hack attempts.


If you have a CRT, you want to purge the cache and then


maybe that if you just said that.


It's weird.


It's working for me.


I went to it's Adrian and then your last name, right?


No.


Oh, OK.


Torontoheadshot.com.


So that site's down?


OK, I'm going to look into that other site.


Sorry, John, one other question before we go.


This is with regards to Search Console.


I have created a custom post type in WordPress


that creates the archive page for reviews, testimonials.


I'm not seeing any of that enhancement show up


in Search Console even though the page has been crawled.


All the schema validates on the schema.org tool.


But yeah, it's just not showing up.


Are they reviews or what did you mark up?


Yeah.


OK, so we probably wouldn't show that as reviews in the search


results because it's more like a testimonial.


So reviews would essentially need to be something


which is based on a specific product on that page.


And the reviews need to be things that users leave


directly on that page.


So if you're kind of like archiving reviews


from other sources and you're posting them,


then we wouldn't pick those up as reviews for the structured


data side.


But you can keep them on the page.


We just wouldn't use the review markup for that.


OK.


Like all the reviews, all the content


I'm taking from the reviews is from my,


it's not called Google My Business anymore.


But it's called reviews there.


So how would I add any kind of schema for you guys to see that?


You wouldn't.


If people are leaving the reviews somewhere else


and you're making a copy of that,


then the original place would be the place


where the structured data would need to be implemented.


Right.


I think it's kind of tricky because we


try to recognize this situation automatically.


And sometimes we don't recognize it properly


and just show it anyway.


So perhaps when you're searching around,


you see other sites that have that shown where it's just,


well, it's like we didn't recognize


that this was actually not left directly on the site.


But from a policy point of view, we


try not to show reviews that are left somewhere else that


are copied over to a website.


OK.


I am getting one review snippet show up.


And it's basically the one that's


tied to the business object.


So I have a section in the business schema


that says review and just has an overall review kind of thing.


But yeah, not the individual ones.


That was my whole plan.


Oh, wow.


I mean, you can still keep them in your pages.


It's just we won't show them with a special treatment


in the search results.


Right.


Cool.


OK, thank you.


All right.


Let me run through some of the submitted questions.


And then we'll get more of the live questions as well.


Let's see.


The first one is about Search Console verification.


We have the code on every subdomain.


But we're also using a tool which requires


domain-level verification.


We're thinking about doing it through DNS records.


Would that invalidate the current setup


and lose the data that we already have?


The simple answer is no.


You can do that.


In Search Console, you can have a site verified multiple times.


You can have different parts of the site verified individually.


It doesn't lose any of the data when


you have it verified separately.


So that's something where you could


have both the host level as well as the domain-level


verification running in Search Console.


We notice when our crawl request from Google


decreases, our crawl request from Bing increases.


Are the two related?


I think that would be a really weird coincidence.


But I'm not aware of any kind of collaboration with Bing


that we swap out the crawl requests.


So I don't see that being related.


What might happen sometimes is on Google's side,


when we recognize that a server is overloaded and kind


of slow and showing server errors,


then we'll tend to crawl less.


And it could be the case that when Bing crawls a lot,


we see the server is generally slower.


So we'll crawl a little bit less.


And when we see that the server has more capacity,


is a little bit faster, then we'll crawl more,


which might be kind of related to when


Bing happens to crawl less.


Theoretically, that's possible.


From a practical point of view, I kind of


doubt that would be happening just because usually websites


have so much capacity for users to come


that some crawling from Bing and some crawling from Google


is not going to slow down the whole website.


Will AMP pages impact the Google requests?


We saw research a few days ago on Twitter


that Google crawl volume on non-AMP pages


increased recently, while the crawl volume of AMP pages


decreased.


So we take into account all crawling


that happens through the normal Googlebot infrastructure.


And that also includes things like updating the AMP cache


on a website.


So if you have normal pages and you have the AMP versions


and you're hosting them on the same server,


then the overall crawling that we do on that website,


that will be essentially what we try to balance out.


And that includes the AMP and the non-AMP pages.


So if your server is already running at its limit


with regards to the normal crawling


and you add AMP on top of that, then of course,


we have to balance and figure out


what we can do there, which part we can crawl at which time.


But again, for most websites, that's not an issue.


It's usually more of an issue when


you have tens of millions of pages


and Google barely gets through crawling all of them.


Then that's something where if you add another duplicate


of everything, then of course it makes it a lot harder.


But if you have, I don't know, thousands of pages,


adding another thousands of pages from the AMP versions,


that's not going to throw things off.


We have a client having almost identical pages per location.


And I thought this was your question, Adrian.


Piano course Birmingham and piano course London,


the only difference in those pages


is the location shall I canonicalize those pages?


What would be the best practice?


Kind of like I mentioned before, if you have a handful of pages,


it's something where purely from a doorway page point of view,


I wouldn't really worry about that.


If you have these two pages with two locations,


the general best practice here is really


to make sure that you're providing


something unique and valuable on those individual pages.


So if you have something that you're offering in one city


and you're offering it in another city as well,


then give some information about why you


have these pages separately.


And make it so that when people are searching for it,


they really find something that matches that.


Not just like, oh, piano courses in the UK,


and like a handful of city pages there.


So that's kind of one approach there.


The other approach is, like I mentioned before,


canonicalizing and just picking one of those pages to show.


I think if you have something distinct


that you want to provide, then having multiple pages


is perfectly fine.


If it's something where you're just offering variations


of the same thing, then sometimes folding them together


is a better strategy.


The advantage of folding them together


is that you make a little bit of a stronger page


rather than two pages that are kind of OK-ish


from a strength point of view.


And the general idea here is that Google


has some information about your website.


And if you kind of dilute that across a whole bunch of pages,


then each of those pages individually


will be a little bit less value than if you concentrate


everything on fewer pages.


So if you think you can make a really strong page just piano


courses, then that might be a good approach,


especially if there is a competitive market there.


If, on the other hand, you think that these individual locations


are distinct enough that you want


to have individual pages, such as maybe you


have really physical addresses in these locations,


or different opening hours, or different kinds of teachers


in these locations, then that definitely


makes sense to keep separate because you're providing


something unique that people can find value on there.


So that's kind of the approach I would take there.


It's not that there's one answer that works for all websites


here.


For some, it makes sense to fold together.


For some, it makes sense to keep separate.


Depending on the website, sometimes you


even combine that.


So if you have an e-commerce site, you might say, well,


this model of shoe is all the same model of shoe,


but you have maybe one unique size that really kind of stands


out, or a unique color variation that really stands out.


Maybe you'll split that off into a separate page.


So those are kind of the thoughts behind that.


In general, does disavowing a 301 redirected domain


have the same impact as disavowing all the backlinks


of the redirected domain?


We have a 301 redirected domain pointing to our home page that


has 18,000 spammy backlinks.


Can we just disavow the domain redirected to us,


or do we need to disavow all the 18,000 backlinks?


We know that in most cases, Google


might already discount these links, but we want to be safe.


So in a case like this, I think it's kind of tricky


because we do try to figure out what


the destination and the origin of individual links are,


and we try to take that into account.


So purely from a kind of a theoretical point of view,


you would probably want to disavow


as many of those individual links as you can.


I think you can probably also get pretty far


by just disavowing that kind of redirect to your home page


and kind of handling it at that level.


If this is the only spammy link that you


can find for your website, probably I


wouldn't even worry about it too much.


I think disavowing that redirect kind of takes


care of a lot of that issue.


So probably that's already enough.


Indexing seems to have changed a lot for me


over the past year or two.


Happens across all of my web properties,


whereas things were quickly indexed before when submitting


them in Search Console.


And nowadays, Search Console submissions


seem to have no effect.


Rather, it appears on its own in the index


after several days or even weeks,


sometimes bouncing in and out of the index.


This happens on well-established sites


with 50,000 organic monthly visitors.


Why is this?


I don't know.


It's hard to say kind of in a broad way to kind of like,


this is why things are not being indexed so quickly


on your website.


But in general, the request indexing tool in Search Console


is something that passes it on to the right systems.


But it doesn't guarantee that things will automatically


be indexed.


I think in the early days, it was


something that was a lot kind of a stronger signal for indexing


system to actually go off and index that.


But one of the problems that happens


with these kind of things is, of course,


people take advantage of that and use


that tool to submit all kinds of random stuff as well.


So over time, our systems have kind of grown a little bit


safer almost in that they're trying to handle the abuse


that they get.


And that does lead to things sometimes being a bit slower,


where it's not so much that it's slower because it's doing more,


but it's slower because, well, we're


trying to be on the cautious side here.


And that can mean that things like Search Console submissions


take a little bit longer to be processed.


It can mean that we sometimes need


to have a confirmation from crawling


and kind of natural understanding of a website


before we start indexing things there.


And from our point of view, that's kind of expected.


One of the things that I think has also


changed quite a bit across the web over the last, I don't


know, last couple of years, probably longer,


is that more and more websites tend


to be technically OK in the sense


that we can easily crawl them.


We can crawl them normally.


So we can, on the one hand, shift to more natural crawling.


On the other hand, that means a lot of stuff that we get,


we can crawl.


And whereas in the past, when something would not get indexed,


you might say, well, maybe there's


something set up incorrectly on the website,


and you try to find that problem from a technical point of view.


Nowadays, most submissions that we get are technically OK.


So we can go off and index them, which


means because there is still a limited capacity for crawling


and also for indexing, we kind of


need to be a little bit more selective there.


So I imagine overall, that's kind


of what you'd be seeing there if we're not picking things up


as quickly as you might have in the past.


I have a domain that hasn't been used for four years.


The blog I had was doing great in its niche,


but because I didn't want to sell it,


I deleted all the content and left the domain part.


I want to revive the content on it,


but I want to take a slightly different approach.


My question is, does Google need to learn about my blog again


as if it was new, or do I have a better chance


to be an authority in my niche faster than usual because


of this old domain?


So if the content was gone for a couple of years,


probably we need to figure out what this site is, essentially


starting over fresh.


So from that point of view, I wouldn't


expect much in terms of bonus because you had content there


in the past.


I would really assume you're going


to have to build that up again, like any other site.


If you have a business and you close down for four years


and you open up again, then it's going


to be rare that customers will remember you and say, oh, yeah,


I will go to this business.


And it looks completely different.


They offer different things, but it used to exist.


I think that situation is going to be rare in real life,


I guess, if you will, as well.


So I would assume that you're essentially starting over here.


This is also one of the reasons why it usually


doesn't make sense to go off and buy expired domains in the hope


that you will get some kind of a bonus out


of using those expired domains.


I'm managing a website with over 5,000 URLs.


We do our best to comply to search ranking and SEO factors,


but we find on average we get around 100 URLs per month


that get de-indexed.


What can we apply to our pages to prevent this from happening?


So it's kind of hard to say with just that information


because for the most part, we don't just


remove things from our index.


We kind of pick up new things as well.


So if you're adding new content at the same time


and some things get dropped along the way from our index,


usually that's kind of normal and expected


because essentially for pretty much no website,


we index everything on the website.


It's something where on average, I


don't know what the numbers are, but what I've seen


is somewhere between 30% and maybe 60% of a website


tends to get indexed.


So if you're adding hundreds of pages a month


and some of those pages get dropped


or some of the older pages or less relevant pages


get dropped over time, that seems kind of expected.


To minimize that, I think it's something where essentially you


need to show to Google or to your users,


I guess, what the value of your website is overall


so that Google says, well, I need


to make sure I keep as much as possible


from this website in my index.


And that's usually less of a technical thing and more


of something where kind of the value that you provide


is such that it's critical for Google


to keep this in our index.


And even in cases like that when we think a website is


fantastic, we still won't index everything.


It's kind of just purely from a technical point of view,


it's almost impossible, partially


because a website tends to have a lot of,


I call it technically duplicated content where you have things


like tracking parameters or maybe pagination that


goes into infinite spaces or filtering parameters or search


form or things like that, where theoretically we


can find an infinite number of URLs on the website.


And if it's possible to find an infinite number of URLs


and we can only index a finite number,


then we'll never be able to get everything indexed.


Let's see, for job posting schema markup,


what are the most important parts for getting indexed


and ranking highly?


We've seen spikes in index jobs that drop off


after a week or two with no understanding as to why.


I don't have much insight into the job search or Google


for jobs, I think it's called, side of things.


But essentially, this also falls into similar categories


of indexing, like what can you do to make sure that


your content is better indexed.


And essentially, the things that I mentioned so far,


I think those are really relevant.


If you have a stronger website structure,


you can also guide things a little bit in that if we can


index, let's say, 100 out of 200 pages on a website using


the website structure, you can guide us a little bit


to focus on which part you want us to focus on.


And the way usually that is done is with internal linking.


For example, you take the important pages of your website


and you make sure that you're linking to the other parts


that you care about.


So that could be like on the home page,


you have a section that says important pages,


or popular products, or new products, or new jobs,


or whatever it is that you want to have highlighted


in the search results.


And from the home page, you link to that part of your website


directly.


And then we'll go and look at that and say,


well, the home page is probably the most important page


for most websites.


And it's telling us that these pages are really important.


So we'll focus more on those pages.


And for example, if you have a jobs website


and you have new jobs that you want


to have indexed as quickly as possible,


then make sure that from the home page


you're linking to those new jobs.


Many sites do that automatically,


kind of intuitively, but you can also


do this kind of in a very determinant way


and clearly kind of tell us exactly what it is that you'd


like to have shown in search.


And it doesn't have to be the newest ones.


It can be things like the pages where you make the most money


out of or the pages where the competition is the strongest


or anything, essentially.


Just tell us what you think is important on your website


through internal linking, and we'll


try to follow that and highlight that a little bit more


in search.


We migrated our site about seven months ago to a new domain.


Should I remove the old site map from the old site?


Probably, yes.


So usually when you migrate a website,


you end up redirecting everything to the new website.


And sometimes you keep a site map file of the old URLs


in kind of in search console with the goal


that Google goes off and crawls those old URLs a little bit


faster and finds the reader.


And that's perfectly fine to do in a temporary way.


But I think after like a month or two,


it's probably worthwhile to take the site map out.


Because what also happens with a site map file


is you're telling us which URLs you care about.


And if you're pointing at the old URLs,


then you're almost saying, well, I


want my old URLs findable in search.


And that can lead to a little bit


of a conflict in our systems in that we say,


well, you're pointing at the old URLs,


but at the same time you're redirecting to the new ones.


Which one do you really want to have indexed?


And at that point, you essentially


want to remove that conflict as much as possible.


And you can do that by just dropping that site map file


and giving us all the signals that you


can that point at your new pages so that once we've


seen that redirect, we can just focus purely on the new pages


that you have on your website or the new domain


or whatever move that you've made.


Google snippets and knowledge graphs


seem to be getting better all the time.


And often users don't have to visit a website


to get the answers they need.


What does the future hold for web publishers, big and small,


in terms of getting SEO traffic?


I don't see the need for websites ever really going away


because that information is really


in detail on your website.


And sometimes people just want something really quick.


Maybe they want the phone number or the address


of your business, and then they go off and go visit


that business directly.


But essentially, for anything more than just


a snippet of information, you want


to find out the full context and get some more information


there.


So I don't really see that going away.


It's also definitely not our goal


to be that one place where everyone goes and just gets


the answers directly because we know


we have to work together with the ecosystem,


together with anyone who's making websites,


to make sure that the things that we're providing in Search,


they provide value for website owners


as well because it's easy for website owners to say, well,


I don't want to take part of this search.


I'd rather just be findable in Facebook or in social media


or somewhere else.


And we want to make sure that there's


kind of an equal deal there in that we can show your content,


we can send users your way.


But you also get something out of that in the sense


that, well, you're getting all of this traffic as well.


And that is something that all of the teams at Google


that work on Search, they care deeply about.


So it's certainly not the case that we're


trying to figure everything out and then just only show them


on our side.


But I understand that this is something


that folks also worry about because they


see these fancy features in Search.


And it's hard sometimes to understand


what the bigger picture there is or what the net effect is


across websites.


And I think also if you run across situations where you're


like, well, I really don't like the way


that Google is showing this because it's really


something that I'd prefer people to look at on my website,


then give us that kind of information.


Contact me on Twitter.


Send me some screenshots of things where you're saying,


well, this is really not the way that I


want to have my content shown.