management services where there have been issues specifically


JavaScript SEO Office Hours April 7th, 2021

with CLS and cookie constant banners.


That I haven't.


LUCIA MRVOVA-MARTINENI.


Yes.


Yes, I agree.


Yeah.


And you are not the first one to report this.


I have an ongoing program or process


of looking into these things.


I wish I could do that more publicly.


But in order to not accidentally shame any providers,


I'm not doing this publicly.


I'm basically looking at this both on the side of what can


we do to avoid these problems.


On the other hand, also what do they need to do.


Because I wish us to have guidelines and guidance


for these tool providers that you and their customers,


as well as their own developers, can refer to to fix the problem.


Because in the end, if it's a problem in the metrics


and it's not just a measurement error,


then that's a problem for actual users.


And we should all work together to do that.


So no finite answers yet.


But we are definitely looking into it.


And are you going to publish something


regarding this on Twitter or something?


Just some kind of conclusions, maybe?


Once.


I don't have any conclusions yet because I'm


early on in the process.


I'm still receiving.


Basically, I'm in the reconnaissance phase


where I'm taking in input.


So if you are, there's a tweet I, gosh, OK,


let's see if I can find my tweet.


Long, long time ago, I tweeted that I am asking for people's,


basically for examples of this happening.


So if I find the tweet, and if not,


then you can just send me a URL where that is happening.


You can use a direct message on Twitter.


My direct messages are open for this kind of stuff.


I don't normally answer direct messages,


but I definitely receive them and look into them.


This is just one of the largest Switzerland classifieds


websites.


So I think that you probably can get an idea of what they are.


I think I know which way to look then.


Yes, OK, fair enough.


Is it the one with T in the name?


Yeah, OK, in that case, I know where to look.


Fantastic.


Yeah, because we've seen the Core Web Vitals going really,


really crazy, and the overall site performance dropping


incredibly low since we have implemented this.


And there has been some confusion also


with the tracking, and we're working on it in analytics,


and everything just seemed to be boiling out.


So, you know.


Yeah.


It's definitely largely from the provider side, I would say.


So I am in the Reconnaissance phase


where I'm basically taking in the information


and looking at it.


And once I have conclusions, I'll


publish them both in the docs and probably also over Twitter.


And who knows, maybe even in the video, who


knows what's going to happen.


This is 2021.


Everything is wild.


Cool, thank you.


Thanks a lot.


Very good question.


Thanks.


Again, a fantastic use of the Raise Hand feature.


I know that, oh, some people that have joined us


have already left again.


Sad.


We do have 15 more minutes, so any further questions?


Yes, Maria, raise his hand the old style way.


I know there is a feature right there to raise your hand.


We'll blame the calls.


I'm a little bit slower.


So I'm going to ask a question from the team


at the new place where I work for.


And probably you've had this question many times,


so here I come.


So the question is, what is the best way


to clean unduced JavaScript codes for a page, obviously?


What is the best method to recommend to the developers?


No, that's actually, I haven't had that question before.


So that's good.


I like that.


That's good.


Nice.


At least I can't remember.


But then again, I can't remember what I had yesterday for lunch.


So I don't have a code that I could blame it on.


I'm just dumb and can't remember things.


That's why I make computers do all my work.


So that's, oh, god.


OK, OK, don't lynch me.


That depends a little bit on how the website's built.


If you are using a framework, the likelihood


that you are also using some sort of bundle or build


tool that combines the JavaScript together in the end.


Because what you have is, OK, stepping back a little bit


so that, because I'm pretty sure most people will actually


benefit from this explanation.


So what usually happens when you build JavaScript web


apps or websites that heavily rely on JavaScript


is that you end up with some sort of dependency tree.


So for instance, you might use, I don't know,


let's say React.


React itself is a relatively small core module


that then has a thing that's templating system is called


JSX.


That's usually a separate module that uses other things.


And basically, you end up with lots of small bits and pieces


of JavaScript that are interconnected


and basically are combining into your JavaScript application.


And then eventually, you write a little bit


of code that makes the thing do what you want it to do.


So the way that it works in most frameworks


is that you have a huge amount of code


that comes from third parties that is not your own.


And then you write a little bit of code


to glue this together roughly.


And then what you end up with is a dependency tree.


So my thing here uses this thing here,


which uses this thing here, which


uses React, which uses this thing, and that thing,


and this thing.


And they all need to be compiled down


to a proper combination of things.


What happens, what easily happens is that some


of these dependencies are larger than others.


Let's say, I don't know, I am using a library that provides


a bunch of basic functions.


Like, it provides 100 useful functions


that you kind of need every day.


But I'm only using two out of these 100 functions.


Then what easily happens is that by bundling all these things


together, it seems like, oh, yeah, so this utility here


that has these 100 functions is needed by this other thing


here.


So I'll drag it into the big blob of JavaScript


that we are shipping to everyone.


Even though 98% of it are unused code, they just download,


they have to be executed, they have to be parsed,


they have to be executed, but they are never


actually running themselves.


It's just like these 98 functions just


sit in your bundle and then don't do anything.


This can be partially automated with tools


like when you're using Webpack, for instance,


you can enable tree shaking.


Basically, the fundamental term that you're


looking for with whatever you're using is you want to bless you.


You want to be tree shaking.


So you basically take the tree and shake off every loose apples


that you don't actually need.


So with tree shaking, these 98 functions


will probably fall down.


Why do I say probably?


Because it depends on how the developers are using it.


That's the tricky bit.


What you can, to a certain degree, do, though,


is in the developer tools, you can, oh, where was that?


Was that sources?


Or was that on the more tools?


I think it's on the more tools.


Yes.


Actually, hold on.


Let me share my screen real quick.


Never a bad idea to share my screen,


but let's do it anyway.


I like to live dangerously.


Why am I doing this to myself?


Everything, I think I never learned.


I think I quoted this.


If you're a good person and you like to help us all.


I don't know.


I think I brought this up a few times already,


but I'll bring it up again.


They did a study with earthworms, earthworms.


Earthworms is probably how you pronounce that.


And they found out that they need 3,000 repetitions


on average to learn something.


Anyway, so why am I bringing this up?


Because I'm an earthworm when it comes to learning things.


Let's zoom in a little bit so that we can actually


see what's happening here.


So I'm on a random website.


You might have heard of it.


It's apparently a social network that has a bird as its logo.


And then in the developer tools, so how did I get here?


Actually, let me get back one more time.


I right click.


I say inspect.


I'm not sure if you're actually seeing this right now,


but I guess you might.


I don't know.


Right click inspect.


Then I get the developer tools.


Then I go to the three little dots


and then say more tools, coverage.


And in coverage, if I am reloading the page


and get the coverage information,


we are seeing that there is a huge bundle of JavaScript


and 60% of the bytes are not actually being used.


Be very, very, very, very, very, very, very, very, very


careful with that.


Because the thing is, if I were to actually log in now and do


something, I could probably try that.


The number would not be accurate because it's very, very likely.


Most of this, OK, it doesn't have my data.


Most of the data that is unused is probably


used later on when I continue to actually work with this thing.


So this number might go up.


But it is a good idea to have a look where this comes from.


Shared core, vendors main.


Vendors main actually has lots of unused stuff apparently.


At least like a third of it is unused.


So I would start looking here because vendors is probably


like some third party code.


And there might be bits and pieces


that you can get rid of.


And you can test this in development.


You don't have to do this in production.


So developers can give it a try every now and then.


And then they need to interpret these results.


The 27% does not mean that 27% of the code is actually unused.


It is unused at the time when I took this trace.


But it still might help them to find out


where to look for things where they can save bytes


and JavaScript.


And then, as I said, tree shaking


in whatever system you're using is a good idea


if that's supported.


If it's not supported, you might want to consider


moving to a system where you can tree shake and import things


more selectively.


But again, there's no like automatic tool


that you just throw your JavaScript in and all the unused


code falls out.


Unfortunately, that would be awesome.


Thank you.


It was a very detailed answer.


I'm trying my best.


I thought there's like context that is worthwhile sharing.


And definitely will help when once the recorder is


ready to go back and kind of look, rewind again.


Have a look.


Yeah.


You're welcome.


All right.


Any other questions?


We already said we have the chat.


You can just say something.


And hey, Gabe raises his hand.


That would be the third option.


I'm coming back for a second one.


I'll try to make it quick because I


know we're short on time.


But just curious, random thought.


Sometimes when you identify a potential JS conflict


with the website, you might then go to Google,


use like a site operator, then check the cache and the index,


and then go to check as like text-based cache.


And then if you see your hyperlink there,


you know, oh, it's rendering.


If you don't, if it's text, you're like, oh,


it's not rendering.


But there's a debate that I had the other day


where with another SEO that was saying, well,


Google is rendering JavaScript.


That technique doesn't necessarily work anymore.


So is A, is that I guess entirely true or kind of true,


not true?


And B, if it is true, is there a more effective way


to kind of cross-check your observations?


MARTIN SPLITTENBERG


So the cache is a really, really old feature.


And the only purpose that it has is to give you something


should the website have been removed


or has gone down for the moment so that you can say,


like, I really want this piece of content.


It's only on this one website.


This website has not been available for the last week.


You may find something in the cache,


which means that the indexing is a pipeline of lots


of different moving parts.


And at some point in this, the cache


might filter out and siphon off the version


that it shows you in the cache.


It's an unmaintained feature.


We have it because I still think it provides value,


even though it is not under development anymore.


But the version of content you get in the cache


is kind of random.


It might just be from a previous crawl.


It might be straight from the crawl.


So we haven't actually run any of the JavaScript yet.


It might have picked up the snapshot


after rendering has happened.


It's luck.


Sometimes it works.


Sometimes it doesn't.


But that doesn't mean that there has been a fundamental issue


or that we haven't been rendering anything.


We are rendering every page.


Every page gets indexed based on the rendered version,


unless edge cases.


But that's really, really unlikely.


And then if you want to check that and see


what we would see in rendering, with certain caveats,


you can see that in Google Search Console Live Test.


That's the rendered version, as well as


in the Mobile-Friendly Test or in the Rich Results Test.


These things are using the same pipeline as Googlebot uses,


the caveat being that the indexing uses very strong


caching and can retry multiple times, which


means that sometimes it might take half an hour for us


to actually get everything downloaded properly


and get everything fresh and everything like that, which


we can't really do when you are sitting in front of a testing


tool.


So the testing tool skips the cache,


because we kind of want to give you the latest version of what


we would get if there wouldn't be any caches involved.


And you might see the infamous other error


in the testing tools, which usually just


means this took too long.


So if we are not able to fetch the JavaScript file,


then it just took too long until we actually


got around fetching it and putting it


into the cache, which for indexing


would not matter, because the indexing would go render,


oh, it failed because one resource wasn't ready.


So we'll basically have us notified once the resource has


been fetched.


Then eventually, the resource is being fetched,


put in the cache, and then we are rendering again.


And then we have the rendered version.


So sometimes you see false negatives in the testing tools,


as in like, oh, this looks like we have a problem.


It's other error.


That usually means it's not an actual problem,


unless it says something else.


If it's a different error source,


then that's a good sign for an actual problem.


But if it's other error, that usually means a resource fetch


took too long, because it hasn't gotten the position in the queue


that it needed to finish within the time limit


that we have set, which is very low for the testing tools,


because we don't want you to sit in front of a loading


bar for half an hour.


But the testing tools are your best bet


to actually see what the rendering has come up with.


Definitely.


But if you don't have access to search console,


say you're analyzing whether it be other websites you're


competing against, trying to determine how they're building


their websites, is there a quick and easy way


to check off on that?


If the cache is tested?


Which results test?


Mobile-friendly test.


But don't mess with the text cache anymore, huh?


I wouldn't, because as I said, it's pretty random


what you'll get.


And drawing conclusions from pretty much random data


is risky.


It's never fun, right?


It makes a good screenshot, but it doesn't tell the story.


It does, yeah.


So just to make sure, if I find a text, which should be a link,


but it's text-based, and it's anchor,


in the text-based cache, that is not a sign


that there is an issue with that link being crawled because it's


blocked by JavaScript.


Correct.


Yeah.


OK.


All right.


I lost the debate then.


I'm sorry.


I'm very, very sorry.


It's just that old-school SEO stuff.


That's just what we used to do.


I'm so sorry.


I can't keep up with you, Martin.


I'm so sorry, Gabe.


I don't always win people's bets, but sometimes I do.


And unfortunately, you are not one


of the one debates in this case.


We'll work this out offline.


We've got to figure this out when we come live.


You know?


So, you know.


Awesome.


Sweet.


Thanks a lot.


And Mihai has raised his hand as well.


Hey, Martin.


Hi.


How's it going?


I'm good, good.


Quick question regarding JavaScript loading priorities.


So and this is mostly regarding platforms


where you don't really have a lot of access to change things.


So for example, I'm working on a site that's


on a popular e-commerce commercial platform.


Brimes with Shopify.


Wait.


I did that wrong.


Obviously, you cannot really change.


You go around and change things.


And how are they loading stuff?


You do have some control, but not like it's your own server


or anything.


But I noticed that for any plugins


that you're using with the platform, they do their best.


They use a JavaScript function to load the third party


JavaScript files after the unload event.


So they're not using defer or async.


They're using that JavaScript function


to basically load everything after the unload event,


which is very nice on their behalf.


So I'm curious if there's any difference in terms


of how Google sees things, whether you're using,


let's say you have a vendor or third party JavaScript file,


whether you're putting it in the footer


and using the defer attribute versus using


this kind of trick to move everything


after the unload event.


No, we don't care.


For us, that doesn't make a difference.


It's just a different moment where


we discover the resource probably, but that's OK.


That's fine.


OK, so as long as the actual content gets loaded the same,


doesn't really matter when the rest of the craft comes up.


I love how you immediately say, that's craft.


There we go.


Well, as you mentioned, tracking things, other things that


are chat bubbles and things like that, things that


are necessary towards actually rendering


the main content of the page.


And as I mentioned, not everybody has the log,


well, not luxury, but not everybody


has the chance to work with their own server,


their own platform.


So they have a lot of websites are using this third party


platform.


So it's, again, very nice that they do not only they defer it,


but they actually defer it all the way


after the onload event.


So I was just curious whether it's the best practice


or if there's any difference in terms of how Google perceives


those or renders those files.


I don't see any reason for us to treat or perceive


this differently.


Maybe because the onload event can be like five seconds,


six seconds away from the load time.


That's true.


Versus the defer, you still see the resource right away.


You just defer it.


Yeah, so the resource discovery, that's a thing.


That's going to be different, but we don't really


care as long as the content is there.


We'll probably trigger a late page, sorry,


a late resource fetch for these resources, but it doesn't hurt.


Yeah.


OK, got you.


All right.


Thanks for the question, Mihai.


Oh, Dave's raising his hand.


Oh, god.


OK.


Ooh.


Oh, my.


That's a bit rough.


I'm not that bad at that.


Yeah, basically, it's just a question


about if you circumvent the cache of certain things


by doing post requests or timestamping or something.


Does it really cause an issue other than just


in crawl budget?


I mean, naturally, you'd want to cache everything you could,


anyway.


Yeah, exactly.


Like, it does impact your crawl budget.


It also has a potential.


You've gone on mute, mate.


How did I mute?


When did I mute myself?


Anyway, it doesn't.


I think it's one of the top sentences, 2020, and probably


2021 as well.


It's like, you're on mute.


Can you hear me now?


No.


I think that was like an ad campaign for a mobile network


at some point, like, can you hear me now?


So besides the crawl budget issue


that you already mentioned, where pretty much


every single time we are requesting


that resource that is cache busting in one way


or the other, you might run into a potential pitfall.


I'm saying potential because I've never seen


this being an actual problem.


But there is a potential for it becoming a problem if you


do it excessively, which is if you are skipping the cache,


you are reducing the resilience against timeouts.


Because if we have a cache resource that we can't fetch


afresh for a rendering run, then we


need to wait for the resource to actually come through live.


And if the crawler does not have the budget


to actually fulfill this within the deadline,


then you might actually not get the resource in time


for rendering.


And then you might see a failing rendering run.


And then we have to retry that again.


And then if the issue persists, it


might increase the time until we actually


get a successful render.


Again, I've not seen this in practice yet.


But I've not seen that many people in practice


using post-to-fetch resources for the website


to avoid the cache.


I think so.


I could imagine it's something more prevalent


once GraphQL grows because that's


caused by the followers in that set.


Yeah.


And I've seen GraphQL being like, oh, GraphQL


doesn't work with Googlebot.


And I'm like, no, it does work.


Why would it not work?


Oh, here is an example where the content doesn't come in.


I'm like, oh, yeah, but that's because it continuously


times out.


And you're posting, so we can't actually cache anything.


So yeah.


But if you were to cache bust, I would probably


do that with query strings that have timestamps that


are more or less predictable.


As in maybe every day, the timestamp changes,


or something like that, if you really, really, really need


to cache bust.


But full disclaimer, I'm not encouraging anyone


to cache bust, inviting whole new scenarios of problems.


You're welcome.


Thank you very much for the question.


Yes.


So repeating myself for the new arrivals,


you can post your questions in the chat as well.


That is, as I learned from Michael,


on the right-hand top corner, I think


I'm running some sort of early access internal version


where it's in the lower right-hand corner.


So eventually, it might move from the right-hand up corner


to the right-hand lower-hand corner.


You can also use the Raise Hand feature, which is somewhere


down there probably.


I think it's down there.


Those who have raised their hands, it's somewhere down here.


Or you can just speak if there is awkward silence


for a long period of time, which there never


is because I talk too much.


So any further questions?


We are already seven minutes over.


We can do another three minutes over.


It won't matter.


Jacob was actively thinking about a question.


I like that.


I have another 10 questions, but I can't just.


We actually have, is it Amin or Amine raising his hand?


So go ahead.


Oh, I think you're on mute.


At least we're not hearing you.


Well, I'm not hearing you.


I don't.


Does everyone else hear him and I just don't?


No, I can't hear him either.


OK, good.


So there's something in your microphone settings


that prevents us from hearing you.


No?


It was very carefully lip-reading,


but I'm not good enough to actually.


Nope.


I know that problem because I have multiple microphones.


And hi, sorry.


I got my microphone, which is external.


And I have to change the settings.


I have two external microphones and an internal one


because the one that I'm speaking to you right now


is not good enough for the podcast.


So I have another one for the podcast,


and then I have the internal one.


It's always fun.


I've got a question with regards to Google Evergreen


and with the recent update on Chrome.


And I was wondering whether you have got any documentation


regarding the JavaScript and how it's handled


and if there was any improvement on that from Google Evergreen


in terms of how it does handle the additional JavaScript,


how it is efficient.


So our web rendering service uses whatever


is the current stable Chrome.


So whenever that gets an improvement,


then we're getting the same improvement.


There's no difference.


There's no special thing.


There are small, specific differences.


They are documented in our documentation


at developers.google.com search.


I think it's now in advanced.


And then JavaScript, there's a bunch of information


on how we're handling JavaScript.


But it's pretty much the same as what we're doing.


It works roughly the same way as it works in the normal Chrome.


So there's no special treatment, no special.


There are certain differences.


For instance, we're not persisting local storage, session


storage, or cookies.


We are behaving different in certain very small but important


ways.


Web sockets are not properly supported.


HTTP2 is just coming now.


It's an experimental feature that we are rolling out slowly.


But the JavaScript processing and parsing and the V8


improvements are not being supported.


The parsing and the V8 improvements


are pretty much the exact same as we are running on rendering.


It's Chrome 89 at this point, I think.


Does that answer the question?


SPEAKER 2


In terms of documentation.


Yeah, hold on.


I can actually share.


OK, let me actually find you the specific link


to that part of our documentation,


because I know that it recently moved.


I think it's in optimize your.


No, that's not the one.


Manage JavaScript content.


Here we go.


So this is the main.


I'm going to share it here in the chat.


This is the main guide.


And from there on out, pretty much everything else is linked.


This is where the differences are.


Most of them, anyway.


I'm really excited that no one has


caught one of the more interesting and surprising


differences yet.


But it doesn't really matter 99.9% of the cases.


Giacomo is smiling because he probably found something.


Cache busting stuff.


I did not.


I think I did not explain that.


No.


We do explain that you shouldn't use the cache, the text-based


cache thing to test.


So that's nice.


SPEAKER 1


Are you going to rub it in, Martin?


All right.


SPEAKER 2 I'm so sorry.


I'm just like, I just read that sentence,


because I think it's on a colored background or something.


That's why it caught me.


Use long-lived caching.


So we do talk about caching in some way, Giacomo.


So just not the cache busting.


I'm talking about the complete opposite, to be fair.


So there's that.


Sweet.


That has been fun.


And we are 30 minutes over.


Do you have further questions?


Because while I'm here, I might as well answer more questions.


There's no further questions.


Feel free to use the Raise Hand button.


Yeah, Gabe.


Gabe.


Why not?


Yeah.


HTTP2 advantages in JavaScript-based websites


maybe that we should be thinking about when


we're talking to clients?


HTTP2 with its ups and downs.


So at this point, I would not.


But I would watch very closely for general availability


of HTTP2 crawling, which has not rolled out yet.


I think we are at 10%, 20%.


I can't remember how many.


What's the percentage of URLs that we're crawling with HTTP2?


The big upside of HTTP2 is that you can multiplex


through a single connection.


So what happened on HTTP1 was you have a website, index HTML,


and then some style.css, and then some image.jpg,


and maybe some header.jpg, and then some app.JavaScript.


Each of these would have been transferred


over a separate connection.


Connection keep alive.


That's what I was looking for.


Connection keep alive would at least


avoid the teardown of the underlying TCP connection.


Yay!


Because what really happens is your browser makes a request.


So it finds a resource like index HTML.


Then it has to actually go to DNS


to get the address behind the URL.


Then it needs to establish a TCP connection.


And then over this TCP connection,


it actually does the HTTP negotiation.


Like, sensor request gets a response back.


HTTPS makes that even slower, because then you


have a TLS connection on top of the TCP connection, on which


you then actually do the HTTP transfer.


And keep alive made at least the TCP connection stay available.


But you would still have to basically, like,


I think it was six connections at some point


to the same thing.


So it would open six connections.


And then if you have more than six things,


like the HTML, the two images that I spoke of,


the JavaScript, the CSS, that's already five.


And then there's like 10 more images.


And you would basically download them in bunches of six.


I think it has at some point been raised to 10.


But that's pretty much it.


With HTTP 2, you have one single connection.


And pretty much everything can just smoothly


flow into one connection.


And then the browser will tear it apart


into the individual different bits and pieces, which


means that domain charting is not


necessary if you use HTTP 2.


Domain charting was exactly avoiding this problem,


where to example.com, there would only


be six connections at a time.


So you would move your images to image.example.com and your CSS


to assets.example.com, same as your JavaScript,


so that you can basically get all of these at the same time.


But you don't need to do that if you have HTTP 2.


HTTP 2 also has the push cache feature,


which is widely recommended a bad idea.


So I would not really invest in that.


Anything else that is really cool at?


I think it has pretty good compression enabled by default.


It is binary, so it doesn't waste as much data.


It pipelines, as I said, like it uses one connection


to get all the assets.


Yeah, that's pretty much it.


HTTP 2 is nice, but I don't think


it's a revolutionary change in terms of SEO for Google Search.


It's not supported for all websites at this point.


It will be in the near future.


Right, and you're not crawling yet in HTTP 2, right?


Is that what you said earlier?


We are crawling HTTP 2 for a small percentage of websites,


and we are about to ramp that up to 100% eventually.


But we haven't done that yet.


So we are still in the experimental phase.


For instance, I'm not sure if it has been resolved,


but Search Console has a glitch when


displaying information for HTTP 2 crawled websites,


which is hilarious.


Gary is working on getting that resolved as far as I'm aware.


Sorry?


Is it a pure coincidence that you guys


are having the page speed update around page speed load times,


and then you're going to be crawling HTTP 2?


That's a coincidence.


The HTTP 2 crawling started in December,


if I remember correctly, or maybe November last year.


So it is already in progress.


The page speed is, or page experience is soon to come.


We might get later than May, by the way.


It's unclear at this point.


We'll see.


And Giacomo adds a tip to the chat,


saying, if you want to keep domain charting for your HTTP 1.1


clients while using HTTP 2, I suggest


to look at connection coalescing.


I haven't actually heard that before.


I need to read up on that.


Cool.


More work.


Thanks, Giacomo, for sharing this tip.


Cool stuff.


All right.


One last question, or we are closing?


I think one last question is good.


OK.


While we are talking about HTTP 2,


that is from Googlebot to the internet,


but internally, there are differences.


There are Chrome instances talking with Googlebot.


And can we know what's the type of connection


between the Chrome instances and Googlebot?


Is HTTP 1.1, ZRPC, or anything else?


It's GRPC, actually.


Well, it's RPC.


Let's call it RPC.


They are basically microservices to each other.


So it's doing an RPC call, one that I'm


very familiar with because I built an internal tool that


uses.


So the infrastructure that is the crawling infrastructure


is called Trawler.


The team is called something different


because they also have another crawler infrastructure thing


that is not what we're using.


So Trawler is a microservice that we are


talking to via RPC calls.


So we basically compile a protobuffer


with all the information for the request that we want,


throw that over to Trawler, and then a Trawler instance


picks it up and works with that and eventually comes back


with a response.


Thank you.


You're welcome.


We have this office hour every two weeks.


And I announce them on youtube.com


slash google search central slash community.


Sweet.


Ladies and gents, it has been a huge pleasure.


Thank you so much for joining, everybody.


Thanks for all the lovely questions.


I hope that you are staying safe and healthy.


And for those who are interested in the technical bits and bobs,


Maria, Dave, Giacomo already know this.


I'm Twitch streaming every Tuesday.


I haven't done that yesterday because it


was a holiday for me.


I just took a day off after Easter.


That was awesome.


And we went diving, and we went in with sun


and came out with snow, which was April, I guess.


Not sure if climate change or just April.


And yeah, see you either at the JavaScript SEO office hours


in two weeks or on Tuesday for Twitch stream time,


where there's live coding.


Awesome.


Thanks a lot.


Have a great time.


Thank you.


Bye bye.


Bye.


Bye, Martin.


Honest mistakes & manual actions #AskGooglebot

Today's question comes from Ashraf, who asks about how Google reacts to honest mistakes.


Now, I can only answer this in regards to our Webmaster Guidelines and web spam policies.


We know nobody's perfect.


It's easy to stumble into bad practices, be it because you saw something wrong online,


or be it from someone else who gave well-meaning advice.


That can happen.


We understand that.


Our systems try to help site owners in two ways.


First, if our systems can automatically recognize and isolate the bad practice, we try to just


ignore it.


We'll focus on the good parts of your website, and most websites have many good parts.


Second, if we do need to take a manual action, we'll let you know about it in Google Search


Console.


This gives you a chance to fix the issue and then to file a reconsideration request.


At that point, we can double-check that everything's fixed and cancel the manual action.


So in short, we do understand that honest mistakes can happen, and we try to deal with


them thoughtfully.


While it's good to learn more about SEO before making changes to your website, a simple mistake


won't ruin your website standing in Google Search forever.


Well, I hope this helps to ease your mind a little bit.


Voice Search in the Google Search Console: #AskGoogleWebmasters

Hi, everyone, and welcome to yet another episode of


Ask Google Webmasters, a video series where


we aim to answer short questions about Google Search


and websites.


Today's question has come from Itamar, as well as from Isha.


Itamar and Isha are asking about voice search in Search Console.


Well, this question comes up from time to time.


It's worth considering that there are multiple ways


to do a so-called voice search.


The way I most frequently use is essentially


to use voice as a special kind of keyboard for a text search


field.


In that case, when I'm doing a so-called voice query,


it's actually just a normal web search


that I enter with the voice keyboard.


These queries are already logged in Search Console in the same


way as all other queries are logged,


just like you'd see queries that are entered


by swiping on a mobile keyboard.


It's slightly different when you use Google Assistant


to ask a question that's answered through a snippet


of text from the page.


In cases like this, we'll send a URL of the page


to the user's mobile phone to make it easier for users


to go to those pages.


However, this data, well, the impressions


of snippets for those queries, that currently


isn't logged in Search Console.


As with other feature requests, we regularly review these


and try to prioritize those that bring the most value to websites


to publishers, webmasters, and SEOs.


If there's something you're keen on seeing in Search Console,


make sure to submit feedback directly in the tool.


Ideally, give us a glimpse of what


you're trying to figure out and how it can help


you to make a better website.


We'd love to hear from you.


Till next episode.


Does URL length matter? #AskGooglebot

Today's question comes from Greg, who asks whether shorter URLs are better.


Does URL length matter? #AskGooglebot

The direct answer is no, the URL length doesn't matter.


Does URL length matter? #AskGooglebot

We use URLs as identifiers. It doesn't matter how long they are.


Does URL length matter? #AskGooglebot

Personally, I try to keep them shorter than 1,000 characters,


Does URL length matter? #AskGooglebot

but that's just to make monitoring easier.


Does URL length matter? #AskGooglebot

The number of slashes in there also doesn't matter.


Does URL length matter? #AskGooglebot

I'm currently only aware of one part of our systems where the URL length plays a role.


Does URL length matter? #AskGooglebot

That part is canonicalization.


Does URL length matter? #AskGooglebot

Canonicalization is what happens when we find multiple copies of a page on your website


Does URL length matter? #AskGooglebot

and we have to pick one URL to use for indexes.


Does URL length matter? #AskGooglebot

If we find a shorter and cleaner URL, our systems tend to select that one.


Does URL length matter? #AskGooglebot

This does not affect ranking. It's purely a matter of which URL is shown in search.


Does URL length matter? #AskGooglebot

So, to sum up, when it comes to search rankings,


Does URL length matter? #AskGooglebot

neither the URL length nor the number of slashes matter.


Does URL length matter? #AskGooglebot

Use a URL structure that works for you and which you can keep for the long run.


Does URL length matter? #AskGooglebot

Well, I hope you found this useful.


Does URL length matter? #AskGooglebot

Let us know how you like these videos by adding a comment below or reach out to us on Twitter.


Does URL length matter? #AskGooglebot


English Google SEO office-hours from January 7, 2022

All right, welcome back, everyone,


to today's episode of the Google Search Central SEO Office


Hours Hangout.


My name is John Mueller.


I'm a search advocate here at Google in Switzerland.


And part of what we do are these Office Hours


where people can join in.


I think, first of all, happy New Year to everyone.


I hope you had a good break and had a chance to recover,


I don't know, whatever you needed to recover


and that things aren't looking too bad where


you are at the moment.


It looks like lots of people have questions.


Lots of people are here.


And we have a bunch of questions submitted as well.


So maybe we'll just go through some of the people that


have raised their hands already, and then we'll


kind of see where we end up.


Let's see, Rogan, I think you're first.


Hi, John.


Yeah, just hopefully relatively quick one.


We've been trying to, we have a book website.


It's a subscription product with about 800,000 books on it.


When you search for a book on Google, on the right hand side,


there's like a get book function,


you get book option on the book component,


on the book overview component.


There's, when we search for some books,


it is up to seven different providers.


And we've been trying to get in as one of those providers.


And we've added the book schema markup


and then submitted a request to be part of that initiative


multiple times.


And I haven't heard anything back.


It seems to be going into a bit of a void.


Do you know if there's a way of getting into that list


of providers, or if there's anything that we


can do different toward it?


I think being submitted with the form


is essentially the right approach at the moment.


My understanding is the team is still


going through those form submissions,


and the plan is to move to a more open model


where whenever we discover the markup,


we just start using it if we feel we can trust it.


But I don't know what the current status is there


at the moment.


If you want to drop maybe a URL here in the chat,


then I can pass that on to the team to have them double check.


Yeah, I've added a comment before the call with all the information there.


We've submitted it maybe like 8, 10 times over the last nine


months and haven't heard anything back.


So if you can give them a nudge, that would be awesome.


Yeah, I can't promise that they'll turn it on


automatically.


But usually, these kind of nudges


help us to make that step to making it a little bit more


open in the sense that we just use it whenever we can trust it.


Yeah, if they can give us any reason why it wouldn't be


working, that's something that we can change.


If the schema markup's wrong or something,


then that would be super helpful.


Cool.


OK.


Thanks.


All right.


The next name I don't think I can pronounce.


If you can pronounce your name and feel free to jump on in.


OK, sir.


Very happy new year and a very good evening


according to our local time.


So my name spells like a showing the geography,


and I'm from India.


So my question is that we all have


heard a term called crawl budget.


And suppose I have a blog in which I post almost one article


every day and another person having a blog in which he posts


about one article in a week.


So according to consistency, I am a bit more consistent


than him.


So will it affect the frequency on how Google will crawl


my website or how Google will handle my website?


Suppose we both publish the same content on the same day,


but he publishes less content in his blog.


So my content will rank in Google on the top,


and his content will rank on Google on down or on low.


Does consistency have any connection with ranking?


Thank you.


I don't think so.


So there are a lot of factors that go into ranking.


And being able to crawl and index a website


is definitely one of those things.


But if we're talking about one page per day or one page


a week, the ability for us to crawl that is trivial.


If we're talking about millions of pages every day,


then sometimes the technical capabilities come into play,


and crawl budget is a topic.


But if we're talking about, I don't know,


a couple of pages a day or 1,000 pages a day


or even 10,000 pages a day, then that's


something that we can usually crawl in a reasonable time


anyway.


So that's less a matter of being able to crawl it on time


and more a matter of all of the other factors


that we use around search.


OK, I have understood.


Another question that when I used to post, as I said,


one article every day in my website,


I used to see that Google almost crawls my website every day.


And when I went to the Search Console,


and when I see this side map, I see that Google


recrawls it every day.


But when I became inconsistent, then I


saw that Google crawls the site once in every two days


or maybe a lesson.


So is this a fact?


That can happen, yes.


So I mean, it's not so much that we crawl a website,


but we crawl individual pages of a website.


And when it comes to crawling, we


have two types of crawling roughly.


One is a discovery crawl, where we


try to discover new pages on your website,


and the other is the refresh crawl,


where we update existing pages that we know about.


So for the most part, for example,


we would refresh crawl the home page, I don't know,


once a day or every couple of hours or something like that.


And if we find new links on their home page,


then we'll go off and crawl those with the discovery crawl


as well.


And because of that, you will always


see a mix of discover and refresh happening


with regards to crawling.


And you'll see some baseline of crawling happening every day.


But if we recognize that individual pages change


very rarely, then we realize we don't have


to crawl them all the time.


For example, if you have a news website


and you update it hourly, then we


should learn that we need to crawl it hourly.


Whereas if it's a news website that updates once a month,


then we should learn that we don't need to crawl every hour.


And that's not a sign of quality or a sign of ranking


or anything like that.


It's really just purely from a technical point of view,


we've learned we can crawl this once a day or once a week,


and that's OK.


SPEAKER 2 OK, my last and final question


is that, suppose I have posted a topic on my blog,


a post on my blog, and due to some need,


I need to regain post that contains on my blog,


again, with possibly maybe that I have posted,


already posted two posts.


And I need to combine them and add something new


and then republish it.


I want to keep the previous posts as same as them,


but I want to republish that combination of the two


new posts.


So will it penalize my blog somehow,


or will Google think that this person is


publishing the same content?


JOHN MUELLER No, I think that's usually fine.


I mean, our systems would recognize


when there is significant duplication on a website,


but for a large part, those are normal things


that happen on a website.


Like if you have a category page and a detail page


for a blog post, then obviously on the category page,


you'll have some part of the blog post itself as well,


which is duplication, but it's also fine.


PRAVEEN SHARMA


OK, so pardon me.


I have got another question.


So every time we often hear that,


Google likes fresh content.


So if I have posted content in a post somehow in late 2020,


and should I again repost it or update it,


or should I repost it?


No, I mean, if it's the same content,


you don't need to repost it.


It's not that we have a ranking boost


for things that are fresh.


So if I need to add something or if I need to update it,


will I update it?


Should I update it or should I repost it?


I would just update it.


I think that's perfectly fine, yeah.


OK, thank you so much.


That's all my query for today.


Thanks.


Nice to talk to you.


Cool.


All right, Ahmed.


Hi, John.


Hi.


My question is about HDFS language tags, actually.


Let's say I have one website that I


target a specific language with that domain


and performs very well in that particular language.


And then I decide to create an English version of that website


to target people in global with new domain.


So should I add HDFS language tags


to connect these two separate domains


or just leave it alone in Google to figure that out?


And can this HDFS language tags impact my website's performance


in a positive or negative way?


So what would you do in a situation like that?


So HDFS language is on a per-page basis.


So it would only make sense if you


have equivalent pages in other languages


or for other countries.


So it's not something that does the whole website kind


of thing.


So if you have some pages that have equivalent versions,


using hreflang is a good way to connect them.


What happens with hreflang is that the ranking


stays the same, but we try to swap out


the URL against the most best-fitting one.


So if someone is searching for your website name,


and we have an English version and a, I don't know,


French version, then if we can tell the user is kind


of in France or searching in French,


then we will try to show the French version of the home


page.


And that works across different domains.


That works within the same domain.


That's essentially a good practice.


AUDIENCE 1. But is it not necessary or is it a must?


JOHN MUELLER It's not necessary.


I mean, it doesn't change the rankings,


but it helps to make sure that your preferred version is shown


to the user.


It doesn't guarantee it, but it makes


it a little bit easier for us to show the preferred language


version.


So if someone is searching in French


and we have your French and your English pages,


we wouldn't accidentally show the English page to them.


AUDIENCE 1. OK, thanks.


Thank you for answering.


JOHN MUELLER Cool.


Let's see.


I think Vahid?


VAHID LABOUZAKHAR Hello, John.


Hi.


Today I'm going to ask about page experience.


We know that Google Search Console reports three factors


for page experience, mobile-friendly HTTPS


and core WebGit talks.


So I'm going to ask you if there is more factors for page


experience or these three are what


Google care about in this case.


JOHN MUELLER I think the only one that is missing


is intrusive interstitials, which we also have documented.


And I think at the moment we don't


have a report specific to that.


So that's not listed separately in Search Console.


VAHID LABOUZAKHAR OK, and I have another question.


What data does Google Chrome collect


from users for ranking?


Can you please tell me by a list?


I don't think we use anything from Google Chrome for ranking.


So the only thing that happens with Chrome


is for the page experience report,


we use the Chrome User Experience Report data,


which is kind of that aggregated data of what users saw when


they went to the website with regards to the page experience


specifically.


And I think that's the only thing that we


use from Chrome within ranking.


OK, what about Google Analytics data?


Do you think Google Analytics data has after ranking?


No, we don't use that at all.


OK, so bounce rate and time on site


are useless for ranking, right?


Well, I mean, we don't use that data.


So I think these metrics are sometimes useful for site


owners to look at.


But that doesn't mean that they're useful for search


to actually use.


OK, thank you, John, for your time.


Sure.


All right, and the next name, I don't


think I can pronounce either.


Maybe if you can pronounce your name


and feel free to get started.


Oh, we can't hear you.


I think you're on mute.


Hello, John.


This is Adeshan.


Hi.


My question is on international targeting.


So like we have set up a business from last 10 to 15


years, and we have.com for the United States.


Recently, in the past two years, we


have started for India, Australia, and Mexico.


And we have purchased some ccTLDs.


And we have set up a different website


for the different websites.


But initially, what happened, we are getting the business


from the.com website.


And we have not set up the hreflang tag


to the other websites.


And they have also no authority for India, Australia,


and Mexico.


So should we add that tag right now?


Or is there anything for the ranking?


Yeah, so the hreflang wouldn't change


the ranking of these pages.


It would just make sure that the preferred version is


shown in the search results.


So especially if you have different country versions


in English in the same language, then the hreflang


makes it a little bit easier for us


to show the best matching version for that user.


So if you're seeing the wrong version shown in the search


results, like if someone is searching in Australia


and they see the Indian version, then that's


a pretty strong sign that the hreflang would be useful.


If they're mostly seeing the right version already


in search, then probably the hreflang is not critical.


Yes, this is the scenario.


Like, our.com is ranking for the Indian version also


in the Indian web region.


But we want to rank like.in website, that's ccdld.


So if we install that hreflang tag right now


for the.in website, so do we get the same ranking


for the.in website?


Yes, it would swap out the URL.


So instead of the.com being shown,


it would show the.Indian version.


Yeah, thank you so much.


Cool.


Let's see.


Hasan, I think.


We can't hear you.


I think you're maybe on mute, Hasan, I think.


Sorry.


OK.


Perfect.


Yeah.


Yeah, one second.


Hi, John.


Hi.


I have just posted my question on the comment section as well.


So just let me tell you again.


I have two questions.


One is that we represent the company expectation.com,


and we are the job portal for international region.


So as of now, like yesterday, we received a reminder


for the new job posting content policy guidelines,


and that it is set like the direct apply markup, which


has been very recently added, and a few policy guidelines.


So in that, it's recommended for us to please review the updates


and if we went the needed changes on our side.


So going through that, we haven't


added the direct apply because most of the jobs


comes from our APS partners and recruitment website.


But however following that, we have


added the direct apply markup for those


which are not direct jobs.


We said it's false, and for direct jobs, we said it's true.


So in this regard, just we want to make sure


that is it a matter of concern for us, that email,


because we want to make sure that we maintain and comply


with Google and maintain their eco-friendly environment.


So all that, that's one of my questions.


I don't know.


I don't know for sure.


I don't know exactly which email you're referring to.


But if it's essentially an informational email


that you received through Search Console.


Yeah, it's an informational email.


Yeah.


Then from what I know, usually those


are situations where it's more that we want to inform you


of maybe this change in the possibilities


that you have with regards to the markup.


It would be different if there were explicitly


a message from referring to a manual action


or something like that, because that would mean


that the spam team or the policy team has taken a look


and seen, oh, there's something really wrong here,


and you must fix it.


So those are usually the two levels of information.


OK.


And one more question I have is that actually we


used to maintain for each country a domain subfolder.


Let's say example for US, Expert in E-Jobs USA.


But then we thought the better idea


is to go on with subdomain for each country.


So we moved from folders to subdomains.


But when we did, we have used the 301 permanent redirect.


But we see there's a lot of fall down in the traffic.


Actually, we did it from a past six months back.


But since we are unable to gain the traffic, what we used


to have on the folder website.


So what we did is, as of now, we have just


enabled both pages, like the folder page for the USA


and subdomain page usa.expertini.


So should we use now the conical link,


rather than what we used for the 301?


Because that was not correct for us.


So we thought the better idea is to use the conical


and keep maintain both the pages till Google fully understand


that we want to prioritize subdomains, rather than


the folder website.


You can do that.


OK.


I think, yeah, I don't see any downside to doing that.


I think the main issue is that if you have those two versions,


we will try to crawl both of them.


And if it's a very large job site, where you're really


posting millions of jobs, then it


might be that the crawling is kind of the limiting factor.


So that's what I might watch out for,


that with the crawling side, we're


actually being able to keep up.


All right.


OK, thanks for the info.


Thank you very much.


All right, Tobias.


All right, so happy new year first.


And yeah, my question is about how


to get another language version of a website into top stories.


So we have a normal site, which is German,


and another site, which is in the section now,


which is French.


And we publish, I think, enough and high quality articles


for almost half a year now.


But we never made it into top stories.


And we wonder what's the problem.


Yeah, so I checked multiple things.


So for example, technically, it's exactly the same.


We implemented like hreflang.


We also added it to a publisher center with a French version.


But we are kind of stuck.


And we know that many that also competitors and other sites


have the same problem with new publishing websites


to get into top stories.


And I wonder if you could talk about that.


And I wonder if you could, if you know something here


about how long do we have to wait, how can we push it?


Because we really think that the content


should be good enough to get at least one ranking in half


a year.


Yeah, I don't know.


So there is nothing technical from our side that really kind


of plays into that.


When it comes to top stories, we don't


require AMP pages nowadays.


So I don't know if that's a factor.


However, we do take into account things like the page experience


metrics, so things around kind of the core web vitals, speed,


those things.


I suspect if you're using the same platform


that you will have similar metrics to your other website.


But I don't know.


That might be one thing to double check.


The other thing is usually just because it's


in a different language wouldn't mean that we


would treat it differently.


So if you have it in something like a subdirectory,


then from our point of view, that would probably


be something that we just treat the same


as the rest of your website.


But it's hard for me to say what exactly might


need to be done there.


If you want to send me the URL, then I


can pass it on to the team for them to double check.


But usually these kind of things just


take time in that we have to understand


the site and all of that.


But I'll take a look and pass it on to the team.


But it's not that there is a technical meta


tag or something that you need to implement to make that work.


Yeah, so all the metrics are basically the same.


So we have the same setup with AMP.


We have the same page experience.


And we see really good performance in Discover


and also in normal search and also with AMP,


sometimes in normal search.


But it's just about these top stories.


And for me, I read some rumors that it's harder


if you started a new platform after December 2019


to get into because there might have something


changed in Publisher Center.


OK, so you don't know.


I don't know.


But that sounds like Google News, though.


That doesn't sound like top stories.


Is it being shown?


Google News is also.


We are in Google News not often.


But it's really top stories is not at all currently.


OK.


Now, I think the Publisher Center changes and things


like that are more around Google News.


But they shouldn't apply to top stories


because any website should be able to be


shown in top stories.


I'll see.


Yeah, thanks.


Cool.


Let's see, Alex.


Hello, John.


And happy new year.


Alex here, coming from Greece.


So about a year ago, Danny Sullivan posted a blog post


on the Google Search blog about how autocomplete suggestions


are generated, which was great.


So it offered a lot of information and insights


about the predictions and how they work.


And then I was actually curious about the other sort


of related feature on the SERP, which are the related searches.


That's on the very bottom of the SERP.


And I actually tried to find some research,


tried to find some sort of similar information,


just like the one that Danny Sullivan


said on the blog post.


I couldn't find anything, to be honest.


And the only thing that I was able to find actually


came from Google Trends, where they have a similar concept


of related searches.


And the way they define it there is


that these are terms frequently searched with the term you


entered in the same session.


So the question is, does this apply also


to the classic Google Search?


And if not, so how are these related searches generated?


And where do they come from?


I don't think we have any documentation on that


specifically.


So I don't have any kind of hot lead for you


in that regard.


And usually, things in other products


are not necessarily the same across that product in Search.


So it's possible that Google Trends


uses something similar to what Google Search uses.


But I would not assume that it's the same kind of algorithm


or same setup for that.


But I mean, intuitively, it sounds


like it's kind of similar, like things search together,


kind of makes sense for related searches.


But I don't really have any details there.


And I think that's similar with a lot of features in Search,


where we tend to document them when they're really


kind of stable-stable, and they've


been working really well like that for a longer time.


But there are just so many different kind


of small aspects that we don't have documented individually.


AUDIENCE 1 OK, so that may be an idea for a future blog post,


then, hopefully.


Thanks very much, John.


JOHN MUELLER Sure.


OK, let me run through some of the submitted questions,


and I'll get to more of the people who've raised


their hands as well later on.


Let's see.


The first one I have on my list, since the number of features


are increasing in search results,


I guess that's similar to before,


I'm wondering if and how Google Search Console is including


rankings.


For example, the Google Map packs, or people


also ask in metrics like average position clicks, et cetera.


If not, what's the best way to see


if my website is ranking in these kind


of different features?


So for the most part, yes, we do include all of that


in the performance report data in Search Console.


So any time, or at least we try to do that.


Any time we show a URL from your website in the search results,


we'll show that as an impression for that website,


for that query.


The position, average position, also goes into play there.


And the average position is not like the average position


on a page, but the average top position.


So if your page, if your website is visible in position 3, 4,


and 5, for example, then we'll track 3


as kind of the position for that individual query.


So all of that kind of comes into there already.


What you don't see is, for a lot of these features,


a breakdown by the feature type.


So you can't go in and say, where is my website always


being shown within Google Business profiles


or within the map searches?


We don't show that.


But we do count that as an impression


for those individual queries.


So you could take those queries, try them out,


and see where your website is being shown,


and kind of try to follow it back like that.


Sometimes the different features in the normal search results


make things tricky to track.


For example, if we show an image from your website


on top in the images thumbnail in a normal search results


page, then we'll also count that as your website appearing


in the ranking for that query.


And if you look at the search results in a textual way,


then you might not see that immediately.


But all of that should come into play.


When we launch new features where we also list the website,


we do try to watch out to make sure that we also include that


in Search Console.


So it shouldn't be the case that we show a link to your website


and not track that as an impression


with the position and the clicks in Search Console.


Let's see.


We see that every JavaScript string starting with a slash


is interpreted as a URL and is followed by Googlebot.


Sometimes the URL is not valid, and we see different crawl


errors in Search Console.


Is there an official recommendation


on how to nofollow such URLs?


We used to split the strings into two or more parts.


Having millions of pages with such strings


may negatively impact the crawl budget.


So I think just kind of like the last question


or the last part of the question there with regards


to crawl budget, that's one thing you definitely


don't have to worry about.


Because when it comes to crawling,


we prioritize things in different ways.


And all of these kind of like random URL discoveries


that we come across where your URL is mentioned in a text


or in a JavaScript file somewhere,


those tend to be fairly low on those.


So if we have anything important that we recognize


on your website, any new pages that you link to,


any new content that you've created,


we'll prioritize that first.


And then if we have time, we'll also


go through all of these random other URL mentions


that we've discovered.


So from a crawl budget point of view,


this is usually a non-issue.


If you're seeing that overall we're crawling too much


of your website, then you can adjust


the amount of crawling in Search Console


with the Crawl Rate setting.


And again, here, we still prioritize things.


So if you set the setting to be fairly low,


then we'll still try to focus on the important things first.


And if we can cover the important things,


then we'll try to kind of go through the rest.


So from that point of view, if you're really


seeing that we're hitting your server too hard,


you can just adjust that.


After a day or two, it should kind of


settle down at that new rate.


And we should be able to kind of keep on crawling.


With regards to no following these URLs,


you can't really do that in the JavaScript files


because we try to recognize URLs in JavaScript


because sometimes URLs are only mentioned in JavaScript.


What you can do, however, is put these URLs


into a JavaScript file that is blocked by robots.txt.


And if the URL is blocked by robots.txt,


then we won't be able to see the JavaScript file


and we won't see those URLs.


So if it's really a critical thing that you're thinking,


oh, Googlebot is getting totally lost on my website,


then you could use robots.txt to block that JavaScript file.


The important part there is to keep in mind


that your site should still render normally


with that file kind of blocked.


So in Chrome, I believe you can just block that individual URL


and test it out.


But especially the mobile friendliness of a page


should still be guaranteed.


We should still be able to see kind


of the layout of the page properly


with that JavaScript file blocked.


So if it's only kind of interactive functionality that


is being blocked by that, then usually that's


less of an issue.


If it blocks all of the JavaScript


and your page doesn't work at all anymore,


then that's something where I'd say maybe you


need to find a different approach to handle that.


For a site that started ranking in multiple languages,


would you recommend changing from subdomain to subfolders


so that SEO juice gets shared?


The main language is English, not in a subdomain


and ranking well.


I don't know.


People argue about subdomains and subdirectories


all the time.


From my point of view, you can use either one.


So I wouldn't necessarily just blindly shift


from one to the other, but rather


think about what makes sense for you for the long run.


Sometimes tracking things is easier


if everything is on the same domain.


So if you use subdirectories, sometimes for legal reasons,


you have to use different subdomains


or you have to use different TLDs even.


These are all kind of considerations


to take into account.


And essentially with regards to the so-called SEO juice


or kind of the links to the website,


usually those get passed on with internal links anyway.


So that's less of an issue.


Can rel nofollow be used as noindex?


For example, when I publish an article on my website


and on every page where this article is mentioned,


I'll just use rel nofollow in the URL with that article.


So no.


It's essentially nofollow tells us not to pass any page


rank to those pages, but it doesn't mean that we will never


index that page.


So if you really want a page to be blocked from indexing,


make sure it has a noindex on it.


Don't rely on us not accidentally running


across a random link to that page.


So I would not assume that those two are the same.


And in particular, with regards to new content on the web,


I think Gary did a blog post maybe a year or so ago


about rel nofollow and the different types of other rel


attributes where he mentioned that we do sometimes


use this for discovery of URLs as well.


So on the one hand, we might see that link without a nofollow.


We might see it with a nofollow and still kind of look at it


anyway.


If you don't want a page indexed,


then make sure it's not indexed.


We published a landing page about a month ago,


and it hasn't been indexed yet.


I tested with the live URL and requested indexing a few times.


I understand indexing doesn't always happen quickly,


but this is the first time a landing page on our site


is not indexed after a couple of days.


So I'm wondering if there might be something I missed.


I also did all the checks I could


find to make sure nothing was preventing indexing, including


the Search Console test.


Everything seemed signed.


It's really hard to say without knowing the individual URLs


there.


We don't index everything on the web,


so it's completely common that for most websites,


we index some chunk of the website but not absolutely


everything on the website.


So that might be something that you're seeing there.


With regards to the amount of content


that we index from individual websites,


sometimes that relies a little bit


on our understanding of the quality of the website itself.


So if we think this is a really high quality


and important website, then maybe we'll


go off and try to crawl and index that content as quickly


as possible.


But there's no guarantee there.


So from that point of view, it's kind of tricky


to see what exactly is happening here.


What I might do in a case like this


is post in the Help Forum just to make sure


that there are no technical issues holding that URL back.


And then otherwise, give it a little bit more time


or see what you can do overall to improve


the quality of the website in general, which is usually


something that's more of a long-term goal


rather than something that you can just quickly tweak and hope


that Google will pick it up.


And tomorrow, everything will be different.


I'm looking at pruning some content on my site,


weak traffic being one of the criteria.


Would you consider the minimum, or what


would you consider the minimum acceptable level of traffic


to keep an article?


So I don't think just purely going off and looking


at the traffic to a page is enough reason


to say this is a good page or a bad page.


Some pages just don't get a lot of traffic,


but they're extremely important.


For example, I don't know, if you're selling Christmas trees,


then you probably expect those pages


to be kind of visible in the search results in December.


So if you look in January or March


and you look at the traffic to your pages,


then you'll say, oh, well, these Christmas trees are terrible.


Like, I should just delete all my Christmas tree pages.


But that's not really the right thing to do there.


I mean, these pages will be relevant at some point


in the future.


And similarly, other kinds of pages on your website


might get very little traffic, but they


might be really good pages.


And they might be really important kind


of pieces of information on the web overall.


So just purely going in and saying,


at this level of traffic, I will delete everything


from my website, I don't think that makes sense.


As a way of kind of fine tuning what you want to focus on


and then manually going through and saying, well, this is good,


this is bad, that's perfectly fine.


But just blindly saying, like, everything


below this traffic level is bad and I'll delete it,


I don't think that makes sense.


So I would not recommend doing that.


I'm curious about breadcrumbs.


What if I don't set the breadcrumb list in the list


element but set it as normal anchor text?


And then I use a breadcrumb list structure,


structure data on the page.


Is there any problem or a chance that my site


will get a manual action?


I don't know exactly what you're trying


to do with the breadcrumb markup there,


but it sounds like you're kind of splitting up the breadcrumb


pieces and marking some of them up with structured data


and some of them with a direct link on the page.


And usually that's fine.


When it comes to structured data,


our algorithms try to recognize what


is visible on the page versus what you just have marked up


for structured data.


And we want to make sure that the structured data is actually


visible on the page.


If some of these elements are linked as well on the page


or not linked, that's usually fine.


When it comes to breadcrumbs in particular,


I don't think the Web Spam team would watch out for this


and do any kind of manual actions anyway.


So that fear of being penalized for marking up breadcrumbs


wrong, you can totally forget about that.


That's not something I would worry about.


I would try to see how breadcrumbs work on your pages,


how you can mark them up properly.


Test things out, see how they look in the search results.


And if you're happy with how they look,


that's perfectly fine.


I would not worry about the Web Spam team taking action


on a website for marking up breadcrumbs incorrectly


because there's just so many bigger problems


to worry about on the web.


With the recent update giving a boost to extensive reviews,


won't this make it difficult for new e-commerce sites


to become competitive?


Also, does Google have a preference for native platform


review functionality or platforms like Yotpo?


I don't know.


So I don't know.


Won't this make it difficult for new e-commerce sites?


I think things are always changing on the web.


So it's really hard to say, will this


make it harder or easier for sites to be competitive?


And it's something where users expect different things


over time, and there are different kinds of sites


available over time.


And this is always a very dynamic relationship, I think.


And it's not so much that we're trying


to make it harder for new websites


to be visible in search, but rather we


see user expectations are going this way.


So we need to make sure that we align with what users want.


And that can sometimes drive what


we show in the search results.


With regards to new e-commerce sites


and being competitive, I think as a new site,


you always need to look at the bigger picture


and see where you can fit in, where you can provide something


that is both valued by users and something that is perhaps


missing from other websites anyway.


So from that point of view, it's always going to be hard,


especially if you're starting out fresh.


Then you have to look at so many different aspects there.


And how things are shown in search is one aspect there.


But there are lots of different things that can come into play.


And with regards to native platform review functionality,


I honestly don't know.


I don't think we would have any preference with regards


to particular providers or particular setups.


Usually, our systems are set up in a way


to be kind of active to kind of respond to the web


as we find it overall.


So if something is visible on your website in a way


that users can see it, that search engines can see it,


then usually that's what we're looking for.


And if that's handled in the background by some third party


or handled by yourself or even maybe hand coded or something


like that, that doesn't really bother us at all.


It's like we look at the pages how they end up on the web,


and we try to index them like that.


Every article on my site has a specific phrase


in the beginning.


Through a Google search for this exact phrase,


I found 3,000 doorway sites that stole my content.


And within six months, I had them


all removed from Google's index through the DMCA.


I did a great job, but it had no effect on my positions.


So question, does a significant increase


in the overall uniqueness of a site's content


have no effect on the site's ranking and visibility


in the search results?


Then is it not worth the effort to fight against content theft?


So as far as I know, there is no aspect in our algorithms


that says, oh, this is something that is


very unique to this one website.


And we will, because there is something very unique here,


we'll rank it higher for all kinds of other queries.


So if you're selling, I don't know, a unique type of shoes


and someone is searching for shoes,


then it's not that we would rank your site because it's


a unique type of shoes, but rather you have shoes.


This person is looking for shoes.


And maybe other sites also have shoes,


and we'll rank them based on the shoe content


that we find there.


So it's not a matter of us going through and saying,


well, there's only something very unique here.


Therefore, we should rank it higher


for this more generic term.


Obviously, if you have something unique


and someone is searching for that unique thing,


then we will try to show your site there.


And that's kind of the reason also


for things like the DMCA complaint process,


where you can say, well, someone else is ranking


with my unique things, and I don't


want them to show up because that's my content


or I have a copyright on it, at least.


And for that, the process makes sense.


But for the generic case where someone


is searching for something generic


and you have unique things that also map


into that generic category, I don't


think we would rank your pages higher just because they're


unique things.


And kind of from that point of view.


So I think if you're seeing that other sites are


ranking for your specific thing, for that unique thing


that you have on your website, and you


have a copyright on your content,


and whatever else aligns that you can use the DMCA process


for that, that's a perfectly fine tool


to try to help clean that up.


But it's not the case that we will rank your website higher


just because we've seen some unique things on your website.


In the case of a software brand using companies


like Medium for branded landing pages, such as brand.medium.com,


should we be worried about possible duplicate content?


If the Medium domain has more strength


for a particular subject or topic


and the page lists exact duplicates to the category


and blog type content on the brand site,


is it possible that Google would canonicalize


to the second domain?


If we differentiate the content, is that enough?


Or will these types of pages always


be boosted for brand results thanks to the apparent domain


strength?


So when it comes to canonicalization,


we use various factors to try to figure out which of these


is the canonical we should show and whether or not


we should even kind of follow a rel canonical


between these pages.


My understanding is that on this platform in particular,


you can set a rel canonical to your pages there.


So on the one hand, you're giving us


some signals that this is the right page to show,


like your original page.


But on the other hand, we might be finding other signals


as well pointing at the other page, kind


of the Medium domain landing page that you have there.


So from that point of view, it's never


guaranteed that we will only show your pages


or only show the other pages.


And it's definitely not the case that just because you're also


hosting it on this other provider's site


that their pages will always be shown.


So it's something where all of that kind of comes together,


and it's possible that we'll show one or the other.


From a practical point of view, usually this


doesn't matter so much because it's your content that's


being shown.


And it's not that your content would


be ranked lower because of these different versions


or the canonical that's chosen.


It's more that we'll either show this URL or the other URL.


And if you're saying they're canonical,


then you're kind of telling us that they're equivalent,


and it doesn't really matter which one of these


is shown in the search results.


So from that point of view, it's not


that you will be penalized for having duplicate content


or that it's a bad thing to host things like this.


It's essentially just it's not guaranteed that it will always


fall one way or the other.


OK.


Wow.


Running low on time already.


Let's see.


Maybe we'll go through some more of the hands that were raised.


I think, Christian, you're next on my list.


Hi, John.


Happy new year.


So I also posted a question before.


So I'm from Telos, and I wanted to know


how backlinks affect the authority of niche pages


with user-generated content.


And so it's about the problem with people


using spam techniques instead of if you're


working hard to generate good content


and how you should do it.


And I also added an example.


I don't know if you have seen it before.


I didn't see it.


OK.


I think it only shows if you started by a newest.


I posted it this morning.


I don't know why.


It looks like a bug.


Then it's like, yeah.


So I posted it five hours ago.


And the example is, so we generated


like a page which was linked from famous German newspapers


and everything.


And people liked this content.


And other companies liked it.


They liked this content.


And other competitors, they are just


like with anchor text of phone number.


And somewhere in the text, there's this number.


But all this stuff seems to be more relevant for Google


for this keyword.


So that's the question.


So how to, as a publisher, how to handle this problem?


I don't know.


I don't quite understand the situation.


So I can't load the questions.


For some reason, the YouTube page isn't loading.


So I don't know.


Can you explain a little bit more?


It's like the content is in the forum?


Or how is it?


No, no.


Sorry.


The content, the question is in the forum.


So I can tell you a little bit more


for a better understanding.


Sorry about that.


So it's like, so for a special search request,


so we are ranking our websites making phone numbers.


And there's like for a special phone number,


they are like competitors.


And they are using spam techniques


like if you just post on dev pages and on university pages


with an anchor link exactly to this keyword.


And it looks like that since December 2020,


Google prefers this kind of pages.


And we have the same keyword, but not exactly


because natural links never have exactly this keyword linking


to this page.


So we have articles doing a lot of insights,


have a lot of reports.


And famous papers are like trustful sources say,


OK, that's the best source to get information on the internet.


But it looks like that it's not helping


to get Google to understand that we are


like an authority in this area.


Now, I don't know.


I'd have to take a look at the examples.


If you want to maybe drop some example URLs or some queries


in the chat, then I can pick that up afterwards


and take a look.


OK, now in the chat.


Yeah, yeah.


OK, I will.


Cool, thanks.


All right.


And Dan.


Hey, hey, John.


So we have a bunch of pages that are dedicated to sports teams


that offer their upcoming schedules.


So there are lists of pages that show up upcoming games.


Like team A will play team B on January 12th,


coming up next week.


But what I'm noticing is Google actually putting


in the date in the search results.


But for example, it's putting like January 12th, 2011,


or 2012.


I'm assuming because that date is kind of in the future.


Google might just be pulling a date from a prior year.


Ultimately, we don't want date snippets to show up


in the search results.


I've noticed this probably like a month ago this started


to happen, and clicks have started to drop on those pages,


assuming that users might think that the content is severely


outdated.


They see a date from like 2013 or 2014.


So just interested overall to hear your thoughts


on how to remove them.


I know the data no snippet tag, if we just wrap the dates


around the dates, because ultimately we


don't want to remove the dates from the page itself.


So just interested to see or to hear your thoughts


if there's anything else other than the no snippet tag.


It's not showing up for every sports page that we have,


probably like half.


But just interested to see if there's any other way


that we can help avoid this problem.


I think that's sometimes tricky, because we pick up


dates on a page in a variety of different ways,


and we try to figure out if this date is relevant to the page


or not.


And I suspect we think it's relevant


for these individual pages, which is why


we show them in the snippets.


But usually if we pick up a completely wrong date,


then that means we're kind of not sure of that date


that we should be showing.


So one of the things you could do, or at least could try out,


is to use the date structure data, specifically


for articles, I think it is, to let us know of either the date


when you publish something and kind of put a publish


date on these pages so that we can pick up that publish date


and use that if we wanted to, or at least to notify us


of the actual date of the event so that if we do choose


that event, that we'll know, oh, this is, I don't know,


next February or whatever, next month kind of thing.


And we could show that in the search results.


So that might also be an option.


My guess is if we're picking up something like a wrong year,


then in some places on your pages


you have the right kind of like day and month.


But there's a number kind of like close by where we think,


oh, this is probably the year.


And you could also try to look at these pages and see,


like, is there a way to make sure that you don't have


other numbers close by so that we can be a little bit easier


with regards to recognizing things?


So I guess those are kind of the three aspects there.


On the one hand, the data and no snippet, like you mentioned,


to kind of hide these dates from the snippet.


Making sure that either the publishing date or the event


date is very kind of easily recognizable for us,


also with the right year.


And perhaps trying to make sure that it's harder


to recognize a random number on the page


as being a part of that date.


OK, cool.


Thank you.


Cool.


OK, let me pause the recording here.


And we can see if we still have more questions coming up.


But thank you all for joining in.


Thanks for submitting all the questions.


Sorry I don't see any of the remaining questions on YouTube.


So if there are questions there that we missed out,


feel free to add them to the next one.


I'll probably do another one of these next week.


And usually I add them a couple days ahead of time.


So feel free to copy your questions over if need be.


Search Console under the hood - Google Search Console Training (from home)

Hi, I'm Daniel Weisberg, Search Advocate at Google.


And I'm Hillel Ma'oz, Engineering Lead for Search Console.


First of all, thanks for joining me today, Hillel.


Thank you, Daniel.


This is the very last episode of the Search Console Training Series,


and we're going into the bits and bytes of the product.


And who better than you to talk about it?


Sure, Daniel. Always great to chat about what we do and how we do it.


Today, we'll talk about what happens under the hood in Search Console.


We'll focus on two areas.


How Search Console actually works and how it fits into the search ecosystem,


and how we think about features and go about building them.


Before we go into the nuts and bolts of Search Console,


it's important to say out loud what the team is trying to accomplish.


Our mission is to provide data and tools to help site owners


improve their websites and optimize their appearance on Google.


The product extracts important relevant information


from Google's search internal systems, reports it to site owners,


and enables them to improve their website performance on Google.


Now, you may be asking yourself, what exactly is a search ecosystem?


Daniel, can you take that one?


Sure.


First, there is the web, where millions of websites publish content all the time,


including you watching this video right now.


Second, there is Google, which crawls the web, finds content,


and stores it in the Google index.


It extracts all the relevant information from every page


so that it can retrieve the best results for user queries


on the Google search results page.


And third, there are search users looking for answers


and navigating from search results to their desired web pages.


Search Console is the main communication channel


between Google Search and site owners.


Basically, you.


The tool provides information on how your site performs on search,


and it provides technical information on potential issues


Google found in your pages.


From working together with the team from some time now,


I know they're always monitoring Google internal systems and data sets


for important website information.


They're also tightly integrated with other Google search teams


to make sure Search Console is ready for new features.


And of course, we keep our ears open to learn what are your needs.


There are numerous reports and features on Search Console.


We organize them into three main groups.


The Search Analytics group displays all the relevant user activity data


related to website performance in Google Search.


The Reporting group focuses on helping you track issues, debug,


and validate fixes on specific pages or entire websites.


The Configuration group manages site and user configurations


as well as alert settings.


Hillel, can you give us a little more information about each of those groups?


Sure, Daniel.


The Search Analytics group is the part of the team that works on retrieving


and reporting data from our internal systems


about users' clicks and impressions on search results.


There are billions of search results every day.


We process that information and store data for every site


for a period of 16 months.


This information is mainly surfaced through the performance reports,


where you can slice and dice the data to learn more about patterns


and anomalies in your Google search track.


The Reporting group is the part of Search Console that surfaces


how your pages are seen by Google Search.


For example, is Google finding all your pages?


Are there any errors that prevented Google from crawling your pages?


Are there issues with your structured data implementation?


Google crawls pages from millions of domains.


It looks for hundreds of signals supporting dozens of search features,


such as AMP, Recipes, and FAQs.


The information is available through actionable reports


that help you debug, fix, and inform Google


without a validation process for pages you just fixed.


We also offer testing tools to enable you to trigger


the entire search stack for any given URL.


For example, when you type a URL in Inspect URL or the rich results test,


we run a simulation of the Google index


and give you the information we find with high fidelity.


This enables site owners to debug any URL on their site easily and effectively.


Accounts is the group responsible for managing site and user configurations.


Their primary goal is to help site owners verify their sites


and get access to their accounts.


And of course, to prevent anyone that is not site owner


to see your Search Console information.


This group also handles all the emails that you receive from Search Console.


They do that for a dozen of other types across all the different reports,


making sure you never miss an important change or update.


Well, now that we know how the Search Console team is organized,


let's talk a little bit about how the team develops the reports


and tools that you use.


The process can be roughly divided into three main steps.


I know, it's the third time in this video that we've grouped something in threes.


Coincidence? Maybe.


The first step is to define what can help a page succeed on search.


We check that Google crawls the pages


and gets all the content the site wants to share with us.


For example, for topics such as mobile friendliness,


Core Web Vitals, or structured data,


we define all the signals that can validate their implementation in every page.


That leads us to the second step, which is to build a pipeline


to go over all the pages in the index periodically


and classify them against these signals.


If we find that a page fails for a specific website,


we collect all the relevant details such as the reason the page couldn't be crawled


or why we're unable to use its structured data.


And finally, the third step is to build reports to share this information with you


using the Search Console interface,


which is where you can see this information for your site.


For a complete picture, we publish dedicated documentation articles


where users can learn about these new reports


and better understand the information being displayed.


Keep in mind that Google is always evolving,


so new opportunities to shine on Search arise frequently.


Our job as a team is to keep up with these changes


and build an effective way for you to leverage new and existing search features


to the fullest.


Daniel, before we forget, there's one last point to discuss before we part.


The team is always looking for feedback from the search community.


You are a valuable source of information to help drive product strategy


and improve Search Console.


If you have feedback, make sure to submit it through the product


or mention us on Twitter.


Thank you so much for joining us today, Lel.


And thanks for all the work you and your team put into this.


It's helping lots of websites succeed on Search.


Hopefully now you will understand a bit more about how


and why the Search Console team builds features in the product.


If you like this video, make sure to check out the Search Console training series


in our YouTube channel.


You'll love it.


Stay tuned.


Oh, I just realized the holidays are coming.


Are you doing anything, Daniel?


No, not much, but I'll definitely take some time to rest a bit.


This year was challenging to say the least.


True.


So let's take this opportunity to wish the SEO community


a wonderful holiday season


and a happy, peaceful, and healthy New Year.


Removing content from Google | WMConf Lightning Talks

Hi, I'm Daniel Weisberg, Search Advocate at Google,


and welcome to another Webmaster Conference Lightning Talk.


Today, I'll talk about the different ways to remove content from Google Search,


including the Search Console Removals tool,


permanent removals, and other solutions.


This talk will be most relevant to website owners,


professional SEOs, or developers.


So here we go.


Removing content from Google


Removing content from Google can take many forms.


The content may be on your site or somewhere else.


And actually, you might want to remove some piece of information


from the entire web or just from Google Search.


Before I dive in, let me give you a quick overview


of what you can and cannot remove.


You can remove content from your website fairly quickly and easily.


You can remove it from Google or the entire web.


You can ask Google to remove content from its properties,


such as YouTube, Blogger, and others, if it violates our policies.


You can ask Google to remove content that violates its policies


if it lives on someone else's website,


but it will be removed only from Google Search results.


To decide how to proceed,


I think it's helpful to look at this as a decision tree.


The first question you need to ask yourself


is whether you control the content or not.


That's an important question because if you do control the content,


it's just a matter of using the right tools.


The second question you should ask


is whether you need to remove it permanently or temporarily.


If you need to remove something urgently,


you should probably start by submitting a request


on Search Console for temporary removal.


Then you can work on a permanent solution.


I'll talk about both options in a bit.


Back to the first question, if you do not control the content,


it might be more challenging to remove it.


As you can see in the tree,


for content hosted on a Google service such as YouTube, Blogger, or ads,


we provide a troubleshooter to help you remove the content.


For other websites, the best option is to contact the website owner,


but you might still be able to remove the content from Google Search


in some special cases.


So let's dive into the options and discuss each of them in more detail.


If you want to remove content that you control from Google Search,


we recommend one of three things.


Return a 404 or 410 HTTP response whenever the content is requested.


Protect your content with a password


so that it cannot be accessed by Googlebot or others on the web.


Or use a noindex meta tag to indicate that the page should not be indexed.


Simply redirecting an old page with a 301 response


or using a robots.txt directive


might not be enough to remove the old content from Google Search results.


Again, this is what you should do to remove the content from Search permanently.


So be careful.


If your intention is just to hide the page until you fix it,


you should use Search Console to do so.


A temporary removal request is a way to quickly remove content on your site


from Google Search results.


A successful request lasts only about six months,


which should be enough for you to find a solution


to either allow the content to be seen or to remove it permanently.


In other words, such a request will only hide the pages from Search results,


but they'll still be crawled and indexed


and will reappear when the request expires.


Before I show you how to submit a request,


I want to highlight some things that you should not use this tool for.


Removing a page won't affect if a page has a manual action,


if your website has been hacked,


if you want to tell Google which version of a page you want us to index,


if you're moving your site to another domain,


or if you want to remove pages that are not helpful or updated.


Don't worry. There are solutions to all those issues.


Check our Help Center to find how to fix them,


but don't use the Removals tool.


When you log into Search Console,


you'll find the Removals report in the navigation bar.


Select the temporary removals tab if it's not already selected


to submit a new removal request.


Then click New Request.


You have two options of requests available.