In general, we will crawl the sitemap file once.


English Google SEO office-hours from October 27, 2020

But if you're looking at the metrics in Search Console


and you want to see the overall metrics


and the individual metrics, then submitting that separately


is perfectly fine.


I don't think it would change anything.


How does Google treat a 302?


What URL does Google show in the search results, the original


or the redirected URL?


What URL will Google Search Console report clicks?


And what happens if we keep the 302 for a long time,


like months?


So these are all good questions.


These all go into the topic of canonicalization.


And when it comes to canonicalization,


essentially, the problem that we're trying to solve


is we know of multiple URLs that lead to the same content.


And it could be that it's exactly the same content, which


is usually the case with a redirect.


It could be that it's mostly the same content, which


could be the case if you have the same article


on different parts of your website, for example.


And with canonicalization, we try


to take into account all of the signals


that we have for those sets of URLs


and pick one of the URLs to be the representative URL


to use in search.


And that's the canonical URL.


And we use things like redirects, 301 and 302 redirects.


We use the rel canonical, internal, external linking


of the website, sitemap file, some other factors


like which of these URLs looks nicer.


And we try to compile all of those with different factors


and figure out which of these URLs


to pick as the representative URL.


And we'll try to use them.


Assuming everything else is the same


and you have one with a 301 and one with a 302 redirect,


probably in the beginning, we will


say 302 redirect means the original URL should


remain indexed.


And 301 would mean the destination page


should be indexed.


However, 302 is a temporary redirect.


And if you make that redirect permanent,


then our systems might say, well, it's


kind of like a soft 301 redirect in the sense


that you're not saying it's a 301,


but you're treating it as a 301.


So maybe we should take the destination page into account.


In practice, all of these kind of subtle things


tend not to matter as much.


It's really more a matter of the bigger picture.


It's like, within the internal linking of your website,


which of these pages are you linking to?


Within your sitemap file, which of these pages


are you actually referring to?


And all of these things add up.


And then usually, we'll try to pick the one where


you're really being clear on what you want to have indexed.


The change that you see in search


is just that we show this URL in the search results.


We will report on it in Search Console as well.


It doesn't mean that there's any ranking change.


So if you have two URLs and we pick the wrong one or the one


that you didn't want, it'll rank exactly the same.


It's just, well, we picked the one that you didn't want.


And if you look at Analytics or if you look at Search Console,


you have to think about the one that you didn't want


instead of the one that you actually did want to have indexed.


So it's more a difference with regards to reporting


rather than with regards to ranking.


SPEAKER 2


In the long term, Google has said evo potential in the long term


treat them the same.


But the question is, if you 301 redirect and undo the 301


redirect after a while, could it be the case


that Google afterwards wouldn't trust a 301 redirect and say,


OK, we've seen in this case, we moved over


and it came back to the original states.


I mean, how does Google treat it in this scenario?


Usually, the canonicalization algorithms


don't think that much to simplify it.


So it's not that they would say, oh, it's


like this guy is trying to be sneaky


and kind of like redirecting back and forth kind of thing.


The canonicalization algorithms tend


to look at the current state.


And they'll just say, oh, well, this redirect


is no longer here.


So this factor from the redirect no longer


accounts for that other URL.


And what do the other factors say?


And based on that, we kind of pick which


one to use as the canonical URL.


So it's not something that, from a canonicalization point


of view, we would see as something


that our algorithms have to kind of second guess.


But it's more we have all of these different factors.


We give them individual weights.


And then we calculate the factors together.


And which one of these URLs has a higher number in the end,


that's the one we'll try to pick.


And one thing that also plays into this a little bit, which


might be something that you would see in a case like this,


is that we try to be a little bit more persistent with


regards to the canonical URL.


So if you redirect somewhere and then you change back,


then our algorithms are not going to just swap back over


to the old one.


But rather, they'll say, oh, it's


like an unsure state at the moment.


But we'll keep the old one first.


And then over time, we'll say, oh, well, it's clear now.


This is actually a change in the preference from the site owner.


AUDIENCE 1 1 question.


And how long does it take for Google


to recognize three or two redirects?


I mean, I know that you can give an exact time,


but approximately how long?


I don't know.


To recognize them as being permanent, you mean?


Or?


Yes, exactly.


I don't know if we have any defined time for that.


OK.


I'm trying to think.


I think in the robots.txt documentation,


we have a similar case documented with regards to,


I think, 403 errors and 404 pages, where, like,


403 say you're not allowed to request this page,


and 404 would say this page doesn't actually exist.


I don't know if we have specified a time there,


but that might be something to look at to see if there's


a time there, kind of as a rough guess,


like what might be the case with 302 and 301 as well.


I don't think it would apply one to one,


but it gives you some idea of the order of magnitude.


My general guess is it's more a matter of months rather than


days, because sometimes 302 redirects just


happen to be in place for a longer period of time.


So it's still 302 redirects.


Thank you.


Sure.


All right.


We're kind of running low on time,


so maybe if any of you all have any questions left,


happy to help.


I have a question.


I joined the meeting about 15 minutes later,


and I don't know if my question, if you had time to see


my question that I said there.


Go for it.


Many questions here are regarding migrations,


and we have about the same issue.


We have two e-commerce websites on two different domains,


and we're migrating to one single domain, more


inclusive domain.


So we're basically migrating all the products, not


the categories.


So we're really interested in having the products.


Well, having a good ranking on the website C,


on the third website where we're moving the product.


So we have been thinking about how to do this,


and we came up with two ways SEO-wise.


So first one was using the Google Change of Address tool


and redirecting all the traffic from the products


from website A and B to website C.


And the second one was just leaving the website A and B


where they are and replacing the e-commerce buttons


with some links to the new websites.


So it is like you find on many websites a button called


Buy from Amazon.


It's just a link to the other page.


Now, my questions here, because we just


thought about these two options to do this migration,


are one, would you propose a better way


of doing this SEO-wise?


So the Change of Address tool, you


wouldn't be able to use there because you have two domains


that you're moving into one.


You would only be able to do it for one domain moving


to a different domain.


So that's kind of the first thing


that you'll notice if you try to implement that in practice.


You can still migrate like that, though.


You can set up 301 redirects from those two domains


to that one central domain.


That's essentially a legitimate site change that you can make.


It's kind of like combining multiple sites together


is always a little bit tricky because you don't know exactly


what the outcome will be.


But it is something where you can make this kind of change.


And usually, it ends up fairly well.


So that's probably the direction I would take.


The other approach with having links to that shared


kind of checkout site that you would have, I mean,


you can go down that direction.


But I feel in the long term, you're


kind of competing with yourself because you would have


essentially three websites for the same content


that you would have otherwise on just one website.


And by having it split out across those three websites,


you're kind of diluting the value of your content there.


So that's what.


I understand that, but here is my reasoning for that.


So if we go the other route and we just do the redirects,


the probability of the product, for example,


we have about two or three products


that are really, really good ranking,


like in the first place for the searches.


So we would not be able, if we would lose that ranking,


to complete the entire traffic just by PPC.


So SEO is really important.


So we can lose that for like 6, 9, 12 months.


It would be the end of the business.


So it's really important for us to do that.


And this was the reason.


This was a loophole that I thought


we could use at least for a while until website C gains


sufficient visibility in order to shut down website A and B.


So this was the reason why I proposed this version,


because many people that use this other approach of just


redirecting traffic report that the traffic just fails.


Yeah, usually it works well.


I think we kind of end up with all of the cases that


don't work well here, and you hear about them.


But usually that works out fairly well.


But I totally understand the worry


that this is something that could be critical.


I think the only thing you'll notice


is if you keep those two websites


and you also set up the third website,


is that the third website will have a really hard time kind


of gaining visibility in search, which


means you'll kind of have a really hard time being


able to make that cutoff change.


Because if you're not redirecting the value


from those existing websites to your kind


of shared common version of the website,


then essentially you're creating another website,


and you kind of have to work to promote that other website


as well.


So my worry there would kind of be that you can do that,


and you can kind of keep those individual pages,


but you will never run into the situation


where that third version, that new version,


is suddenly more visible in search


than the existing previous ones, because you never


kind of pass that value to that new version.


GIO NAKPILO, M.D. Yeah, we were thinking if those are the first


two A and B remain there, just like some blogs that just


redirect some traffic to the new one,


hoping that the new one will gain a better position.


And for example, if we see the third website getting closer


to the positions that the other one had,


we just take them up, take the first website down,


and do redirect to the new website.


And in the end, Google would just look,


probably Google would just look at the data,


we'll see that there's nothing there, just a redirect.


There's no data on the old website


because we just deleted the website.


And probably in the end, the new website


will take place, will replace that ranking,


and we will kind of keep our position,


but I'm not sure about that.


Now, I mean, this also seems like something


that you can test individually, where you can kind of take


that more kind of safe, I don't know,


I don't know if it's really safer,


but kind of the more kind of slow approach


of keeping those two websites and then doing,


on a per product basis, kind of setting up redirects.


And that would kind of give you a little bit more confidence


in is this new domain picking up the value from the old one


or not.


AUDIENCE 1 OK, so this is a good idea.


So I can try just to redirect half of the products


and maybe keep half of the products


with the buy from website C option


and see how that plays out.


Yeah, I mean, it's like when we talk with the engineers


about these kind of situations, usually they just say,


it's like, oh, you should just redirect everything


to the right version.


But I totally understand if people are a little bit,


I don't know, worried about a big change like this.


Because you can't really test it ahead of time.


Like you could test something from a usability point of view,


like do users like this one or the other one better.


A redirect is kind of like this one shot thing,


and then it's there or it's not there.


OK, thank you very much for your time and your answer.


Sure.


John, are you certain that you cannot move multiple sites,


you cannot use the change of address for multiple sites


towards a given domain?


Because I thought it was possible.


I think it was possible with the old version,


but with the new one that's there,


I think for like a year now, you can really just


have one change of address set up for a specific site.


OK, I do have a site in Search Console


that says two of your other sites are moving to this site.


Oh, OK.


I see.


But to be honest, this is the case for HTTP and HTTPS,


I think, or www and non-www or something


like that for the same domain.


So maybe it doesn't apply for different domains,


but just saying that it's possible.


Oh, you're using it for www and HTTP?


That's interesting.


Because I thought we kind of tried to check for that


and prevented people from doing that,


because it doesn't make that much sense as a site move.


But OK, interesting.


But I think that doesn't change much with regards


to this question where it's like, should I do it


or should I not do it?


Yeah.


Cool.


OK.


Well, thank you all for joining in.


Thanks for all of the questions that were submitted.


I hope you found this useful.


And the next one in English, I think, is lined up for Friday.


German one is lined up for Thursday.


A JavaScript one is tomorrow.


Wow, yeah, so many things happening.


All right.


Thank you all, and wish you all a great day.


Stay safe.


Bye.


Thank you all.


Thank you.


JavaScript page performance problems

Hello and welcome to another episode of SEO Fairy Tales.


JavaScript page performance problems

With me today is Bartosz Goralowicz.


JavaScript page performance problems

Did I not butcher your name this time?


JavaScript page performance problems

You're better every time.


JavaScript page performance problems

I keep getting better.


JavaScript page performance problems

Who is CEO at Onely and recently became a Texan by choice


JavaScript page performance problems

and is now learning to barbecue properly to blend in


JavaScript page performance problems

with the neighbors, aren't you?


JavaScript page performance problems

I am.


JavaScript page performance problems

The smells every evening are just overwhelming.


JavaScript page performance problems

So I can imagine.


JavaScript page performance problems

Another thing that might be overwhelming for people


JavaScript page performance problems

is JavaScript SEO, right?


JavaScript page performance problems

It's mine.


JavaScript page performance problems

Ooh.


JavaScript page performance problems

So I know that you have been into JavaScript SEO quite a bit,


JavaScript page performance problems

but I don't actually know the origin story of that.


JavaScript page performance problems

And I think it's linked to an interesting SEO case


JavaScript page performance problems

that you might want to talk to me about, no?


JavaScript page performance problems

In a way, Martin.


JavaScript page performance problems

That's a story that got us together for the first time.


JavaScript page performance problems

Right.


JavaScript page performance problems

Yeah, there has been exchanges beforehand.


JavaScript page performance problems

Yeah.


JavaScript page performance problems

So I felt like this would be a cool origin story episode.


JavaScript page performance problems

All right.


JavaScript page performance problems

And settling in.


JavaScript page performance problems

So I picked this story because, first of all, it


JavaScript page performance problems

got me into JavaScript, which is not something


JavaScript page performance problems

that you get by choice most often.


JavaScript page performance problems

Hey, whoa, whoa, how dare you?


JavaScript page performance problems

So like, JavaScript is amazing.


JavaScript page performance problems

JavaScript SEO, mate.


JavaScript page performance problems

OK, mate.


JavaScript page performance problems

Maybe JavaScript SEO.


JavaScript page performance problems

OK, I'll give you that.


JavaScript page performance problems

Maybe not in 2015 or 2016.


JavaScript page performance problems

Like, setting a scene back to back.


JavaScript page performance problems

Once upon a time.


JavaScript page performance problems

In a galaxy far, far away.


JavaScript page performance problems

No, that's a different piece.


JavaScript page performance problems

I had to do a quick math seven years ago-ish.


JavaScript page performance problems

Yeah, ish.


JavaScript page performance problems

We had a client.


JavaScript page performance problems

She came to us.


JavaScript page performance problems

She was a founder of this very kind of modern space,


JavaScript page performance problems

like modern fast startup, the fashion space.


JavaScript page performance problems

Ooh, fashion.


JavaScript page performance problems

Yeah, and I think I have an ask, but I


JavaScript page performance problems

think she was very innovative, judging


JavaScript page performance problems

by the choice of technologies and people around.


JavaScript page performance problems

And she had a very dynamic team of developers


JavaScript page performance problems

in some of the modern frameworks back then.


JavaScript page performance problems

And it seemed like a very small, exciting project.


JavaScript page performance problems

It sounds exciting, yeah.


JavaScript page performance problems

And it was like, maybe it's not the biggest client.


JavaScript page performance problems

Maybe it's not the biggest case.


JavaScript page performance problems

If it's an interesting one, whatever.


JavaScript page performance problems

Yeah, so.


JavaScript page performance problems

But the vibe was there.


JavaScript page performance problems

You know what I mean?


JavaScript page performance problems

It was like one of those.


JavaScript page performance problems

It felt right at the time.


JavaScript page performance problems

You would hop on Hangouts, and you would feel like,


JavaScript page performance problems

we get along well.


JavaScript page performance problems

Nice.


JavaScript page performance problems

OK.


JavaScript page performance problems

We were fairly, I could even say cocky about the project.


JavaScript page performance problems

It seemed very easy at the time.


JavaScript page performance problems

Just doing a bit of information architecture,


JavaScript page performance problems

polishing the website, web performance, things like that.


JavaScript page performance problems

How hard can it be?


JavaScript page performance problems

Web performance, one of the easiest topics out there.


JavaScript page performance problems

It was just a brand new website.


JavaScript page performance problems

So you didn't have all of those.


JavaScript page performance problems

Greenfield projects are fantastic, yes.


JavaScript page performance problems

So how did that go?


JavaScript page performance problems

I'm not sure if you're either, but yeah.


JavaScript page performance problems

So basically, we started.


JavaScript page performance problems

Everything was going well.


JavaScript page performance problems

We relaunched the website after a few months.


JavaScript page performance problems

I don't remember exactly months of work.


JavaScript page performance problems

So some of the implementations are done, and nothing.


JavaScript page performance problems

Like crickets.


JavaScript page performance problems

After two months, we're getting 700 visits per month


JavaScript page performance problems

to a startup that got quite a lot of investing.


JavaScript page performance problems

The metrics are not there.


JavaScript page performance problems

It got fairly awkward throughout the calls.


JavaScript page performance problems

And there was a lot of people advising her,


JavaScript page performance problems

saying this is a brand new website.


JavaScript page performance problems

You just bought the.com domain.


JavaScript page performance problems

You bought the history, obviously.


JavaScript page performance problems

I don't think it's possible to find one without one.


JavaScript page performance problems

And people are like, you need links.


JavaScript page performance problems

You need content.


JavaScript page performance problems

You need all of these aspects.


JavaScript page performance problems

Yeah.


JavaScript page performance problems

It's true.


JavaScript page performance problems

Yeah, it's true.


JavaScript page performance problems

They are wrong.


JavaScript page performance problems

On most websites, you can say that.


JavaScript page performance problems

But I wasn't very proud of the 700 visits.


JavaScript page performance problems

Which is reasonable, I guess.


JavaScript page performance problems

I took it to heart.


JavaScript page performance problems

And I was looking for clues.


JavaScript page performance problems

I was looking for something that would make it pop.


JavaScript page performance problems

Something that would.


JavaScript page performance problems

You know yourself, probably knowing you, Martin,


JavaScript page performance problems

that sometimes you quickly become obsessed with something


JavaScript page performance problems

when something just doesn't add up for you.


JavaScript page performance problems

Then it's a rabbit hole, and you have to go down.


JavaScript page performance problems

All right.


JavaScript page performance problems

I went very deep into that rabbit hole.


JavaScript page performance problems

Literally.


JavaScript page performance problems

And I started looking into technologies.


JavaScript page performance problems

Everything was there, so I went into Google Search Console.


JavaScript page performance problems

So you're here, too.


JavaScript page performance problems

And I used Fetch as Googlebot.


JavaScript page performance problems

It's working.


JavaScript page performance problems

Everything was OK.


JavaScript page performance problems

And to be honest, I have no clue how I got to this point.


JavaScript page performance problems

I tried to think that I was very logical


JavaScript page performance problems

and assumed that because of the tech-savvy developers


JavaScript page performance problems

and things like that.


JavaScript page performance problems

Should be fine.


JavaScript page performance problems

Tech side should be fine.


JavaScript page performance problems

It's all very innovative.


JavaScript page performance problems

And I spotted the problem of this rotating wheel that


JavaScript page performance problems

would show up when the website loading,


JavaScript page performance problems

the code was used on thousands of different websites.


JavaScript page performance problems

And I was like, what if I slowed down the Chrome


JavaScript page performance problems

as much as possible?


JavaScript page performance problems

And then I saw that the rotating wheel, when it shows up,


JavaScript page performance problems

not all of the code is sent to the browser.


JavaScript page performance problems

I was like, it works well in Fetch as Googlebot back in 2015.


JavaScript page performance problems

Now it wouldn't happen.


JavaScript page performance problems

And I started looking for problems around that.


JavaScript page performance problems

It was AngularJS.


JavaScript page performance problems

I went to the dev team.


JavaScript page performance problems

I was talking to them and saying,


JavaScript page performance problems

maybe this is a problem.


JavaScript page performance problems

And you can imagine what I heard back then.


JavaScript page performance problems

It was as close as I would state that the Earth is flat


JavaScript page performance problems

and just cancel all the files.


JavaScript page performance problems

MARTIN SPLITTENBERG


JavaScript page performance problems

It means nothing to search, but sure.


JavaScript page performance problems

That's what they heard.


JavaScript page performance problems

And they were very hesitant to even try it


JavaScript page performance problems

because I don't remember.


JavaScript page performance problems

I'm assuming something didn't look very well when


JavaScript page performance problems

the website was loading.


JavaScript page performance problems

But after a little bit back and forth,


JavaScript page performance problems

they kind of bought into this idea,


JavaScript page performance problems

especially that they became quite desperate for traffic


JavaScript page performance problems

with a large team.


JavaScript page performance problems

Long story short, after doing that, within days, literally


JavaScript page performance problems

days, the traffic went up.


JavaScript page performance problems

MARTIN SPLITTENBERG When you say doing that, what is that?


JavaScript page performance problems

I skipped a few steps.


JavaScript page performance problems

Let's dig into these steps.


JavaScript page performance problems

We disabled the wheel completely.


JavaScript page performance problems

So we just took out the code.


JavaScript page performance problems

That was a separate element.


JavaScript page performance problems

I'm not a dev, especially in Angular,


JavaScript page performance problems

but basically removed that part of it, all the dependencies.


JavaScript page performance problems

And we saw a huge uptick in traffic within days.


JavaScript page performance problems

I don't remember the numbers, but it was significant enough.


JavaScript page performance problems

Orders of magnitudes, basically, suddenly.


JavaScript page performance problems

Our calls were not awkward.


JavaScript page performance problems

That's the specific metric of how I described it,


JavaScript page performance problems

the volume of traffic.


JavaScript page performance problems

So there was some.


JavaScript page performance problems

But I wasn't happy.


JavaScript page performance problems

And I'm assuming you know why.


JavaScript page performance problems

MARTIN SPLITTENBERG Yeah, because it feels like magic.


JavaScript page performance problems

It's like smoke and mirrors.


JavaScript page performance problems

MARTIN SPLITTENBERG It shouldn't have worked.


JavaScript page performance problems

MARTIN SPLITTENBERG Exactly.


JavaScript page performance problems

This should not be.


JavaScript page performance problems

What is happening here?


JavaScript page performance problems

So you wanted to look behind the curtain,


JavaScript page performance problems

behind the spinning wheel, to find out what's happening.


JavaScript page performance problems

So what did you do then?


JavaScript page performance problems

MARTIN SPLITTENBERG I don't remember a lot of details,


JavaScript page performance problems

because I was in the rabbit hole still.


JavaScript page performance problems

The success didn't really help.


JavaScript page performance problems

I think it even pushed me further into that.


JavaScript page performance problems

MARTIN SPLITTENBERG Because it's frustrating to have something.


JavaScript page performance problems

It's bad to have something not working and not knowing why.


JavaScript page performance problems

But that's kind of part of the journey.


JavaScript page performance problems

Whereas if it works and you don't know why,


JavaScript page performance problems

it's scary and weird.


JavaScript page performance problems

It's scary because you're like, what if I anger the gods again?


JavaScript page performance problems

MARTIN SPLITTENBERG Yeah, especially if this is your daily job.


JavaScript page performance problems

And you're an agency, and we are so innovative.


JavaScript page performance problems

We believe we can solve all of the problems.


JavaScript page performance problems

MARTIN SPLITTENBERG Exactly.


JavaScript page performance problems

And now you have found one that you solved,


JavaScript page performance problems

but you don't understand why or how.


JavaScript page performance problems

MARTIN SPLITTENBERG We didn't really solve it.


JavaScript page performance problems

I didn't feel like it.


JavaScript page performance problems

MARTIN SPLITTENBERG You made it go away-ish,


JavaScript page performance problems

but you don't know why.


JavaScript page performance problems

So OK, so where did you go then?


JavaScript page performance problems

What did you do?


JavaScript page performance problems

MARTIN SPLITTENBERG I went to Tomek, our R&D.


JavaScript page performance problems

So our head of research and development.


JavaScript page performance problems

And we went into that rabbit hole together,


JavaScript page performance problems

which in retrospect wasn't the best idea.


JavaScript page performance problems

It took us a while.


JavaScript page performance problems

Anyhow, we've built a number of experiments.


JavaScript page performance problems

We've built multiple experiments, but one did stick.


JavaScript page performance problems

I believe it was due to a fancy graphic that was easy to follow.


JavaScript page performance problems

And if you say you build experiments,


JavaScript page performance problems

what would these experiments look like?


JavaScript page performance problems

How did you?


JavaScript page performance problems

OK.


JavaScript page performance problems

MARTIN SPLITTENBERG I'm glad you asked.


JavaScript page performance problems

I didn't know how deep into the woods we can get.


JavaScript page performance problems

MARTIN SPLITTENBERG I want to, like.


JavaScript page performance problems

MARTIN SPLITTENBERG I love it.


JavaScript page performance problems

So it was very simple.


JavaScript page performance problems

So we had jsseo.expert, so JavaScript seo.expert,


JavaScript page performance problems

basically a free domain that wasn't very expensive.


JavaScript page performance problems

And we've built every single subpage in different technologies.


JavaScript page performance problems

Oh, so you could see which ones get picked up


JavaScript page performance problems

and which ones work and which ones don't.


JavaScript page performance problems

I think the websites should still work.


JavaScript page performance problems

If there is something weird going on,


JavaScript page performance problems

it's Tomak and his experiments.


JavaScript page performance problems

But the website should still be up.


JavaScript page performance problems

Probably some modifications.


JavaScript page performance problems

Anyhow, there was like a placeholder of like a text


JavaScript page performance problems

for each subpage in bare HTML, a red frame.


JavaScript page performance problems

And everything within the red frame was JavaScript-generated.


JavaScript page performance problems

So you had to go through JavaScript


JavaScript page performance problems

and render that into HTML to see the content.


JavaScript page performance problems

And at the very end, there was a link within that red frame.


JavaScript page performance problems

So we would look into server logs


JavaScript page performance problems

and see if Google fell into the honeypot.


JavaScript page performance problems

OK, so when you say honeypot, basically


JavaScript page performance problems

the link is only injected with JavaScript.


JavaScript page performance problems

And it points to a thing that hasn't been pointed out


JavaScript page performance problems

anywhere.


JavaScript page performance problems

So it's the only place we can find out


JavaScript page performance problems

about this specific URL is in the JavaScript-generated link.


JavaScript page performance problems

So you know that if this page gets accessed by a Google bot,


JavaScript page performance problems

the link has been rendered, has been seen, has been understood,


JavaScript page performance problems

and is now being crawled.


JavaScript page performance problems

OK, got it.


JavaScript page performance problems

OK, interesting.


JavaScript page performance problems

Wow.


JavaScript page performance problems

It was, you know, now it might seem like super basic.


JavaScript page performance problems

But come on.


JavaScript page performance problems

You were probably one of the first people to do that.


JavaScript page performance problems

It worked.


JavaScript page performance problems

Yes.


JavaScript page performance problems

Call it a new project!


JavaScript page performance problems

It was veryBL jingle.


JavaScript page performance problems

We repeated multiple different experiments.


JavaScript page performance problems

And then, you know, after all that we said OK, we know more.


JavaScript page performance problems

And throughout that process, I've


JavaScript page performance problems

reached out to a lot of your colleagues,


JavaScript page performance problems

ilja Grigorik, sorry, Ilja, if you're ever watching this


JavaScript page performance problems

We've been we've been deep in the rabbit hole with Tomic and nobody could explain what's happened


JavaScript page performance problems

So at some point we're like this is weird and the most important thing I should have mentioned before


JavaScript page performance problems

Back then Google's documentation stated that Google renders JavaScript just like any modern browser


JavaScript page performance problems

so we're like


JavaScript page performance problems

Looking into that and you know, I just don't think it as I'm not me blaming anyone


JavaScript page performance problems

It was just very early days of job. Yeah, and the other thing is I think like generally that is not wrong


JavaScript page performance problems

It's just an oversimplification as usual


JavaScript page performance problems

We have to balance out how much into detail do we go and like where do we end up?


JavaScript page performance problems

And yeah, I see where this is. I like to believe that back then it was pre Martin times


JavaScript page performance problems

It was pre Martin times I believe


JavaScript page performance problems

Maybe this slipped through the crux maybe because it was very fresh with the new


JavaScript page performance problems

Frameworks probably like someone got very excited that we now run JavaScript like a normal modern browser would and


JavaScript page performance problems

ignored the fact that there are caveats and


JavaScript page performance problems

What's actually just to excuse Google just a tiny bit?


JavaScript page performance problems

Probably I'm gonna make some enemies with that but just to excuse you a bit


JavaScript page performance problems

It's it was fairly


JavaScript page performance problems

And fairly simple to miss because our experiment did show that you are the only back then search engine and their JavaScript


JavaScript page performance problems

But just not all of the frame. Yes. Yeah, I'm assuming there were some kings that needed to be I don't still are


JavaScript page performance problems

But what was this? Yes, this is difficult. Yeah


JavaScript page performance problems

Long story short we knew like what's up?


JavaScript page performance problems

Okay, and we came back to the client, right? So what exactly was happening there?


JavaScript page performance problems

We came back to the client saying some of your content still depends on this new modern


JavaScript page performance problems

technology called JavaScript


JavaScript page performance problems

angular JS and we wanted out yeah as an experiment and


JavaScript page performance problems

You didn't land well, let's like, you know, I come in saying we write


JavaScript page performance problems

Yeah, I was I was about to say like so we did all the work


JavaScript page performance problems

We're very happy with what we got from the users perspective and now a random person


JavaScript page performance problems

Over video call tells me burn it all to the ground. Yeah and build it again


JavaScript page performance problems

I'd probably murder you and just to add


JavaScript page performance problems

Drama


JavaScript page performance problems

All right, I should actually lead with that this was all happening in in Poland


JavaScript page performance problems

So it wasn't a very modern


JavaScript page performance problems

market where technical SEO would be


JavaScript page performance problems

Appreciated appreciated that much. Everyone was really focusing on bucklings back then and I was this kind of you know


JavaScript page performance problems

hipster guy who was like use


JavaScript page performance problems

Use this approach and just stop building links


JavaScript page performance problems

Anyhow after that call


JavaScript page performance problems

We


JavaScript page performance problems

Pushed them we pushed them very high. They would ask like what can we gain with that?


JavaScript page performance problems

We couldn't promise them anything. It's like it's gonna be better, but we have no


JavaScript page performance problems

No documentation, no case studies


JavaScript page performance problems

No, I'd like nothing to do to fall back on but it was like I really want you to do it


JavaScript page performance problems

Because I know you're gonna


JavaScript page performance problems

It's gonna bring you some kind of ROI. Yes, but also I want to see the result


JavaScript page performance problems

So long story short we we've pushed them we've pushed them


JavaScript page performance problems

That's actually the best word to use it because we kind of push them into doing it. It took them a while


JavaScript page performance problems

I don't want to tell me that it was like two weeks later. I'm assuming six seven months


JavaScript page performance problems

I don't go a while. Yeah a while


JavaScript page performance problems

And it was a slow process of just removing some of the dependencies. It took them a while. They relaunched the website


JavaScript page performance problems

We helped them with with migration to a new technology


JavaScript page performance problems

Nothing happens no and this was this was


JavaScript page performance problems

Like the awkwardness from the first calls was back like was when the twice yes


JavaScript page performance problems

It was squared or whatever. So like it went up and it went down and it went back up or didn't went down


JavaScript page performance problems

It's stagnated


JavaScript page performance problems

I think this would be the end of our our


JavaScript page performance problems

friendship


JavaScript page performance problems

it just didn't improve that much maybe like 2% something like that and


JavaScript page performance problems

We were looking heavily into like server logs


JavaScript page performance problems

We saw that you crawl them and you visit and we saw that Google is


JavaScript page performance problems

Indexing some of the like each ones you didn't index before some of the missing elements of like comments things like that


JavaScript page performance problems

So we saw that it's working. Mm-hmm


JavaScript page performance problems

But it took I don't like like three six months something like that few months for this change to be reflected fully


JavaScript page performance problems

Yeah, and that


JavaScript page performance problems

cross-frequency


JavaScript page performance problems

That was really rough. I'm assuming it's that you crawled the website fairly quickly. I mean, I think you recall the whole website within


JavaScript page performance problems

We didn't know which elements but we're looking server logs fairly quickly. It wasn't a huge website


JavaScript page performance problems

It was like let's say fifty hundred thousand URLs. Yeah, okay, which wasn't enormous


JavaScript page performance problems

but it just took a while but basically, you know with


JavaScript page performance problems

With the communication with how we approach them at some point and I think this is the biggest success we had with them


JavaScript page performance problems

We got them


JavaScript page performance problems

into this rabbit hole of


JavaScript page performance problems

You yeah, no, we know we want to know yeah, and they at some point they were like, you know


JavaScript page performance problems

We don't care about like let's just look into it like a bit more


JavaScript page performance problems

We're gonna explain that to investors at some point, but but that was that was quite interesting


JavaScript page performance problems

And


JavaScript page performance problems

Did you eventually?


JavaScript page performance problems

See more traffic coming in. So that's that's unfortunately the end of the story


JavaScript page performance problems

With the website they still are growing. Yeah, I think I've


JavaScript page performance problems

I've I think I've got some of the charts in the notes this video so we can show them they're still growing


JavaScript page performance problems

They're doing very well. We're very friendly with them good


JavaScript page performance problems

That they're a part of something that was quite important but until this day in our team


JavaScript page performance problems

this is something built into the culture into the core values of the company like I


JavaScript page performance problems

think


JavaScript page performance problems

with the wealth of knowledge we have right now because


JavaScript page performance problems

Us technical SEOs all it takes nowadays in 2022 is like tweet Martin with John and we can you know


JavaScript page performance problems

Find out quite a bit and it's we're always gonna be looking, you know checking back and there are some things, you know yourself


JavaScript page performance problems

We we don't fully agree on


JavaScript page performance problems

Probably it's gonna be like that and it's healthy in this way. Yeah


JavaScript page performance problems

but


JavaScript page performance problems

We still do that. We still experiment quite a bit. We here we still have a new client


JavaScript page performance problems

Something doesn't work as it should and I think this curiosity this kind of maybe


JavaScript page performance problems

It's not even curiosity, but it's like being very stubborn. Yeah, you believe in your data


JavaScript page performance problems

Yes, because technical is data-driven. It's not about I believe


JavaScript page performance problems

Yeah, exactly. Yes, you should try it out


JavaScript page performance problems

I think this is like the key takeaway from from your story really is


JavaScript page performance problems

If you're not sure what you're seeing or why you're seeing what you're seeing


JavaScript page performance problems

Experiment and I don't I find it very sad that I don't see many people experimenting


JavaScript page performance problems

It's easy to I I did an experiment once. Oh god


JavaScript page performance problems

I'm not sure if I should even because I remember when I started my career a lot of people were not just you other people


JavaScript page performance problems

As well, it came to me saying like it takes ages for JavaScript to be picked up. I don't know what's happening


JavaScript page performance problems

So I built two identical websites one was a fake photography agency like photographer specializing in dog photography


JavaScript page performance problems

So I had one website built with HTML all about dog photography and I had a fake photo. Yeah


JavaScript page performance problems

Come on, I'm not gonna talk about my site hustles. Then I had a fake photographer that was specializing in cat photography


JavaScript page performance problems

Okay, and that website was all built with JavaScript JavaScript frameworks


JavaScript page performance problems

everything was dynamically injected everything was was coming from an API and


JavaScript page performance problems

I was very surprised to see that a both were picked up equally quickly and B actually the link discovery


JavaScript page performance problems

So like the the spreading throughout the the site in terms of when pages were called sub pages were called


JavaScript page performance problems

That was a lot faster on the HTML website than it was on the JavaScript driven website


JavaScript page performance problems

but the JavaScript driven website


JavaScript page performance problems

Usually got indexed quicker and I I was like what and then I figured out okay


JavaScript page performance problems

Ah, so there's like processes running in parallel. How can we best explain that and that a lot of the documentation?


JavaScript page performance problems

Improvements have been based on that and other data that I found but from this you have you have to build experiments


JavaScript page performance problems

We still do like this is so easy. It's so cheap by the way, and so many people don't do it


JavaScript page performance problems

It's just yes, so very simple and with with ours. We didn't even create a copy


JavaScript page performance problems

We used there was like this weird software content machines like something that generates like something that generates like machine learning


JavaScript page performance problems

There was an article about that on like Bloomberg or something like that


JavaScript page performance problems

I was like, I'm gonna use it doesn't have to be good. It's just experiment if it burns and crashes doesn't matter


JavaScript page performance problems

It's an experiment and I think this is an important thing


JavaScript page performance problems

But what you said for every single experiment we publish there like 20 or 50 or more


JavaScript page performance problems

But we don't because they didn't work there embarrassing sometimes. Yes, but and it's right like exactly it doesn't it doesn't matter


JavaScript page performance problems

It doesn't have to be good


JavaScript page performance problems

Yeah, if someone finds them like you might link out from JavaScript as you JSS you that expert to some weird places


JavaScript page performance problems

That's okay


JavaScript page performance problems

We're open to you know collaborating to do that like but long search what they what I've learned with that and what we building


JavaScript page performance problems

If you feel like this should work


JavaScript page performance problems

This should be like the data tells you something and everyone else is like, you know advising you something that somehow


JavaScript page performance problems

Just my links like no


JavaScript page performance problems

And you need to know your priorities. This is what I'm trying to get. I'm sorry


JavaScript page performance problems

This is kinda a lot of things a lot of thoughts going through my head, but just every conversation


JavaScript page performance problems

Basically if you start with something it has to be the technical base you cannot start with links you cannot serve content. However


JavaScript page performance problems

They're crucial. I'm not saying like yes, you're gonna rank without those things. You probably won't


JavaScript page performance problems

But at some point I wouldn't feel like I'm gonna give the technology side a pass like the quality score the quality


JavaScript page performance problems

Whatever check mark we would we should start doing at one link. But basically I wasn't feeling that this is it and


JavaScript page performance problems

Most of the good things I've learned myself and I think most of our conversations come from place of deep frustration


JavaScript page performance problems

Yeah of place where we hate at times what we do. I


JavaScript page performance problems

Do sometimes?


JavaScript page performance problems

So frustrated with it that I'm willing to


JavaScript page performance problems

To to kind of go into that rabbit hole for a few weeks. Yes, baby


JavaScript page performance problems

Yes, not very social but solve the problem. So it's the problem whatever gets the job done


JavaScript page performance problems

But this that's really really an interesting point of view. Yes, you need to experiment


JavaScript page performance problems

You need to be aware that the technical foundation should be solid and you just look at the data


JavaScript page performance problems

Don't listen to what random people on the internet say look at the data or Google sometimes


JavaScript page performance problems

We are part of the random people on the internet group. So I don't exclude Google from that. So


JavaScript page performance problems

Yeah, that's a that's a very very good takeaway. So everyone out there do build experiments very important


JavaScript page performance problems

Bartosz it has been such a great pleasure to have you here. Thanks you so much for for joining us for this series


JavaScript page performance problems

Thank you, I'm just happy to see you face to face like likewise it has been covered I was like should I hug you?


JavaScript page performance problems

Should I say if you want to hug me?


JavaScript page performance problems

So nice to meet you again, thanks so much for


JavaScript page performance problems

inviting me to this


JavaScript page performance problems

I hope this is gonna be helpful if you struggle with experiments reach out to me or this guy


JavaScript page performance problems

He was super helpful when everything went out. We're doing our best


JavaScript page performance problems

Martin's patience every public interaction. I'll try my best. Okay


JavaScript page performance problems

But I definitely enjoyed the conversation you clearly did. I hope that you all enjoyed it too and


JavaScript page performance problems

With that I'd like to say thank you so much for watching


JavaScript page performance problems


English Google Webmaster Central office-hours from Nov 26, 2019

All right.


Welcome, everyone, to today's Webmaster Central Office Hours


Hangout.


My name is John Mueller.


I am a webmaster trends analyst here at Google in Switzerland.


And part of what we do are these Webmaster Office Hours,


where people can join in and ask any web search, website-related


question that might be on their mind.


A bunch of questions were submitted already on YouTube.


Not as many as sometimes.


I guess the holidays are here.


So if any of you want to get started with the first question,


you're welcome to jump in.


Otherwise, I'll just go down the list


and see what we have that was submitted.


OK.


Let's see.


I guess a complicated one here.


You explained there are several core algorithm changes released


in early November.


After those were allowed, there were many different types


of sites impacted.


So several updates make sense to me.


But as a result of several updates,


many smaller bloggers in certain niche categories


were impacted heavily, like recipe sites.


And when checking those specific sites,


you could clearly see many unnatural links


via recommendation widgets and other link building tactics.


For those sites, is it even worth disavowing those links?


It really looks like Google just simply devalued those links.


And if so, it seems disavowing would be useless.


Thanks for any information.


So I didn't take a look into any specific site there.


So it's kind of hard to say what exactly is happening there.


In general, if you look at your site in the way


that it's embedded in the web, and it


seems like there's really a clear pattern


of unnatural links associated with your site, that


could be because maybe you've been doing link building


in some kind of, I don't know, weird way


if you've been using widgets to build links,


all of the usual kind of things, then


that's generally something I'd recommend


trying to clean up regardless of any updates that happened.


And cleaning up link-related issues


usually involves either cleaning up those links


so that they're no longer out there.


In general, that's the best approach.


If these are widget links, for example, then sometimes it's


as simple as improving the widget


so that it doesn't have these links that are in there.


That's really the ideal way, is like removing those links


if you think they're problematic.


If you can't remove those links, then using the Disallow file


is an option.


That's one way for us to kind of drop those links from being


used.


And essentially, a third approach


that you could also take, depending on the type of link,


is to remove or block the page on your side


from being linked.


So what generally happens is when we have links to a website,


we associate them with individual pages.


So we have kind of the source of the link


and the destination of the link.


And with those two endpoints, we know which way these links go.


So if any one of those signs is removed from our index,


if they no longer exist, then essentially that link


loses effectivity.


So that could be another approach


that you could take there.


In general, usually, though, people


try to focus on either removing the links from the source site


or using the Disallow tool.


With regards to kind of this general situation


where you assume that an update has been affecting your site


based on the links to your site, that's


something where I would tend to be a little bit cautious


before jumping to conclusions and really take


a look at the links for your site


to make sure that there's really kind


of a pattern of unnatural links there that is really


problematic.


It's very easy to look at any website that's


been on the web for a longer period of time


and to find a handful of kind of weird and unusual links.


So that's something where just because you find something


weird doesn't necessarily mean that those links are negatively


affecting your site.


It might just be that these are kind


of the usual kind of crufty links that


get collected over the years.


But if you do find a pattern of really significant unnatural


links to your site, then that's something


I'd recommend cleaning up before maybe someone from the website


team takes a look at it manually and then applies


a manual action to clean it up for you.


So like a significant amount of links,


if it's like only like maybe 10, 20, it doesn't matter, right?


It's hard to give an exact number.


It's really something where if you look at the website overall


and you see that this is a really


significant and a clear pattern there that's


repeated across most of the links there,


then that's something I would take action on.


SPEAKER 2


stats in the old Search Console.


When I compare the URL counts from crawl stats


in the old Search Console to my access logs,


there are lots of differences.


So lots of details there.


And why is that?


How are these stats counted?


What's typically included?


What about rendering?


What about other kind of bots that access pages?


Yeah, good question.


I find it really hard to kind of reproduce


those counts because of the way that they're compiled.


And I suspect that's what you're seeing there, too.


So that might be something where it's really


tricky to try to understand the kind of exact connection


that you're seeing now.


In general, what happens there is


we include all of the accesses that


go through the infrastructure that Googlebot uses,


which does include Googlebot.


It does include rendering, includes robots.txt access,


or includes site map accesses.


It also includes some of the other bots


that are out there, like the ad spot, for example.


So it includes a lot of different things.


It's not just HTML pages that we request.


In general, I find looking at the aggregated logs like this


is kind of useful in the sense that it gives you


a bigger picture view of your website overall,


how many requests are being made,


how fast those requests are being responded to,


which is something that you often


don't see as clearly when you look at just the HTML files,


for example.


So that's something where I find it useful overall


to look at these, though it makes it trickier


if you're also looking at your log files


and trying to compare them.


I imagine, in general, if you're looking at the log files


and you see what kind of requests fall into this,


then that's something where you'll see the trends are


generally the same.


Depending on what you're tracking,


you might also see the speeds or the file sizes


that are involved there, which help you a little bit


to understand what all is really happening there.


So that's something where it's hard to track that back


one-to-one to specific requests that are made.


And for the most part, it's something


which probably doesn't make too much sense.


Let me see.


All of the things that you mentioned there,


JavaScript API calls, rendering CSS images, all of that


is in there.


What is not included in there, you mentioned


in one of the questions, is with different domains.


So that would not be included there.


So for example, if you have one HTML page


and it has all of the hundreds of images,


but the images are hosted on a different domain


or a different host name, then those


would not be counted there.


Those would be counted on the other domain.


So similarly, if you turn it around,


if you're the CDN for a number of websites


and they include lots of images that are hosted on your CDN


and you don't have any HTML pages at all,


then we'll still show those counts


as being requests made to your website


and show that in the crawl stats data.


So that's kind of from that.


One thing I thought I would mention just


because it's the crawl stats, someone put together


a really nice bookmarklet that lets you download


these stats as CSV files, which makes it a little bit easier


to track them over time.


It still doesn't make it easy to map one-to-one


to individual requests, but at least you


can download the numbers and put them into spreadsheets.


Hello, John.


Thank you.


Thank you for answering all my question.


A lot of information, thank you very much.


I'm not much smarter about this, but thank you for your time.


Can you tell me if you plan to, or you or your team,


plan to take crawl stats to new Google Search Console?


Or will it be there in some different, I don't know,


report or something?


I don't know.


I personally, I'd like to keep everything


that's still available in the old Search Console,


but I imagine the Search Console team


has to make some hard decisions along the way.


What might happen is that we could take these reports


and integrate them into something


a little bit easier to understand or a little bit


clearer actionable.


So one thing that people have been asking about a lot,


for example, is crawl budget.


Maybe it makes sense to create some kind of a report


around crawl budget.


And if we did that, then maybe it


makes sense to take these crawl stats


and include them in a way that makes them a little bit easier


to understand.


I don't know if that's something that is lined up


or will happen, be happening anytime soon,


but these are always discussions that we have.


And part of the directions that we end up


going for these kind of features,


it really depends on what kind of feedback


you get back from you.


So if that's something where you see this is really useful


and you can give us information on how you're improving


the web overall by using this feature


or by using this kind of data, then that


makes it a lot easier for us to go back to the engineering


team and say, hey, this is really critical.


We need to make sure that we build this out rather than


that we build it back.


So any kind of feedback or tips or ideas


that you have around that is always useful.


SPEAKER 2 OK, I will definitely email you


that information for me.


Thank you.


One information, just a note about the bookmarklet.


I really love it.


And I use it for the data in my comments.


So it's really useful for me too.


And can I have one additional question about the topic?


Sure.


I would love to know if how do you or I can tell


that the website or some website has


problems with the crawl budget.


Typically, I can see that from or I can find it out


when the mean of the duration time


is something like very high or something


or a lot of URLs which aren't useful are downloaded to crawl.


And maybe in some report, a discovery report,


I think it's called discovery report,


there is some column with URLs which are discovered but not


crawled.


I think I'm not sure about it, about the right name.


So are there any other possibilities


how to find out if the website has some crawl budget issues?


Yeah.


In general, most websites don't have issues


with the crawl budget.


So that's always the tricky part in that everyone


is kind of worried about this, or at least


the people who know about it.


And most websites don't need to worry about this.


Usually, they're, let's see.


So one way to kind of see what is limited there is essentially


if you see that the amount of pages that are being crawled,


doesn't match the amount of content


that you think you've been changing on your website.


So for example, with a news website,


if you look at your server logs and you


see Googlebot is crawling one third of all of the news


articles that I put out in any given day,


then that's pretty clearly a sign


that we're not able to keep up.


But that's sometimes really hard to determine.


So the two approaches that I usually recommend


is, on the one hand, looking at the speed


that it takes to download individual pages,


like you mentioned.


If that seems fairly high, then that's


often a sign that we'd like to be able to crawl more,


but your server is kind of slow, so we're not


able to do as much as we could.


The threshold there for fairly high is hard to say.


So it's something where smaller websites,


if they're fairly slow, then that


doesn't matter so much, because we get as much content


as we need anyway.


But larger websites, if they're fairly slow,


then it does matter quite a bit.


The numbers I see when I look across various websites,


usually the websites where we tend


not to have that much trouble with crawling,


I see speeds between 100 and 500 milliseconds per request.


So that's not per page that is rendered in Chrome,


but rather per request made to the website, which


includes things like the CSS files, JavaScript files, et


cetera.


And sites where I see issues with regards to the crawl speed,


usually the time per request is towards two seconds or higher.


So those are kind of, I mean, it's not


that we have fixed thresholds there.


This is more just anecdotal from what I've seen.


So that's kind of the one thing.


And the other aspect is server errors.


That also plays quite a bit into when we start to slow down.


So if we see a lot of 500 errors or 503 errors,


then usually what happens is over time we tend to slow down.


So that's something you can pretty clearly


see in your server logs.


In Search Console, it's a bit tricky to see as clearly.


But definitely in your server logs,


you can track the number of 500 type errors


and see if that grows or if you have spikes there


or if that's a fairly high threshold.


Because ideally, your server shouldn't be returning


500 errors regularly.


That should really be something that's more


of an exceptional thing.


So those are kind of the two aspects that play


into kind of how much we can crawl.


And the other aspect with regards


to how much we want to crawl, that's


something you can also control, which kind of goes into,


I think, in the coverage report, what you mentioned,


the discovered but not crawled URLs.


And usually looking at those, one way


to tell if there's an issue is to kind of look


at the pattern of those URLs.


If those are all URLs that seem reasonable,


that are kind of like clean URLs in the way


that you have your canonical shape,


then that seems like something where probably we could


be crawling a little bit more.


If those are all URLs where it looks from the URL structure


that they're kind of non-canonical URLs,


so maybe you have parameters attached to them,


maybe you have filtering and sorting parameters


or just generally long and complicated URLs


that you know are not the ones that you would have specified


as canonicals, then if we don't get to crawling those,


that's usually less of a problem.


But that could be a sign that you're linking to those


somewhere within your website.


And yeah, all of that kind of flows


into this complicated topic of crawl budget.


So on the one hand, if we can't crawl everything


that you're producing, that's definitely a bad sign.


If you're linking to a lot of URLs


that don't necessarily need to be crawled,


that's something that you can improve, kind of reduce


the number of URLs that you push to Google.


If you're seeing that the speed is fairly low,


that the latency to fetch a single request,


to process a request is fairly high,


that's something you can do.


And if you see a lot of server errors,


that's also a sign that we're probably


crawling less than we could.


So yeah, lots of complicated things.


And like I said, for the most part,


I'd say small to mid-sized websites,


this is not something that you really need to worry about.


It's really hard to do.


SPEAKER 2


Is that right now on AI and just kind of understanding what's


happening with websites, where they are in terms of seconds,


milliseconds, and then you're going to take that


as a ranking factor?


That's kind of the different kind of speed.


That's the speed to actually view a page.


And I believe that's just based on the Chrome User Experience


Report data, which is essentially what users


have seen in the wild.


So that wouldn't be related to the crawl budget type


questions.


That would be something different.


And especially on mobile, we do take speed


into account for ranking.


So that's something if your pages are really slow


and you see that in the speed report,


then that might be worth trying to find ways to improve.


But the speed in the speed report


wouldn't be related to the speed of or wouldn't


be related to the crawl budget type questions.


I have another question to the throttling topic.


OK.


You said about crawl budget.


When I see that you're crawling our page


and we don't get any 500s, does that


mean that you crawl as fast and as much as you want


and at some point you just stop?


Is that an indicator we can look at on our server logs?


Not necessarily.


So usually what happens is we make


a plan to crawl a collection of URLs from your website per day.


And I don't know.


These might be, let's say, 10,000 URLs


that we'd like to crawl.


And we don't try to crawl them all at once,


but rather we spread them out over the whole day.


So it's not that you would see a pattern of us crawling


in the morning and then stopping, saying,


but rather we would crawl kind of spread out.


And the main reason for that is because we don't


want to overload your server.


So if we were to crawl all of these URLs at once,


then that would be a pretty high load on your server.


Whereas if we can spread them out more,


that makes it a lot easier.


And especially if the server is a little bit slow,


so that we end up having multiple concurrent requests,


like requests that happen at the same time,


then that's something we want to avoid.


So we tend to spread things out if we


see that things are just slower than we would have expected.


Yeah, I'm thinking about it.


Because we are scaling based on our load.


So when you try or would try to crawl us,


I guess we would just scale.


And so the question would be if you would notice that


and stop then, or it would struggle down,


or would it still go up?


So we try to do this on a per day basis


and look at the whole day, kind of looking back


to see how the day went, and then


based on that plan for the next day.


So that's something where if you're


kind of scaling up in between in the day,


and some periods are faster, some periods are a bit slower,


that's generally OK as long as the aggregate


over the whole day is kind of within that range


where we'd say, well, this is still reasonable.


We can still crawl more.


And you'll see kind of day by day kind of the speed


go up when we think we can crawl more


and when we want to crawl more.


In a 2017 blog, you stated that basically increasing the speed


will increase the crawl rate.


If we have enough demand for more URLs,


then we will try to crawl more.


Yeah.


So.


But so that's kind of the one thing.


The one thing I do want to caution here,


though, is that it's very easy to focus on this


and to say the crawl rate is really important.


For ranking, it's not necessarily the case


that being crawled more often means


that you will be ranked better.


So I wouldn't see this as a ranking factor,


but really mostly as a technical kind of thing


where if you're changing your content


and you want that change to be reflected in the search


results, then being crawled fast enough for that


is kind of what you need.


Whereas if you're not changing your content,


then it really doesn't matter how quickly you're


being crawled.


So you don't need to kind of push Google to crawl more just


because more crawling is better type of thing.


It's really not the case that more crawling


means higher ranking.


OK, thank you.


Thank you for answering.


I saw a few websites which started to use AMP.


And this is the thing about the current budgets, too.


And the stats in crawl budgets looks


like the count of AMP growth typically grew a lot.


Like it looks like the crawl budget for AMP


is different to the crawl budget of the website.


It looks for me like this.


Can you say something on the topic


like AMP and the crawl budget?


That wouldn't change anything.


So those would be normal requests that we would make.


I could imagine that maybe AMP pages are smaller files,


so they can be transferred faster,


which means we could crawl a little bit better.


It could also be the case if you have a separate AMP subdomain,


and you're looking at just the crawl stats for that subdomain,


then those would look better because it's easier


to serve maybe static AMP content versus the dynamic full


HTML content.


But essentially, when it comes to crawling,


we don't special case anything around AMP or normal HTML


pages.


OK.


It's fair to me.


Thank you.


Sure.


All right, let's see.


Question about Search Console.


On the one hand, we have for our property about 3,500 pages,


valid pages, 1,900 index, and 1,600 index.


In coverage under mobile usability,


we only have 1,400 valid pages and 55 errors.


As we had until the beginning of October 2,


a desktop and a mobile site, then


we switched to dynamic serving.


I would expect that this should clear out


the difference in counting the valid pages,


but it's still there.


Can you explain to me where this difference is coming from?


OK.


So I think one of the big challenges


with the aggregate reports in Search Console,


the mobile usability one, other reports as well,


like the rich results or the AMP reports that are there,


is that they focus on a relevant sample of the URLs


from your website.


So if you look at the coverage report,


that includes all of the URLs that we


have indexed for your website.


So not just kind of a relevant sample of them,


but rather really everything across the whole index


that we have.


And that means that it's hard to compare the total numbers.


So the coverage report might say, like in your case,


maybe 4,000 valid and indexed pages.


And if you look at the mobile usability report,


the total might be 2,000 or 1,000 valid pages.


And that's not the case that the difference is not


mobile-friendly or kind of matching the mobile usability


criteria.


But rather, we took a total of 1,000 pages from your website,


and we reviewed those specifically


for these criteria.


So that's something where you might see kind of like the 4,000


that we have indexed, 1,000 we check for mobile usability


issues.


And of those 1,000, some of those


will be kind of valid with regards to mobile usability.


And some of those might have individual errors.


With regards to mobile usability,


so that's something where adding up


those pages from an aggregate report


generally won't give you the totals of the actual crawl


or the actual indexing report.


In general, since we try to take a relevant sample


of those pages, if you're seeing minimal amounts of errors


on the sample that we show in that aggregate report,


then that's generally signed you're OK.


So that's not something where I would worry about and say,


well, what about those other URLs that are not checked?


Does that mean that maybe they're all bad?


And usually what happens is that that relationship kind of


stays the same, and most of them, like in your case,


are OK.


Maybe a small part are not OK.


The other thing that is specific to the mobile usability report


is that we determine the mobile usability based


on rendering those pages.


We have to render those pages in a way that


matches what a user would see in their device.


And sometimes that doesn't work out.


So sometimes things like the CSS file we can't touch properly,


or the JavaScript file we temporarily can't fetch.


And even if that's just a sample of all of your index pages,


sometimes that can happen.


And with that, you'll always see kind


of a small amount of issues with regards


to mobile usability in that report, which are probably


mostly based on just kind of temporary fluctuations


in our ability to fetch individual pieces of content.


And you can double check that by going


to the mobile usability report and taking some of the errors


that you see there, do a live test.


And if you do a live test and it says, oh, it's OK,


then that's generally a sign that this was probably


just some kind of fluke with regards to the testing,


some kind of flaky testing that was happening


that isn't really something that is actionable for your site.


So in particular, specifically with regards


to the mobile usability report, I


would not assume that this is something where you can


trivially get the number of errors


that we show down to exactly zero,


just because some amount of flakiness with regards


to requests made to a server, that is kind of natural.


That happens anyway.


That doesn't affect the indexing.


If it were to affect the HTML pages,


we would just refresh those and try to get that again.


So it's really only with regards to the mobile usability report


there.


So that's one thing to keep in mind there,


that some of these errors might not be actual errors as much


as they could be just fluctuation in the network


connections that can sometimes happen.


Let's see.


OK.


An easy linking question.


In your opinion, which ultimately


is best to use when doing internal linking,


absolute or relative URLs?


Ultimately, I mean, your site has correctly implemented


canonicals and has a single uniform domain being used


and no duplicate domain issues.


So in that theoretical case, where


you have a theoretically perfect website,


then it doesn't matter at all if you use


absolute or relative URLs.


So from that point of view, use whatever is easier for you.


Oftentimes, relative URLs make it easier


to test things locally.


So maybe that's better.


That's not something I would really worry about there.


So I really would leave it up to you.


And if I were working on this website,


I would just see which of these ones


were easier in this specific case with whatever CMS


I'm working on and just use whatever makes sense there.


In the case where your website is not


this theoretically perfect structure, which probably


most websites are not, then working with absolute URLs,


if you can make sure that they really


point at the canonical versions of all of the URLs


that you have, probably makes a little bit more sense


because then you don't have to worry about things like,


what if Google or some user ended up


accessing the non-dub-dub-dub version of your website


and it was loaded?


With absolute URLs, we always find our way back


to your preferred version.


In practice, you can also work around this


by using the rel canonical.


And we can generally figure that out there anyway.


So in the theoretical perfect situation,


use whatever makes sense.


In the kind of realistic situation,


I'd still say use whatever makes more sense for you.


One thing that sometimes comes up


is that people try to use absolute URLs as a way


to kind of fight against scrapers.


And from my point of view, that doesn't really


work that well in that most scrapers know


how to deal with URLs.


So absolute or relative, they'll get around that anyway.


And probably there are smarter things


that you can do to kind of work against scrapers


if you're seeing that's happening with your website.


Let's see.


How to remove the content of the expired domain from Google.


If the user publishes expired domain's content


on his own site, another one, how will Google treat it?


So I'm not quite sure which direction


those questions are going.


In general, if you take over an expired domain


and you want to kind of start fresh with your new content,


then essentially you just make sure


that the old content returns 404.


And over time, as we recrawl those old URLs


and see that there's none of the old content left anymore,


then that will be reflected in search.


I would not recommend using the URL removal tool for this.


In particular, don't use the URL removal


tool for your whole domain because it will not


reset your domain.


It will just hide the whole domain


from search, which means as you publish new content,


that will be hidden as well.


So don't use the URL removal tool


if you're just taking over an existing domain.


So that's kind of the one side there.


On the other hand, if your domain expires


and you forgot to renew it and you'd


like to keep publishing the content, then ideally what


you would do is, I mean, in the ideal situation,


you would set up redirects to the newly hosted version


of that content.


If the domain has expired in the meantime,


then you can't set up redirects.


And in a case like that where you can't set up redirects,


then essentially your copies of the old content


are essentially new URLs on the web.


And our systems have to discover them first.


We have to find links to those pages.


We have to crawl them.


We have to index them like any other piece of new content.


So there is really no way for you


to say, without setting up redirects


or without using things like rel canonical,


that this new version of the content replaces the old one.


Instead, what we will see is, well, there


is a version of content here, and the old version is gone.


And we see something new.


Something has gone away.


It's not really clear to us that the something new


should replace the thing that went away.


So that's kind of an unfortunate situation


if you let your domain expire and then


decide that actually you do want to keep the content indexed.


I have maybe a question related to that.


OK.


It's now about the whole web page.


What is when a lot of URLs get indexed, but they shouldn't?


And afterwards, the whole folder is blocked by robots.txt.


How can you get those URLs indexed without changing


the robots.txt?


So you want to get those URLs indexed,


or you don't want to get those URLs indexed?


No, they are indexed, but I don't want them to be indexed.


And that's quite a lot of URLs.


So with the robots.txt, we would not crawl those pages.


So if you have a noindex there or if you


have server-side authentication, then we would not see that.


So that's kind of one problematic thing there.


What generally happens is if these URLs are blocked


by robots.txt, then over time, we


will only index the URL, not the content anymore.


So if someone searches for something


that used to be kind of indexable under that URL,


then at most what might happen is


we show the roboted URL in the search results.


Usually, this is less of an issue


because if these are topics that you actually care about,


then you would have a version of content


on your website that is indexable with more information.


So for example, if you have a page about, I don't know,


blue cards, and you have one version blocked by robots.txt


and one version that is still live, if someone searches


for blue cards, we might know, based on maybe links


to the roboted page, that this is also a blue card page.


But we definitely know that the version


that you allow us to crawl and index is about blue cards.


So that would probably be the one


that we would prefer anyway.


So if there is a version of content


that is crawlable and indexable and one that


is blocked by robots.txt, then probably we


would just show the one that is crawlable and kind of indexed


already.


If you need to get those old URLs out completely


so that if someone does a site query


and explicitly looks for those URLs,


then you would need to use the URL removal tool.


The URL removal tool can be done on a folder level


or kind of a prefix of a URL level


or on an individual URL level.


For a lot of URLs, if you have several thousand URLs,


then you would try to find a folder or a prefix that would


work to remove all of those URLs.


If this is a really urgent situation


and you have some URLs that are actually indexable


within that folder as well, then that's kind of a call


that you might have to make where you would say,


well, maybe I removed the whole folder,


even though I know there's some content I actually still want,


just to make sure that the critical content that I


definitely don't want to have indexed


is also removed from search.


So that's kind of a, I don't know, rare situation


that we don't see that often.


But that can happen as well.


OK, thanks. Another question related to that,


would that go also on the crawl budget?


If it's blocked by robots.txt, no.


If the URL is blocked by robots.txt,


then we would not count that as something that we try to crawl.


OK, cool.


And that means over time, let's say half a year or something,


those URLs would also be automatically on noindex


because you can't crawl them.


Not necessarily automatically noindexed,


especially if we find links to those pages.


So if we find a lot of links to those pages, what will happen


is we will index just the URL.


And we might show kind of as a title in the search results


something from an anchor to that page.


So that's something that can happen.


But like I mentioned, if you have normal content


for that topic, then we would probably


prefer that over the robot page.


All right, thank you.


Sure.


I noticed that Chrome console is listing scripts


that are not using the same site attribute,


that those are blocked.


And recommending more about it here.


So there's a link to Chrome status, I think, blog post.


Can you explain what the harms are of not doing anything


in this case?


So I had to look at this post as well to kind of see


what exactly is changing here.


But it looks like it applies mostly to cookies.


And that's something that generally from a search point


of view doesn't play a big role.


We don't keep cookies for individual pages.


So we wouldn't forward those cookies or reuse them.


So that's something from a crawling and indexing


point of view, essentially it doesn't play a role for us.


It can affect how users see your pages, of course.


Any change in a common browser can


affect how users see your pages.


So it's not something I would completely ignore.


But I wouldn't see it as something


that is specific to search.


One thing also to keep in mind here


is that for Google rendering, we're


using kind of the evergreen Googlebot now,


which means we update the Googlebot Chrome version


regularly as well.


So similar to you, when you get that warning


that there's a new version of Chrome available,


Googlebot gets updates as well.


And if there are changes in Chrome


that affect how a page is viewable,


then that would affect Googlebot rendering as well.


So for example, the kind of, what is it,


one of the changes recently about HTTPS mixed content,


that's something that generally I


think has been like that in Chrome


for a longer period of time.


But if that were to come completely unexpected from one


day to the next, that users in Chrome


would be able to see your images, for example,


then that would also mean that Googlebot would not


be able to see those images and would not


be able to render those pages that


use kind of that functionality, which


is now blocked in the browser.


That would also be blocked for Googlebot as well.


So on the one hand, these things primarily play a role


with regards to your users.


On the other hand, if this functionality


is critical for finding content that is indexable


for your pages, then that could play a role for rendering


as well.


I feel that's kind of a confusing answer.


Maybe I need to rephrase that at some point.


I am looking to improve the visibility of my product pages.


Many of our PDPs include large number


of images, which each include large amounts


of explanatory text with them.


So my concern is that the associated text, while great,


might be diluting the identity.


In a lot of cases, the H1s are generally too generic.


And the text below, in terms of source delivery,


provides better clarity about what the product actually is.


Do you think moving the images down and the relevant text up


would help Google to better interpret


the focus of each of these pages?


So just shifting the location of content within an HTML page,


I don't think that plays a big role at all.


Headings are useful in that we can take a heading


and see which images and which text


kind of applies to that heading.


But just shifting around things with HTML or with CSS,


I don't see that playing a big role there at all.


So in that regard, I wouldn't really worry about this.


One thing I did notice when looking at that example page


that you linked to is that when I load it up


after a certain period of time, it


switches to kind of a country picker interstitial.


And I don't know how you're kind of triggering this


and if you trigger this in all locations.


But for example, if you were to trigger this


when Googlebot crawls and renders your pages,


that might also result in Googlebot not being able


to index your pages properly.


So that's one thing you might want to double check.


In general, I recommend using kind of like a banner instead


of an interstitial when it comes to things like country


or language pickers.


Because if you're using a banner, then


even if that does end up being rendered in Google systems,


then it wouldn't block the indexing


of the rest of your content.


Whereas if you have an interstitial that,


in the worst case, kind of takes out all of the old content


and replaces it with this kind of country or language picker,


then we might not have much content


left on that page for indexing.


OK.


Let's see, one more question because it's a Swiss-related


one, I guess.


We're fixing a website with a ch domain.


We have a well-running clone under a.com domain.


If we take the ch domain offline for a few weeks


and redirect all its traffic to the.com domain,


what effect does this have on search and ranking?


So I assume taking it offline and redirecting


happens at the same time.


Whereas if you take a domain offline,


and then after a certain period of time,


then you finally end up redirecting,


then that's a little bit trickier.


But generally speaking, if you just redirect,


then we would see that as a domain move


and try to move all of the signals


that we have for the old domain, for the ch domain,


to the.com domain.


And that's something where if you


don't change the structure of your website,


then generally that's easy for us to process.


You can use the site move tool in Search Console


to tell us a little bit more about that.


But generally, we can do that fairly quickly.


The one thing I would watch out for here


is you're going from a country-level domain


to a generic domain.


So the ch is specific for Switzerland,


which means there is geotargeting happening


specifically for users in Switzerland


when they search for local content.


And with the.com, by default, that's kind of generic.


So that could be geotargeting any specific country.


So what I would recommend doing here,


if you're moving from a country domain to a generic domain,


make sure you definitely have the geotargeting settings


set in Search Console.


And like with all kinds of moves,


double-check all of the other things


that we have listed in the Help Center as well.


So there are some things which are kind of obvious that


get forwarded there as well.


But things like the ownership, make


sure that you have all of those transferred,


all of the settings transferred to the new domain.


If you have the parameter handling settings set up,


if you have any specific removals that are set up,


make sure you have all of that moved over to your new domain


as well.


So that's kind of what I would watch out


for there with regards to the overall effect on search


and ranking.


In general, a kind of one-to-one domain migration


is pretty easy for us to process and shouldn't


have a visible effect.


You might see a few days where things are shuffling around.


But generally speaking, that should settle down


fairly quickly.


But it's not guaranteed.


So in particular, if the new domain


has a really different history associated with it


or really different settings associated with it that


take a longer period of time to settle down again,


then you could see effects where we would shift over


to your new domain.


But your site overall on that new domain


is not as visible in search as it used to be.


On the other hand, it can also happen the other way,


that maybe you have a really good domain


or your old domain is really bad and you're moving things over.


Then maybe things are a little bit better.


OK, so maybe it's time to switch over to questions from you


all as well.


Looks like there's a bit of back and forth happening


in the chat.


What can I help with?


So go for it.


So can I still ask the same question regarding that site?


Like, if it ranks for its own name, is that still OK?


I mean, it ranks for its own name,


but doesn't rank for keywords.


So it obviously is very suspicious.


And at the same time, I've also noticed that Cloudflare


has been blocking certain things from the CDN.


And I also recall a chat that you had on Twitter with Eric


Wu, I think, mentioning that Cloudflare does


block Googlebot.


And so I've disabled all that stuff,


but I'm noticing now that the CDN is blocking particular things


CSS within the site.


But I'm seeing now the site ranking for its own name


under its own.


So I don't know.


I mean, what is happening here?


Could it be also Cloudflare that's blocking?


I say probably not, because these kind of technical issues


with regards to the access are more specific with regards


to indexing the actual content.


But it's very frustrating, because I'm


seeing other major local brands that I look at ranking fine.


Within mobile and desktop, it can't


be that this is the only one, John.


Of course not.


There's lots of, I mean, that's something.


So I think maybe just generally, in a case like this,


where you're seeing the content being indexed,


and it's the same content that's used


for the mobile and the desktop versions,


we don't have an m-dot version of the page,


then I wouldn't worry about those technical indexing


blocking type issues, because that essentially means


we're able to index the content.


And we index the content once for desktop and mobile search.


So it's not that we have a desktop version


indexed and a mobile version indexed.


If we can't get one, then suddenly it


doesn't rank in that one anymore.


If it's indexed, then essentially it's indexed.


That doesn't mean that maybe we could crawl better


if the hosting environment was set up differently.


That's kind of a different question.


For smaller sites, like I said, usually that's


not a problem anyway.


And what I recall from Cloudflare,


I don't know which thread you're referring to,


but what I recall from W.


F. There was a W. A. F.


What I recall there is that they do


block bots that act like Googlebot that are not actually


Googlebot.


And from our point of view, that's fine.


So in particular, if you're using some kind of tools


to crawl a website and you're using a fake Googlebot user


agent, then if the hosting environment blocks that,


then that doesn't really affect us


as long as the real Googlebot is able to get through.


So that's kind of that side there.


But in general, when it comes to desktop and mobile,


especially for things like local businesses,


there are sometimes significant differences in the way


that they're visible in the search results.


And sometimes that plays a role with the elements on a page,


things like how many maps entries


can we show for these queries.


That's something that you would also see in Search Console,


where the ranking would essentially


take those elements and be accounted as well.


So I think it's very important.


SPEAKER 2 How is it OK, though, for their competitors


to rank both on mobile and desktop with very even


positions, which is one to one?


There's no differences between their competitors' search


and mobile and desktop.


I've been doing this 17 years.


I live this life, right?


I do believe it's still a technical error.


I literally


Yeah, I mean, I would definitely take a look at that again.


But it's really something where there are just


differences with regards to how people search


on desktop and mobile.


And that is something that is reflected in the search


results.


So kind of the elements that are shown, sometimes the ranking,


the understanding of the query, it can differ quite a bit.


And especially when you're looking


at queries that don't have a lot of impressions,


then looking at the data in Search Console


can be a bit tricky.


Because when there is an impression,


because someone was searching for it,


then suddenly there's data there.


When there is no impression or people are searching for it


in a non-local way, then that might


be that there's a different kind of data there, which makes


it really hard to double check.


So especially with local businesses,


if you're looking at things like local-related queries,


then that can be really tricky to check when you're not


in that specific location.


So take a query like, I don't know,


some kind of business near you.


That's something that people might do on mobile,


where they expect things that are really close by.


But if you're in a different town or in a different country


and you try that query, then that's


something where you would see drastically different results.


And that's kind of normal.


And in Search Console, if we see that site mentioned somewhere


in the search results, we would still


track that as an impression, even though it's not


necessarily an impression from a user that you would care about.


So even if you take a query like, out of 55,300,000


like Flowers Toronto, and I see the first result on mobile


number one, and the desktop is also number one.


And that's a very competitive query.


That query is less than a million search.


It's a very small town.


But anyhow, I see it as an issue.


And I don't know, it's frustrating.


When everything is done correctly,


the speed is correct, even though I'm not sure why,


but the speed is showing low.


The desktop pages are showing low.


I mean, but they're not fast.


Clearly in Page Insights, you see they're in milliseconds


and also in Lighthouse as well.


I don't know.


I'm just frustrated.


This is the first time I've been frustrated in, I think,


literally 10 years.


Yeah.


Yeah, it's always frustrating to try to chase these differences


down.


Totally see that.


Well, you try to be a good webmaster, right?


I mean, like, and it's just upsetting.


That's all.


Cool.


OK.


So we're a bit over time.


Let's take a break here.


It's been great having you all here.