If there is like $19.95 on one and then $21.95 on the other,


English Google Webmaster Central office-hours from August 4, 2020

it might be that we would look at that and say, well,


we just crawled it again.


Maybe the price will change next week.


It's not a matter of which URL we crawl,


but rather when we crawl it.


But if there are other local signals on that page,


then that makes it something where,


at least from my point of view, I


would pass that on to the team.


That's something definitely worth bringing up to us


as well to kind of make sure that we treat them


as separate pages.


OK, is it legitimate with hreflang as well just to try


and simplify it, just to keep the theme going there


a little longer?


If you sort of just used hreflang,


we had a question in the forums.


Someone had multilingual, once again, Germany, Swiss.


If you just used the hreflang amongst language groups


and left different languages to Google to decide,


so all the English ones are there,


all the German ones are there, just makes life a lot easier


in terms of an awful lot less URLs to cross-link.


Yeah, that's something you can do.


I mean, if we don't have an hreflang pair that


maps specifically to something that you have on your pages,


then we kind of have to guess at that anyway.


So if you have German and English,


and someone in France searches for your site,


then you didn't really specify which one you want.


But we can tell they want to go to your site


because they search for your site name or something.


So we'll just kind of try to pick one.


And John, can we have some hreflang tools


within Search Console, please?


Hreflang tools?


I wish we would just have something a lot easier.


But yeah.


I think hreflang is one of those, I don't know,


awkward things where since a lot of people at Google


work from the US, it's kind of like in the US,


you tend not to worry about hreflang as much because you


just make a global site.


And global means all of the US and Canada,


and it's kind of the same thing.


Whereas in Europe, you drive an hour,


and you're in a different country,


and suddenly hreflang is a really big thing.


So that's something where because so much is happening


around search in the US, it's kind of like one


of those neglected topics where we always


have to keep pushing and kind of making sure


that, OK, don't forget about this, and keep this supported.


People need a way to make international sites work well.


Running with the US as well, you have Spanish language.


So you've got South America plus Northern America


with large Spanish volumes.


So I know more and more larger sites


are doing both English and Spanish versions.


So I think it's something that's becoming more important.


Yeah.


I think different language versions is a lot easier


than same language different countries


because with different language versions,


it's usually the case that someone will search in Spanish,


and those Spanish words will be on the Spanish version


of your site.


It's not so much that we would accidentally show the English


site if someone is searching in Spanish.


So different languages are almost one of those situations


where I'd say, well, maybe you don't even


need to use hreflang because people are automatically


searching in your language, and that matches


that specific language version.


But in South America, where you have different countries that


all speak Spanish, if someone's searching in Spanish in,


I don't know, Argentina, it would


be good to make sure that we show them


whatever kind of local version would fit for them.


It's interesting.


I was going to ask.


I don't want to sound like that dumb New Yorker or something.


All you think about is you live in New York, you live in the US,


you don't think about outside the world.


But it seems like most of the nice significant portion


of your questions that you get are around hreflang.


But how much of the web is using hreflang?


Is there some data point out there


that you could share even from a non-Google source?


I don't know.


I think it's more than 10%.


I think.


I don't know, probably less than 10%.


But I think at some point, we did a search on archive.org.


Or what is it?


HTTP Archive, where you can do BigQuery searches


across the web to see what kind of content is there.


And at some point, I did one for the different hreflang


versions.


But I need to dig that up.


Maybe I can tweet about that.


Cool.


Thanks.


Yeah, I think usually what happens with hreflang


is these questions show up in the German hangout,


because that's where all of the German for different countries


in Europe kind of thing happens.


And in the English hangout, it's, I don't know,


every now and then, but not as regular.


OK, let's see.


Handful of questions left, I think.


Let me refresh, see if anything more shows up.


Let's see.


Is a Facebook comment section on blog news portion


of a website good or bad for medical sites EAT?


We recently removed ours and saw a small dip in traffic.


I'm wondering if there is a correlation.


I suspect this is totally unrelated to any change


in traffic that you're seeing.


I don't actually know if Facebook comments


are indexable.


That might be something kind of first off


to double check to see if Google can even


render those comments.


So that's probably what I'd double check first.


And if we can't render those Facebook comments,


then probably they have no effect at all.


We have a news website accepted into Google News


and a social discussion forum located in different directories


but on the same base URL.


We believe the content in the forum


is valuable but hurting the overall search


ranking of our news.


How do we tell Google to treat these sections separately


since the news section is quality in-depth content


and the forum is mostly user generated?


It's hard to say how you need to separate those.


I don't know, actually, from the Google News side


if there's something specific where


you need to say everything for Google News


is in a specific subdirectory.


That's something where you might want


to double check in the news publisher help forum


or in the help center for news publishers.


Otherwise, if you really want to keep these separate,


then I would make it as clear as possible


that these are separate parts of your website, which


might be something like using subdomains to separate that


out or to have a clear slash news section and a slash forum


section on your website, just something where it's clear


that one is not a subset of the other part of your website.


So that's kind of the direction I would head there.


In general, having user generated content, forum,


discussions on a site is not something


that I would see as being negative per se,


but it does depend on how you kind of manage


that part of your website.


Because ultimately, if you're publishing content


on your website, then that's what we


try to index for your website.


And if you're publishing random user generated content


from random people from across the web,


and it's totally unmoderated, and sometimes it's great,


sometimes it's crazy, sometimes it's something mixed


in between, then it will be really hard for us


to understand what we should focus on with your website


and how we should treat it.


So that's something where I would


try to make sure that the content that you're providing


from a user generated content point of view


matches the kind of content that you


want to provide in the news section of your website too.


Is it actually cloaking or forbidden


if I deliver a faster website to Googlebot?


For example, if I leave out all 20 web trackers and retargeting


pixels so that it gets through the page faster,


or you can render the pages faster.


I don't think we would necessarily see this as cloaking,


but I suspect you're not going to get a lot of value out


of it.


On the one hand, when it comes to speed,


we use the Chrome User Experience report data,


and that's something that is based on what users actually


see.


So if you're making a page faster for Googlebot,


that's kind of nice, but it would


be nicer to make the page faster for users


because that's what we would kind of pick up on.


With regards to rendering, in general,


doing something like this is similar to pre-rendering


a page on the server side.


And from our point of view, that would be OK.


From a practical point of view, if you're


doing this for a website that doesn't need to be pre-rendered,


then you're just introducing a lot of extra functionality


that could break.


So that's something, just purely from a maintenance point


of view, I'd recommend trying to find a way


to avoid having to do this kind of special version for search


engines versus the version that everyone else gets.


MIKE PILAPILO-MARTINETTI.


Sure.


Last year, Barry covered a tweet that you


had said about doorway pages.


Basically, the person had asked, oh, I'm


going to automatically create 1,300 city-based landing pages


to rank for phrase plus city name.


Now, that obviously sounds like mass spam


and potential doorway pages.


Can that strategy actually be useful and valuable


and not considered a doorway page


done in a much smaller context, like maybe a handful of cities


where you're actually adding content that is valuable


to somebody in that city?


For example, car dealers love to do this, right?


They want to rank for cities and all around their area.


And somebody in a city that's maybe 100 kilometers away


or 50 kilometers away, maybe you offer them something special


in order to come to the dealership and the business.


So if you actually provide value,


do you see that as something that's beneficial?


Yeah.


That can make sense.


I mean, obviously, finding that balance there is tricky.


And that's probably true if you would do it for ranking,


of course.


If you're serving the user at the same time,


I think we can all be friends.


I mean, it's also something where it's more than just,


well, I have something unique for that city kind of thing,


where one thing that we often see


is just general information about the city.


So you're a car dealer, and you're people from the city


can come and visit us.


And the schools in the city are like this.


And population is like this.


And all of these things.


It's essentially auto-generated content.


It's unique, but it's essentially still a doorway


page.


On the other hand, something like you


mentioned where you say, well, I really like this city,


and I'm doing this special deal.


So if you come all the way from this city or this region


to visit me, then I'll have something special for you


or whatnot.


Here are reviews from people from your city.


Here are the top models that may have been purchased


in your city.


Something that maybe they actually care about, perhaps.


So that you might consider, OK.


It's when you're kind of just spamming it out and just trying


to do things from a ranking perspective.


Then that would be considered a doorway page.


Does Google have a penalty for doorway pages,


or is it just you just might flag an individual?


JOHN MUELLER I think we at least we


used to have something for doorway pages, a manual action.


But I don't know if that's something


that we still do from a manual action point of view


or if it's just our systems trying to pick that up


automatically.


A lot of these manual actions have evolved over time


where we would say, well, we need to do this manually.


And at some point, we figure out how to do it algorithmically.


And it kind of evolved into that direction.


But I don't know in particular with regards


to doorway pages.


It's at least something that I haven't seen people


talk about that much recently.


SPEAKER 2 Could we get a list of those that have switched from


manual actions to algorithmically?


Thank you.


Appreciate it.


No.


I mean, if you're working with sites


that get hit by manual actions, and maybe you


have a list already, or maybe, I don't know,


in Black Hat world, people will have.


There was a time where we thought


you stopped sending out manual actions for links


because of Penguin and stuff like that.


And it was quiet for a while.


And we're like, oh, no more manual actions


because you just ignore these links.


And then a whole boatload of manual actions


was released over link spam and stuff like that.


I don't know in particular around the link spam stuff.


But it is something where, from our point of view,


it makes sense to find ways to do it algorithmically rather


than just purely manual because the web is just gigantic.


And we can't manually review the whole web.


We have to find ways to do as much as possible algorithmically.


And in many cases, there's still weird things out there


that we don't catch algorithmically


that maybe we do have to take manual action on.


Cool.


OK.


We're kind of out of time.


So I'll pause here.


Thank you all for joining in.


I hope you found this useful.


And if you have a scheme that can replace hreflang, that


is easier to understand and implement, let me know.


And we'll see what we can do.


Otherwise, I wish you all a great week


and maybe see some of you again on Friday.


Bye, everyone.


Bye-bye.


Thank you.


SEO summer: Index waves, sandboxes, crawling, and more! | Search Off the Record podcast

Welcome, everyone, to the next episode of Search


Off the Record, a podcast that we're trying out.


Our plan is to talk a bit about what's


happening at Google Search, how things work behind the scenes,


and maybe have some fun along the way.


My name is John Mueller.


I am a search advocate on the Search Relations team


here at Google in Switzerland.


I'm joined here today by Martin and Gary,


who are also on the Search Relations team.


Good morning.


What do you mean good morning?


Do you wish me a good morning or mean


that it is a good morning, whether I want it or not,


or that you feel good this morning,


or that it is a good morning to be good on?


Utram dobry mneeb vayet.


Yes, yes.


There's no such thing as a good morning.


You said yes.


What does that mean?


What does that mean, John?


You gave me a bunch of options with or in between,


so the logical answer is yes if one of those is true.


That's the opposite of logic.


This is the worst day of my life.


So far.


And it's 4 AM.


So far, Gary.


In Gary's time zone, that is, right?


All right, Martin, do you want to take us off?


Sure thing.


Speaking of logic and fantastic things,


do you remember when we were introducing


the two waves of indexing two years ago?


Yes, yes.


I was involved with a lot of that back then.


We worked together, I think, with Tom Greenaway, who


is also on the developer relations team,


at the time to talk about rendering and indexing


for the first time.


And I think we introduced the two waves metaphor there.


Thanks for that.


That's great.


I think I understand where that's coming from.


And if I remember correctly, basically, I


joined the team right before you did that presentation at IO


And I was convinced back then that, yeah, that's


a fantastic way of explaining how things work in rendering


and indexing and crawling, because it's


such a complicated process with lots of things happening


in parallel.


But I got to say, I kind of have to deal with a fallout


from that metaphor, because it invites misunderstandings


and people are basically relatively frequently asking,


so how long does it take for the second wave to happen?


Or how do I deal with my website being indexed


before the second wave happened?


And these kind of questions, which I think,


given the metaphor, makes sense.


But given the way that the process actually looks like,


is not helpful for them, or is not


leading to the right results.


And I really tried to phase out this metaphor,


but it keeps coming back at me.


And yeah, I mean, the way that we are seeing it, most cases,


and basically that's nearly 100% of the cases,


your website gets crawled, then it gets rendered,


and then it gets indexed.


There are certain situations where


that isn't true, like when the rendering fails multiple times,


or when we have other signals that we can pick up


from the initial HTML and stuff.


So it isn't necessarily that everything gets rendered,


but pretty much practically every website


gets rendered before it gets indexed.


And yeah, so I wonder how long I'll have to, I don't know,


swim through these two-wave metaphors to get to safer land.


It's going to be fun times.


JOHN MUELLER-GORDONITO So do you think we should not


have introduced it like that?


Or do you think we just need to be clear in what


is the current status?


MARTIN SPLITTENBERG I think we should have introduced it


like that, but explained that it's a simplification.


It's like a mental model for people to look at,


and that it's not literally that.


I think people took it quite literal,


and that causes a few confusing moments, I think,


where I would say if you look for a simple mental model,


assume crawl render index.


And then if you are seeing really weird things,


then you might actually look into what's


going on more specifically, or ask


us more specific questions for your specific case,


where that might not be the case.


But I think the metaphor was OK.


It's just people took it a little too literal.


JOHN MUELLER-GORDONITO Yeah, I mean, all of rendering


and JavaScript is pretty complicated.


So it's probably hard to find a middle ground that


explains where the problems might be coming from


and what they need to be doing.


MARTIN SPLITTENBERG That is true.


JOHN MUELLER-GORDONITO But I think you've been here a couple


of times, and you see how when you say things once on our side,


it sticks around for a really long time.


So sometimes that's challenging.


MARTIN SPLITTENBERG And things just keep changing,


and that's kind of fine, because in the end, most of the changes


are implementation details that have not much impact.


But then people kind of latch onto them and be like, ooh,


so is this a big thing?


And I'm like, no, it's fine.


It's OK.


Don't worry about it.


And then also there's ranking involved.


It's like, oh, my website didn't get indexed.


And I'm like, actually, it is indexed.


It's just not ranking for anything.


Yeah, fun times.


It's always fun.


And then people are like, ooh, is this


because the website only gets out of the sandbox


once the JavaScript has been processed?


I'm like, no, that's not.


Yeah, it's tricky.


JOHN MUELLER-GORDONITO Sandbox.


Yeah.


Well, I guess since we talked about waves and sandbox,


it's almost like a visit to the beach, you know?


I don't know.


MARTIN SPLITT


Ooh, vacation theme.


So I recently did this tech SEO Reddit AMA, where people were


asking me questions.


And not only did the two waves come up multiple times,


I mean, it's vacation time.


I think people are taking their mind to the beach.


So that's that.


But also, the sandbox and honeymoon periods came up.


And it struck me as weird, because kind of one


precludes the other.


It's like, if my website is new and I get all this traffic,


why does the traffic then drop off eventually?


And then, yeah, is this the honeymoon period kind


of situation?


And then the next person is like, well, my website is new


and I don't get any traffic or don't get any ranking in Google.


Is this the sandbox?


And I'm like, so what is it?


Do we have a sandbox?


Do we have a honeymoon period?


What's going on?


Yeah.


I always find it interesting when someone on the team


asks these kinds of questions as well,


because it's not that you can go into the internal Google


documentation and look for the sandbox.


Does it exist?


Because you probably won't find a lot of useful things there.


But I think the general problem there is really


more a practical thing.


And the names are a bit, I don't know, simplified.


But the practical problem is really


that if you have a new website, we generally


don't know a lot about it if it's new,


because we can look at the content.


We can kind of see what's written on there.


But we don't really know how it's accepted


within the whole web ecosystem.


So from a practical point of view,


our systems essentially make some guesses.


And we make some assumptions trying


to figure out where should we position


this site in the search results, how competitive is kind


of the search results area in general for the queries


that we think we might be showing the site for.


And based on all of these different factors,


we try to figure out, where could we position this website


until we know more about it?


And that's essentially this period of time


which people externally sometimes simplify


into sandbox or honeymoon period, where maybe we


will position it in a way that is very optimistic and say,


oh, wow, this looks really good.


I guess it'll perform really well in search.


And we'll get lots of really good signals over time.


Or we could look at that and say,


well, it's a very competitive environment.


There are other sites that have been working on this area


for a really long time.


And they're really well accepted on the web.


We might need to be a little bit more critical or, I don't know,


watch out a little bit more with regards


to how we position the site.


And that's something that is not like an algorithm that's


built in that says, we should hold the site back


or we should show it even more visibly


for a certain period of time.


It's really just, we don't know how


to show this site in search.


So we have to make some guesses.


We have to make some assumptions.


And over time, as we figure out how we should be showing it,


kind of when we have more signals to show,


and we use lots of signals in search, of course,


then we'll be able to show it a little bit more reasonably.


And that could mean that maybe we


were showing it overoptimistically


in the beginning.


And as we learn more about how it's accepted,


we have to kind of pull that back a little bit.


It could also mean that we showed it a little bit too


pessimistically in search, almost,


and we need to show it a little bit more visibly.


But these are all things that happen over time anyway.


And in particular, when a site is new,


that transition from not knowing a lot about the site


to knowing more about the site is sometimes a bit jarring.


It's a bit more visible compared to kind


of the traditional changes that happen with the site over time.


So that's kind of, I guess, where


this sandbox and the honeymoon periods come from.


Usually, when people externally bring up one or the other,


they focus on just that one aspect.


And telling them or showing them how other people are talking


about exactly the other side or the exact opposite of that,


that makes it a little bit clearer to them


that it's not just like one thing or the other.


It's actually kind of a balance between the two things.


So I don't know.


I don't think it's a topic that will go away,


because there will always be new people who


make websites on the web.


And they'll always run into this situation


where they find someone saying, oh, you


have to write about this topic, and you will rank well.


And they do that.


They follow those instructions, and they don't rank well.


Or they rank particularly well, and they're


really enthused about doing more on the web.


And the next website they do doesn't rank that well.


So I suspect that'll continue to follow us around for a while.


Maybe longer than the two waves of indexing.


I don't know.


I think it makes sense that these kind of things come up,


and I think that's also what happens with the two waves of indexing.


Except that two waves of indexing is a much more niche topic,


I guess.


So that's helping.


Also, if I remember correctly, the sandbox thing


that did have this weird effect, because how was it?


I think we rebuilt the index every month,


but only once every month.


And then that meant that if you created a website


at the wrong time, as in farthest away from when


we built the index, then you had this long wait period.


And during that time, you couldn't do much with that site


because, well, we were not indexing things from your site.


And then somehow that indexing sandbox, I guess,


transformed into a ranking sandbox in people's minds


as well, which was probably not that helpful.


And then we just got to enjoy talking about it for 20 years.


MARTIN SPLITTENBERG


Stop laughing, Martin.


Sorry.


And if I publish on another section some good stuff,


something that I do think that should show up in the index,


it's not gibberish, then basically,


Googlebot is wasting time on the gibberish side


because it's auto-generated gibberish content that is


linking infinitely to other pages.


And it's spending time on that stuff


instead of going to the good section of the site.


And if you think about it, then that kind of makes sense.


Like if you remove, if you chop, if you prune from your site


stuff that is perhaps less useful for users in general,


then Googlebot will have time to focus on higher quality pages


that are actually good for users.


Backout signals or back-off signals,


if you send us back-off signals, then that


will influence Googlebot crawl.


So if your servers can handle it,


then you want to make sure that you don't send us


like 42950x status codes and that your server responds


snappy, fast.


Yeah, I guess that's it.


But in general, I would, unless you need to,


I wouldn't worry about the crawl budget.


There are better things to worry about.


Like if your site is still not mobile friendly, for example,


then you could catch up with 2003


and have a mobile site already.


Oh, 2013, sorry.


So Martin, does crawl budget also play a role


with JavaScript?


A little bit, because if you are already crawl budget sensitive,


and as Gary said, that's not very many websites.


Like most websites that talk about crawl budget issues


turn out to not have crawl budget issues.


But if you do, then depending on how you build your JavaScript,


you might actually end up having more requests


than if you do like a server-side rendered version


or a static version of your website.


So as an example, if you have a client-side rendered website,


the website loads, it loads its JavaScript,


and then the JavaScript makes five API requests


to fetch the actual content.


Then all of these five extra API requests


do count against your crawl budget in a way.


And that can then easily scale.


If like every page that you have on your website


and you have 10 million of these pages,


then there's like 10 API requests,


then that does add up quite quickly.


But normally, that should be fine.


As Gary said, most websites have other lower hanging fruit


than crawl budget to reap first.


But yeah, if you are crawl budget sensitive,


then the way that you build your JavaScript or architect


your web application, that does might have an impact.


GARY ILLYESISIOSAKE LOUISHAVYREVICIOUYASU


That was a very good point, the JavaScript and resources


and stuff.


Because every single URL that we crawl on the site


will chip away from crawl budget.


So basically, if we have to crawl alternate versions


of the site, let's say that you have 170 language


variations of a page, then all those


will chip away from your crawl budget.


It's not like, oh, these are just a variation


and we don't count it in the crawl budget.


We do have to crawl them.


And if we have to crawl them, then that


means that it will chip away from your crawl budget.


Basically, we will have less time on something else.


Yeah, does that also include things


like CSS files or images, which usually don't change?


Yes.


But we do have caching in place.


Oh, that's what I wanted to say.


Oh, sorry.


Damn it, Martin.


Sorry.


Now you go.


Right.


So let's say you have one style sheet for all your pages.


And then we crawl one URL.


And then we make the request to that URL.


That's one request in your crawl budget gone.


Then we crawl the CSS file.


Let's say, I don't know, main.css or something


like that.


That's the second one.


And then maybe there's one image on this one page


that we just crawled.


That's the third request.


Then we see the next page.


That's the fourth request.


But we would not request the CSS again because we already


have it in the cache.


So that would not count again against your crawl budget.


And the cache is relatively aggressive.


Aggressive cache.


OK.


It kind of makes sense to use cache


when you're talking about a crawl budget, right?


You didn't.


There are rabbit holes.


You can dig quite deep into each of them.


But I think it makes it even more important to just,


at the surface level, understand how your site is doing


and what your site's doing and where to spend the time


to look further.


If your site has 10 pages, it's very unlikely


that you will suffer crawl budget issues per se.


If your site uses a regular JavaScript framework


and you are not ranking well, then that's


very unlikely to be primarily a JavaScript issue


unless you see that we are not rendering your content.


So you want to very carefully find out


what's causing the problem, if it even is a problem.


Because sometimes what looks like a problem


really isn't a problem, depending


on the tools that you use.


It's like, this website doesn't rank for the top 100 keywords


that this tool thinks are important.


It's like, yeah, but your website


is in a completely different industry or niche.


So why do you care?


Yeah, I think it's interesting because there are always


new people that jump into SEO, and they


start working on small sites sometimes in the beginning.


And they hear all of these complicated things


and wonder what they should be watching out for.


But it sounds like the two waves of indexing and crawl budget,


if you're working on a smaller site,


you probably don't have to care about it all.


And you can kind of grow into that over time


if you start working on different kinds of sites, more


complicated JavaScript sites, or really large websites.


Yeah.


Cool.


Start with the basics.


Always start with the basics and then build up from there.


So when it comes to crawl budgets,


I heard you have to give Gary cookies,


and then you get more crawl budget.


Is that right?


Or how does that work?


I haven't gotten more crawl budgets,


but I have gotten fantastic cookies.


I'm really looking forward to the next batch.


Oh, wait, he gives you cookies.


You don't have to give him.


Gary had a ridiculous amount of oatmeal in his pantry,


apparently.


And I was like, what do I do?


And I'm like, make cookies.


And so I got two batches of cookies,


and they were delicious.


Somehow I still have, I think, maybe


one pound worth of oatmeal.


Well, actually more.


Like, one was two pound.


Yeah, actually, one was two pounds of oatmeal.


How much is that in real measurement units?


It's like 1 kg.


OK.


Is that like 27.5 cups?


Yeah, let's not talk about that.


How many caterpillars is that?


At least seven.


I think that's right.


Yeah, I have no idea how I ended up with so much oatmeal,


but it was occupying way too much space


in my pantry, in my cupboards.


And then I was trying to figure out what to do with them,


and I just searched for something like what


to do with lots of oatmeal.


And then one of the first results was, well, make cookies,


you dumb idiot.


And then I actually have no idea how I ended up with,


because I'm making vegan cookies now.


But I don't remember how I ended up with vegan cookies,


based on the oatmeal.


I think it was like most of the recipes


that I found that were using oatmeal for cookies were vegan.


And also, I could never in my whole life


manage to make good vegan cookies.


I can make vegan food, but I could never


make good vegan cookies.


That's no longer true.


Yeah, no longer true.


But I always had problems with that.


Like either the fat part of the cookie was not right,


or it had an acquired taste that was like, well, disgusting.


A acquired taste.


I don't know how to say it nicely.


It was disgusting.


So yeah, this is the first time I actually


managed to create good vegan cookies,


and I'm very happy about it.


That's not true.


I'm just like OK about it.


You are very happy about it.


And it's also actually technically,


it's probably the second time, because there


are in the three batches that I got so far,


there were two different types of cookies.


And the first type of cookie was great.


That was the lemon cookies that you made.


Oh, the lemon cookies were excellent.


Those were really, really good.


I wish you would publish these recipes somewhere, Gary.


Yes.


Spamiguy.com.


No, no, no, I have a new domain name for the cooking site,


or not cooking site, recipe site.


Cookieguy.com.


Cookie Guy.


Quick, register that.


Cookie overlay.


Yeah, maybe I should have asked you.


You have better ideas than I do, apparently.


Yeah, I will not tell you that domain name.


But yes, the cookie site is going to happen eventually.


I actually didn't have time to work on it.


Now I have two interns that we will maybe


introduce externally as well.


And they keep me very busy nowadays, somehow.


So cool.


Which is good.


What are your interns working on?


They are working on the robots.txt parser.


I don't want to spoil it yet, but they


are doing some really interesting work


on basically enabling others to build on top of the parser.


That sounds good.


Cool.


Is this like the second wave of robots.txt processor ring?


OK, I'm out from here.


So sorry.


OK, maybe we should take a break here


before things go even more downhill.


Thanks, you two, for joining in.


It's been fun and entertaining, for me at least,


hopefully for those of you who are listening along as well.


Stay tuned.


I guess for our next episode, we'll


have more of these over time.


And thanks for listening in.


Hope to see you next time.


Well, hear you.


Wait, you will hear us next time.


Yeah.


At one point, you will have to figure this out, John.


I'm working on it.


This is fine.


Cool.


Bye, everyone.


Bye.


Good day.


How to improve your ecommerce websites presence in Search results (6 tips)

Google Merchant Center


Google Merchant Center is a great way to share data about your e-commerce business with Google.


Hi, my name is Alan Kent and I'm a Developer Advocate at Google.


In this episode, I'm going to share six tips on how to get the most out of Merchant Center for your presence in search results.


The most common use for Merchant Center is to upload product data via structured feeds.


Because feeds are designed to be read by computers, data is extracted more reliably than Googlebot crawling your site and extracting data from web page markup.


If you are familiar with structured data, you may be wondering whether to embed structured data in web pages or provide a feed to Merchant Center.


Google's recommendation is to do both. Google may cross-check feed data against your website.


So product structured data in web pages is still recommended even if you also provide Merchant Center feeds.


If you have physical stores, you can also share inventory location data with Google.


This can then be used by Google when answering queries for products near me.


Note that there are other e-commerce related non-product feeds.


For example, if your business creates goods, you can provide authoritative product data via Manufacturer Center.


Finally, please note that Merchant Center product data feeds are also used by experiences other than search, such as Google Ads.


In this episode, I focus on the relationship between Merchant Center and search results.


Let's dive into 6 tips for using Merchant Center to improve your presence in search results.


The Googlebot web crawler attempts to locate all products on your site by following links between pages.


Googlebot, however, may miss pages in some circumstances.


For example, you may have some products only reachable from on-site search results.


Google typically does not enter search terms into your on-site search box to discover new pages.


If you have a product page and you are not sure if it is indexed, you can use the URL inspection tool.


This will report what Google search knows about your page.


You can also use site colon URL as a search term to search for that specific URL.


In a previous episode, I described creating a sitemap file to list the important pages to index on your site.


The sitemap file is used by the Googlebot crawler to find pages on your site without relying solely on links between pages.


But there is another way.


Creating a Merchant Center product feed will help Google discover all the product pages on your website.


These product page URLs are shared with the Googlebot crawler to potentially use the starting points for crawls of additional pages.


It is, however, important to note that this and some other Merchant Center features are not available in all countries.


Please refer to the Merchant Center Help Center for up-to-date lists of countries features are available in.


The second tip is to check the accuracy of product pricing data used by Google.


If Google incorrectly extracts pricing data from your product pages, it may show your original price instead of your discounted price in search results.


To check if Google is extracting price data accurately, quickly test a sample of results.


You can search for a product page and check the price displayed if rich results are displayed.


Search using site colon URL for your product page to return the web page as a search result.


To accurately provide product information such as list price, discounts and net price,


it is recommended to add structured data to your web pages and provide Merchant Center with structured feeds of your product data.


This will help Google correctly interpret pricing shown on product pages.


Tip number three is to minimize inconsistencies in pricing and availability data between your website and Google's understanding of your site due to timing lags.


For example, Google crawls web pages on your site according to its own schedule.


Changes on your site may not be noticed until the next Googlebot crawl.


On the other hand, Merchant Center feeds can be updated on a more consistent schedule, such as once a day or even once an hour.


These delays can result in Merchant Center and search indexes lagging behind site changes, such as when a product goes out of stock.


I described how to check Google's understanding of your pricing data in the previous tip using a site colon URL query.


In addition, Merchant Center may identify products that have different pricing data according to your website due to delays in processing.


This can negatively impact your products in search results until the discrepancy is resolved.


Merchant Center also allows you to download all pricing data in bulk if you want to do a more exhaustive reconciliation of pricing data in Merchant Center against your website.


To reduce lag, you can request Merchant Center to process your feeds more frequently.


This can reduce the time lag between the product data changing on your website and Google being aware of it.


Another approach is to enable automated item updates in Merchant Center.


This causes Merchant Center to automatically update collected pricing and stock level data based on web page contents when discrepancies are detected.


This is based on the assumption that your website is updated real time when pricing or availability changes.


Tip number four is to check your products are getting rich results treatment in search results.


Rich results are displayed at Google's discretion but rely on Google having rich product data.


To check if your product pages are receiving rich results presentation treatment, you can use a site colon URL query to search for a specific web page.


If not found, the page may not be indexed.


You can also use the Google search URL inspection tool to verify if Google is indexing your product page.


To get the special rich products presentation format, it is recommended to provide both structured data in your product pages and a product feed to Merchant Center.


This will help ensure that Google correctly understands how to extract product data from your product pages needed for rich text product results.


Also check for error messages in Google search console and Merchant Center.


Tip number five is to ensure if you have physical stores that your products are being found when users add phrases such as near me to the queries.


To test if locality data is being processed correctly, you may need to be physically near one of your physical stores and then search for your product with near me or similar added.


Register your physical store locations in your Google business profile and then provide a local inventory feed to Merchant Center.


The local inventory feed includes product identifiers and store codes so Google knows where your inventory is physically located.


You might also like to check out Pointy from Google.


Pointy is a device that connects to your in-store point of sale system and automatically informs Google of inventory data from your physical store.


The final tip is related to the shopping tab.


You may find your products are available in search results but not appearing.


To see if your products are present, the easiest way is to go to the shopping tab and search for them.


To be eligible for the shopping tab, provide product data feeds via Merchant Center and opt into surfaces across Google.


Structured data and product pages alone is not sufficient to be included in the shopping tab search results.


This is the final episode in the current series on improving the presence of your e-commerce website in search results.


If you have topics you would like seen included in the future series, please leave a comment.


If you have found this series useful and you would like to see more similar content, make sure to like and subscribe.


Google Search Central publishes new content every week.


English Google Webmaster Central office-hours from January 31, 2020

All right, welcome, everyone, to today's Webmaster Central


Office Hours Hangouts.


My name is John Mueller.


I'm a webmaster trends analyst here at Google in Switzerland.


And part of what we do are these Office Hour Hangouts,


where folks can join in and ask their questions around websites


and web search.


And we try to find answers.


Bunch of stuff was submitted, even though it's


like just a few days, but lots of things lined up.


But kind of as always, if any of you


want to get started with the first question,


you're welcome to jump in.


Hi.


I have actually two questions.


So first one is about the URL link.


Recently, we have got a client.


So when I checked his website, his website is about Twitter.


So his category URL are like this,


domain URL slash header slash food header


slash free-standing food header.


It's very long, and he used the keywords for a lot of time.


So is it bad for SEO?


What do you suggest in this case?


That's fine.


I mean, we use URLs to identify the content.


So if it's long or short, that's essentially up to you.


When we have multiple URLs that have exactly the same content


and we have to pick a canonical, then usually we


tend to prefer a shorter one.


But usually, that's within your own website.


So if you have kind of the short version and the long version,


and they both show the same content,


then we try to pick the shorter version.


But it's not that the shorter version has a ranking advantage


or anything like that.


And the second question is about the backlink source.


So if a English website gives backlink


from another English website and another backlink


from non-English website, do you give same importance


to both URL, or it varies based on the language?


That's all the same to us.


So it doesn't matter the language of the website.


I mean, what happens is we try to understand the anchor text


and how that belongs within the context


to understand the page they're linking to a little bit better.


But it's completely normal for websites


to have links from all kinds of other websites


from different countries, different languages.


Thank you.


John, perhaps I may chime in before you


go over the other comments, if that's OK.


OK.


And I just wanted to ask a trivial question,


or perhaps not that trivial, regarding interstitial pop-ups.


Or what we are currently having is


a discussion between our sales team and the SEO people


regarding how we actually get content from our users.


I tried to go on.


Ah, sounds better.


So we are having a discussion with our sales team


And there is one option, which is this kind of bar


at the bottom of the page.


And the other one, which like says people more,


is to have an interstitial, which actually


is overlaying the content and even trying to gray out


the background content.


And you said on a recent tweet that content as such


is OK if you have the cookie bar at the bottom.


However, it becomes a problem if you really


have these interstitials, which are overlaying any content.


And I wanted to actually perhaps you


can elaborate a bit more on what could be possible when


it comes to interstitials.


Perhaps we don't gray out the background,


because it's basically a means of increasing the people


to give content to allow us to display advertisements.


What's your advice when it comes to this?


So I guess first of all, I don't know


what the legal requirements are that you


might need to watch out for.


So that's kind of like on the side.


I don't know if my suggestions would be compliant


with what you're looking for.


In general, the kind of a bar on the bottom of the page,


that's something that's really useful for us,


because then we can still see the rest of the page.


So in particular, we can recognize how much of the page


is mobile friendly.


We can recognize which are the prominent elements on the page,


all of that.


That's really kind of the thing that


makes a lot of the normal web search


processes just kind of work by default.


That's kind of the one side.


The other, I guess, extreme that we sometimes see


is when people redirect to a different URL


for kind of confirming that they're


allowed to access the content or age interstitial or something


like that.


And there the problem is when you redirect


to a different URL, Googlebot will follow that redirect.


And we will only have that interstitial content to index.


So there is no other content on the page.


It's not even the right URL.


So what usually happens in those cases is we index interstitial.


And we think all of the pages from the website


are the interstitial.


And basically, we do eliminate all of the pages


towards that interstitial URL, which is kind of the thing


that you really, really want to avoid


unless you want to remove your website from search.


No, we don't.


And I guess the alternative that you're kind of considering


is somewhere in between in that you have a full page


interstitial but you remain on the same URL.


And there it's generally more a matter


of how you implement this in a way


that Google can still see the normal content.


So in particular, if you're using JavaScript


to display like a div on top of the page


and the rest of the page is still there,


then that's something where we can still


see the rest of the page.


And we can kind of work with that.


So those are kind of the different approaches there.


One other thing that I think I've seen some sites do


is to go into the area of, what is it,


flexible sampling almost, where they kind of request


that a user signs in in order to see the content.


And by signing in, they also have some kind of cookie


consent or whatever attached with that.


And for us, that's usually less of a problem


because we would see that as flexible sampling.


Like you're saying, well, my content is here.


You just have to take these steps


to actually see the content.


So that's perhaps another approach


that might be worth reviewing.


Perfect.


Makes sense.


Thanks a lot, John.


Sure.


Hello, John.


Hi.


Yeah, at the German descent, actually, I


do a lot of problems.


I mean, I wanted to ask from you,


but I can only ask some things from you.


Actually, recently, I have got rejected


reconsideration request.


I have filed reconsideration request for my client.


So Google has sent us two sample links.


I mean, you guys are sending us two sample links


as an example of a natural link.


So I found that one of the links carries no follow tag.


And I don't think Google is considering


with no follow tag links as an important link.


You understand my point?


I don't think how you guys can send no follow


tag in a sample URL.


Can you please explain this thing?


That could be something that went wrong on our side.


But generally speaking, if you did a reconsideration request,


especially for link spam, and you got it back


saying that it's not good enough,


then I would focus on the bigger picture,


not kind of worry too much about that one no follow link.


But really think about what kind of bigger picture things


have you been doing with regard to links


that you still need to clean up.


And sometimes what can happen is that they will flag an issue


and say, well, this is the kind of thing


that you should be watching out for.


It's not like these are the two remaining links


that you need to fix, but rather, here


are some ideas of issues that you need to clean up.


So I would take it more in that direction


and really kind of try to clean those links out


as completely as possible.


Sometimes the folks on the forum can


be helpful for these kind of things, where they can say,


oh, you still have all of these millions of links.


And they can guide you towards the things


that you really need to kind of remove.


SPEAKER 3 OK, so John, we have also identified,


I mean, many of the links are coming from the feed pages,


right?


I mean, after checking all the external links that we are


getting, the backlinks we are getting in the master.


So a lot of feed pages are generated.


And what should we do about them also or what should we do?


Feed pages.


JOHN MUELLER What kind of links did you mean?


I didn't understand it.


SPEAKER 3 Feed, feed, feed, I mean, feed URLs.


Like, how do you mean feed URLs?


I mean, we cannot able to open that kind of URLs.


They are just, if I open that kind of URL,


that will download, it will be downloaded


in our PC, in our system.


OK, so you can examples also.


Yeah, that's correct.


RSS feed.


OK.


Usually, that's less of a problem.


But usually, the RSS feeds are generated based on the content


on a blog, for example.


So that's usually where, if there's a link in an RSS feed,


then probably that's due to a link in a blog post somewhere.


And cleaning up the blog post fixes the RSS feed


automatically, too.


So what do you say to this?


Should we consider a feed URL also in order to disavow them?


I don't know if you would agree.


No, I wouldn't worry about disavowing the feed URL.


So it's something where, if you've kind of gone out


and placed blog posts on other people's sites,


and those links also show up in the feed,


then maybe it's best to just disavow the whole domain,


to make sure that you have everything covered.


Because if you've been placing links on other sites,


and they show up in the feed in addition to the site,


then it's not the feed that's the problem.


It's kind of the general practice


of going out and dropping links on other sites.


Great.


John, one last question.


I mean, John.


Sure.


So I mean, if we will place naked URL in building,


naked URL, in any place in the website,


so will it be considered as unnatural?


I mean, if I was placing any naked URLs?


So without a link, or?


Without a link, without any anchor text.


Without any?


Well, if it's not a link, it's not a link.


You should be fine.


Yeah.


OK, fine.


Thanks a lot, John, for your help.


Let me run through some of the submitted questions,


and we can get to more live questions afterwards,


just so that we don't completely ignore all the things


that people lined up.


Because a lot of times, people just


can't join because of the time zone.


So I want to make sure that we can try to get


some of their questions through.


OK.


The first one is about an AMP website.


They changed all the internal links


to point to Google's cache URLs.


So basically, even on desktop, it always goes to the cache.


A few months later, I had some negative results


in terms of rankings and change back.


So the question is, how should I do it, essentially?


So first of all, I think pointing to the cache URL


is generally a bad practice, because these URLs


can change over time.


It's not that those URLs will always remain the same.


So personally, I would link within your website,


link to your own URLs.


And if it's loaded on the AMP cache,


and they can swap out the URLs against the AMP cache URLs


with the cache where it's displayed, that's fine.


But I wouldn't generate a website structure


by pointing at AMP cache URLs.


The other reason why I wouldn't do that is that, as far as I


know, a lot of these caches are blocked by robots.txt.


So if you're changing all of your internal navigation


to point at URLs that are blocked by robots.txt,


then that's going to definitely have a strong effect


on your website.


So my recommendation there would be to link within your site


and keep those links within your site.


And if it's loaded on the cache and it swaps it out


for the cache, then that's kind of on the user side.


That's not something you need to worry about.


But from crawling and indexing, definitely


make it so that we can index your site


and not have to kind of worry about the cache.


I'm seeing random errors of Google Tag Manager


verification failures for Search Console for our sites.


And you link to a thread.


I'm not aware of anything specific


happening there with regards to Tag Manager verifications.


But I'll definitely take a look at the thread


and see if there's anything more that we need to figure out.


So try to find something there.


We often receive DMCA violation notifications.


We're not pirates, so disputing complaints and dismissing them


often, but it takes time.


If it weren't for the fear of Google discontentment,


then we wouldn't be doing it.


Does the number of DMCA complaints


affect the ranking of a site?


Is it possible to get rid of a bad reputation?


So I double checked on this because I


wasn't sure what we actually announced at the time.


But I think 2012 around, we did a blog post


saying that the number of valid DMCA complaints


is something that we would take into account for ranking.


So if you're dismissing them, if they go away


when you kind of flag them as being wrong,


then that's perfectly fine.


But if your site is collecting them over and over again,


then that's something where our systems might pick up on that.


If we have breadcrumbs only on the desktop site,


how will it affect this snippet on Google?


So in general, when we switch to mobile-first indexing,


we only take the content from the mobile site into account.


So if you only have some kinds of markup


or some kinds of content on the desktop version


and it's not available at all on the mobile version,


then we will not have that available for indexing.


So breadcrumbs, if they're only on desktop and not


on the mobile and we switch to mobile-first indexing,


we will not have your breadcrumb markup anymore.


With regards to putting breadcrumbs


on the bottom of the main content on mobile,


that's perfectly fine.


It's also possible to use kind of responsive design elements


where you're saying, well, on desktop we have room


and we can display it.


And on mobile, maybe we don't have room


or we need to display it in a different way.


That's also fine.


But the markup needs to be there on mobile


if you want us to use that markup.


We migrated our website on the 13th of December


from an old platform that wasn't mobile-enabled.


It was a major move to a new server, new platform,


new category, new URL structure.


All the pages and categories have been indexed


and the URL redirect set up.


We had some technical issues with 404s,


which have been cleaned up.


We're working on speed, which was fine in testing,


but now too slow.


Taking all of this into account, is it


reasonable to assume that the rankings won't have settled


down yet?


Or could the site speed be the cause of the low traffic?


So I took a quick look at this before.


And I think what is happening is you


moved within the same domain to completely different URLs.


And usually, these kind of moves tend


to take more time than when you move to a different domain.


And that sounds a bit paradox almost,


but when you move to a different domain,


we can take everything one to one


and just copy it over, essentially.


Whereas when you restructure your website,


then we really have to relearn your whole website


and understand, again, how all of this


fits in together, what the context is of individual pages,


how to kind of connect things, what the internal links are


like, the internal anchor text.


And all of these are things that take


a significant amount of time for us to settle down and figure it


out again.


So I wouldn't assume that the speed is kind of the issue


here.


We do take speed into account, but that's


something that's generally a little bit lower with regards


to priority and probably takes a little bit longer


to get picked up.


But rather, on the one hand, the restructuring takes time.


And on the other hand, if you change the design of a website


significantly, then that can also affect


how we see that for ranking.


So for example, if you had a website


with a lot of good textual content on the pages


and a clear internal navigation, then that's


kind of a good thing.


And if you redesign into a really minimal version


of the website with the same URLs,


maybe, but the internal navigation is a lot less


or there's a lot less content on the pages,


then obviously we would rank it based on the less information


that we have there.


One thing that points in the direction of this not being


really a speed-related problem is when


you look at Search Console.


So what I tend to do in cases like this


is take a period of time before the change


and after the change where you're looking at roughly


the same kind of days.


So I try to pick the same weekdays.


I try to skip over holiday seasons


because that always skews the numbers a little bit.


And I compare the queries where the site was ranking


before and after and the URLs that


were ranking before and after.


By comparing both of those, you can fairly quickly


see if there is any problem with the old URLs maybe not being


redirected, if the most important old URLs don't show


up at all in the new time period.


And that's kind of a sign.


And with regards to the queries, you


can often understand a little bit better


what kind of changes am I seeing.


Is this like a change across the board


where maybe all of the queries before and after maybe


dropped by 20% or some number, is it


specific to the branded queries?


Or is it specific to the long tail, the non-branded queries?


And by looking at those differences,


one thing I noticed with your website is essentially


the branded queries are all more or less the same, which


to me points in the direction of, well, technically


your website is kind of OK.


Overall, we're seeing it as generally being OK.


Probably speed, which would be something


that would apply across the whole website,


is less of an issue here.


But the non-branded queries are the ones where


there's kind of a drop there.


And to me, on the one hand, that suggests


that maybe these are URLs that we haven't recrawled


and re-indexed that well yet, which


could be that kind of from a restructuring,


it just takes time.


Or alternately, maybe these are pages


that look significantly different before and after,


where we're kind of ranking the page on what we have indexed


now, and the indexed version that we have now


is significantly less useful for us for whatever reasons.


Maybe it doesn't have enough text on it.


Maybe it doesn't have a clear heading structure on it.


Maybe the images are not that accessible for us,


all of these things.


So that's kind of the direction I would go there.


And kind of look at it, I would double check this


in your own Search Console account


to kind of see if that matches what you would see as well.


And if it's really more the non-branded queries, more


the long tail queries that were going to your site


and are less visible for your site,


then I would kind of, on one hand,


give it more time to settle down, and on the other hand,


really significantly review the pages themselves


to see what were they like before,


what do they look like now.


If you're not sure what they looked like before,


there is archive.org, which is a free service where


you can check a lot of URLs to see older versions of those.


So you can check kind of the old versions


that used to be shown in search and see what they look like


and compare that to the current version that you have yourself.


All right.


We have a few thousand sites that show up in Search Console.


Report, index, and not submitted in sitemap.


I think sites probably is pages.


Not sure if in German, we tend to use the same word


for websites and web pages, which makes things a bit


confusing sometimes.


But I think probably they mean pages.


So a few thousand pages that show up in Search Console.


Report, index, not submitted in sitemaps.


Most of them are non-canonical pages,


which point to their respective canonical version via rel


canonical.


When we check one of these with a site query,


it shows up in the search results.


When we check them with the URL inspection tool,


it says canonical URL provided by the user


with the correct canonical and canonical URL selected


by Google with verified URL.


Why are these non-canonical pages indexed?


Although there is a clear signal with a canonical,


the pages are almost 100% identical.


So let's see.


So I think there might be multiple things happening here.


It's kind of hard to tell without some example pages.


But in general, when it comes to canonicalization,


we take multiple factors into account.


We do use the rel canonical.


That's a fairly strong signal for us.


We also use things like redirects.


We also use internal linking, external linking.


We use sitemap URLs as well.


Kind of the internal linking structure


with things like hreflang, all of that plays a role as well.


And essentially, we take these multiple URLs


that show the same content, and we


try to figure out which one of these URLs


is the one that we should pick.


And it can happen that we pick URLs


that have a rel canonical on them pointing to a different


one.


So that's not impossible.


If you're seeing this on a large scale,


then I would try to figure out why Google might be doing this.


So for a few individual pages, I wouldn't care about this.


It's like sometimes it goes this way.


Sometimes it goes that way.


From a ranking point of view, it doesn't change anything.


It's purely just which of the URLs is shown.


So again, if you're seeing this on a large scale,


I would try to dig into those situations


and see what might you be doing that could be confusing Google.


So that could be things like internal linking.


Are you linking to the exact URLs


that you have in the set for rel canonical?


If you have something like hreflang,


are you using the exact URLs there as well?


Sometimes things like upper lowercase plays a role.


If you have a sitemap file, does that


match exactly what you have submitted as well?


So all of these things ideally would align.


If they align, then we'll go with what


you suggest, essentially.


If they don't all align, then Google


will try to figure it out and try


to come up with a canonical version that works in general.


And again, it doesn't change anything from ranking.


It's just purely you have a preference,


and ideally, Google would show that preference.


And the other place where this plays a role


is, of course, with tracking.


If you use things like analytics,


then you want to be able to track the individual URLs


that you care about.


But that's the direction I would go there and try to figure out


what might be happening there.


If you have URL parameters in these URLs as well,


so something with a question mark and then some,


I don't know, words equal something else,


then the other place I would also look


is the URL parameter handling tool,


because it can happen that you have a setting there


that you forgot where it's like Googlebot should


ignore these parameters.


And that's why those URLs are not being picked as canonicals.


I'm looking at log files from the past several months,


and about 75% of the 404s we send Google


happen when Google requests very old embedded content.


So we're talking about old CSS, JavaScript image files


that haven't been on site for up to five years in some cases.


I've seen URLs from a site we migrated three years ago


in the refer.


I verified the request as genuine Googlebot.


Do these kinds of requests happen


when Google renders old pages?


If so, why are you rendering really old versions


of our page?


It seems like a sign of something going wrong.


Is there something we can do about this?


So there's generally nothing you need to do about this.


It's essentially just our rendering setup,


our crawling and indexing setup, checking pages again.


And when it comes to crawling, it's


really common for us to recrawl URLs that are kind of old,


where we think probably they're going to be a 404.


And if you return a 404 for us, then that's perfectly fine.


It's not going to cause any problems there.


So I think if you look at your log files in detail,


you'll find lots of these weird quirks.


But they generally don't cause any problems.


I think it's kind of weird that we would re-render really


old pages.


But I don't see it as being weird enough


that I would contact the rendering team


and tell them to stop doing this,


because usually they have good reasons


to try to check things out.


Sometimes they're experiments.


Sometimes it's just double-checking


how the old rendering worked versus how


the current rendering works.


Lots of reasons why that might be happening.


But again, I generally wouldn't worry about this.


If the number of requests is causing a too high load


on the server, then that's something


where you can adjust the crawl rate setting in Search Console


to let us know to crawl less.


And when we crawl less, it's not that we'll kind of crawl


percent-wise less of all of the URLs,


but rather that we'll focus more on the URLs


that we actually do care about.


So that's kind of an approach you could take


if you wanted to go that direction.


Our competitor is now making all the external links


on their site with help of Gasket page


with JavaScript redirection.


He links from site A to site B not directly,


but through an additional page on the site


with some kind of a JavaScript.


Is something like a link from Facebook?


So the question is, do these JavaScript Gasket


URL links convey weight and signals?


Does Google not perceive them as a real intermediary?


So a lot of times, we can crawl through those,


and it's absolutely fine.


We get to the destination page, and that works for us.


So usually, that's less of a problem.


In general, if we can't crawl through them,


then it'll look to us like they're just not


linking to those pages.


But I don't see this as being particularly


useful for a website.


So if a website just doesn't want


to pass any signals through a link,


then I would just use a nofollow on that link.


All of this extra complexity just


means more things can break.


So personally, I would avoid trying


to go through all of these extra steps to try to hide links.


If Google's search result page is intermittently


ranking two pages for the same keyword,


how can we help Google to rank only one of them?


We have two pages that keep alternately


ranking for the same term in the search result.


And I wonder if the competition is ultimately


drawing down ranking.


Both pages are important to the site,


so we can't just delete or redirect one.


So the cases where I have seen something like this happen


is usually when we're unsure what the intent of a query is.


So it can happen that this kind of fluctuates over time.


For example, we might not be sure


if the user is searching for a specific product


or a specific kind of product.


And depending on how we would judge


the intent of those queries, we might say, well,


maybe a category page would be more useful because they're


looking for this type of product.


Or alternately, we might say, well, they're


looking for the exact product, so the product detail page


would be more important.


And these kind of fluctuations can happen over time.


It can happen that, I don't know, from day to day,


it changes.


It might happen that with queries,


like we're so far on the edge that we might show


both of these pages because we're just not sure


what exactly the user wants.


And from our point of view, that's usually normal


in that it settles down over time usually,


or maybe it'll settle down in the sense of, well, we figured


out we can't tell what the exact intent is,


so we have to show both of those pages


because one of them will work, but we're not sure which one.


So that's something where, from my point of view,


I wouldn't worry about it too much.


If you have really strong feelings about those two pages,


then I would just make sure one of those pages


is really well targeted for what you're trying to aim for,


and maybe the other page is really well targeted


for a different, clearly separate facet


that people would be searching for.


But for the most part, I don't see this as being problematic.


Question about implementing schema markup for the future.


Future looking is always tricky.


Can it be easier to maintain?


Can it need less reliance on multiple steps and parties


for extensible code?


Can we make it easier to instruct developers


and designers, maybe make Google Tag Manager more efficient?


I don't know.


I think it would be nice to have it easier.


But to be totally honest, I think in the future,


at least in the near term future,


we will have more types of structured data markup,


and it will continue to get more complicated, probably,


in the sense of there are just lots of requirements


for different search features.


And in order to do something really fancy in the search


results, where we can really highlight your site really


well, or if there is a way to also include information


that we can show on Google Assistant,


or rather, I guess, speak on Google Assistant,


then all of these things, they currently


rely a lot on structured data markup.


So that's something where, as these areas expand,


I would expect to see some amount of additional markup


coming in.


And any time you have more structured data markup,


kind of like the interactions get trickier,


and the requirements get a little bit harder in the sense


that, oh, you have to have, I'm just making something up,


like a name with 17 characters.


You can't just have a name with 15 characters.


Sometimes these things change over time.


So I suspect it'll get harder in the near future, at least.


Maybe in the really long term, it'll be like, oh,


the machine intelligence can figure it out for you.


You just write a text file, and everything else


will happen automatically.


I don't see that happening in the next couple of years.


So if you're kind of on, I don't know,


on the hill with regards to, like,


should I learn more JSON-LD to figure out how to do markup


myself, I think that's a good approach.


I also think, at the same time, a lot of these things


can be made easier by content management systems and plugins,


extensions that you have for content management systems.


So if you're using something like WordPress,


then maybe there's a plugin that does all of this structured data


markup for you automatically.


I know there are a bunch of recipe plugins, for example.


So instead of creating all of the kind of special markup


yourself, if you have a plugin that does it for you,


then those plugin developers will


continue to kind of stay on top of things,


ideally, as the markup evolves over time,


and probably make that easier for you.


But purely from a markup point of view,


I don't see this getting any easier, unfortunately.


We had two products.


So the page structure was obvious.


A home page, domain.com, with links to both products.


And now we dropped one product.


Which is better, put the main product landing page directly


on domain.com, or redirect domain.com


to the main product landing page?


So both of these are kind of the same thing,


in that you're folding one product


and putting it together with your home page.


From my point of view, if you're reducing


the scope of your website and really going to just a one page


website, then I would focus on the home page


directly, rather than redirect from the home page


to a lower level page.


If this is something that is very temporary,


in the sense that maybe next month you'll


have another product to sell, and then you


go back to having multiple products on your site,


or maybe next month you have 100 products to sell,


then I would stick to the structure


where you have multiple product URLs and the home page


separate.


But if you're really sure that you


want to decrease the scope of your website to just one page,


then put it on the home page.


I think that just makes everything a lot easier


for tracking and for crawling and indexing.


The rich result update in September


changed a lot of search results.


I want to understand why, for some home pages,


stars are still appearing.


In my opinion, it could be the use of misleading markup.


Or is there an actual issue with understanding markup


technically, and will it be solved in the near future?


So not exactly sure which update you mean from September.


We've had a lot of things over time.


But I think, in general, when it comes to structured data


and the rich results, sometimes we


do get tricked by people who are implementing the wrong markup.


And then we think, oh, this should be showing review stars


because we think it makes sense here.


And actually, when someone manually looks at it,


they're like, no, that's implemented incorrectly.


Or another common one that we see


is recipe markup, where maybe you'll


have a recipe for a fantastic blog post on your website.


And that's not really a recipe.


So our algorithms might look at that and say,


oh, it looks like you have a recipe with multiple steps.


But actually, it's a blog post.


And it's definitely not a recipe.


So when someone manually looks at it,


they'd say, well, this is wrong.


And from a manual point of view, we


do try to take action on a lot of these.


So if there's something where you're


seeing sites showing up with rich result types that


don't make sense for that site or those pages,


then let us know.


Feel free to use the spam report form.


If you're seeing this on a larger scale,


you can also drop me a note directly.


Sometimes what I've also seen is that some plugins, like I


said, they make it easier.


But plugins can also be wrong sometimes.


Sometimes the plugins are just set up incorrectly,


which can result in us, on a larger scale, seeing an issue.


And that's something where sometimes we'll just


send a note to the plugin maintainer.


And it's like, hey, you have this bug.


You can fix it.


And then your users won't be upset.


So I'd use the web spam report form for most of these.


And if you're seeing things on larger scale,


feel free to drop me a note.


If some URLs are blocked by robots.txt,


is this still considered a waste of Google's crawl budget?


No.


If they're blocked by robots.txt,


we're not going to crawl them.


So it doesn't change anything.


In particular, crawl budget is something most sites


don't need to worry about.


So for the most part, I wouldn't worry too much


about that particularly and try to optimize for that,


unless you're really a website with millions and millions


of URLs.


And sometimes small things on a website like that


can lead to us suddenly having two or three or 10 times


as many URLs as you actually have.


And with a small website, that doesn't really matter.


But if you have millions of pages from the start


and suddenly you have 10 times as many pages,


then that's where we really get bogged down with crawling.


Now that deduplication is in effect,


does a web page need to rank in position one


to be a featured snippet?


Before deduplication, I was under the impression


that the results 1 through 10 were eligible,


but mostly 1 through 3 earned the snippet.


Is that or was it ever the case?


I don't think that was really ever the case.


So I'm not sure what all has been changing


with the deduplication that you're mentioning.


I think this is with regards to the featured snippet


and the organic results sometimes appearing


on the same page and some of the experiments we've done there


to try to simplify that.


But in general, we can pick up featured snippets


from all kinds of URLs.


It's not that they have to rank number four first.


And just purely from a practical point of view,


it's pretty common that we run across a page where we think,


well, this is a really good page for this result,


so we'll rank it first.


But at the same time, it's hard for us


to determine what would be a good featured


snippet for this page because maybe the structure of the page


is a little bit tricky or maybe the query


is hard to match with the content on the page.


And then it can happen that we say, well,


this is a good first result, but we cannot pick any featured


snippet from this page because we're just so unsure about that.


So we won't.


And then maybe we'll pick a featured snippet


from one of the other results.


Maybe we'll just say, well, we don't have a featured snippet


for this query, and that can also be fine.


And then the cookie consent thing.


I'll try to see if I can find some more


information on what variations would work well


for cookie interstitials.


So I also know that some folks on Twitter pinged me about this,


so maybe we can find something there.


How does Google treat timestamps in URLs?


For example, if you have slash month, slash year, slash today,


would that be the same as slash month, slash year,


slash yesterday?


So when it comes to dates, we use various signals


to pick up information about a date that is relevant for a page


but the URL can be a part of that.


Sometimes it's a pretty clear signal


that a page was published on a day if it has a URL like that.


But we also have the, I think, in the article markup,


you can specify a date on the page


to let us know there as well, as well as


kind of visible dates on a page and all kinds of other signals.


And with regards to dates in general,


the question, I guess, is, is something


that looks like it's from today better than something


that was from yesterday?


And the answer is no.


It's not the case that something that is fresher


will be considered better than something that is older.


And that's kind of obvious when you think about it.


There are lots of topics where the information has


been published and there's evergreen information out there


and there's really good reference material out there


that doesn't change from day to day


because nothing has changed there.


And that's really good content that


deserves to rank well in search.


On the other hand, it might happen


that maybe some recent event takes place


where suddenly everyone is looking for something new


on this topic.


For example, if you have geographic regions,


then maybe geographic background information for a country


is really kind of useful as a reference material.


But if something critical happens in that country,


then people don't want that reference material.


They want the news.


They want the newest information.


So in those cases, we probably would prefer something fresher


rather than kind of the reference material.


So it's not so much that fresher is always better,


but it really depends on the situation.


And our algorithms change there fairly quickly


when we see that something kind of new is happening.


If domain A links to domain B with do follow links,


but domain A later adds a no follow to the link,


does Google discount the link domain B got in the first place?


So when it comes to links, when it comes to content in general,


we essentially rank the pages based on the information


we have at that time.


It's not the case that we would say, well,


it used to be better, but now it's bad.


Therefore, we'll still treat it as something better.


But rather, we base our decision on the current situation.


So if a link gets changed to a no follow,


then we treat it as a no follow.


If a link gets changed to have a no follow removed,


we treat it as a link without a no follow.


If the content on a page changes,


then we treat it as that new content.


So all of these things are essentially


tied to the current state of the web page of the website,


not so much to the previous states.


And usually, that's a good thing because that means


that you can improve things.


So if you have a website that's not ranking first for all


of the queries that you care about, you can improve things.


And over time, as we see that the new content,


the new website, deserves to rank better,


then we'll rank it better.


It's not so much that we say, oh, this website


didn't rank well last year.


Therefore, it will never rank well in the future.


We have a site that ever since March, April 2018,


we've seen a big drop in traffic that hasn't recovered.


So that's a really long time.


I don't know if so.


This might be my question.


Oh, that's your site.


OK.


I think so, yeah.


There were two sites on the same day.


OK, go for it.


OK, so yeah, back in April 3, both our sites


were doing really well.


And then it dropped through the floor by 50%, 60% down


on each site, two separate domains, just overnight.


So yeah, we've had people in.


We've had technical audits.


We've, as sort of journalists, we pride ourselves


on what we do anyway.


But I mean, we've doubled down.


We've been cleaning up.


We've been, I mean, the list, I've


got 25 technical and content-based points.


We've improved site structure.


We've really been through everything, really,


and nothing.


No improvement, really.


They're flat, both of them.


Like, it's eerily flat.


It's the same traffic every day since April the 2nd.


So I don't really know where to turn now.


I mean, we're talking about good pieces of content


that rank second or third, page four.


No, so it's not just a sort of just general distaste.


Or I don't feel like, oh, people are doing everything


significantly better than us, that we


went from three to page four.


So yeah, we've lost staff.


I don't really know where to turn now.


If you want, you can drop your domain in the chat.


And I can take a look at that afterwards.


I think, in general, if it's been so long since a change,


then that sounds like our algorithms


are pretty sure that we should be ranking the site like this.


And I mean, changes can happen in our algorithms.


So it's not completely impossible


that we would see changes like that.


But it's something where usually that


means we think we're ranking it in the right place.


And I can take a quick look after we kind of pause


the recording to see if there's anything specific that I


can point out, like offhand.


But usually that means that there's not a situation where


we're on the edge.


We're not sure how we should rank it.


But rather, it's like we're pretty confident that we


should rank it like this.


If you had other people look at it,


then probably from a technical point of view, it's OK.


I know Glenn Gape sent you a note about this,


because he's been through it and sort of getting


to head scratching time.


I don't know.


I can take a quick look after we pause the recording


and see if I find anything offhand.


But usually it's more a sign that we're


kind of OK with the ranking that we figured out there.


And these things do change over time.


So it's theoretically possible that something in April


happened where we're like, OK, we


need to re-evaluate how we rank some of these sites


that we have indexed.


AUDIENCE 1 A curious part of it is


that the site's rankings actually


return for exactly one week in July.


And then disappeared again.


Yeah.


It's hard to say.


So I don't know your sites at all.


So it's really hard for me to say,


like, oh, it's a terrible site.


And it's like it deserves to not be ranked.


It sounds like you're pretty sure that it's not.


I'd let you take a look.


And yeah, I know I see a lot of these sorts of queries


on the web.


And there's always something sort of nefarious


in the background.


And as you say, you guys know what you're doing.


But things have been pretty good.


And yeah, I don't know.


I'd like to see if you could take a look.


That'd be amazing.


Just I think.


So I have a bit more time after we end the official Hangout.


And if you want to stick around, I can take a quick look.


Yeah.


To see.


Thank you.


Sure.


Maybe we'll just switch to more questions from any of you


in the meantime.


It looks like there is also some in the chat here.


Let's see.


I have a question about sites using heaps of microsites


to link back to the main site.


A client or competitor has over 100 of these sites.


The site's linking over helped prop up


the rankings of the main site.


Is this business breaching any Google guidelines at all?


Anything I can do.


So that just sounds like a PBN, which is kind of like the games


that people play.


If you want, you're welcome to send me,


I don't know, the site in question.


And I can have someone from the web spam team take a quick look.


Sometimes it's OK to have multiple sites


and they link together.


Sometimes the multiple sites are essentially just there


as a network to prop up the ranking of another site.


And sometimes it's useful to kind of dig


into those details.


So if you want to drop in some URLs, then feel free to do that.


Additional to the product question.


I guess with different domains.


I don't know.


More questions from you all.


Maybe something live.


Probably a bit easier.


Nothing.


Oh my gosh.


OK.


Then someone pointed out an hreflang question.


Let me see if I can find that.


So maybe scrolling down some of the other questions


that were submitted.


And like I mentioned, if any of you


want to jump in, ask a live question, go for it.


Yeah.


I was just looking for mine here.


Go for it.


So after December, I started getting some weird messages


that my robots.txt was blocking important URLs.


And in fact, it apparently was because it was not


very supposed to do that.


Those URLs were like our main URLs,


which makes me think that probably somewhere in time


someone dropped a disallowed slash there


or something like that.


But the thing is, I went through all of our logs,


all of our internal systems, and I didn't see anything


changing our robots.txt.


We also have robots.txt monitoring alerts


that we basically ping it.


And if it's different than we expect, we get an alert


and we didn't.


When I went to Google Search Console, old version,


it allows me to see the previous version that Google has


in cache, something like that.


And it did not block any of those.


What I've been doing is basically I've been just


marking all of them as fixed.


But I'm afraid that it might happen again


in some time, and I don't know what could have happened here.


It's hard to know without looking at the site.


Double-checking the old versions in Search Console


is a great way to start.


I would try to figure out if this was a one-time event


or if this is something that is ongoing.


So in particular, if you submit those URLs with the Inspect URL


tool, does it say that it's still blocked,


or is it OK now?


They're all OK now.


No.


So if it was a one-time event, it


could have been something as simple as the robots.txt file


returning 403 for a brief period of time


or returning a 500 server error for a brief period of time.


Because both of those cases are situations where we say,


well, we can't access the robots.txt file,


therefore we won't crawl anything from the website.


And usually what happens is we don't


and usually what happens is the next time we


check the robots.txt file, which is maybe a couple hours later


or the next day, then we can crawl it normally again.


And we get all of the normal content,


and we can crawl the website again normally.


But during that period of time where we're essentially


blocked by the robots.txt file that we can't reach,


then we would assume that all of the URLs that we would


otherwise try to crawl are blocked by robots.txt,


and we might flag that in Search Console.


So if this is like a one-time thing, it happened once,


and it was a fairly short period of time and just a bunch


of URLs because we tried to crawl a bunch during that time,


then maybe that was just something weird that happened


on the server very briefly.


If you see this happening more than once,


then I would definitely try to determine where


this might be coming from.


One place that I usually go to debug things like this


is the server logs and, in particular, the robots.txt


request.


So filter out all of the robots.txt requests


that you got, in particular, from Google,


and look at things like the response size and the response


code that you gave back, and try to see


if there are differences in the sizes


or if the response code has varied over time,


and use that as a way to kind of narrowing


down where things went wrong.


Sometimes it's as simple as someone


pushed a new version of the website,


and the robots.txt file was made inaccessible during that push,


and we tried to crawl just during that short window.


Then the next time things go, it'll be better.


That might also be a sign that maybe kind of you're


pushing things to production process


could be improved so that it doesn't block Google


when it tries to access the robots.txt file.


SPEAKER 2 OK, now I think what might have happened


is we allowed Google to index a bunch more URLs


in the beginning of December, and it


had been hitting us really hard, like six times the usual.


And it was overloading our servers,


and we started returning four, two lines to Google.


And after some digging, I figured


that you received that as 503 or something.


So maybe that's just a message that was wrong.


It's not that the robots.txt was blocking,


is that you could not read it.


Now we would differentiate between 503 and us thinking


that a website is blocked.


Also, if the robots.txt file returns a 503,


then that's a temporary thing for us.


Then we'll try to get the next version of the robots.txt file


first before we stop crawling.


But if you return a 500, then that


would be a sign that the robots.txt file is blocked


for us.


SPEAKER 1 OK, but do you treat four, two lines as 500s?


SPEAKER 2 I don't know where we would treat four, two nines.


I don't know for sure.


SPEAKER 1 Yeah, because I remember the only thing


that I found around this was a post because of some Gary


answer on Twitter that it said that you might treat four,


two nines at the same way, meaning it will just slow down


crawling.


But what I saw here on my logs was


that we were not returning any 500s to Google, only four,


two nines.


And at the same time, the number of server errors


in our Google Search Console properties,


they started increasing by tens, even though we were not


returning 500s.


SPEAKER 1 OK.


I don't know.


I'll double check.


That sounds like it might be, it might fit.


But I'll double check on our side


to see how we would treat four, two nines.


SPEAKER 2 All right, thank you.


SPEAKER 1 OK, we're a bit over time.


So I'll pause the recording here.


If any of you want to stick around a little bit longer,


you're welcome to do that.


But otherwise, thank you all for joining.


It's been great having you all here.


It's nice to have so many people jump in, even on short notice.


So thanks for all of that.


And I wish you all a great weekend and a good start


next week in the meantime.


Bye, everyone.


SPEAKER 2 Thanks, John.


URL inspection: What SEOs need to know #AskGoogleWebmasters

All right.


Welcome, everyone, to today's special episode


of Ask Google Webmasters, here today with a special guest,


Martin.


MARTIN SPLITTENBERG Hi, everyone,


and thanks for hosting us, John.


JOHN MUELLER Oh, it's great to have you here.


So in the Ask Google Webmasters series,


we take questions that were submitted on Twitter


using the hashtag AskGoogleWebmasters.


Today's questions are on the topic of Search Console.


So let's get started.


Gev asks, Googlebot seems to be rendering only a part


of our pages according to the Search Console screenshots,


even after the recent evergreen Googlebot update.


Question, is it just the screenshot,


or is there something faulty that I need to worry about?


MARTIN SPLITTENBERG That's a good question.


And in order to not just answer with it depends,


let me go a little bit into more details.


So if it's content that you care for that is really


important to you, you should make sure that it's actually


rendered.


But you should also check the rendered HTML source code


if it's not in the HTML, because maybe sometimes it's


in the HTML.


It's not showing up on the screenshot


because it's below the fold or something.


So make sure that it's in the DOM in the source code


that we show as rendered source code.


And also, sometimes things just go wrong.


So try again, I guess.


JOHN MUELLER So if it's below the fold,


would it just not be shown, but still be used for indexing?


MARTIN SPLITTENBERG Yeah.


It's just we have to cut the screenshot somewhere.


So we're going to cut it where normally, especially


if it's a mobile crawler, where the screen for a mobile phone


would end.


JOHN MUELLER Cool.


Thanks.


MARTIN SPLITTENBERG All right.


Our next question comes from Twitter user TwitterInofficial.


And they ask, why does Google Search Console always


tell me that there's JavaScript error when I use arrow


functions?


Why does it not support them?


Well, that's an easy one, so thanks for asking me


that question, Martin.


The short answer is, we support them now.


So if you try again, it'll probably work.


For our last question, Merlinox on Twitter


asks, why does the URL inspection tool report


a status code 200 for a page that absolutely returns a 301


redirect?


Nowhere in Search Console does there


appear any indication that the URL tested gives a 301.


That's correct, yes.


The reason behind that is that the URL inspection tool


is designed to show you what happens when we index pages.


And what happens when we get a 301?


We make a new request to the URL that you redirected us to.


So we are not ending up with a 301.


We are ending up with whatever happens on the page


that you redirected us to.


In that case, a 200.


And that's by design, basically, because the URL inspection


tool, as I said, is meant to show you


what actually happens when Googlebot processes


your website.


So I can see how that might confuse someone who has


double-checked the result code, but that definitely


makes sense.


So all of these are about the URL inspection tool, which


is a cool part of Search Console.


I heard you even made some code changes there.


Is that correct?


I did.


Yes, that is correct.


I made a tiny little change in the code.


Wow, so what's involved there?


Can I just go in there and add a bunch of images of cheese,


or is someone actually watching out for?


There are people watching out.


So I worked with a team.


We agreed on building a certain feature.


And then they guided me through the process of making


that feature actually happen.


I wrote the code.


They made sure that it's tested properly.


And we are now running what's called an experiment.


So basically, new features are undergoing a testing phase


first.


And only if I pass that testing phase,


my code will actually run for your inspections


in Google Search Console.


So you can't just add new stuff to it or break existing things.


Wow.


On the one hand, I'm kind of sad I just


can't add pictures of cheese.


But on the other hand, it's good to know


that we're sending tools that actually work to our users.


Yes.


I hope you found these answers useful.


Thank you for submitting them all.


If there are more questions on the topic of Search Console


or anything else, feel free to submit them on Twitter


using the hashtag AskGoogleWebmasters.


All right.


Thanks, everyone.


Thank you, Martin.


And goodbye.


Thank you.


Bye.