Resolving rogue robots directives

We have a good selection of tools out there.


Resolving rogue robots directives

Yes.


Resolving rogue robots directives

And that's just one.


Resolving rogue robots directives

There's lots of other types of tooling you can do to understand if a page has changed,


Resolving rogue robots directives

which is really useful for SEO.


Resolving rogue robots directives

Lots of people have hands on a website.


Resolving rogue robots directives

You may not know everything that's happening.


Resolving rogue robots directives

It's very useful to know if something has changed and that way you can prevent potentially


Resolving rogue robots directives

long term effects.


Resolving rogue robots directives

Yeah.


Resolving rogue robots directives

And especially the change detection is a tricky one because usually when a client comes to


Resolving rogue robots directives

you and says, this is broken.


Resolving rogue robots directives

And you say, so what did you change?


Resolving rogue robots directives

And they say, nothing.


Resolving rogue robots directives

They never changed anything.


Resolving rogue robots directives

It was like that when I found it.


Resolving rogue robots directives

I didn't touch it.


Resolving rogue robots directives

And then if you have a tool like that, you can be like, well, actually, this has changed


Resolving rogue robots directives

last week.


Resolving rogue robots directives

Why?


Resolving rogue robots directives

And then the interesting investigation goes on as to why it has changed.


Resolving rogue robots directives

Exactly.


Resolving rogue robots directives

Good one.


Resolving rogue robots directives

Interesting.


Resolving rogue robots directives

So would you say like a robots.txt issue is one of the things that a lot of companies


Resolving rogue robots directives

fall into, like a problem that they have experienced?


Resolving rogue robots directives

I think so.


Resolving rogue robots directives

Because just for that very reason, a lot of times either they may have not had one configured


Resolving rogue robots directives

at all or a product development team may have thought they put the best effort robots.txt


Resolving rogue robots directives

and had no one really review it.


Resolving rogue robots directives

And that can have super detrimental effects on the site itself.


Resolving rogue robots directives

So it is pretty common and it can be tricky.


Resolving rogue robots directives

You can configure it if you want to block crawling for certain parts of the site.


Resolving rogue robots directives

And that's often a useful tactic to do.


Resolving rogue robots directives

So if you have like these very tricky filter pages that you actually don't care for or


Resolving rogue robots directives

like a bunch of content that is important for your customers but not really important


Resolving rogue robots directives

for people searching for your brand, then you want to disallow that from crawling.


Resolving rogue robots directives

That makes sense.


Resolving rogue robots directives

Robots.txt is such a fascinating but also slightly frightening thing because it is a


Resolving rogue robots directives

very powerful tool.


Resolving rogue robots directives

But as such, with great power comes great responsibility.


Resolving rogue robots directives

And that's an interesting one.


Resolving rogue robots directives

Would you say it's usually a good idea to just not touch your robots.txt if you know


Resolving rogue robots directives

it works?


Resolving rogue robots directives

Or what would you do if I have to change something?


Resolving rogue robots directives

If I come to you and say, we have these bunch of pages here that we want excluded from search,


Resolving rogue robots directives

what's the best way to do a change in robots.txt without potentially breaking it?


Resolving rogue robots directives

I would use the robots.txt testing tool, which allows you to test out how something would


Resolving rogue robots directives

be excluded or included before you make that change live.


Resolving rogue robots directives

So I can try things out before it potentially breaks my website's experience in search.


Resolving rogue robots directives

Okay, that's pretty nice.


Resolving rogue robots directives

Because there's lots of different types of regular expressions you can use.


Resolving rogue robots directives

It can get really tricky.


Resolving rogue robots directives

Oh yeah.


Resolving rogue robots directives

And if you do a slight mistake, the computer is going to do what you told it, not what


Resolving rogue robots directives

you wanted to tell it.


Resolving rogue robots directives

So that's an interesting one.


Resolving rogue robots directives

That's nice.


Resolving rogue robots directives

So we have like a set of tools there that you went through.


Resolving rogue robots directives

And I think we both agree that robots.txt is a fantastic tool at what it does, but it


Resolving rogue robots directives

needs to be treated with care.


Resolving rogue robots directives

And it's just always interesting to me to see how people are like debugging these kind


Resolving rogue robots directives

of things when they appear or like even spot them.


Resolving rogue robots directives

I mean, you were the ones who spotted it, right?


Resolving rogue robots directives

The product team wasn't even aware that this was going on.


Resolving rogue robots directives

And you mentioned monitoring.


Resolving rogue robots directives

And it's interesting because monitoring didn't come up so far.


Resolving rogue robots directives

And that's why I want to know from you, are you using monitoring?


Resolving rogue robots directives

And if so, what are you using to do the monitoring?


Resolving rogue robots directives

I know there's a bunch of different tools, but how do you monitor your website?


Resolving rogue robots directives

What tools are you using?


Resolving rogue robots directives

What services are you using?


Resolving rogue robots directives

How do you configure them?


Resolving rogue robots directives

What do you specifically look for?


Resolving rogue robots directives

We heard robots.txt and changes on the website.


Resolving rogue robots directives

Maybe you have other setups out there.


Resolving rogue robots directives

Let us know in the comments below.


Resolving rogue robots directives

That would be very interesting.


Resolving rogue robots directives

So for now, I'd like to say thank you so, so much, Jason, for being on the show.


Resolving rogue robots directives

Of course.


Resolving rogue robots directives

Glad to be here.


Resolving rogue robots directives

And I hope that you all enjoyed it as much as I did.


Resolving rogue robots directives

And yeah, stay safe.


Resolving rogue robots directives

Take care and bye-bye.


Resolving rogue robots directives

Boop.


Resolving rogue robots directives

Okay.


Resolving rogue robots directives


Welcome to the Webmaster Conference Product Summit (WMConf MTV '19)

Hello and welcome to our first ever Google Webmaster Conference Product Summit,


because no conference is good unless you can have five words to go with it.


And that's how we describe it.


I am Danny Sullivan, the public liaison for Search.


Some of you may have known me from my previous life.


And let me tell you, it has been an interesting journey where


I did not expect to be kind of doing a conference type of thing again.


And yet, here we are.


So I will try to reclaim all my older skills


and see if I can bring those back in with it.


We have a really great day planned.


I am super excited about this.


We have had events at Google for many times.


We've had different SEOs, Webmaster Content Producers and such come in.


But I don't think we've ever had anything to this degree


involving this many Googlers, as you will see as we get started.


And we're really excited to have all of you as part of this.


Why are we having this event?


So it's really that we want to better increase


our two-way communication between people who work at Google Search


and you folks and whoever's calling on this phone.


That was totally fine because it happens to me as well.


So we have lots of feedback.


Like there's no end of the feedback we hear.


Here's John Mueller just getting feedback.


Feedback, feedback, back handing the feedback.


So we have our office hangouts.


We have feedback that's coming in there.


We have feedback that comes internationally,


multiple languages all over the world.


Lots of, lots of feedback.


And we really are thinking about the ecosystem, the open web,


et cetera, et cetera, all the time.


And it really does come up.


In my time that I've been here, people are like,


what's it like working at Google?


Well, once they put me in the machine, they brainwashed me.


Everything's been great since then.


No.


There's a small thing like in Clockwork Orange,


and they held my eyes open.


And then it was all fine.


No, no.


It wasn't like that at all.


It's actually been like what I'd known


Google was like to be with, which


was a very thoughtful company that really does


care about these kinds of issues.


And it does come up in meetings.


And we had this one thing that came up


where we had launched something, and then people


were slightly concerned.


It almost never happens.


And I was like saying, well, OK, I'll come in.


And by the way, we really do think about this stuff.


And everybody totally believed me.


They're like, oh, thank you.


I felt really reassured.


No, people are like, yeah, I believe it.


So no, but so no one really believed it.


But as an example of this is the origin of this entire event.


So this is an email that Danielle Marshak sent out


who you will see later on.


And by the way, there have been a lot of people involved


this event.


And we will not be thanking all of them


because it would go on for ages.


But Danielle has been like a real hero of everything.


She has roped us all in.


And every week said, get it.


And so now we are all here.


So she sent this email out, I think it was March,


maybe it was April, saying, look, we want to have this


meet the ecosystem type of thing.


And Emily, Emily of the Emily Moxley,


who you'll hear from later, one of our senior people


I mentioned, thought this would be a good thing.


And it's just like coming down, it's coming out just


organically, organic, from within Google saying,


we would like to have this type of thing.


It's not like, OK, let's.


It's part of that thing that's already in with what's


going on with all the products.


So yeah, we do think about it all the time.


And this is one of those ways that it's coming out.


And then going back to that earlier thread


that I mentioned, so it came up and I said, look,


we really do think about the ecosystem.


And in fact, it's important.


If Google wants to support the ecosystem,


because we really do recognize that if we do not do that,


if the ecosystem isn't thriving, we can't thrive.


So everybody, it's that virtuous loop,


or however you want to describe it.


It really is that our concerns are your concerns,


and your concerns are our concerns, and kumbaya,


whatever.


But the other, I think, big challenge


is all this is that search has to keep evolving.


One nice thing is you all have job security, as you know,


because every day we'll just roll something else out.


And you'll go, oh, that stuff has changed.


It's all different.


Bert changes everything.


Everything, SEO, it's dead.


No, it's none of those things.


So just, again, you can attribute to me


that it's not dead.


But search has to keep evolving, because you remember,


you turned on your computer and you waited three minutes.


And then after it booted, you plugged in your modem


and then you waited another three minutes.


Are there any children that don't remember what a modem is?


Because you're very lucky.


So that's how we searched, and that's


how people got their stuff.


And this is how people are searching today.


They're on the back of bicycles with their phones.


I don't know if you've seen that, but this is truth.


That's everywhere out there.


I see it all the time, people on the back of bicycle.


No, people are searching.


They're on the move.


They're all ages.


They're constantly searching all sorts of ways.


So we have to keep search evolving,


because if we don't keep doing it,


then people aren't going to be searching with us.


Now, how do we know as we evolve whether or not


we are still doing the right thing by the ecosystem?


And one thing could be, well, we'll look at traffic.


And one metric that we have of just one thing in terms of news


is, well, we send like 24 billion visits


per month to news sites.


So that's a lot of visits.


So you think we're supporting news, as we hope that we are.


But if you just look at traffic, then you're


kind of dealing with an imperfect measure.


Because let's take Bert here.


And here's before Bert, and someone did this search for,


I want to get medicine for someone at a pharmacy.


And before we had Bert, we would say, all right,


here's how you get medicine for yourself,


which sent that site traffic, but wasn't really


answering what the person was looking for,


wasn't really providing what they were.


And then afterwards, we're sending to a different page


at a different site that is actually saying, well,


this is how you can get medicine for someone else.


So traffic's been lost.


And you could say, well, now the ecosystem's been harmed.


And particularly, that site's been


harmed because they didn't get the visit they should


have gotten in the first place.


I was getting this visit.


Now I'm not getting this visit.


It's horrible.


But you were getting a visit.


But was it actually a visitor that was useful for you?


That's just one of the things you


have to consider when you look at all this stuff.


And I don't know all the answers to this.


And we don't know all the answers to this.


That's why we invited 200 of you all


to come because we figure you know all the answers to this.


What is it?


No, anyway, you can tell us later.


But one of the things we would like to get


is a sense of, well, how should we


assess if we're supporting the open web and the ecosystem?


What are some of the metrics we might want to be considering?


What are some of the ways that, to you, are important to know


whether or not we're doing the right thing by both


your sites and that you would think of, I know this is hard,


but sites in general?


What are the ways that Google can improve our support


overall?


We definitely want to understand that as well.


So that's part of all this that's going on here.


And really, anything that's on your minds at all,


we want to know.


That's why you're here.


So don't be shy, as I know that so many of you can often be.


I have seen your tweets.


I don't really want to say this.


And then you hold back, so I won't.


No, we really want to hear everything from you.


So we are so, so glad to have you out here today.


How often does Google re-index websites?

Today's webmaster question was submitted by Jim


from Vancouver in Washington.


He asks, how often does Google re-index a website?


It seems like it's much less often than it used to be.


We add or remove pages from our site,


and it's weeks before those changes are


reflected in Google search.


You too might be wondering, how long does it


take for Google to recognize bigger changes on a website?


And from there, what can you do to speed that up?


Looking at the whole website all at once,


or even within a short period of time,


can cause a significant load on a website.


Googlebot tries to be polite and is


limited to a certain number of pages every day.


This number is automatically adjusted,


as we better recognize the limits of a website.


Looking at portions of a website means


that we have to prioritize how we crawl.


So how does this work?


In general, Googlebot tries to crawl important pages more


frequently to make sure that most critical pages are


covered.


Often, this will be a website's home page or maybe


higher level category pages.


New content is often mentioned and linked from there,


so it's a great place for us to start.


We'll recrawl these pages frequently, maybe


every few days, maybe even much more frequently,


depending on the website.


Thank you for watching.


Subscribe to the channel to be notified about our latest


English Google Webmaster Central office-hours from December 13, 2019

All right.


Welcome, everyone, to today's Webmaster Central Office Hours


Hangouts.


My name is John Mueller.


I'm a Webmaster Trends Analyst here at Google in Switzerland.


And part of what we do are these Office Hour Hangouts,


where people can join in and ask their questions all


around their websites and search.


A bunch of things were submitted already,


so we can go through some of that.


But as always, if any of you want


to get started with a question, feel free to jump on in.


Hi, John.


Hi.


Recently, we have faced one issue with one


of our clients online ranking.


So most of the time, it is a gift basket or gift camper


related website, e-commerce website.


And it ranks very well after a few, four, or five years.


But this year, we made some changes on the website.


The blog we have, we moved it to the subdomain.


So now blog is the blog URL is like blog.domainname.com.au,


something like this.


That means all the URL of the blog post are changed.


Now, our strategy was writing good content for the blog


and abstract backlink.


And we give link from our blog to the product category


and product page.


But after changing this thing, after moving the blog


from the domain to the subdomain,


the ranking dropped a lot.


So is this the reason?


Because we did 301 redirection, we


submit the subdomain to the Google search console.


But do you think it is because of the moving the blog


from the domain to the subdomain?


It might be.


It's hard to say kind of absolutely.


But what might be happening there


is that we start seeing this as a separate site.


And we see essentially you going from one site


to splitting it up into two sites.


And that takes a while for us to understand.


And it takes a while for us to understand


kind of the context of those two new sites,


even if it's the same content as before.


So that's something that could be coming from that.


But it is really kind of hard to say.


And it's something where there is not really


a technical indicator that would tell us this is because of this


or because of that.


It's essentially just kind of the new ranking.


And if this is something that was recently


done back then in the last couple of months,


then I could imagine that it just takes a little bit longer


to settle down.


SPEAKER 1 OK.


And the next question is the structured data.


So if I add two types of structured data in one page,


is it OK?


For example, if it is an e-commerce website


and local business, so if I add one structured data


for the product and another structured data


for the local business, is it OK?


JOHN MUELLER Yes.


Yes.


SPEAKER 1 And my last question.


Does Google consider blog comment as main content


or supplementary content?


JOHN MUELLER It depends.


I mean, we see it as a part of the page.


So that's something where we would probably


see it as part of the main content in the sense


that when we look at a web page, we


have things like the navigation all around, the footer,


the sidebars, these things, which are kind of the parts


that we would see as boilerplate content, so not as critical.


But the blog post and kind of the changing content


on a blog page, which would include the comments,


is something that we would see as the primary content.


We would try to understand that these are blog comments.


But essentially, these are a part of the primary page.


And sometimes what can happen is that maybe you


don't have a lot of content in your blog post,


but there are lots of really good comments on there.


So it's important for us to say, well,


this could be something that is really


useful for this particular page.


So we should not just ignore it by default.


SPEAKER 1 Thank you.


JOHN MUELLER Sure.


All right.


Let me run through some of the questions that were submitted.


And as always, you're welcome to jump in in between.


And we should have some time towards the end


as well to go through other questions that come up.


Can editing the robots.txt on the main domain


affect crawling of images that are served by a separate CDN


domain?


What happens if the robots.txt for the CDN returns a 503?


So robots.txt is per host name and protocol.


So if we're looking at a web page that's hosted on, say,


www.example.com, and that includes content


from a different subdomain or from a different domain


completely, then we would use the primary robots.txt file


on www.example.com for that page.


And for the embedded content, we would


use wherever that embedded content is hosted.


So that's something where, essentially,


for that request that we want to make to the server,


we check for that subdomain, for that host name,


whether we are allowed to crawl it.


So blocking something on the www version


would not block it from being crawled


from a different host name or different subdomain.


What happens if the robots.txt returns 503?


So 503 is a temporary error that basically tells us


we should try again later.


It's a bit of a tricky one, probably the question


because of the question as well, in the sense


that we see these errors as blocking in that, by default,


when we see a server error, we say,


we don't know what the robots.txt file is.


So therefore, we will not crawl anything from this host name.


That's kind of the default situation.


However, sometimes we see that these kind of server errors


are more like a permanent thing.


They're not a temporary situation


where we just temporarily can't get access to that file,


but rather we see this as a permanent error.


And then in cases like that, we often


fall back into kind of the state that, well, maybe


the server is just sending us the wrong response code


and this should be a 404 page.


So what generally, I think we have this documented


in the robots.txt documentation as well.


What generally happens is we see the 500 or 503 error,


we stop crawling completely, and then


after a certain period of time, I don't know,


maybe a couple of months or so, I


don't know if we have anything specifically documented there


with the time frame.


After that period of time, we think, well,


this is a permanent error, so therefore we


should treat it like a permanent file I found


and try to see what we can crawl normally.


So that might be happening there.


If it's been 503 for a really long time,


maybe we're crawling it because of that.


Hi, John, that was my question.


Part of the reason I asked it was


there was a bit of misinformation going around


and people giving incorrect advice,


so I kind of wanted a bit of a backup on it.


And the scenario was interesting.


It was a client with Google Shopping,


and their shopping campaign got completely switched off


because of the 503 error, and they couldn't access


the images, and two days ago, the company fixed the problem,


and instantly the images came back,


so confirming that's your behavior.


Yeah, that seems kind of the way that it should be working now.


So a 503 is also a good way of stopping crawling completely.


If there's some technical reason that you really


need to stop crawling of your website,


then you can return 503 for your robots.txt file,


and we'll stop crawling as soon as we reprocess that,


which is usually within a day.


So sometimes you need to have it stop crawling.


Sometimes you don't want to have it stop crawling.


Cheers.


Cool.


Then question about thin content.


Several months ago, we removed a lot of thin pages


and redirected them to a single page, which


consolidates the content from these thin pages into one page.


The thin pages are still being indexed in search,


even though they redirect to the new page.


The URL inspection tool shows the thin pages


as returning 200, but the screenshot


shows the correct redirected page.


What's happening here?


So I think there are a few things combined here.


On the one hand, redirecting thin pages


and making a better version of a page perfectly fine.


I think that's always a great idea.


On the other hand, the URL inspection tool,


we try to show the content that we would end up indexing there.


So we silently follow redirects on pages like this.


So if you look at a page that is redirecting to a valid page,


then the URL inspection tool will say,


well, there's a valid page here because that's


what we would index under that URL.


So you don't see that redirect there.


That's kind of confusing if you're


trying to diagnose a redirect like this.


The last thing that almost always happens with redirects


is that we continue to understand where the page used


to be hosted, kind of the old URL.


And if someone explicitly looks for something


that seems like they're looking for the old URL,


we'll try to show it to them.


So a really common way to see this


is if you do a site query for an old domain,


if you've moved to a different domain,


you'll continue to see a lot of results there.


And that's not because we kind of


didn't process those results, but rather because we


think you're looking for the old domain,


so we'll show it to you, kind of trying


to be helpful when probably you don't want us to be helpful.


So with a site query, that's pretty common.


But what I imagine might also be happening in your specific case


is if someone is looking for a query


or looking at a query that looks very similar to your old URL,


like taking the words from the old URL


and they're looking for that, then it


might be that we would show that old URL in search as well.


A good way to double check that we're picking up the new one


is the URL inspection tool, kind of checking out


which one we have indexed, looking at the cache page.


So usually if you look at the cache page,


you'll see the URL of the new page, which with that example


that you have there, that's what's happening there.


So from my point of view, I don't


think you need to do anything different here.


It's just kind of maybe it's good to be aware


that sometimes we show the old URL in search,


even though we've switched everything over to the new URL.


And I think you also mentioned the URL removal


tool in the question.


You definitely don't need to do that.


The URL removal tool would just hide that page in search.


So in that case, if someone is looking for the words that


would trigger your old URL to be shown


and you have the URL removal tool active,


then we would not show anything.


We wouldn't even show the new one.


So the removal tool doesn't affect


the choice of canonical or the choice of the visible URL.


I have a website that was of mediocre quality,


but I improved it a lot in terms of content.


Now it's suddenly appearing on pages four to six


and sort of stuck there for the last two months.


I read you once mentioned that the effects


of refining websites can take more than six months


to reflect in search.


Does that still hold true?


If yes, then it wouldn't make much more sense


to start with a new domain instead of kind


of improving the old one.


So six months is not a fixed time.


That's essentially just something


that I see from time to time.


And it's particularly something that with larger websites


you tend to see.


If you make really significant changes on a larger website,


then there's a lot of content that


needs to be recrawled and reprocessed,


and that takes a lot of time.


So that's kind of where I pulled that six months number out.


It doesn't mean it will always take six months.


It doesn't mean that it will always


be complete in six months.


Sometimes it takes longer.


A lot of times for smaller websites,


it usually ends up happening a little bit faster.


So in your case, it sounds like it's more of a smaller website.


If you kind of significantly improve


the content of the whole website,


then that seems like something that you can kind of,


I don't know, like I'm guessing on the order of maybe 100,


couple hundred pages.


And that's something that usually we would be able to


crawl and index a lot faster.


With regards to the ranking in general,


there is no kind of fixed rule that we


would rank pages significantly higher after you've


improved the content.


These things do take time as well,


and sometimes improving the content,


there's a lot involved with improving a website.


It's not just like rewriting some pieces of text


and making them look a little bit better.


So that's something where I'd continue


to work on this to try to find ways to improve that.


If you're active in a competitive area,


then it might also make sense to kind of think


about how you can differentiate yourself from other sites,


not just with regards to kind of the content that you have


there, but kind of with your offering in general.


Like, what can you provide that is significantly


different than everyone else?


That is such that when people see your stuff


for the first time, that they'll be able to,


or they'll want to come back specifically to your site.


Hi, John.


Hi.


Yo, how are you?


Pretty good.


Yeah.


And I have a question regarding Google Search Console.


Like, one of the websites we have in Google Search Console


account, it's showing that the site map is not


able to be pitched.


So when I cross-check the in detail error,


what is it showing that HTTP 301 error.


But in fact, there is no 301 redirection has been defined.


And the URL is accessible without any problem.


I just want to know, like, will it


be any temporary kind of thing, or do we


need to do anything specific?


And right now, there is no preferred domain setting also


is not there right now in the new search console.


Like, how can fix this?


Yeah, so preferred domain you wouldn't need for a site map


file.


But like with any of the other features in Search Console


that are not migrated over to the new Search Console yet,


that's something you can continue to use, those features.


Just because it's in the old Search Console


doesn't mean you can't use them.


So I would continue to use that.


With regards to site map file, it


might be good if you could send me a link to the site map file.


Maybe you can drop it in the chat here.


And then I can take a look at that with the site maps folks


afterwards.


Sure.


Thank you.


Cool.


All right.


Let's suppose we have two authors.


One is an expert, and one is a no name.


They write exactly the same piece of content.


Will there be any difference in ranking


based on which name is shown as the article author?


I don't know.


This feels like a very theoretical question


in the sense that in practice, if you're writing as an expert,


then you highlight your expertise.


And you show to users, you show everyone


that you know what you're talking about.


You kind of have background information.


You can back your information up.


So that's something where I would expect, even


from a user's point of view, to understand


that there's a difference between this kind of content.


And in general, when you're making a website,


I would take steps to make sure that you're highlighting


why your version of this content is a good version


and why people should be able to trust you because you have


background information or you kind of have


all of these references that you can point at


or you're an expert in this field.


All of these things are things that I


would try to highlight as much as possible.


And that's something that we would


be able to pick up directly or indirectly sometimes.


It really depends on the pages themselves.


But it's something I wouldn't kind of treat


as a theoretical thing, but rather,


this is a practical way of showing


why people should trust your website and kind of come to you


rather than to random other people.


So that's something where I would expect things to,


all that to kind of make sense and help out a bit.


Oh, wow.


So many people jumping in.


Does Google consume structured data better


if it's coded in an at graph versus separate snippets


in JSON-LD?


And there's a link to, I think, a plugin


that does structured data.


So I'm not aware of any difference with regards


to how we process this structured data.


In general, what happens with structured data


is either we can read it or we can't read it.


It's not that there is a better or a worse.


Or kind of like, if you do it like this,


provide the exact same information in structured data


as you would do it in a different way,


that one would be better and one would


be treated with more weight or given a ranking advantage


or anything like that.


If the structured data is valid, if it


works in the testing tools, if it shows up


in Search Console as something that we've processed,


then it works.


So there is no better or worse from our point of view


if we can process it.


And I know Yoast, specifically, that you linked to there,


they spend a lot of time to make sure that their structured data


works.


So that's something where if it works for you by using


a plugin like this, then go for it.


I wouldn't say you have to manually code it


or you have to do it exactly the same way as we have


in the developer documentation, but rather if it works,


if it's technically correct, if it can be processed,


then it can be processed.


I'm having an issue where Google is ignoring 301 redirects


when selecting canonical URLs.


So I think this probably falls into a similar situation


as one of the earlier questions.


So one thing maybe I didn't add with the other question


is when it comes to canonicalization,


we use multiple factors to try to figure out which URL we


should be showing in search.


So which one we should pick as a canonical URL.


A 301 redirect is definitely a really strong sign.


There is also the rel canonical that you


can put on a page, the meta tag or the header tag.


Internal linking plays a really big role for us.


If you're saying this page is canonical


but you're not treating it as a canonical,


then that would be kind of tricky for us.


External linking plays a role as well.


Sitemap files, hreflang, all of these references


of individual URLs within your website,


they do play a role for us.


So that's something where sometimes these signals all


align and we try to follow that.


And sometimes they're a bit confusing and we have to guess.


And if you make it so that Google has to guess,


then Google will guess and maybe it


won't guess what you want to have happened.


So that's kind of just as background


with regards to canonicals.


The other thing that I think is also really important


to realize is that these things are generally not


a ranking issue in the sense that if Google picks


a different canonical, it doesn't mean


your pages will rank worse.


It just means that we will show that page


with that specific URL in search.


And usually the ranking would be exactly the same


as any other canonical page.


So for us, it's more a matter of a practical thing,


like which URL should we show in search, which URL should we


be crawling the most.


That's kind of the thing there.


So if we're not picking the canonical that you want,


at least it's still ranking in the same place.


It's not critical that you urgently


need to jump in and change something


to make Google pick that up.


So let me look at the examples that you have there.


You have a redirect from a page that moved from one domain


to another.


Google selected the old domain as canonical


despite visiting the new page in October.


So to me, there might be two things happening here.


On the one hand, maybe you're doing a site query


for the old domain.


If you do that, then we will definitely


try to show the old domain, even for a really long period


of time.


Sometimes multiple years we'll know


that the old domain used to be here


and that it's now redirecting.


And doing a site query for old domain will still show that.


So like October of this year, that


seems like something definitely we could still


be showing if you explicitly look for that old domain.


The other thing is if you're saying


this page was visited in October,


that sounds like a page that we don't crawl that often.


So that's probably something where


maybe it's just a matter of us getting to all of these pages


as well.


But kind of like I said in general,


this is not something that would be affecting


the ranking of this page.


It's really just the URL that is shown for that page in the search


results.


Redirect on a category page, adjusting the URL structure,


now using parameters.


That seems like something where it


might be that you're explicitly looking


for those with a site query.


It might also be that there's still


kind of conflicting signals that we're


seeing for these kind of things.


So if you're changing the internal URL


structure of a website, then often we'll


still see a lot of internal links


that are going to the old pages.


And as a user, when you're clicking around on the website,


you don't necessarily immediately see that.


But what you can do is use a local crawler, something


like Screaming Frog or Deep Crawl or these various crawler


tools, and run it over your website


to really make sure that there are no links still going


to the old version of the page that end up being redirected.


So if you can really make sure that your internal navigation


is clean and only points at the new URLs,


then that helps us quite a bit.


MIHAI APERGHISLAYA-SINGH CAMPBELLO-MURPHYASAKIYANU


Hi, John.


Couple of quick questions on canonical topic.


One is related to that experiment


I'm running with changing the URL to see if the page will


start ranking better.


You mentioned last time just show a 404 code to Google,


while also showing the page to users


so that Google drops it completely out of the index,


while users can still access it.


Because Google did the canonical from our old URL


when we dropped it to the new URL,


despite showing a 404 for the old URL.


So the idea is that we did that.


We did the 404 code, and the page completely


dropped out of the index.


But the old URL is still showing a canonical to this one


that we 404.


So is it normal that I can see in Search Console


a URL that shows a canonical to another page that's 404 now?


I think that could happen.


But that's more like a temporary state,


in the sense that we need to propagate


all of this information back in the index as well.


I imagine over time that that will just


be seen as a 404 everywhere.


So if I change the URL to a new one now,


and let Google index it, would I have any issues in those older


URLs changing and pointing as a Google choosing


a canonical for this new one?


I don't know.


It seems like a very confusing state at the moment.


Yeah.


So should I just wait a bit more?


I would let that settle down a little bit first.


OK.


You're testing all of these weird things


that are like internal anecdotes almost of how we store things.


So it's kind of cool to see.


Living dangerously, yeah.


Yeah, I'll just let the holidays pass then and see afterwards.


One other thing is related to the URL parameters tool.


So the documentation doesn't really cover one case where,


so you can choose which page should Google crawl,


and you have the representative URL there.


So the recommendation doesn't cover that case.


So I'm not sure exactly what representative URL is.


I mean, would Google crawl that URL as the representative URL


or the one that it canonicalizes to?


Or what exactly is that?


No, we would try to pick that one that you specify.


So for example, if you have, let's say


you have multiple product variants,


and you have a parameter or something


that specifies which variant.


And you have, I don't know, different color shoes,


for example.


Why not?


Like black, green, blue, and the parameter


is color equals black, color equals blue.


Then you could say this color parameter is one


that you don't want to have crawled,


and the representative one is always black.


It's like black shoes fit everywhere kind of thing.


So that's how you could tell us this


is the one representative URL that


replaces all of these others.


But in the URL parameters, you don't


get to choose the parameter value, though.


You get to choose the.


So let me give you an exact example.


Maybe I'm misremembering that setting.


But that's what I thought.


OK.


I'll have to click into that to double check, though.


Just to give you a quick example,


let's say you're using UTM tracking parameters,


and you have that parameter there, UTM campaign,


and you choose representative URL.


Does that mean that Google will try


to pick those URLs that contain UTM campaign


as the canonical version if you choose representative URL?


I don't know.


You're confusing me now.


I would need to.


I'll double check afterwards.


If we still have time afterwards,


maybe we can double check together.


Sure.


Cool.


OK.


Does Google throttle or reduce link value


to content that is significantly updated?


For example, you may have very old content


with lots of links that's seriously out of date.


If you rewrite the content, does Google kind of say,


well, this isn't what people link to,


so we're not going to pass value to those links anymore?


No.


We don't really do that.


The one thing where I see that sometimes happening


is when it comes to things like expired domains.


So if you just buy some expired domain


and you put your own content up there,


then from our point of view, that's a different site.


It's not like suddenly the old expired domain is passing


everything onto your site.


So that's the one case there.


But with normal changes within a website,


it's absolutely no problem.


Sometimes content changes.


Sometimes it changes significantly.


That's perfectly normal.


I used a Korean character on a URL of my posts.


But when I share it on Facebook, the URL


appears in this format, kind of the escaped URL version.


The URL becomes so long.


Does it also affect my SEO, or does it change anything,


essentially?


It doesn't change anything.


You can use an escaped version of the URL.


It's essentially treated as exactly the same URL


from our side internally.


If we see a link that goes to your website that


uses an escaped URL, we will kind of


unescape that when it comes to indexing.


Similarly, internal linking, you can use escaped URLs,


or you can use direct Unicode characters in the URL.


Both of those are completely equivalent.


It's not that there's a redirect in place there.


It's essentially the exact same URL, just different ways


of writing it.


Can you tell me whether schema.org slash course markup


is supported outside of the USA?


I don't really know.


But I will check for you.


My understanding was it's supported everywhere.


But you're saying it's not.


So I'll double check.


You earlier recommended not to block URLs with parameters,


since Google can't canonicalize the pages correctly,


and that the value from links to those pages gets lost.


For example, we should use rel canonical instead of robots.txt


or meta noindex.


My question is, on an e-commerce website with many pages,


do you also recommend nofollowing links


to filters which create URL parameters for crawl budget


reasons?


I would do that.


I mean, that's something where using nofollow probably


makes sense.


If you create a lot of filter variations for URLs,


then that could make sense.


But you can also just link, essentially, normally


with normal URL parameters.


And you can also use the URL parameters settings tool


to help us with this.


We have a really, really long blog post


on faceted navigation with, I think,


it's like five good practices and five bad practices, which


covers all of this.


So I would really double check that.


The thing with robots.txt in particular,


I think I mentioned this, is we don't know


what is behind that URL at all.


So we can't canonicalize things.


If there's a link there, we don't know what to do with it.


That's kind of the one thing there.


The other thing that I realized when I first


read this question, when I looked at it before,


is our crawling and indexing team,


they have never told me to go to a website


and tell them to use robots.txt to block something


like filters or pagination on an e-commerce website.


It's always the opposite cases.


So it's something where if there is an issue with regards


to crawling and indexing content,


the crawling team is more likely to tell


me to go and contact the website and tell them,


hey, you have something incorrect with your robots.txt


file, you need to unblock things so that we understand


your site better.


And they've never sent me out to contact any e-commerce site


and tell them to add robots.txt blocks because we


get lost in the filters or we get lost in the pagination.


So that's kind of as an anecdote there.


They've also never told us to go off and tell people


to use nofollow for internal navigation.


So that's another interesting part there,


in that for the most part, we should


be able to deal with e-commerce sites


and the normal navigations that they provide.


For AMP search results, does Google


gather the article structure data from the AMP page itself


or from the canonical page?


Will the AMP rich results work if only the canonical page


has the structured data?


So this is a confusing one that confuses me, at least,


apparently also you, in that we need


to see the markup on both versions of the page.


It's not just that we would say, well,


the AMP version canonicalizes to the desktop version,


so we'll just use that one.


But rather, we would show the AMP version


in the search results.


So we need to make sure that the markup there


is also that what we would show in the search results.


So things like the article markup, that matters.


Also things like metadata, like robots.txt meta tags,


they matter there too.


If you have a noindex on the AMP version


and an index on the desktop version,


then should we show the AMP version


or should we not show the other version?


It gets really confusing.


So ideally, make sure that the AMP version


has the same structured data, the same metadata


as the primary version of the page.


Or when I talk with the AMP team,


they're always like, can't you encourage people just


to use AMP?


If they're already using AMP for the content


and able to use it, sometimes it makes sense


to just use AMP as a web framework


and make the whole page AMP.


Does a website's cache configuration


have anything to do with the time it takes Google


to index it?


No.


Not at all.


So the cache configuration usually


applies to how a browser would be


able to load that page with regards


to being able to cache CSS files or things like that.


And that doesn't affect us for indexing the web page at all.


For rendering, we also pretty much


ignore all of those cache parameters


because we do things so differently in rendering


than a normal browser would do.


So we can cache things like CSS files for a really long time


and just reuse that, and it works well for rendering.


Whereas in a browser, maybe you don't


have that much storage to actually cache things that long.


Machine learning has been a part of Google Search algorithm,


and I can imagine it's getting smarter every day.


Do you, as an employee with access to the secret files,


know the exact reason why pages rank better than others?


Or is the algorithm now making decisions


and evolving in a way that makes it impossible for humans


to understand?


We get this question every now and then.


And we're not allowed to provide an answer


because the machines are telling us


not to talk about this topic.


So I really can't answer.


No, just kidding.


It's something where we use machine learning in lots


of ways to help us understand things better.


But machine learning isn't just this one black box


that does everything for you.


You feed the internet in on one side, the other side,


come out, search results.


It's a tool for us.


It's essentially a way of testing things out a lot


faster, trying things out, figuring out


what the right solution there is.


So for example, we use machine learning for canonicalization.


So what that kind of means is we have all of those factors


that we talked about before, and we give them individual weights.


That's kind of the traditional way to do it.


And we say, well, rel canonical has this much weight,


and a redirect has this much weight,


and internal linking has this much weight.


And the traditional approach would be to say, well,


we will just make up those weights, those numbers,


and see if it works out.


And if we see that things don't work out,


we will tweak those numbers a little bit.


And with machine learning, what we can essentially do


is say, well, this is the outcome


that we want to have achieved.


And machine learning algorithms should figure out


these weights on their own.


So it's not so much that machine learning does everything


with canonicalization on its own,


but rather it has this well-defined problem.


It's working out what are these numbers that we should have


there as weights, and kind of repeatedly trying


to relearn that system and understanding,


like, on the web, this is how people do it,


and this is where things go wrong,


and that's why we should choose these numbers.


So when it comes to debugging that,


we still have those numbers.


We still have those weights there.


It's just that they're determined by machine learning


algorithms.


And if we see that things go wrong,


then we need to find a way, like,


how can we tell the machine learning algorithm that


actually, in this case, we should have taken into account,


I don't know, phone numbers on a page


more rather than just the pure content


to kind of separate local versions, for example.


And that's something that we can do when we kind of train


these algorithms.


So with all of these machine learning things,


it's not that there's one black box


and it just does everything and nobody knows why it does things,


but rather we try to apply it to specific problems


where it makes sense to automate things a little bit in a way


that saves us time and that helps to pull out patterns


that maybe we wouldn't have recognized manually


if we looked at it.


Yeah, I know it's been on the news a couple of weeks


ago that Google has developed a set of tools


to be able to kind of explain what's going on in machine


learning models.


And you gave that example with weights


to pages where which canonical version should they pick.


So are you using those kind of tools


to kind of understand what weights are actually


being chosen and whether those are the correct ones


to choose and adjust afterwards and things like that?


I don't know specifically the setup that you're referring to,


but we do try to understand as much as possible what


is happening there.


And that's something where the search quality team,


they do have ways of debugging all of these things.


So on the one hand, you can do that


by separating out these problems a little bit more.


And on the other hand, it's like you just


with a complex system like Google Search,


there are just always a lot of individual pieces


that are moving around.


So even without machine learning, it's not trivial,


but it's something that we need to be able to debug.


Because people come to us with weird ranking issues.


Like, I get them from here every now and then.


You've sent me some really weird ones.


And it's something that we need to be able to figure out


why is this happening?


And is this happening at a larger scale?


Is it just happening to this one page or this one site?


What can we do to improve it?


So that's something where you always


need an understanding of what happened here,


how did we get to this state, how can we improve it.


Yeah, I think it's called explainable AI.


I think that's how Google coined it.


OK.


Yeah.


I don't know if we use that exact thing.


I mean, there's so many different AI machine learning


systems out there nowadays.


I could imagine we use something like that


for some aspects of search or some of our systems


and other aspects use other things.


So no.


But in any case, also the meta question here,


does anyone really know how the algorithm works?


And that is something where we do


have a lot of people in search quality that


are able to debug pretty much any query


and understand what is happening here.


Why are things ranking the way they are?


That's something that, for example, Gary has spent


a lot of time on as well.


And he's been working on some of these search things too.


So it's not limited to this elite set of priests


somewhere in Mountain View that know what is happening,


but rather engineers can learn it.


And it's something that we need to have at that level.


It's not something that we need to have at that level.


It's not like it shouldn't be a big black box


if we want to be able to improve it.


OK, really long question on videos.


So it goes into video structure data markup.


And the questions are, let's say I host my video on YouTube.


Would structure data and video sitemap


help Google to choose my landing page to be


ranked over my YouTube page?


Not necessarily.


So the structured data gives us more information


about how you're embedding the video,


but it doesn't necessarily mean that your page will


be better suited than the YouTube landing page.


However, it all depends on the query and the type of content


you have on these pages.


So if you have really good content that matches well


what the user's intent is for a query,


then we'll try to show your page.


And if otherwise the same content


is available on the YouTube page and it's


really visible and prominent there


and it's kind of hidden away on your page,


then maybe the YouTube page is a better landing


page for that specific query.


It's not that we would always show one or the other.


It's that we try to understand which of these pages


is more relevant.


And then, let's see, how does Google


make sure that the page, including the video structured


data or video site map, belongs to them,


that it's the same entity?


I believe we don't necessarily do that in the sense


that we're trying to figure out what the most relevant page is,


but we try not to make any algorithmic evaluation on who


the owner of the video is and is this person who's


embedding the video allowed to embed that video.


In practice, this is something that you


can specify on YouTube where you can say,


I want to allow embedding of the video


or I don't want to allow embedding of the video.


And if the video doesn't work on a page,


then chances are it's not going to perform that well in search


in the sense that if users don't see that video ever,


they're not going to recommend that page as a video landing


page anyway.


Finally, if the goal of my video campaign


is to drive more traffic to my website instead of brand


awareness, some SEOs suggest to use


self-hosting instead of YouTube.


Do you agree with this point of view?


You can do that if you want.


So I think one of the big advantages of using


a platform like YouTube or Vimeo or any of the other big video


platforms is that they have infrastructure


that makes sure that your videos just


work really well everywhere.


They have caches and content delivery nodes,


pretty much worldwide, to make sure that it just


works really quickly.


It doesn't have to go through your one server.


So that's something where I think


there's a lot of value in using these kind of setups


as CDNs for video content.


So that's something where I wouldn't necessarily


say you should avoid it or you should host it yourself.


But rather, if you want to host it yourself,


that's perfectly fine.


That's totally up to you.


But just keep in mind that sometimes


the technical requirements can be quite tricky.


Hi, John.


Can I add a question four to that?


Sure.


I've been doing quite a bit of video markup lately.


And what I find is quite a few of the parameters


that you want to do in the markup


will not be visible on the page.


Things like duration of the video,


even maybe the description because the video kind


of describes itself.


So is there any issue if you are marking that up in meta tags


and things like that?


Does Google understand it's kind of in the video?


I mean, what we wouldn't do is extract


kind of the content of the video to try to figure out


how we should rank it.


So things like transcribing the video


and understanding what the text is inside


and then trying to rank that video based on that,


I don't think we would do that.


From YouTube, that's something where


I believe they show the transcription


as a part of a page.


So theoretically, that could be on the page there.


But when it comes to descriptions and things


like that, I would try to put those on the page somewhere


or in the video markup itself.


That's also one of the things where a lot of these things


come to me when it comes to mobile-first indexing


in that the mobile-first indexing team will come


and say, well, this site is not performing as well on mobile


or when we would switch it to mobile-first indexing


because the videos are not able to be picked up properly.


And often, that comes down to video structure data.


It comes down to the positioning of the video on the page.


Is it a prominent part of the page or not?


Where on mobile, maybe you have a completely different layout


of the page, and suddenly the video, instead of being


like in the middle and center and large,


is like the small aspect in the corner somewhere,


then that's something where we would not


be able to send as much video traffic to a site like that


because we don't think it's really the same kind of video


landing page as the desktop version would be.


So all of these things are things


that the mobile-first indexing team kind of brings up


and we talk about with regards to videos.


And that does include the structured data.


It does include things like the description


in the structured data.


That kind of does play a little bit of a role there.


It might be that we don't show it directly in the search


results, but it's still useful for us for ranking purposes.


Cheers.


All right.


I just noticed we're kind of at time.


So if there's anything left from your side


that you want to ask right away, feel free.


And otherwise, I'll pause the recording in a few minutes


and stay online for a little bit longer


if you just want to chat about other things as well.


Hi, John.


I've got a question, if that's OK.


I have a client at the moment who


appears to have some sort of a penalty.


We've acquired a number of high-quality editorial links


over the last couple of years with a little movement


in SERPs.


And we are getting a new website built,


and we have the opportunity to start fresh


on a new domain.


And I'm just wondering, based on your Twitter comment


last week about building on shaky foundations,


if you would recommend not performing a 301 when


we build the new site in order to not pass and enforce


any issues.


So you'd be starting on a new domain


with completely new content?


Or, yeah?


Yeah.


But we do have the opportunity.


I know we could 301 and, I guess, take the risk.


But it's very hard to understand what's going on.


It's just very little SERP movement


over the last two years.


I'm putting a significant amount of work


into trying to move things.


We did a lot of technical work.


We tried this at those for old shady links


that other agencies have built in the past,


but it doesn't seem to be lifting.


So I just want to give the client the best


advice going forward.


It's really hard to say offhand.


I mean, on the one hand, starting


with a new domain with a new website


is a chance to start fresh.


On the other hand, that also means you do start fresh.


So all of the recommendations, the links, everything


that you've built up over the years


would essentially be gone.


And you'd need to take your knowledge of that niche


and go out and present yourself as a fresh website,


essentially.


So it has pros and cons.


It feels like a lot of work compared


to trying to improve something old.


But sometimes, especially if there's really weird stuff that


has been happening with your old website,


sometimes it makes sense to start fresh and try it


like that.


Just having a look at the links, some of the links


just appear to be guest post outreach type


links that they've done.


But there was a significant use of author boxes.


And that, to me, seems like the only issue


that there could be from a link perspective.


And I'm just wondering if that could be a factor.


It is a bit of a risk starting from scratch.


They've got locations up and down the country.


So they've got a lot of location pages and several Google My


Businesses and stuff.


So it would be a lot of work.


So I'm just wondering if you're a bit confused.


I think offhand, I'd try to avoid starting from scratch,


especially if there are lots of local listings as well


and all of that.


It seems like that's a lot of value


to throw away for something like this.


So that's kind of my gut feeling there.


It's also something where if you don't see something


like a manual action in Search Console,


then probably it's not that tricky with regards


to the links there.


But if you want to stick around after the recording stops,


I can take a quick look to see if I see anything obvious.


I mean, it's kind of hard.


It has a two-minute check of a site,


but maybe there's something obvious.


And it sounds like you're in that almost desperate situation


of like, ah, throw everything away and start over


or continue working.


The last part of the question is just


if we did decide to new domain it,


would a 301 possibly just pass along the problem?


I guess the other part of the question


is should we 301 or not and just stick with the current domain?


Yeah, so if you 301 from one domain to another,


then you're forwarding all of those signals.


So links would be forwarded.


All of that would essentially be forwarded.


And we'd essentially see the new domain


as a variation of the old one.


So I don't think that would be that useful, which also kind


of makes it even harder, I guess, in a case like yours.


But I'm happy to take a quick look afterwards.


If you want to drop the link in the chat,


then I can take a quick look.


That's great, John.


Thank you.


Sure.


Hi, John.


Hi.


I have a question related to the Google paths.


Like, I have seen so many Google listings which have not


optimized their sites and still they are ranking.


So is there any specific parameter


which we need to optimize on our sites for the local paths


basically?


For Google local, I don't know how


Google local does the ranking.


So that's really hard.


But in general, what I sometimes see,


not specific with regards to the local business listings


but with regards to sites in general,


is that there are lots of sites that do things wrong


or that do things in ways that is not optimal.


And they can still appear well in search


because they have lots of other signals that are telling us


this is actually a pretty useful site.


So that's something that might be happening there,


where if you look at it, you say, well,


they're missing all of these obvious optimizations,


but they're still ranking.


Then it might be that maybe they're


doing other things really well.


And that kind of evens things out.


So it's not that you have to do everything.


You have to do everything perfect.


It's like we try to pick the factors that play a role


and to rank them accordingly.


Thank you.


All right.


Let me stop the recording here.


And if you want, you're welcome to hang around


for a little bit longer.


And we can chat about sites or anything else.


And otherwise, I want to thank you all for joining in.


Thanks for submitting so many questions.


I hope you have a great weekend.


And looking forward to seeing you all again


in one of the future Hangouts.


Bye, everyone.


Bye, John.


Spilling the beans on Caffeine (Google's indexing system) and more! | Search Off the Record podcast

Welcome, everyone, to the next episode of Search


Off the Record, a podcast that we're trying out.


Our plan is to talk a bit about what's


happening at Google Search, how things work behind the scenes,


and maybe have some fun along the way.


My name is John Mueller.


I am a search advocate on the Search Relations team


here at Google in Switzerland.


And I'm joined here today by Martin and Gary,


who are also on the Search Relations team.


We've done a number of episodes now,


and I think we're kind of getting a hang of it.


But it's still all kind of exciting and cool and awkward


sometimes.


But anyway, lots of cool stuff happening.


And I hope you enjoy this episode.


Martin, do you want to take us off?


Yeah, sure.


Sure, why not?


I was on vacation recently.


And while I was on vacation, we announced a cool new thing


that we have been working on for quite a while, which


is the virtual webmastering conference.


And I was really, really surprised,


because one of the things that I was worried about upfront


was the fact that I knew it would


be a very different format.


And I was like, no one's going to join that.


But that's not what happened.


It filled up really, really quickly.


That was wild.


I'm not sure how the event's going to go.


John, what's going to happen if the event goes wrong


and doesn't go the way that we expected it to go?


Will this be OK?


It'll be OK, Martin.


Don't worry.


I think, I mean, by the time this episode is live,


we'll know how it went.


But in general, it's good to try things out.


And I think, especially when it comes to online events,


people have tried various things already.


And some people are kind of tired


of the traditional things.


So doing something different and taking a risk,


trying something new out, I think that is fantastic.


I really, really hope that it goes well.


Also, if it goes well, I think we will definitely


turn it into a series.


We can definitely run these multiple times


and with different things.


People have asked me, oh, can you record it?


Can we listen in?


And I'm like, no, I don't think that makes sense.


But maybe we can try that with later events, just not


in this pilot.


Yeah.


And I think one of the things we also notice


is there's just such a high demand for any kind of event


from our side.


And I think the big takeaway we picked up there also


is that we kind of need to do something more


traditional along the way, too.


Definitely.


So I don't know exactly how we'll do that.


I think timing-wise, we'll aim more towards end of the year


so that we have a bit of time to get everything organized,


to make sure that we have everything set up properly


for the speakers, that we have sessions that are interesting


and setups that are a little bit, I hope, interactive,


and where we can get some feedback from people beforehand


and maybe along the way as we go in there.


What kind of sessions do you think we should do there,


Martin?


Do you have any ideas?


I would love to have some sort of maybe Q&A panel.


I think these are very, very useful.


I think your site clinics are generally also super useful.


I'm guessing that we have probably


new things to communicate.


So the classical front-centric sessions where we have a talk


is probably also a good idea.


What I'm wondering about is, so on one hand,


I want to make sure that we are not limiting capacity


because people don't like that.


And I think the beauty of virtual events


is that we don't have to limit capacity.


But then we have to figure out how we can mix


the interactive parts with the less interactive parts.


And with the non-interactive things, like the regular talks,


I prefer them pre-recorded, to be honest.


I don't know, how do you feel about pre-recording


and screening with a live chat like we


do with Webmaster Lightning Talks?


Or do you prefer live presentations,


as in live-live?


I kind of like both.


I think for information-rich talks,


doing something pre-recorded makes it a little bit easier


because then you have a chance to really fine-tune


the message and to bring it across in a way that


is easier to understand, where you can match your slides,


maybe if you're doing slides as well.


But the live-live talks, I think what's interesting there


is just it's a whole different energy because you have all


of this pressure and adrenaline from doing something live,


which means it's not perfect.


But it also, I think, brings across a little bit more


the human touch when it comes to content.


So especially for Q&A, for maybe a small panel or something


like that, doing something live, I think is pretty cool.


Doing some of that in a chat format is also an option.


I kind of like the video for live as well,


but sometimes chat works just as well.


And especially considering that not everyone speaks English


as a first language, then sometimes having something


written makes it a little bit easier to understand


what is actually happening.


Definitely, and you can also get transcriptions for the videos


or could we do something that's spread out a little bit more?


I don't know.


Yeah, that's a good question.


Maybe we could do a two-day event, one in one time zone


and then the other in the other time zone,


but it's similar content.


We'll have to figure this one out.


This is going to be interesting.


Or we could just do it as a podcast.


Then we don't have to worry about the video format at all.


I approve of this idea.


Of course, yeah.


Because, Gary, we want you on video,


but you really, really don't want to get on video, huh?


That is correct, sir.


But your class on how search works was so good.


I would love this content to be more widely available.


And I know that you are OK with being on stage in person,


just not on video, right?


So we were actually talking about this


with our producer, with Anna.


But I really want to do it properly, not from a home


recording, but in a studio.


And we have, well, Anna has ideas


about how to make it happen.


But you will be on camera?


But when we have a studio, and I would be on camera.


Basically, if it's worth it, then I


can deal with being on camera.


But for events, for example, I just


feel so awkward about it that I just don't want to do it.


If someone tricks me into being on camera, like, for example,


there was this Aircoats podcast the other week


where I was invited to talk on a podcast, again, Aircoats.


And then when I joined in into the meeting,


then they started recording the video feed.


And I'm like, hang on.


What's happening?


And what do you mean, what's happening?


It's like, I thought this is a podcast.


It's like, well, it's actually a webcast.


What does that mean?


It's like, you are recording me?


Yeah, could you also change your t-shirt?


Because it's like you're in a home t-shirt.


But don't you think you should have told me this before?


Oh, my.


So yeah, I will not cancel on the spot, because I don't know.


I feel that it's unprofessional.


But yeah, that was kind of awkward and not something


that I would prefer to repeat.


But if we have a studio, and we have our producer and her ideas,


then I'm very happy to do it for something like Life


of Query or How Search Works or whatever it's called externally.


That is fantastic news.


So Gary, since we're in a podcast format now,


would you be interested in going through some


of those details around how search works,


how indexing works here?


What kind of question is that?


Tell us a little more about indexing, Gary.


Come on, let's do it.


Man, I wish I could, but I really don't want to.


You should do it.


Tough luck, buttercup.


You have to.


Let's do it.


Well, it's kind of, well, fine.


OK.


Let's talk about caffeine.


Actually, that's a good topic, because there was some confusion


about that on Twitter as well.


So maybe I can shed a light or two on that topic.


Actually, we could do a breakdown


of caffeine on these podcasts.


Yeah.


We should.


OK, let's do that.


So yeah, we have caffeine.


That's our indexing system.


Only external is cold caffeine.


Internal, it has some other name.


But that doesn't really matter.


And it does many, many things.


And I think that's actually not very clear externally,


that it does many things.


For people, it's just like we have the crawler, which


is Googlebot, and then that goes to something, something,


Google magic.


Well, people know that it gets rendered.


And then something, something, Google magic,


and then we have an index.


Now, we can actually break down that Google magic.


And people, in general, know that Google magic


or could figure it out if they wanted to.


But that Google magic is essentially


what Caffeine is doing, basically ingesting, picking up


whatever is produced by Googlebot,


which is a protocol buffer.


You can look it up on your favorite search engine,


what a protocol buffer is.


And then that protocol buffer is picked up by Caffeine.


And then we collect signals, blah, blah, blah.


And then we add the information that Caffeine


produced into our index.


Now, what's happening inside Caffeine?


Well, the very first step is that protocol buffer ingestion.


Basically, it picks up the protocol buffer


and starts processing it.


The very first step after ingestion is conversion.


MARK MANDEL-TALBOTTAO.


What does that mean?


GARY ILLYES-SCHREIBER.


Well, it's conversion.


Basically, it converts, right?


What to what?


Do you have a problem with the word?


Or?


No, I'm just like, OK, we have a protocol buffer which has all


the information that it needs.


Or does it convert there?


Does it convert the protocol buffer into a different format?


Well, that too.


But first, we have to, for example, normalize the HTML.


Because as you may have heard or noticed,


the internet is generally broken, HTML-wise.


But we still try to make sense of it.


Now if you have really broken HTML, then that's kind of hard.


So we push all the HTML through an HTML lexer.


Again, search for the name.


You can figure out what that is.


But basically, we normalize the HTML.


And then it's much easier to process it.


And then there comes the hot stepper, H1, H2, H3, H4.


I know all these header tags are also normalized.


Through rendering, we try to understand the styling that


was applied on the H tags.


So we can determine the relative importance of the H tags


compared to each other.


Let's see what else we do there.


JOHN MUELLER.


Do we also convert things like PDFs?


Or?


GARY ILLYES-SCHREIBER.


Oh, yeah, yeah, yeah.


So Google Search can index many formats, not just text HTML.


We can index PDFs.


We can index spreadsheets.


We can index Word document files.


We can index what else?


Lotus files for some reason.


MARTIN SPLITTENBERG.


Seriously?


Yeah, I know.


Wow.


Can we index floppy disks?


What's your obsession with floppy disks?


I love them.


They are such a beautiful medium.


I wish they would come back.


GARY ILLYES-SCHREIBER.


No, they are not.


They are actually the opposite of beautiful, whatever that is.


No.


They get corrupted just by, like, literally,


you blow some air on them, and they are corrupted.


But they are iconic because they are the safe icon.


Ha ha ha.


MARTIN SPLITTENBERG.


Oh my god.


I'm so not sorry about this.


OK, sorry.


Yeah, so we convert these.


We put out the HTML tokens.


We normalize the HTML tree.


Wait, wait, wait.


So going back to PDF.


So PDF is a binary format.


It's not that easy to process.


So for that, as far as I remember,


we license decoder from Adobe that we


use to basically convert the PDF to HTML.


And then from there on, we are just working with HTML.


And this happens with all the other binary formats


that we can index in Web Search.


Of course, those are also normalized.


So the HTML eventually will be very well formed.


We then start looking at meta tags


because there are a few meta tags that we deeply care about.


For example, the meta name robots.


GARY ILLYES-SCHREIBER.


Keywords.


MARTIN SPLITTENBERG.


No, not the keywords.


We don't care about the meta keywords at all, like at all.


I'm sorry.


I'm not sorry.


I changed my mind.


But the meta name robots, that's something


that we deeply care about.


And for example, if we find with a value no index,


then we know that we have to stop processing that document,


or at least don't put in the index.


What else do we do here?


Something that confuses some people


is that if there are HTML body-related elements


inside the head tag.


Like iframe.


That belongs in the body.


GARY ILLYES-SCHREIBER.


Like iframe, or div, or p, or span, or whatever,


then the HTML lexer will close the head right before those


tags and starts the body from there on.


And you can also look at this, like how


it works in practice on the W3C HTML validator, which


will do something similar.


But that's an important detail if you're injecting stuff


in the head and you want stuff to be picked up from there.


I think that's pretty much that's it.


One system that is related to converter


is something that we call the Collapser, which


is essentially the system that's doing error page handling.


As we said previously, the internet


is generally broken HTML-wise, also HTTP-wise


for some web servers.


But we still try to make sense of the internet.


And one thing that is broken on the internet sometimes


is error pages.


For example, there might be not found page,


404 page that comes back with an HTTP 200 status code.


It's actually quite common.


And what we call soft 404s.


MARTIN SPLITTENBERG


Oh, really?


Oh, wait, you're saying we have to do this for JavaScript sites?


Not just for JavaScript, OK?


Other people screw this up in different ways, too.


Are you saying that JavaScript sucks?


No, I'm saying people don't know how


to use it without breaking the internet sometimes.


So JavaScript sucks, according to Martin Splitt.


No.


How dare you, Gary?


So you're saying JavaScript is an error page.


OK, continue.


Why am I even working with you people?


So going back to error pages, which


are generated by JavaScript.


Gary.


OK, OK, fine.


Not every error page is generated by JavaScript,


but some are.


Yes, thank you.


OK, now that we have Martin's blessing,


error pages, soft error pages, we


don't want those in our index.


So we try to detect when they happen,


when they show up in our processing pipelines.


And that's what this error page handling thing does.


Basically, we have a very large corpus, well, actually,


for PORPORA, of error pages.


And then we try to match text against those.


This can also lead to very funny bugs, I would say,


where, for example, you are writing an article about error


pages in general, and you can, for your life, get it indexed.


And that's sometimes because our error page handling systems


misdetect your article based on the keywords


that you use as a soft error page.


And basically, it prompts Caffeine


to start processing those pages.


And of course, error page handling also


works on other kinds of error pages, not just 404s.


For example, if the server sends I'm overloaded message HTML


page, but the 200 status code, then we


might be able to understand that.


We have redirects that are not so obvious,


and we can detect those as well.


What else?


We also try to detect login pages here.


I'm not sure why is that useful, but we know about login pages.


So with error pages, would we also


count something like an e-commerce site that


has a product out of stock?


Would that be seen as a soft 404 page,


or would we try to index that page anyway?


I think generally that would depend on the words


that you are using on the page.


So should people try to hide that then, or is it?


Well, I think this is a philosophical question.


And I'm sure that there is someone on our team who's


trying to figure this out.


But in my opinion, you don't want out of stock pages


in the index.


So you would probably want to get rid of them.


Again, I think it's a philosophical question


because you can definitely provide some functionality


on the page so people can, I don't know,


subscribe for updates about that product or something like that.


And then it might become more useful maybe


to have those pages in the index.


But if you just have an out of stock page, why?


Why would you want to do that?


Questionable value there.


Except, I have an idea.


Except if you are trolling the users.


Basically, you are advertising that you have this product.


But then when they end up on your site,


then it's like, ha ha, just kidding, out of stock.


That doesn't seem very nice.


Yeah, OK.


So I guess the answer is it depends.


It depends.


OK.


It's one of your favorite words I know, or phrases.


Yeah.


It's the catch all phrase, I think.


Yeah, I think that's really tricky.


The out of stock situation is something


that people, when I talk about e-commerce, it always comes up.


It's like, how should we handle these?


And it feels like, on the one hand,


you still want to be findable for some of these,


especially if it's a vintage item.


There is only one in the world, and you no longer have it.


But you have all of the documentation and all of that.


And you kind of want that to be findable.


But if it's something that everyone else has,


and you just don't have on stock at the moment,


then for the user, it's more useful


if they just go to some shop that does have it.


Yeah, I think this it depends is very,


it depends phrase is very applicable here,


and on many things in SEO.


I know that people don't like it, but it's very useful.


And we should use it more.


It's true.


It's true, yes.


It is just most of the things depend on the context, yeah.


Oh my gosh, we should use it more.


OK, fine.


So one of the places where I kind of ran


into some of these struggles with it depends


myself is when I had a friend who got in touch with me


because they heard I was at Google


and doing some stuff on search.


And of course, they have a website.


And could I just help them to make it number one kind


of question?


Which of course, I can't.


I can't go there and give you that secret keyword meta


tag that will make you rank number one because Gary just


said my secret doesn't work, the keyword meta tag.


Wait, the keyword meta tag is your secret?


Your secret sucks.


I mean, everyone knows about it.


Oh, man.


I thought we'd just tell people not to use it because it's


one of those things that makes the magic happen,


and we just don't want them to know about the magic.


No, that's the patron meta tag.


Oh.


Oops.


You mean meta rank, name rank, content cheese?


I don't know.


Well, anyway, so on the one hand,


I was putting together all of the general SEO information


that we have that we can give people and tell them what to do


like our SEO starter guide, also some of the other third party


SEO starter guides that are out there.


There's some, I think, really useful stuff.


But there is a ton of information out there.


And if you don't know anything about SEO,


it's really hard to get started.


So that's something where they also came to me and was like,


so how do I find someone to do SEO for my website?


And we have a video on that.


We have, I think, some Help Center pages on that.


So that was kind of useful to point them there.


But it's still one of those situations where it's like,


well, I don't really feel comfortable just recommending


someone that I know, even if I know there are some really


good people out there.


And it was more that I was trying


to find ways to give them advice on what they should be looking


for when it comes to an SEO.


And in the past, when people asked me about this,


usually I told them to find someone local


that you can work together with in person


and kind of discuss the problems and questions


that you have about your website.


Nowadays, you don't really do a lot of business in person


anymore because of all the coronavirus stuff.


But it's still something where being at least in the same time


zone and being able to do some kind of a phone call,


video conference, makes it a lot easier to discuss.


And it makes it a lot easier to kind of figure out


what kind of needs you really have


and what kind of things the SEO side could be helping with.


So that was really interesting to kind of point them at that.


One of the things that I ended up doing more for my own sake


was to crawl their website.


So that's something that, before I joined Google,


I wrote a site map generator that basically went off


and crawled websites.


Of course it did, which was really fun.


Like how you mentioned, Gary, on the one hand,


the content on the web is really broken.


On the other hand, all of the URLs and crawling on the web


is just so broken.


And it's sometimes a surprise that Google get through it all.


But I picked up some third-party website crawler tool


and I played around with it.


And it's really cool to see how you can just


crawl your website and see what the current status is.


And especially when you compare it


to something like Search Console, which


is based on what we index, it is really interesting to see


that immediate feedback, where you can just


crawl your whole website in an hour


and see what kind of comes out versus in Search Console,


where you wait a month and see what comes out.


So I mean, it's not to say Search Console is not


useful in that regard.


But it's just, well, search takes a long time


to update everything across all of its indexes.


And understanding those differences,


I thought was really useful.


So I pointed them at a website crawler like that as well


and said, if you're going to make


big changes on your website, here's


one way to test that before waiting to see how Google


actually figures things out.


So I thought that was pretty cool.


It was fun.


Yeah, I always find it awkward when


people are asking for recommendations about SEO,


and especially if they are close friends and you are like, well,


I can't really tell you.


And then they are like, what?


You work on search?


It's like, yeah, that's exactly the problem.


And then you point them to the documentation,


and they are like, are you really just sending me


to some documentation that you wrote for the noobs?


You are a noob.


What?


And then when someone asks for an SEO recommendation,


like recommend an SEO company, then it gets even more awkward.


Because I do know tons of good SEO companies


that I could recommend.


But since I am working on search,


plus we are the face of search for SEOs and marketers


and webmasters and whatnots, how do you single out


one SEO company and recommend that it's just so weird


or it would be so weird?


So yeah, that's also really weird.


Yeah.


It's an interesting struggle because we're


so involved with the whole community.


And at the same time, we really don't


want to point at any particular people


and say, well, we know them really well,


and you should go and do business with them.


I don't think that would be fair.


So yeah.


It's true.


Another cool thing I found, not while looking at their website,


but just on Twitter in general, is this really cool


presentation from Topias Vuhlmann.


He's also from Zurich.


So woohoo, Swiss people.


I hope he's Swiss.


I don't know, actually, for sure.


Schwitt, local community.


But he's in Switzerland.


He did this really cool presentation


and kind of some research on GIF SEO.


Have you ever heard of that?


I heard GIF.


Isn't that like a peanut butter brand or something in the US?


No, no, GIF.


No, that's the image format.


GIF.


Animated images.


Yeah, GIF.


Oh, you mean GIF.


GIFs, yeah.


No, GIF.


I know that the inventor got it wrong


how his own invention is called, but yeah, it's definitely GIF.


OK.


Am I called Gary or Jerry?


You're called Gary.


Jerry.


I think I'm Jerry.


So cool.


Sometimes you're George.


I made a terrible mistake.


Now we know you're Jerry.


Cool.


Well, I mean, people never have problems pronouncing your name


anyway, so.


Sorry of my luck.


Anyway, so one of the cool things in that presentation


was kind of a side note that actually some of these GIF


search engines are really popular.


And apparently, one of them is actually the second most


popular search engine.


What?


Yeah, like I never realized.


Let me repeat that.


I mean, I don't know if it's really


like the second most popular.


Maybe Google is second most popular.


I don't know.


Jugl.


Google?


No, wait.


What's up, Jerry?


All right.


Well, anyway, it's really obvious


once you think about where these search engines go,


namely on your phone, on your keyboard,


when you're looking for an animated image,


it goes to one of these GIF search engines, right?


Yeah.


It's like you're looking for a meme.


You're looking for something funny to add to a chat,


to Twitter, or whatever.


You go there, and you search for something specific,


kind of like a Martin reaction GIF or something.


And you include that in your message.


And obviously, people are just going off and searching


for funny images and not always going


to Google to search for something useful


or informative.


And John, to be fair, we both are meme material


when it comes to that.


I know that I have a few GIFs of myself


that I use as reactions sometimes.


And people have seen them in slides and enjoyed them.


And I think we need more of these.


Izzy Smith has a bunch of SEO memes and SEO funny images.


And I think also GIFs, we should probably produce more of that.


I'm pretty sure in our video material,


there's so much that you can use for reaction GIFs or GIFs.


It's GIFs.


We should support that.


We should do more of that.


We should do more reaction GIFs.


Yeah, that's pretty cool.


Or maybe we can do stickers, like Googlebot stickers,


like the virtual kind.


Oh, that's good stuff.


How do you actually make these stickers?


It's in his presentation.


I'll send you a link.


It's pretty cool.


OK, I need to check that out.


Also, the way to do SEO for these is hashtags,


basically keywords.


Wow, so meta keywords would work there?


Meta keywords is how you rank your GIFs.


That's interesting.


That worked out really nice for early search engines, web


search engines.


Well, I think, I mean, I don't know.


Maybe people search differently when they're looking for GIFs.


But who knows?


Maybe all of that will change, and it's more


kind of a generational question.


And at some point, people will be like, what do you mean,


people searched for text?


Wait, but you know what the difference is?


There is no commercial intent.


It's true.


Not yet.


And how would spam look like?


I mean, if there's advertisement in the GIF or GIF results,


then you would just not click on it.


That's it.


Well, I don't know.


I'm sure there are pictures of spam


that you can include in GIFs.


I love spam.


Hashtag spam.


Spam, spam, spam.


Spam, spam, spam, spam, spam.


OK, wait.


Wait, wait.


Guys, guys.


I think we should just cut it off here.


This is getting out of hand.


Spam.


Spam, beans, and egg.


Wait, stop.


Spam, spam, spam, spam.


No.


Eggs and spam.


Where is the mute button here?


Anyway, thank you all for listening


in to our episode of Search Off the Record.


We'll be back with more useful and insightful information


in one of the next episodes.


Hope you subscribe and listen in when we come back.


Thank you all.


Bye bye.


Go sei cho.


Arigato gozaimashita.


Whatever goes there.


Yes.


I mean, Jerry.


Jerry.


JavaScript SEO office hours November 25th, 2020

Hello, and welcome to the JavaScript SEO office hours


here at Google Search Central.


If you haven't seen them beforehand,


we have office hours for general SEO questions,


as well as JavaScript SEO questions, as well as


Japanese office hours, and we have e-commerce office hours.


So if you have questions, drop over to our YouTube channel,


youtube.com slash c slash google search central,


and check the community tab for upcoming office hours.


I'm super happy to see that a few people have joined today,


and a few people have submitted questions on the YouTube


thread.


So we'll go through the YouTube questions first,


and then I'll open the floor for questions from the audience


here.


Excellent.


So Jeff James asked, if part of a page


is rendered server-side initially,


then uses client-side rendering, does


that influence Google's decision to render the JavaScript


component of the page?


We can see crawl logs where only about 20%


of the server-side requests from Googlebot on a certain page


show up for client-side versions or assets.


Generally speaking, that should not make a decision


or not influence our decision to render pages.


We are pretty much rendering all the pages.


But sometimes your JavaScript might time out,


or there might be a caching issue,


which might mean that we are not seeing all your content.


Unless you have an indexing issue,


I wouldn't really think about this.


As long as your content is showing up in searches,


then you should be fine.


It's not really something that you need to actively monitor.


Oh, hold on.


The question is actually longer.


Is this in contrast to a page where rendering is done


exclusively to JavaScript?


No.


There is a heuristic that tries to understand this,


but this heuristic is very rarely used,


and only for certain legacy domains.


So it is not really a problem that you need to work with.


Just make sure that you version your assets properly,


and it should be fine.


Also, the amount of crawling does not really mean much.


This might just be a crawl.


What's it called, a crawl rate-related thing


that you might be seeing in server logs.


Also be aware that some fake bots pretend to be Google bots.


So server logs need to be taken with a grain of salt.


You need to verify that it's the Google IP


address that the crawl came from,


because anyone can say, yeah, I'm Google bot.


And we see lots of fake Google bots out there.


So I wouldn't worry about that, unless you


see specific issues with content not showing up in search.


Philip asked, how do you prefer to measure CPU and RAM


consumption when the browser loads a URL or a website?


And what are acceptable ranges?


This may be more general, but in most cases,


the problem is on the JavaScript side,


and high CPU RAM consumption could


lead to a partial indexing, because Google bot will not


process all content on the URL.


That is true.


We do limit CPU consumption mostly.


It's CPU time, actually, that we are limiting,


but mostly to catch things such as infinite loops


and other issues.


So I wouldn't be worried too much specifically about this.


I have personally seen very few cases


where this was a problem.


And all of the cases I've seen, that was incorrect code


or basically broken code that ran into infinite loops


and then wasn't indexing the rest of the content,


because the code never actually reached it.


The general thing is I'd rather not give a specific time


or range or whatever kind of measures,


because A, these are implementation details that


can change at any point in time.


B, we are also using heuristics, so it's not really


a clear cut number or range that I can give you.


The general recommendation is make it as fast as you can


with as few CPU resources as you can.


If it takes a MacBook two minutes of spinning the CPU


at 100%, that may be not great for users either.


So think about it from a user's perspective.


If you are consuming CPU resources on end


for lots of time, your users will probably be unhappy,


especially on mobile phones.


That is really a killer.


So try to optimize as much as you can,


but be reasonable about it, because in the end,


if users are seeing your content and everyone's happy


and it's also showing up in search, you shouldn't worry


about this.


Again, this is something that you can go into and profile


and debug when you are seeing an issue.


I wouldn't proactively look too much.


Unless you really hear your fans spinning up


at 100% after a couple of minutes of trying


to load a website, that's usually a really bad sign.


And those are the two questions we have from YouTube,


which means we can now open up the floor for questions.


And I've seen various people here in the call today,


so that's pretty cool.


If you have any questions, feel free to just ask them.


You can also use the chat function


on the right-hand side of this Google Meet recording


if you don't want to ask a question by voice.


Any questions?


Hi, Martin.


Are you OK?


Hi, Dave.


Just a quick one.


Content visibility, the new CSS property